WOLFRAM

Wolfram Language & System Documentation Center

ApacheLog

  • Import can read various standard log formats, as well as user-customized log formats.

Background & Context

    • Log files.
    • Common variants are Common Log Format, Extended Log File Format and Apache Access Log Format.
    • Is used by MathLM, the Wolfram System license manager.
    • Plain text format.
    • Stores information about web server transactions.
    • Represents each record as a line of text.
    • Allows user-customized log formats.
    • Developed by Apache.

Import

  • Import["file","ApacheLog"] imports a log file as a Tabular object.
  • Import["file",{"ApacheLog",elem}] imports the specified element.
  • Import["file",{"ApacheLog",elem,subelem1,…}] imports subelements subelemi, useful for partial data import.
  • The format "ApacheLog" must always be specified when importing log files.
  • See the following reference pages for full general information:
  • Importimport from a file
    CloudImportimport from a cloud object
    ImportStringimport from a string
    ImportByteArrayimport from a byte array

Import Elements

  • General Import elements:
  • "Elements" list of elements and options available in this file
    "Summary"summary of the file
    "Rules"list of rules for all available elements
  • Data representation elements:
  • "Data"two-dimensional array
    "Dataset"table data as a Dataset
    "EventSeries"table data as an EventSeries
    "Tabular"a Tabular object
    "TimeSeries"table data as a TimeSeries
  • Import by default uses the "Tabular" element.
  • Subelements for partial data import for the "Tabular" element can take row and column specifications in the form {"Tabular",rows,cols}, where rows and cols can be any of the following:
  • nnth row or column
    -ncounts from the end
    n;;mfrom n through m
    n;;m;;sfrom n through m with steps of s
    {n1,n2,…}specific rows or columns ni
  • Column specifications can also be any of the following:
  • "col"single column "col"
    {col1,col2,…}list of column names coli
  • Data descriptor elements:
  • "ColumnLabels"names of columns
    "ColumnTypes"association with data type for each column
    "Schema"TabularSchema object
  • Metadata elements:
  • "ColumnCount"number of columns stored in file
    "Dimensions"data dimensions
    "RowCount"number of rows stored in file

Import Options

    Import Options
  • General Import options:
  • "DataFormat"Automaticformat specification, given as a string
    "Schema"Automaticschema used to construct Tabular object
    "TimeColumn"Automaticcolumn to use for times in "EventSeries" and "TimeSeries" elements
  • Possible settings for the "DataFormat" option include:
  • Automaticautomatically determine format specification (default)
    "ApacheErrorLog"Apache error log format for server issues and diagnostics
    "CommonLog"Common Log Format (CLF) for basic web traffic
    "CombinedLog"common log with referrer for web analytics and SEO
    "CommonLogVHost"common log with virtual host for multi-site hosting environments
    "RefererLog"only referer log for marketing and SEO
    formatarbitrary fields and record separators
  • To import a custom log format, use "DataFormat"->format to specify the format of each record consisting of data elements and record separators such as "`Field1` [`Field2`] `Field3`".
  • The format specification can be a list of strings and OptionalElements to denote elements that may be absent in some lines.
  • Possible settings for the "Schema" option include:
  • schemaa complete TabularSchema specification
    propvala schema property and value (see reference page for TabularSchema)
    <|"prop1"val1,…|>an association of schema properties and values

Examples

open all close all

Basic Examples  (2)

Import a log file as a Tabular:

Wolfram Language code: Import["ExampleData/apache_error.log", "ApacheLog"]

Import the file summary:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Summary"}]

Scope  (2)

Import metadata from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Dimensions"}]
Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "ColumnTypes"}]

Import a sample row of a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Data", 5}]

Import Elements  (25)

"ColumnCount"  (1)

Get the number of columns from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "ColumnCount"}]

"ColumnLabels"  (1)

Get the column labels from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "ColumnLabels"}]

"ColumnTypes"  (1)

Get the inferred column types from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "ColumnTypes"}]

"Data"  (7)

Import a log file as a 2D list of values:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Data"}]

Import a single row from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Data", 5}]

Import multiple rows:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Data", {1, 5, 7}}]

Import the first 5 rows:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Data", 1 ;; 5}]

Import a single element at a given row and column:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Data", 5, 5}]

Import a single column:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Data", All, 5}]

Import selected columns using column names:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Data", All, {"Source", "Referer"}}]

"Dataset"  (2)

Import a log file as a Dataset:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Dataset"}]

Import selected columns using column names:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Dataset", All, {"Source", "Referer"}}]

"Dimensions"  (1)

Get the dimensions from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Dimensions"}]

"EventSeries"  (1)

Import a log file as an EventSeries:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "EventSeries"}]

Import a single row from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "EventSeries", 5}]

Import some specific rows from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "EventSeries", {1, 5}}]

Import the first 10 rows of a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "EventSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "EventSeries", All, {"Date", "Module", "LogLevel"}}]

"RowCount"  (1)

Get the number of rows from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "RowCount"}]

"Schema"  (1)

Get the TabularSchema object:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Schema"}]

"Summary"  (1)

Summary of a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Summary"}]

"Tabular"  (7)

Import a log file as a Tabular object:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Tabular"}]

Import a single row:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Tabular", 5}]

Import multiple rows:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Tabular", {1, 5, 7}}]

Import the first 5 rows:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Tabular", 1 ;; 5}]

Import a single element at a given row and column:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Tabular", 5, 5}]

Import a single column:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Tabular", All, 5}]

Import selected columns using column names:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "Tabular", All, {"Source", "Referer"}}]

"TimeSeries"  (1)

Import a log file as a TimeSeries:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "TimeSeries"}]

Import a single row from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "TimeSeries", 5}]

Import some specific rows from a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "TimeSeries", {1, 5}}]

Import the first 10 rows of a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "TimeSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "TimeSeries", All, {"Date", "Module", "LogLevel"}}]

Import Options  (7)

"DataFormat"  (4)

Import automatically infers format specification from data stored in a log file:

Wolfram Language code: Import["ExampleData/apache_error.log", "ApacheLog"]

Specify predefined log format:

Wolfram Language code: Import["ExampleData/apache_error.log", "ApacheLog", "DataFormat" -> "ApacheErrorLog"]

Use a custom format:

Wolfram Language code: Import["ExampleData/apache_error.log", "ApacheLog", "DataFormat" -> "[`Date`] [`Module`:`LogLevel`] [pid `ProcessID`:tid `ThreadID`] `Source`: `Error`: [client `Client`] `Message`, referer `Referer`"]

Import log data with optional fields:

Wolfram Language code: log = "Monday [temp = 5C] [precipitation: rain] Tuesday [temp = 0C] Wednesday [temp = 2C] [precipitation: hail]";
Wolfram Language code: ImportString[log, "ApacheLog", "DataFormat" -> {"`Day` [temp = `Temperature`]", OptionalElement["[precipitation: `Precipitation`]"]}]

"Schema"  (2)

Import automatically infers column labels and types from data stored in a log file:

Wolfram Language code: tab = Import["ExampleData/apache_error.log", "ApacheLog"]
Wolfram Language code: tab["ColumnTypes"]

Use "Schema" option to specify column labels and types:

Wolfram Language code: tab = Import["ExampleData/apache_error.log", "ApacheLog", "Schema" -> {"ColumnKeys" -> {"Date", "Module", "LogLevel", "ProcessID", "ThreadID", "Source", "Error", "Client", "Message", "Referer"}, "ElementType" -> {"Date" -> "Date", "ProcessID" -> "Integer32", "ThreadID" -> "Integer64"}}]
Wolfram Language code: tab["ColumnTypes"]

"TimeColumn"  (1)

By default, the time column is selected automatically for "TimeSeries" and "EventSeries" elements:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "TimeSeries"}]

Use the "TimeColumn" option to specify the time column:

Wolfram Language code: Import["ExampleData/apache_error.log", {"ApacheLog", "TimeSeries"}, "TimeColumn" -> "ProcessID"]