WOLFRAM

Wolfram Language & System Documentation Center

CommonLog

Background & Context

    • Common Log Format.
    • Also known as NCSA Common Log Format.
    • Used for logging web traffic.
    • Plain text format.
    • Represents each record as a line of text.
    • Developed by NCSA.

Import

  • Import["file","CommonLog"] imports a log file as a Tabular object.
  • Import["file",{"CommonLog",elem}] imports the specified element.
  • Import["file",{"CommonLog",elem,subelem1,…}] imports subelements subelemi, useful for partial data import.
  • The format "CommonLog" must always be specified when importing common log files.
  • See the following reference pages for full general information:
  • Importimport from a file
    CloudImportimport from a cloud object
    ImportStringimport from a string
    ImportByteArrayimport from a byte array

Import Elements

  • General Import elements:
  • "Elements" list of elements and options available in this file
    "Summary"summary of the file
    "Rules"list of rules for all available elements
  • Data representation elements:
  • "Data"two-dimensional array
    "Dataset"table data as a Dataset
    "EventSeries"table data as an EventSeries
    "Tabular"a Tabular object
    "TimeSeries"table data as a TimeSeries
  • Import by default uses the "Tabular" element.
  • Subelements for partial data import for the "Tabular" element can take row and column specifications in the form {"Tabular",rows,cols}, where rows and cols can be any of the following:
  • nnth row or column
    -ncounts from the end
    n;;mfrom n through m
    n;;m;;sfrom n through m with steps of s
    {n1,n2,…}specific rows or columns ni
  • Column specifications can also be any of the following:
  • "col"single column "col"
    {col1,col2,…}list of column names coli
  • Data descriptor elements:
  • "ColumnLabels"names of columns
    "ColumnTypes"association with data type for each column
    "Schema"TabularSchema object
  • Columns or identifiers typically present in a common log file are:
  • "RemoteHost"remote hostname or IP number
    "ClientID"remote logname
    "RemoteUser"username used for authentication
    "Date"date and time of the request
    "RequestLine"request as given by the client
    "StatusCode"the HTTP status code returned to the client
    "ByteCount"byte size of the tranferred document
  • Metadata elements:
  • "ColumnCount"number of columns stored in file
    "Dimensions"data dimensions
    "RowCount"number of rows stored in file

Import Options

  • General Import options:
  • "Schema"Automaticschema used to construct Tabular object
    "TimeColumn"Automaticcolumn to use for times in "EventSeries" and "TimeSeries" elements
  • Possible settings for the "Schema" option include:
  • schemaa complete TabularSchema specification
    propvala schema property and value (see reference page for TabularSchema)
    <|"prop1"val1,…|>an association of schema properties and values

Examples

open all close all

Basic Examples  (2)

Import a log file as a Tabular:

Wolfram Language code: Import["ExampleData/common20.log", "CommonLog"]

Import the file summary:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Summary"}]

Scope  (2)

Import metadata from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Dimensions"}]
Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "ColumnTypes"}]

Import a sample row of a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Data", 10}]

Import Elements  (25)

"ColumnCount"  (1)

Get the number of columns from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "ColumnCount"}]

"ColumnLabels"  (1)

Get the column labels from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "ColumnLabels"}]

"ColumnTypes"  (1)

Get the inferred column types from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "ColumnTypes"}]

"Data"  (7)

Import a log file as a 2D list of values:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Data"}]

Import a single row from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Data", 5}]

Import multiple rows:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Data", {1, 5, 7}}]

Import the first 5 rows:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Data", 1 ;; 5}]

Import a single element at a given row and column:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Data", 5, 5}]

Import a single column:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Data", All, 5}]

Import selected columns using column names:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Data", All, {"RemoteHost", "RequestLine"}}]

"Dataset"  (2)

Import a log file as a Dataset:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Dataset"}]

Import selected columns using column names:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Dataset", All, {"RemoteHost", "RequestLine"}}]

"Dimensions"  (1)

Get the dimensions from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Dimensions"}]

"EventSeries"  (1)

Import a log file as an EventSeries:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "EventSeries"}]

Import a single row from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "EventSeries", 5}]

Import some specific rows from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "EventSeries", {1, 5}}]

Import the first 10 rows of a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "EventSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "EventSeries", All, {"Date", "RemoteHost", "RequestLine"}}]

"RowCount"  (1)

Get the number of rows from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "RowCount"}]

"Schema"  (1)

Get the TabularSchema object:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Schema"}]

"Summary"  (1)

Summary of a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Summary"}]

"Tabular"  (7)

Import a log file as a Tabular object:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Tabular"}]

Import a single row:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Tabular", 5}]

Import multiple rows:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Tabular", {1, 5, 7}}]

Import the first 5 rows:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Tabular", 1 ;; 5}]

Import a single element at a given row and column:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Tabular", 5, 5}]

Import a single column:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Tabular", All, 5}]

Import selected columns using column names:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "Tabular", All, {"RemoteHost", "RequestLine"}}]

"TimeSeries"  (1)

Import a log file as a TimeSeries:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "TimeSeries"}]

Import a single row from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "TimeSeries", 5}]

Import some specific rows from a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "TimeSeries", {1, 5}}]

Import the first 10 rows of a log file:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "TimeSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "TimeSeries", All, {"Date", "RemoteHost", "RequestLine"}}]

Import Options  (3)

"Schema"  (2)

Import automatically infers column labels and types from data stored in a log file:

Wolfram Language code: tab = Import["ExampleData/common20.log", "CommonLog"]
Wolfram Language code: tab["ColumnTypes"]

Use the "Schema" option to specify column labels and types:

Wolfram Language code: tab = Import["ExampleData/common20.log", "CommonLog", "Schema" -> {"ColumnKeys" -> {"RemoteHost", "ClientID", "RemoteUser", "Date", "RequestLine", "StatusCode", "ByteCount"}, "ElementType" -> {"Date" -> "Date", "StatusCode" -> "UnsignedInteger32", "ByteCount" -> "Integer64"}}]
Wolfram Language code: tab["ColumnTypes"]

"TimeColumn"  (1)

By default, the time column is selected automatically for "TimeSeries" and "EventSeries" elements:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "TimeSeries"}]

Use the "TimeColumn" option to specify the time column:

Wolfram Language code: Import["ExampleData/common20.log", {"CommonLog", "TimeSeries"}, "TimeColumn" -> "ByteCount"]