WOLFRAM

Wolfram Language & System Documentation Center

ArrowIPC (.arrow, .arrows, .feather, .ftr)

  • Import supports ArrowIPC file and stream formats as well as Feather Version 1 and Version 2.
  • Export supports ArrowIPC file and stream formats.

Background & Context

    • Registered MIME types: application/vnd.apache.arrow.file, application/vnd.apache.arrow.stream
    • Arrow IPC columnar data format.
    • Used for efficient serialization of large columnar datasets.
    • The primitive unit of serialized data in the columnar format is called record batch.
    • Arrow IPC file format is used for serializing a fixed number of record batches and supports random access.
    • Arrow IPC streaming format is used for sending an arbitrary-length sequence of record batches.
    • Feather version 2 is a file format represented as the Arrow IPC file on disk.
    • Feather version 1 is a legacy file format distinct from Arrow IPC files.
    • Developed by the Apache Software Foundation.
    • Binary file format.
    • Supports multiple compression methods.

Import & Export

Import Elements

  • General Import elements:
  • "Elements" list of elements and options available in this file
    "Summary"summary of the file
    "Rules"list of rules for all available elements
  • Data representation elements:
  • "Data"two-dimensional array
    "Dataset"table data as a Dataset
    "EventSeries"table data as an EventSeries
    "Tabular"a Tabular object
    "TimeSeries"table data as a TimeSeries
  • Import by default uses the "Tabular" element.
  • Subelements for partial data import for the "Tabular" element can take row and column specifications in the form {"Tabular",rows,cols}, where rows and cols can be any of the following:
  • nnth row or column
    -ncounts from the end
    n;;mfrom n through m
    n;;m;;sfrom n through m with steps of s
    {n1,n2,…}specific rows or columns ni
  • Column specifications can also be any of the following:
  • "col"single column "col"
    {col1,col2,…}list of column names coli
  • Data descriptor elements:
  • "ColumnLabels"names of columns
    "ColumnTypes"association with data type for each column
    "Schema"TabularSchema object
  • Metadata elements:
  • "ColumnCount"number of columns stored in file
    "Dimensions"data dimensions
    "RowCount"number of rows stored in file
    "MetaInformation"metadata

Options

  • General Import options:
  • IncludeMetaInformationAllmetadata types to import
    "Schema"Automaticschema used to construct Tabular object
    "TimeColumn"Automaticcolumn to use for times in "EventSeries" and "TimeSeries" elements
    "UseMemoryMappedFile"Truewhether to use memory-mapped reader
  • Possible settings for the "Schema" option include:
  • schemaa complete TabularSchema specification
    propvala schema property and value (see reference page for TabularSchema)
    <|"prop1"val1,…|>an association of schema properties and values
  • General Export options:
  • "Compression"Nonecompression method
    CompressionLevelAutomaticcompression level
    "Streamable"Falseif true, then Arrow IPC streaming format is used
  • The following settings for "Compression" are supported:
  • Noneno compression
    "LZ4Frame"LZ4 Frame compression
    "ZSTD"ZSTD compression

Examples

open all close all

Basic Examples  (3)

Import Tabular object from Arrow IPC file:

Wolfram Language code: Import["ExampleData/USstates.arrow"]

Import the file summary:

Wolfram Language code: Import["ExampleData/USstates.arrow", "Summary"]

Export Tabular object to Arrow IPC file:

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow"];
Wolfram Language code: Export["file.arrow", tabular]

Scope  (3)

Import  (3)

Show all elements available in the file:

Wolfram Language code: Import["ExampleData/USstates.arrow", "Elements"]

By default, a Tabular object is returned:

Wolfram Language code: Import["ExampleData/USstates.arrow"]//TabularQ

Import column types:

Wolfram Language code: Import["ExampleData/USstates.arrow", "ColumnTypes"]

Import Elements  (19)

"ColumnCount"  (1)

Get the number of columns:

Wolfram Language code: Import["ExampleData/USstates.arrow", "ColumnCount"]

"ColumnLabels"  (1)

Read column names:

Wolfram Language code: Import["ExampleData/USstates.arrow", "ColumnLabels"]

"ColumnTypes"  (1)

Import column types:

Wolfram Language code: Import["ExampleData/USstates.arrow", "ColumnTypes"]

"Data"  (3)

Get the data from a file:

Wolfram Language code: Import["ExampleData/USstates.arrow", "Data"]//Short

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Data", 1 ;; 3}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Data", All, {1, 3}}]//Short

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Data", All, {"Name", "Area"}}]//Short

"Dataset"  (3)

Get the data as a Dataset:

Wolfram Language code: Import["ExampleData/USstates.arrow", "Dataset"]

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Dataset", 1 ;; 3}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Dataset", All, {1, 3}}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Dataset", All, {"Name", "Area"}}]

"Dimensions"  (1)

Import data dimensions:

Wolfram Language code: Import["ExampleData/USstates.arrow", "Dimensions"]

"EventSeries"  (1)

Export a Tabular object to an ArrowIPC file:

Wolfram Language code: file = Export["file.arrow", ResourceData["Sample Tabular Data: Sales Data"]]

Import an ArrowIPC file as an EventSeries:

Wolfram Language code: Import[file, "EventSeries"]

Import a single row from an ArrowIPC file:

Wolfram Language code: Import[file, {"EventSeries", 5}]

Import some specific rows from an ArrowIPC file:

Wolfram Language code: Import[file, {"EventSeries", {1, 5, 7}}]

Import the first 10 rows of an ArrowIPC file:

Wolfram Language code: Import[file, {"EventSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import[file, {"EventSeries", All, {"Product", "Date", "Quantity"}}]

"MetaInformation"  (1)

Import metadata:

Wolfram Language code: Import["ExampleData/USstates.arrow", "MetaInformation"]

"RowCount"  (1)

Get the number of rows:

Wolfram Language code: Import["ExampleData/USstates.arrow", "RowCount"]

"Schema"  (1)

Get the TabularSchema object:

Wolfram Language code: Import["ExampleData/USstates.arrow", "Schema"]

"Summary"  (1)

Get the file summary:

Wolfram Language code: Import["ExampleData/USstates.arrow", "Summary"]

"Tabular"  (3)

Get the data from a file as a Tabular object:

Wolfram Language code: Import["ExampleData/USstates.arrow", "Tabular"]

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Tabular", 1 ;; 5}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Tabular", All, {1, 3}}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/USstates.arrow", {"Tabular", All, {"Name", "Area"}}]

"TimeSeries"  (1)

Export a Tabular object to an ArrowIPC file:

Wolfram Language code: file = Export["file.arrow", ResourceData["Sample Tabular Data: Sales Data"]]

Import an ArrowIPC file as a TimeSeries:

Wolfram Language code: Import[file, "TimeSeries"]

Import a single row from an ArrowIPC file:

Wolfram Language code: Import[file, {"TimeSeries", 5}]

Import some specific rows from an ArrowIPC file:

Wolfram Language code: Import[file, {"TimeSeries", {1, 5, 7}}]

Import the first 10 rows of an ArrowIPC file:

Wolfram Language code: Import[file, {"TimeSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import[file, {"TimeSeries", All, {"Product", "Date", "Quantity"}}]

Import Options  (4)

IncludeMetaInformation  (1)

By default, all metadata stored in a file is imported and embedded in the Tabular object:

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow"]; tabular["Metadata"]

Do not import metadata:

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow", IncludeMetaInformation -> None]; tabular["Metadata"]

"Schema"  (1)

Wolfram Language code: file = Export["out.arrow", Tabular[Association["RawSchema" -> Association["ColumnProperties" -> Association["A" -> Association["ElementType" -> "String"], "B" -> Association["ElementType" -> "String"]], "KeyColumns" -> None, "Backend" -> "WolframKernel"], "BackendData" -> Association["ColumnData" -> DataStructure["ColumnTable", {{TabularColumn[Association["Data" -> {{0, {0, 11, 22, 33, 44, 55}, "Jan 03 2006Jan 04 2006Jan 05 2006Jan 06 2006Jan 09 2006"}, {}, None}, "ElementType" -> "String"]], TabularColumn[Association[ "Data" -> {{3, {0, 5, 10, 15, 20, 25}, "11.8212.0412.0911.8812.43"}, {}, None}, "ElementType" -> "String"]]}}]]]]];

By default, column labels and their types stored in a file are used when Tabular or Dataset objects are imported:

Wolfram Language code: tabular = Import[file]; tabular["ColumnTypes"]

Use "Schema" option to specify column labels and types:

Wolfram Language code: tabular = Import[file, "Schema" -> {"ColumnKeys" -> {"Date", "Value"}, "ElementType" -> {"Date" -> "Date", "Value" -> "Real32"}}]; tabular["ColumnTypes"]

"TimeColumn"  (1)

Export a Tabular object to an ArrowIPC file:

Wolfram Language code: file = Export["file.arrow", Tabular[Association["RawSchema" -> Association["ColumnProperties" -> Association["Date" -> Association["ElementType" -> TypeSpecifier["Date"]["Integer32", "Day", "Gregorian", None]], "Value" -> Association["ElementType" -> "Real32"]], "KeyColumns" -> None, "Backend" -> "WolframKernel"], "Options" -> {}, "BackendData" -> Association["ColumnData" -> DataStructure["ColumnTable", {{TabularColumn[Association["Data" -> {5, {{NumericArray[{13150, 13151, 13152, 13153, 13156}, "Integer32"], {}, None}}, None}, "ElementType" -> "Date"["Integer32", "Day", "Gregorian", None]]], TabularColumn[Association[ "Data" -> {NumericArray[{11.819999694824219, 12.039999961853027, 12.09000015258789, 11.880000114440918, 12.430000305175781}, "Real32"], {}, None}, "ElementType" -> "Real32"]]}}]]]]];

By default, the time column is selected automatically for "TimeSeries" and "EventSeries" elements:

Wolfram Language code: Import[file, "TimeSeries"]

Use the "TimeColumn" option to specify the time column:

Wolfram Language code: Import[file, "TimeSeries", "TimeColumn" -> "Value"]

"UseMemoryMappedFile"  (1)

By default, memory mapping is disabled. "UseMemoryMappedFile"->True to enable memory mapping:

Wolfram Language code: Import["ExampleData/USstates.arrow", "UseMemoryMappedFile" -> True]//TabularQ

Export Options  (6)

"Compression"  (2)

Compression is disabled by default:

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow"]; Export["out.arrow", tabular]//FileSize

Compare supported compression methods:

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow"]; AssociationMap[(FileSize@Export["out.arrow", tabular, "Compression" -> #])&, {"LZ4Frame", "ZSTD"}]

CompressionLevel  (2)

By default, Automatic value of CompressionLevel is used. It corresponds to a different default value for each compression method.

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow"]; AssociationMap[(FileSize@Export["out.arrow", tabular, CompressionLevel -> Automatic, "Compression" -> #])&, {"LZ4Frame", "ZSTD"}]

Use maximal compression for each method:

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow"]; AssociationMap[(FileSize@Export["out.arrow", tabular, CompressionLevel -> 1.0, "Compression" -> #])&, {"LZ4Frame", "ZSTD"}]

"Streamable"  (2)

By default, Export uses Arrow IPC file format:

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow"]; Export["out.arrow", tabular]//FileSize

Use "Streamable" option to generate Arrow IPC streaming format:

Wolfram Language code: tabular = Import["ExampleData/USstates.arrow"]; Export["out.arrow", tabular, "Streamable" -> True]//FileSize