WOLFRAM

Wolfram Language & System Documentation Center

ORC (.orc)

Background & Context

    • Efficient, general-purpose, column-oriented data format.
    • Developed by the Apache Software Foundation.
    • ORC is an acronym for Optimized Row Columnar.
    • Binary file format.
    • Supports multiple compression methods.

Import & Export

Import Elements

  • General Import elements:
  • "Elements" list of elements and options available in this file
    "Summary"summary of the file
    "Rules"list of rules for all available elements
  • Data representation elements:
  • "Data"two-dimensional array
    "Dataset"table data as a Dataset
    "EventSeries"table data as an EventSeries
    "Tabular"a Tabular object
    "TimeSeries"table data as a TimeSeries
  • Import by default uses the "Tabular" element.
  • Subelements for partial data import for the "Tabular" element can take row and column specifications in the form {"Tabular",rows,cols}, where rows and cols can be any of the following:
  • nnth row or column
    -ncounts from the end
    n;;mfrom n through m
    n;;m;;sfrom n through m with steps of s
    {n1,n2,…}specific rows or columns ni
  • Column specifications can also be any of the following:
  • "col"single column "col"
    {col1,col2,…}list of column names coli
  • Data descriptor elements:
  • "ColumnLabels"names of columns
    "ColumnTypes"association with data type for each column
    "Schema"TabularSchema object
  • Metadata elements:
  • "ColumnCount"number of columns stored in file
    "Dimensions"data dimensions
    "RowCount"number of rows stored in file
    "MetaInformation"metadata

Options

  • General Import options:
  • IncludeMetaInformationAllmetadata types to import
    "Schema"Automaticschema used to construct Tabular object
    "TimeColumn"Automaticcolumn to use for times in "EventSeries" and "TimeSeries" elements
  • Possible settings for the "Schema" option include:
  • schemaa complete TabularSchema specification
    propvala schema property and value (see reference page for TabularSchema)
    <|"prop1"val1,…|>an association of schema properties and values
  • General Export options:
  • "Compression"Nonecompression method
    "CompressionStrategy""Speed"compression strategy
  • The following settings for "Compression" are supported:
  • Noneno compression
    "LZ4"LZ4 compression
    "GZIP"GZIP Hadoop compression
    "Snappy"Snappy compression
    "ZSTD"ZSTD compression
  • The following settings for "CompressionStategy" are supported:
  • "Size"optimize size of file
    "Speed"optimize the speed of export

Examples

open all close all

Basic Examples  (3)

Import Tabular object from ORC file:

Wolfram Language code: Import["ExampleData/USstates.orc"]

Import the file summary:

Wolfram Language code: Import["ExampleData/USstates.orc", "Summary"]

Export Tabular object to ORC file:

Wolfram Language code: tabular = Import["ExampleData/USstates.orc"];
Wolfram Language code: Export["file.orc", tabular]

Scope  (3)

Import  (3)

Show all elements available in the file:

Wolfram Language code: Import["ExampleData/USstates.orc", "Elements"]

By default, a Tabular object is returned:

Wolfram Language code: Import["ExampleData/USstates.orc"]//TabularQ

Import column types:

Wolfram Language code: Import["ExampleData/USstates.orc", "ColumnTypes"]

Import Elements  (19)

"ColumnCount"  (1)

Get the number of columns:

Wolfram Language code: Import["ExampleData/USstates.orc", "ColumnCount"]

"ColumnLabels"  (1)

Read column names:

Wolfram Language code: Import["ExampleData/USstates.orc", "ColumnLabels"]

"ColumnTypes"  (1)

Import column types:

Wolfram Language code: Import["ExampleData/USstates.orc", "ColumnTypes"]

"Data"  (3)

Get the data from a file:

Wolfram Language code: Import["ExampleData/USstates.orc", "Data"]//Short

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Data", 1 ;; 3}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Data", All, {1, 3}}]//Short

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Data", All, {"Name", "Area"}}]//Short

"Dataset"  (3)

Get the data as a Dataset:

Wolfram Language code: Import["ExampleData/USstates.orc", "Dataset"]

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Dataset", 1 ;; 3}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Dataset", All, {1, 3}}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Dataset", All, {"Name", "Area"}}]

"Dimensions"  (1)

Import data dimensions:

Wolfram Language code: Import["ExampleData/USstates.orc", "Dimensions"]

"EventSeries"  (1)

Export a Tabular object to an ORC file:

Wolfram Language code: file = Export["file.orc", ResourceData["Sample Tabular Data: Sales Data"]]

Import an ORC file as an EventSeries:

Wolfram Language code: Import[file, "EventSeries"]

Import a single row from an ORC file:

Wolfram Language code: Import[file, {"EventSeries", 5}]

Import some specific rows from an ORC file:

Wolfram Language code: Import[file, {"EventSeries", {1, 5, 7}}]

Import the first 10 rows of an ORC file:

Wolfram Language code: Import[file, {"EventSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import[file, {"EventSeries", All, {"Product", "Date", "Quantity"}}]

"MetaInformation"  (1)

Import metadata:

Wolfram Language code: Import["ExampleData/USstates.orc", "MetaInformation"]

"RowCount"  (1)

Get the number of rows:

Wolfram Language code: Import["ExampleData/USstates.orc", "RowCount"]

"Schema"  (1)

Get the TabularSchema object:

Wolfram Language code: Import["ExampleData/USstates.orc", "Schema"]

"Summary"  (1)

Get the file summary:

Wolfram Language code: Import["ExampleData/USstates.orc", "Summary"]

"Tabular"  (3)

Get the data from a file as a Tabular object:

Wolfram Language code: Import["ExampleData/USstates.orc", "Tabular"]

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Tabular", 1 ;; 5}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Tabular", All, {1, 3}}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.orc", {"Tabular", All, {"Name", "Area"}}]

"TimeSeries"  (1)

Export a Tabular object to an ORC file:

Wolfram Language code: file = Export["file.orc", ResourceData["Sample Tabular Data: Sales Data"]]

Import an ORC file as a TimeSeries:

Wolfram Language code: Import[file, "TimeSeries"]

Import a single row from an ORC file:

Wolfram Language code: Import[file, {"TimeSeries", 5}]

Import some specific rows from an ORC file:

Wolfram Language code: Import[file, {"TimeSeries", {1, 5, 7}}]

Import the first 10 rows of an ORC file:

Wolfram Language code: Import[file, {"TimeSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import[file, {"TimeSeries", All, {"Product", "Date", "Quantity"}}]

Import Options  (3)

IncludeMetaInformation  (1)

By default, all metadata stored in a file is imported and embedded in the Tabular object:

Wolfram Language code: tabular = Import["ExampleData/USstates.orc"]; tabular["Metadata"]

Do not import metadata:

Wolfram Language code: tabular = Import["ExampleData/USstates.orc", IncludeMetaInformation -> None]; tabular["Metadata"]

"Schema"  (1)

Export Tabular object to Parquet file:

Wolfram Language code: file = Export["out.parquet", Tabular[Association["RawSchema" -> Association["ColumnProperties" -> Association["A" -> Association["ElementType" -> "String"], "B" -> Association["ElementType" -> "String"]], "KeyColumns" -> None, "Backend" -> "WolframKernel"], "BackendData" -> Association["ColumnData" -> DataStructure["ColumnTable", {{TabularColumn[Association["Data" -> {{0, {0, 11, 22, 33, 44, 55}, "Jan 03 2006Jan 04 2006Jan 05 2006Jan 06 2006Jan 09 2006"}, {}, None}, "ElementType" -> "String"]], TabularColumn[Association[ "Data" -> {{3, {0, 5, 10, 15, 20, 25}, "11.8212.0412.0911.8812.43"}, {}, None}, "ElementType" -> "String"]]}}]]]]];

By default, column labels and their types stored in a file are used when Tabular or Dataset objects are imported:

Wolfram Language code: tabular = Import[file]; tabular["ColumnTypes"]

Use "Schema" option to specify column labels and types:

Wolfram Language code: tabular = Import[file, "Schema" -> {"ColumnKeys" -> {"Date", "Value"}, "ElementType" -> {"Date" -> "Date", "Value" -> "Real32"}}]; tabular["ColumnTypes"]

"TimeColumn"  (1)

Export a Tabular object to an ORC file:

Wolfram Language code: file = Export["file.orc", Tabular[Association["RawSchema" -> Association["ColumnProperties" -> Association["Date" -> Association["ElementType" -> TypeSpecifier["Date"]["Integer32", "Day", "Gregorian", None]], "Value" -> Association["ElementType" -> "Real32"]], "KeyColumns" -> None, "Backend" -> "WolframKernel"], "Options" -> {}, "BackendData" -> Association["ColumnData" -> DataStructure["ColumnTable", {{TabularColumn[Association["Data" -> {5, {{NumericArray[{13150, 13151, 13152, 13153, 13156}, "Integer32"], {}, None}}, None}, "ElementType" -> "Date"["Integer32", "Day", "Gregorian", None]]], TabularColumn[Association[ "Data" -> {NumericArray[{11.819999694824219, 12.039999961853027, 12.09000015258789, 11.880000114440918, 12.430000305175781}, "Real32"], {}, None}, "ElementType" -> "Real32"]]}}]]]]];

By default, the time column is selected automatically for "TimeSeries" and "EventSeries" elements:

Wolfram Language code: Import[file, "TimeSeries"]

Use the "TimeColumn" option to specify the time column:

Wolfram Language code: Import[file, "TimeSeries", "TimeColumn" -> "Value"]

Export Options  (4)

"Compression"  (2)

Compression is disabled by default:

Wolfram Language code: tabular = Import["ExampleData/USstates.orc"]; Export["out.orc", tabular]//FileSize

Compare supported compression methods:

Wolfram Language code: tabular = Import["ExampleData/USstates.orc"]; AssociationMap[(FileSize@Export["out.orc", tabular, "Compression" -> #])&, {"LZ4", "GZIP", "Snappy", "ZSTD"}]

"CompressionStrategy"  (2)

By default, "Speed" value of "CompressionStrategy" is used:

Wolfram Language code: tabular = Import["ExampleData/USstates.orc"]; AssociationMap[(FileSize@Export["out.orc", tabular, "CompressionStrategy" -> "Speed", "Compression" -> #])&, {"LZ4", "GZIP", "Snappy", "ZSTD"}]

Use "Size" compression strategy:

Wolfram Language code: tabular = Import["ExampleData/USstates.orc"]; AssociationMap[(FileSize@Export["out.orc", tabular, "CompressionStrategy" -> "Size", "Compression" -> #])&, {"LZ4", "GZIP", "Snappy", "ZSTD"}]