WOLFRAM

Wolfram Language & System Documentation Center

Parquet (.parquet)

Background & Context

    • Registered MIME types: application/vnd.apache.parquet
    • Efficient, general-purpose, column-oriented data format.
    • Developed by the Apache Software Foundation.
    • Binary file format.
    • Supports multiple compression methods.

Import & Export

Import Elements

  • General Import elements:
  • "Elements" list of elements and options available in this file
    "Summary"summary of the file
    "Rules"list of rules for all available elements
  • Data representation elements:
  • "Data"two-dimensional array
    "Dataset"table data as a Dataset
    "EventSeries"table data as an EventSeries
    "Tabular"a Tabular object
    "TimeSeries"table data as a TimeSeries
  • Import by default uses the "Tabular" element.
  • Subelements for partial data import for the "Tabular" element can take row and column specifications in the form {"Tabular",rows,cols}, where rows and cols can be any of the following:
  • nnth row or column
    -ncounts from the end
    n;;mfrom n through m
    n;;m;;sfrom n through m with steps of s
    {n1,n2,…}specific rows or columns ni
  • Column specifications can also be any of the following:
  • "col"single column "col"
    {col1,col2,…}list of column names coli
  • Data descriptor elements:
  • "ColumnLabels"names of columns
    "ColumnTypes"association with data type for each column
    "Schema"TabularSchema object
  • Metadata elements:
  • "ColumnCount"number of columns stored in file
    "Dimensions"data dimensions
    "RowCount"number of rows stored in file
    "MetaInformation"metadata

Options

  • General Import options:
  • IncludeMetaInformationAllmetadata types to import
    "Schema"Automaticschema used to construct Tabular object
    "TimeColumn"Automaticcolumn to use for times in "EventSeries" and "TimeSeries" elements
  • Possible settings for the "Schema" option include:
  • schemaa complete TabularSchema specification
    propvala schema property and value (see reference page for TabularSchema)
    <|"prop1"val1,…|>an association of schema properties and values
  • General Export options:
  • "Compression"Nonecompression method
    CompressionLevelAutomaticcompression level
  • The following settings for "Compression" are supported:
  • Noneno compression
    "Brotli"Brotli compression
    "GZIP"GZIP compression
    "LZ4"LZ4 compression
    "LZ4Hadoop"LZ4 Hadoop compression
    "Snappy"Snappy compression
    "ZSTD"ZSTD compression

Examples

open all close all

Basic Examples  (3)

Import Tabular object from Parquet file:

Wolfram Language code: Import["ExampleData/USstates.parquet"]

Import the file summary:

Wolfram Language code: Import["ExampleData/USstates.parquet", "Summary"]

Export a Tabular object to Parquet:

Wolfram Language code: tabular = Import["ExampleData/USstates.parquet"];
Wolfram Language code: Export["file.parquet", tabular]

Scope  (3)

Import  (3)

Show all elements available in the file:

Wolfram Language code: Import["ExampleData/USstates.parquet", "Elements"]

By default, a Tabular object is returned:

Wolfram Language code: Import["ExampleData/USstates.parquet"]//TabularQ

Import column types:

Wolfram Language code: Import["ExampleData/USstates.parquet", "ColumnTypes"]

Import Elements  (19)

"ColumnCount"  (1)

Get the number of columns:

Wolfram Language code: Import["ExampleData/USstates.parquet", "ColumnCount"]

"ColumnLabels"  (1)

Read column names:

Wolfram Language code: Import["ExampleData/USstates.parquet", "ColumnLabels"]

"ColumnTypes"  (1)

Import column types:

Wolfram Language code: Import["ExampleData/USstates.parquet", "ColumnTypes"]

"Data"  (3)

Get the data from a file:

Wolfram Language code: Import["ExampleData/USstates.parquet", "Data"]//Short

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Data", 1 ;; 3}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Data", All, {1, 3}}]//Short

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Data", All, {"Name", "Area"}}]//Short

"Dataset"  (3)

Get the data as a Dataset:

Wolfram Language code: Import["ExampleData/USstates.parquet", "Dataset"]

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Dataset", 1 ;; 3}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Dataset", All, {1, 3}}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Dataset", All, {"Name", "Area"}}]

"Dimensions"  (1)

Import data dimensions:

Wolfram Language code: Import["ExampleData/USstates.parquet", "Dimensions"]

"EventSeries"  (1)

Export a Tabular object to a Parquet file:

Wolfram Language code: file = Export["file.parquet", ResourceData["Sample Tabular Data: Sales Data"]]

Import a Parquet file as an EventSeries:

Wolfram Language code: Import[file, "EventSeries"]

Import a single row from a Parquet file:

Wolfram Language code: Import[file, {"EventSeries", 5}]

Import some specific rows from a Parquet file:

Wolfram Language code: Import[file, {"EventSeries", {1, 5, 7}}]

Import the first 10 rows of a Parquet file:

Wolfram Language code: Import[file, {"EventSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import[file, {"EventSeries", All, {"Product", "Date", "Quantity"}}]

"MetaInformation"  (1)

Import metadata:

Wolfram Language code: Import["ExampleData/USstates.parquet", "MetaInformation"]

"RowCount"  (1)

Get the number of rows:

Wolfram Language code: Import["ExampleData/USstates.parquet", "RowCount"]

"Schema"  (1)

Get the TabularSchema object:

Wolfram Language code: Import["ExampleData/USstates.parquet", "Schema"]

"Summary"  (1)

Get the file summary:

Wolfram Language code: Import["ExampleData/USstates.parquet", "Summary"]

"Tabular"  (3)

Get the data from a file as a Tabular object:

Wolfram Language code: Import["ExampleData/USstates.parquet", "Tabular"]

Import only selected rows:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Tabular", 1 ;; 5}]

Import only selected columns:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Tabular", All, {1, 3}}]

Import only selected columns using column names:

Wolfram Language code: Import["ExampleData/USstates.parquet", {"Tabular", All, {"Name", "Area"}}]

"TimeSeries"  (1)

Export a Tabular object to a Parquet file:

Wolfram Language code: file = Export["file.parquet", ResourceData["Sample Tabular Data: Sales Data"]]

Import a Parquet file as a TimeSeries:

Wolfram Language code: Import[file, "TimeSeries"]

Import a single row from a Parquet file:

Wolfram Language code: Import[file, {"TimeSeries", 5}]

Import some specific rows from a Parquet file:

Wolfram Language code: Import[file, {"TimeSeries", {1, 5, 7}}]

Import the first 10 rows of a Parquet file:

Wolfram Language code: Import[file, {"TimeSeries", 1 ;; 10}]

Import only selected columns using column names:

Wolfram Language code: Import[file, {"TimeSeries", All, {"Product", "Date", "Quantity"}}]

Import Options  (3)

IncludeMetaInformation  (1)

By default, all metadata stored in a file is imported and embedded in the Tabular object:

Wolfram Language code: tabular = Import["ExampleData/USstates.parquet"]; tabular["Metadata"]

Do not import metadata:

Wolfram Language code: tabular = Import["ExampleData/USstates.parquet", IncludeMetaInformation -> None]; tabular["Metadata"]

"Schema"  (1)

Export Tabular object to Parquet file:

Wolfram Language code: file = Export["out.parquet", Tabular[Association["RawSchema" -> Association["ColumnProperties" -> Association["A" -> Association["ElementType" -> "String"], "B" -> Association["ElementType" -> "String"]], "KeyColumns" -> None, "Backend" -> "WolframKernel"], "BackendData" -> Association["ColumnData" -> DataStructure["ColumnTable", {{TabularColumn[Association["Data" -> {{0, {0, 11, 22, 33, 44, 55}, "Jan 03 2006Jan 04 2006Jan 05 2006Jan 06 2006Jan 09 2006"}, {}, None}, "ElementType" -> "String"]], TabularColumn[Association[ "Data" -> {{3, {0, 5, 10, 15, 20, 25}, "11.8212.0412.0911.8812.43"}, {}, None}, "ElementType" -> "String"]]}}]]]]];

By default, column labels and their types stored in a file are used when Tabular or Dataset objects are imported:

Wolfram Language code: tabular = Import[file]; tabular["ColumnTypes"]

Use "Schema" option to specify column labels and types:

Wolfram Language code: tabular = Import[file, "Schema" -> {"ColumnKeys" -> {"Date", "Value"}, "ElementType" -> {"Date" -> "Date", "Value" -> "Real32"}}]; tabular["ColumnTypes"]

"TimeColumn"  (1)

Export a Tabular object to a Parquet file:

Wolfram Language code: file = Export["file.parquet", Tabular[Association["RawSchema" -> Association["ColumnProperties" -> Association["Date" -> Association["ElementType" -> TypeSpecifier["Date"]["Integer32", "Day", "Gregorian", None]], "Value" -> Association["ElementType" -> "Real32"]], "KeyColumns" -> None, "Backend" -> "WolframKernel"], "Options" -> {}, "BackendData" -> Association["ColumnData" -> DataStructure["ColumnTable", {{TabularColumn[Association["Data" -> {5, {{NumericArray[{13150, 13151, 13152, 13153, 13156}, "Integer32"], {}, None}}, None}, "ElementType" -> "Date"["Integer32", "Day", "Gregorian", None]]], TabularColumn[Association[ "Data" -> {NumericArray[{11.819999694824219, 12.039999961853027, 12.09000015258789, 11.880000114440918, 12.430000305175781}, "Real32"], {}, None}, "ElementType" -> "Real32"]]}}]]]]];

By default, the time column is selected automatically for "TimeSeries" and "EventSeries" elements:

Wolfram Language code: Import[file, "TimeSeries"]

Use the "TimeColumn" option to specify the time column:

Wolfram Language code: Import[file, "TimeSeries", "TimeColumn" -> "Value"]

Export Options  (4)

"Compression"  (2)

Compression is disabled by default:

Wolfram Language code: tabular = Import["ExampleData/USstates.parquet"]; Export["out.parquet", tabular]//FileSize

Compare supported compression methods:

Wolfram Language code: tabular = Import["ExampleData/USstates.parquet"]; AssociationMap[(FileSize@Export["out.parquet", tabular, "Compression" -> #])&, {"Brotli", "GZIP", "LZ4", "LZ4Hadoop", "Snappy", "ZSTD"}]
Wolfram Language code: BarChart[%, {...}]

CompressionLevel  (2)

By default, Automatic value of CompressionLevel is used. It corresponds to a different default value for each compression method:

Wolfram Language code: tabular = Import["ExampleData/USstates.parquet"]; AssociationMap[(FileSize@Export["out.parquet", tabular, CompressionLevel -> Automatic, "Compression" -> #])&, {"Brotli", "GZIP", "LZ4", "LZ4Hadoop", "Snappy", "ZSTD"}]
Wolfram Language code: BarChart[%, {...}]

Use maximal compression for each method:

Wolfram Language code: tabular = Import["ExampleData/USstates.parquet"]; AssociationMap[(FileSize@Export["out.parquet", tabular, CompressionLevel -> 1.0, "Compression" -> #])&, {"Brotli", "GZIP", "LZ4", "LZ4Hadoop", "Snappy", "ZSTD"}]
Wolfram Language code: BarChart[%, {...}]