Market Datasets

Market Type	Feature Type	Sample Files
Spot	Order Book Snapshots	Download
Spot	Order Book Events	Download
Spot	Tickers	Download
Spot	Trades	Download
Spot	OHLCV	Download (minutely) Download (hourly) Download (daily)
Futures	Funding Rates	Download
Futures	Insurance Funds	Download
Futures	Liquidations	Download
Futures	Long/Short Ratio	Download (minutely) Download (hourly) Download (daily)
Futures	Order Book Snapshots	Download
Futures	Order Book Events	Download
Futures	Open Interest	Download
Futures	Tickers	Download
Futures	Trades	Download
Options	Liquidations	Download
Options	Open Interest	Download
Options	Order Book Snapshots	Download
Options	Order Book Updates	Download
Options	Tickers	Download
Options	Trades	Download

Market Data Fields & Descriptions

Spot - Order Book Snapshots

Field	Description
exchange	The name of the exchange.
pair	The name of the asset pair.
exchangeTimestamp	The time at which the order book snapshot took place.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
isBid	Indicates if the order is a bid or ask: true for a bid and false for an ask.
timestamp	The time at which the order book snapshot took place.
receivedTimestamp	Timestamp when Amberdata received the order book snapshot.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
metadata	The metadata associated with the order book data.
sequence	The sequence number provided by the exchange (equal to null if it is not provided by the exchange).
data	The order book data corresponding to the columns fields.
maxPrice	The maximum price for the asset pair. Any buy orders you submit higher than this price, will be clamped to this maximum.
minPrice	The minimum price for the asset pair. Any sell orders you submit lower than this price will be clamped to this minimum.

Spot - Order Book Events

Field	Description
exchange	The name of the exchange.
pair	The name of the asset pair.
exchangeTimestamp	The time at which the order book event took place.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
isBid	Indicates if the order is a bid or ask: true for a bid and false for an ask.
receivedTimestamp	Timestamp when Amberdata received order book event.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
timestamp	Timestamp when Amberdata received order book event.
metadata	The metadata associated with the order book data.
sequence	The sequence number provided by the exchange (equal to null if it is not provided by the exchange).
data	The order book data corresponding to the columns fields.

Spot - Tickers

Field	Description
exchange	The name of the exchange.
pair	The name of the asset pair.
timestamp	The time at which the event occurred.
timestampNanoseconds	The nanosecond part of the timestamp where applicable.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	Timestamp when Amberdata received the ticker data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
metadata	The metadata associated with the ticker data.
sequence	The sequence number provided by the exchange (equal to null if it is not provided by the exchange).
ask	The ask price of the market pair.
askVolume	It represents the requested order size of all best asks.
bid	The bid price of the market pair.
bidVolume	It represents the requested order size of all best bids.
last	The last price of the market pair.
mid	The mid price of the market pair.

Spot - Trades

Field	Description
exchange	The name of the exchange.
pair	The name of the asset pair.
exchangeTimestamp	The time at which the trade took place.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable. NOTE: This is an optional field whose value may or may not be provided by the exchanges.
tradeId	The exchange provided id of the trade.
receivedTimestamp	The time Amberdata received the trade data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
metadata	The metadata associated with the trade data.
isBuySide	Indicates if the trade is a buy or sell: true for a buy and false for a sell.
price	The price at which the asset was traded.
quoteSize	Quote size at the moment of trade NOTE: This is an optional field whose value may or may not be provided by the exchanges.
size	The total amount of that asset that was traded.

Futures - Funding Rates

Field	Description
exchange	The name of the exchange.
instrument	The name of the instrument.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	Timestamp when Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
fundingInterval	The funding interval for which data is available.
fundingRate	The funding rate for which data is available.
nextFundingRate	The next funding rate for which data is available.
nextFundingTime	The next funding time for which data is available.

Futures - Insurance Funds

Field	Description
exchange	The name of the exchange.
instrument	The name of the instrument.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	Timestamp when Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
fund	The fund of insurance fund.
underlying	The underlying asset for the instrument.

Futures - Liquidations

Field	Description
exchange	The name of the exchange.
instrument	The name of the instrument.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
id
receivedTimestamp	Timestamp when Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
timestamp	The time at which the liquidation occurred.
action
orderId
price	The price of the instrument at the time of the liquidation.
side	The direction of the trade.
status	The status of the liquidation.
timeInForce	How long the order is to remain active before it is executed or expires, for example: - IOC: immediate-or-cancel - FOK: fill-or-kill - GTC: good-'till-canceled - etc.
type	The type of liquidation.
volume

Futures - Long/Short Ratio

Field	Description
exchange	The name of the exchange.
instrument	The name of the instrument.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	Timestamp when Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
period	The number of the period.
longAccount	The long account number ratio of all traders
ratio	The long/short account number ratio of all traders
shortAccount	The short account number ratio of all traders

Futures - Order Book Updates

Field	Description
exchange	The name of the exchange.
instrument	The name of the instrument.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
isBid	`true` if the order is a bid, `false` otherwise.
receivedTimestamp	Timestamp when Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
timestamp	The time at which the event occurred.
sequence	The sequence number (equal to `null` if it is not provided by the exchange).
data	The order book data corresponding to the columns fields, aggregated by exchange.
status

Futures - Open Interest

Field	Description
exchange	The name of the exchange.
instrument	The name of the instrument.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	Timestamp when Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
type	The type of instrument.
value	The total outstanding number of contracts.

Futures - Tickers

Field	Description
exchange	The name of the exchange.
instrument	The name of the instrument.
timestamp	Timestamp when Amberdata received the data.
timestampNanoseconds	The nanosecond part of the timestamp.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	Timestamp when Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
sequence	The sequence number (equal to `null` if it is not provided by the exchange).
ask	The ask price for instrument.
askVolume	It represents the requested order size of all best asks.
baseVolume
bid	The bid price for instrument.
bidVolume	It represents the requested order size of all best bids.
last	The last price for instrument.
mid	The mid price for instrument.
quoteVolume
markPrice
lastVolume

Futures - Trades

Field	Description
exchange	The name of the exchange.
instrument	The name of the instrument.
exchangeTimestamp	The time at which the event occurred.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
tradeId	The exchange provided id of the trade.
receivedTimestamp	Timestamp when Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
timestamp	The time at which the event occurred.
isBuySide	`true` if the trade is a buy, `false` otherwise.
price	The price at which the asset was traded.
quoteVolume
volume	The total amount of that asset that was traded.
sequence
isLiquidation

Options - Liquidations

Field	Description
exchange
instrument
exchangeTimestamp
exchangeTimestampNanoseconds
id
volume
sequence
receivedTimestamp
receivedTimestampNanoseconds
metadata
action
orderId
price
side
status
timeInForce
type

Options - Open Interest

Field	Description
exchange
instrument
exchangeTimestamp
exchangeTimestampNanoseconds
receivedTimestamp
receivedTimestampNanoseconds
type
value

Options - Order Book Snapshots

Field	Description
exchange
instrument
exchangeTimestamp
exchangeTimestampNanoseconds
sequence
receivedTimestamp
receivedTimestampNanoseconds
metadata
stats
state
asks
askIv
bids
bidIv
bestAskAmount
bestAskPrice
bestBidAmount
bestBidPrice
estimatedDeliveryPrice
greeks
indexPrice
interestRate
lastPrice
lastPrice
markIv
markPrice
maxPrice
minPrice
openInterest
underlyingIndex
underlyingPrice

Options - Order Book Updates

Field	Description
exchange
instrument
exchangeTimestamp
exchangeTimestampNanoseconds
isBid
receivedTimestamp
receivedTimestampNanoseconds
timestamp
metadata
sequence
data
status

Options - Tickers

Field	Description
exchange
instrument
exchangeTimestamp
exchangeTimestampNanoseconds
sequence
receivedTimestamp
receivedTimestampNanoseconds
stats
state
ask
askIv
askVolume
baseVolume
bid
bidIv
bidVolume
quoteVolume
lastVolume
estimatedDeliveryPrice
greeks
indexPrice
interestRate
mid
last
markIv
markPrice
maxPrice
minPrice
openInterest
settlementPrice
underlyingIndex
underlyingPrice
open24H
high24H
low24H

Options - Trades

Field	Description
exchange
instrument
exchangeTimestamp
exchangeTimestampNanoseconds
tradeId
isBuySide
receivedTimestamp
receivedTimestampNanoseconds
metadata
sequence
price
quoteSize
size
tickDirection
markPrice
iv
indexPrice

CME Only

Futures - Open Interest (CME only)

Field	Description
exchange	The name of the exchange.
symbol	Instrument Name or Symbol.
exchangeTimestamp	Timestamp provided by CME.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	The time Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
sentTime	Time CME MDP gateway sends the message (UTC).
tradeStatistics	Object - view more details here
tradeStatistics.openInterest	The total open interest for the market at the close of the prior trading session.
tradeStatistics.clearedVolume	Cleared volume quantity.
tradeStatistics.settlementFinal	Final settlement price
tradeStatistics.settlementPrice	Settlement price.
tradeStatistics.openInterestDate	Open interest trade date Format: "YYYY-MM-DD"
tradeStatistics.settlementActual	Actual settlement price
tradeStatistics.clearedVolumeDate	Cleared volume date.
tradeStatistics.settlementRounded	Rounded settlement price
tradeStatistics.settlementPriceDate	Date of trade session corresponding to a statistic entry. Format: "YYYY-MM-DD"
tradeStatistics.openInterestTimestamp	Open interest update time.
tradeStatistics.clearedVolumeTimestamp	Cleared volume time.
tradeStatistics.settlementPriceTimestamp	Time of trade session corresponding to a statistic entry. Format: "yyyy-MM-dd'T'HH:mm:ss.n"
instrument	Object - view more details here
instrument.id	Unique instrument ID as qualified by the exchange per market segment. The unique instrument ID value will not be reused until the next trade date following an instrument expiration or deletion.
instrument.symbol	Instrument Name or Symbol.
instrument.periodCode	This field provides the calendar month reflected in the instrument symbol. Format YYYYMM (e.g., 201912) For futures spreads, this field contains the first leg's calendar month reflected in the instrument symbol.
instrument.exchangeMic	Exchange used to identify a security.
instrument.productCode	String field that indicates the underlying asset code (Product Code). Example: SR1 (SOFR), ES (E-Minis).
instrument.productType	Product Type
instrument.productGroup	Product Group Code.
instrument.marketSegmentId	Identifies the market segment. Populated for all CME Globex instruments.
instrument.definitionSource	Identifies user-defined instruments. If the tag is not present, the instrument is not user-defined.

Futures - Tickers (CME only)

Field	Description
exchange	The name of the exchange.
symbol	Instrument Name or Symbol.
exchangeTimestamp	Timestamp provided by CME.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	The time Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
sentTime	Time CME MDP gateway sends the message (UTC).
tradingStatus	Identifies the trading status applicable to the instrument or product group.
instrument	Object - view more details here
instrument.id	Unique instrument ID as qualified by the exchange per market segment. The unique instrument ID value will not be reused until the next trade date following an instrument expiration or deletion.
instrument.symbol	Instrument Name or Symbol.
instrument.periodCode	This field provides the calendar month reflected in the instrument symbol. Format YYYYMM (e.g., 201912) For futures spreads, this field contains the first leg's calendar month reflected in the instrument symbol.
instrument.exchangeMic	Exchange used to identify a security.
instrument.productCode	String field that indicates the underlying asset code (Product Code). Example: SR1 (SOFR), ES (E-Minis).
instrument.productType	Product Type
instrument.productGroup	Product Group Code.
instrument.marketSegmentId	Identifies the market segment. Populated for all CME Globex instruments.
instrument.definitionSource	Identifies user-defined instruments. If the tag is not present, the instrument is not user-defined.
askLevel	Object - view more details here
askLevel.qty	Order quantity.
askLevel.price	Price of the MD Entry.
askLevel.orderCnt	Aggregate number of orders at the given price level.
askLevel.lastUpdateTime	Last update time for ask price Format: "yyyy-MM-dd'T'HH:mm:ss.n"
bidLevel	Object - view more details here
bidLevel.qty	Order quantity
bidLevel.price	Price of the MD Entry.
bidLevel.orderCnt	Aggregate number of orders at the given price level.
bidLevel.lastUpdateTime	Last update time for bid price. Format: "yyyy-MM-dd'T'HH:mm:ss.n"

Futures - Trades (CME only)

Field	Description
exchange	The name of the exchange.
symbol	Instrument Name or Symbol.
exchangeTimestamp	Timestamp provided by CME.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	The time Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
sentTime	Time CME MDP gateway sends the message (UTC).
tradeSummary	Object - view more details here
tradeSummary.orderQty	Object - view more details here
tradeSummary.orderQty.orderId	Unique ID assigned by CME Globex to identify orders.
tradeSummary.orderQty.lastOrdQty	Quantity of trade
tradeSummary.tradeQty	Total traded quantity.
tradeSummary.tradePrice	Trade price.
tradeSummary.aggressorSide	Indicates which side is aggressor of the trade. If there is a zero value present, then there is no aggressor. Trades without aggressors occur: - at Market Open - after a Pre-Open or after a Pause - when the event includes customer order participation in a trade with a CME Globex-generated implied bid or offer.
tradeSummary.mdTradeEntryId	Common Trade ID that links each trade execution.
tradeSummary.tradeOrderCount	Identifies the total number of non-implied orders per instrument that participated in a match event.
tradeSummary.tradeUpdateAction	Trade market data update action.
instrument	Object - view more details here
instrument.id	Unique instrument ID as qualified by the exchange per market segment. The unique instrument ID value will not be reused until the next trade date following an instrument expiration or deletion.
instrument.symbol	Instrument Name or Symbol.
instrument.periodCode	This field provides the calendar month reflected in the instrument symbol. Format YYYYMM (e.g., 201912) For futures spreads, this field contains the first leg's calendar month reflected in the instrument symbol.
instrument.exchangeMic	Exchange used to identify a security.
instrument.productCode	String field that indicates the underlying asset code (Product Code). Example: SR1 (SOFR), ES (E-Minis).
instrument.productType	Product Type
instrument.productGroup	Product Group Code.
instrument.marketSegmentId	Identifies the market segment. Populated for all CME Globex instruments.
instrument.definitionSource	Identifies user-defined instruments. If the tag is not present, the instrument is not user-defined.

Options - Open Interest (CME only)

Field	Description
exchange	The name of the exchange.
symbol	Instrument Name or Symbol.
exchangeTimestamp	Timestamp provided by CME.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	The time Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
sentTime	Time CME MDP gateway sends the message (UTC).
tradeStatistics	Object - view more details here
tradeStatistics.openInterest	The total open interest for the market at the close of the prior trading session.
tradeStatistics.clearedVolume	Cleared volume quantity.
tradeStatistics.settlementFinal	Final settlement price
tradeStatistics.settlementPrice	Settlement price.
tradeStatistics.openInterestDate	Open interest trade date Format: "YYYY-MM-DD"
tradeStatistics.settlementActual	Actual settlement price
tradeStatistics.clearedVolumeDate	Cleared volume date.
tradeStatistics.settlementRounded	Rounded settlement price
tradeStatistics.settlementPriceDate	Date of trade session corresponding to a statistic entry. Format: "YYYY-MM-DD"
tradeStatistics.openInterestTimestamp	Open interest update time.
tradeStatistics.clearedVolumeTimestamp	Cleared volume time.
tradeStatistics.settlementPriceTimestamp	Time of trade session corresponding to a statistic entry. Format: "yyyy-MM-dd'T'HH:mm:ss.n"
instrument	Object - view more details here
instrument.id	Unique instrument ID as qualified by the exchange per market segment. The unique instrument ID value will not be reused until the next trade date following an instrument expiration or deletion.
instrument.symbol	Instrument Name or Symbol.
instrument.periodCode	The calendar month reflected in the instrument symbol. Format YYYYMM (e.g., 201912) For futures spreads, this field contains the first leg's calendar month reflected in the instrument symbol.
instrument.exchangeMic	Exchange used to identify a security.
instrument.productCode	String field that indicates the underlying asset code (Product Code). Example: SR1 (SOFR), ES (E-Minis).
instrument.productType	Product Type.
instrument.productGroup	Product Group Code.
instrument.marketSegmentId	Identifies the market segment. Populated for all CME Globex instruments.
instrument.definitionSource	Identifies user-defined instruments. If the tag is not present, the instrument is not user-defined.

Options - Tickers (CME only)

Field	Description
exchange	The name of the exchange.
symbol	Instrument Name or Symbol.
exchangeTimestamp	Timestamp provided by CME.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	The time Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
sentTime	Time CME MDP gateway sends the message (UTC).
tradingStatus	Identifies the trading status applicable to the instrument or product group.
instrument	Object - view more details here
instrument.id	Unique instrument ID as qualified by the exchange per market segment. The unique instrument ID value will not be reused until the next trade date following an instrument expiration or deletion.
instrument.symbol	Instrument Name or Symbol.
instrument.putOrCall	Indicates whether an option instrument is a put or call.
instrument.periodCode	This field provides the calendar month reflected in the instrument symbol. Format YYYYMM (e.g., 201912) For futures spreads, this field contains the first leg's calendar month reflected in the instrument symbol.
instrument.exchangeMic	Exchange used to identify a security.
instrument.productCode	An exchange-specific code assigned to a group of related securities, which are concurrently affected by market events.
instrument.productType	Product type
instrument.strikePrice	The strike price.
instrument.productGroup	Product Group Code.
instrument.marketSegmentId	Identifies the market segment. Populated for all CME Globex instruments.
instrument.definitionSource	Identifies user-defined instruments. If the tag is not present, the instrument is not user-defined.
instrument.underlyingSymbol	Underlying Instrument Symbol (Contract Name) * this value will be the same as that contained in Leg Instrument's Security Definition Tag 55-Symbol.
askLevel	Object - view more details here
askLevel.qty	Order quantity.
askLevel.price	Price of the MD Entry.
askLevel.orderCnt	Aggregate number of orders at the given price level.
askLevel.lastUpdateTime	Last update time for ask price Format: "yyyy-MM-dd'T'HH:mm:ss.n"
bidLevel	Object - view more details here
bidLevel.qty	Order quantity
bidLevel.price	Price of the MD Entry.
bidLevel.orderCnt	Aggregate number of orders at the given price level.
bidLevel.lastUpdateTime	Last update time for bid price. Format: "yyyy-MM-dd'T'HH:mm:ss.n"

Options - Trades (CME only)

Field	Description
exchange	The name of the exchange.
symbol	Instrument Name or Symbol.
exchangeTimestamp	Timestamp provided by CME.
exchangeTimestampNanoseconds	The nanosecond part of the exchangeTimestamp where applicable.
receivedTimestamp	The time Amberdata received the data.
receivedTimestampNanoseconds	The nanosecond part of the receivedTimestamp.
sentTime	Time CME MDP gateway sends the message (UTC).
tradeSummary	Object - view more details here
tradeSummary.orderQty
tradeSummary.orderQty.orderId	Unique ID assigned by CME Globex to identify orders.
tradeSummary.orderQty.lastOrdQty	Quantity of trade
tradeSummary.tradeQty	Total traded quantity.
tradeSummary.tradePrice	Trade price.
tradeSummary.aggressorSide	Indicates which side is aggressor of the trade. If there is a zero value present, then there is no aggressor. Trades without aggressors occur: at Market Open after a Pre-Open or after a Pause when the event includes customer order participation in a trade with a CME Globex-generated implied bid or offer.
tradeSummary.mdTradeEntryId	Common Trade ID that links each trade execution.
tradeSummary.tradeOrderCount	Identifies the total number of non-implied orders per instrument that participated in a match event.
tradeSummary.tradeUpdateAction	Trade market data update action.
instrument	Object - view more details here
instrument.id	Unique instrument ID as qualified by the exchange per market segment. The unique instrument ID value will not be reused until the next trade date following an instrument expiration or deletion.
instrument.symbol	Instrument Name or Symbol.
instrument.putOrCall	Indicates whether an option instrument is a put or call.
instrument.periodCode	The calendar month reflected in the instrument symbol. For futures spreads, this field contains the first leg's calendar month reflected in the instrument symbol.
instrument.exchangeMic	Exchange used to identify a security.
instrument.productCode	String field that indicates the underlying asset code (Product Code). Example: SR1 (SOFR), ES (E-Minis).
instrument.productType	Product Type.
instrument.strikePrice	The strike price.
instrument.productGroup	Product Group Code.
instrument.marketSegmentId	Identifies the market segment. Populated for all CME Globex instruments.
instrument.definitionSource	Identifies user-defined instruments. If the tag is not present, the instrument is not user-defined.
instrument.underlyingSymbol	Underlying Instrument Symbol (Contract Name) * this value will be the same as that contained in Leg Instrument's Security Definition Tag 55-Symbol.

FAQs

Why is delivery via S3 important?

With data in S3, our customers can bulk download historical data in an analytics friendly format. This allows them to dig deep into the data and perform their own proprietary research and test trading strategies without being limited by our REST API throughput.

How does a customer get access to these datasets?

Customers will need to have their own AWS credentials in which we will provision for S3 access. If you are interested in downloading data via S3, please contact your Account Executive.

Why Parquet format instead of a GZIP compressed JSON file?

Parquet is a columnar storage format for structured data that is optimized for querying and analysis. In Parquet format, data is stored in columns rather than rows, allowing for more efficient compression and encoding of data. This can result in significant performance improvements for analytical workloads that involve large datasets and complex queries. Parquet is widely used in big data environments for data warehousing, analytics, and machine learning applications and can be easily integrated into existing data pipelines.

How do I download a parquet sample file and open it to see which fields are returned?

If you only want to see the fields, simply download the sample parquet file, load it as a pandas dataframe in Python and use dataframe.dtypes, that'll give you a quick output of the field types. Here is the code available for you to try out:

#Import the pandas library
import pandas as pd

# Replace 'your_parquet_file.parquet' with the path to your Parquet file
parquet_file = 'your_parquet_file.parquet'

# Load the Parquet file as a pandas DataFrame
df = pd.read_parquet(parquet_file)

# Display the data types of the DataFrame
print(df.dtypes)

Now if you wanted to actually read the parquet data, once you've downloaded the sample parquet file, you can run the following Python code:

#Import the pandas library
import pandas as pd   

# Replace 'your_parquet_file.parquet' with the path to your Parquet file
parquet_file = 'your_parquet_file.parquet' 

pd.read_parquet(parquet_file, engine='pyarrow')