DeFi Data - S3
We offer Amazon S3 bulk downloads to retrieve massive historical datasets for select data types, delivered in Apache Parquet format.
DeFi Datasets
DeFi Data Fields & Descriptions
Ethereum - Protocol Events (Aave v2 & v3)
| Field | Description |
|---|---|
| account | The EOA that triggered this event |
| action | The event that the EOA triggered in the smart contract |
| amountNative | The amount of the asset in native units, normalized with the asset's decimals |
| amountUSD | The amount of the asset in US dollars |
| assetId | The smart contract address of the asset |
| assetSymbol | The human readable, abbreviated name of the asset |
| blockNumber | The integer value identifying the block |
| borrowRate | The interest rate for borrowing the asset |
| borrowRateMode | Indicates whether the borrowRate is stable or variable |
| caller | |
| collateralAmountNative | The amount of the asset in native units, normalized with the asset's decimals |
| collateralAmountUSD | The amount of the asset in US dollars |
| collateralAssetId | The smart contract address of the asset. This is the asset that was used as backing for the borrowing activity of liquidatee |
| collateralAssetSymbol | The human readable, abbreviated name of the asset |
| initiator | |
| liquidatee | The EOA being liquidated because they are under-collateralized on their borrowed amount |
| liquidator | The EOA that is triggering the liquidation on liquidatee |
| logIndex | |
| market | |
| marketId | |
| principalAmountNative | The amount of the asset in native units, normalized with the asset's decimals |
| principalAmountUSD | The amount of the asset in US dollars |
| principalAssetId | The smart contract address of the asset. This is the asset that was borrowed by the liquidatee |
| principalAssetSymbol | The human readable, abbreviated name of the asset |
| profitUSD | The amount in US dollars that the liquidator earned from triggering a liquidation |
| repayer | The EOA that repaid the load |
| reserveAsCollateralEnabled | Indicates if assets deposited into the smart contract can be used as collateral |
| target | |
| timestamp | Indicates the datetime or epoch milliseconds of when the event took place |
| totalFee | The fee for the FlashLoan |
| transactionHash | The unique identifier of the transaction indicating that the transaction was validated and added to the block |
Ethereum - Protocol Events (Compound v2 & MakerDAO)
| Field | Description |
|---|---|
| account | The EOA that triggered this event |
| action | The human readable name of the event that the EOA triggered in the smart contract |
| amountNative | The amount of the asset in native units, normalized with the asset's decimals |
| amountUSD | The amount of the asset in US dollars |
| assetId | The smart contract address of the asset |
| assetSymbol | The human readable, abbreviated name of the asset |
| blockNumber | The integer value identifying the block |
| borrowRate | The interest rate for borrowing the asset |
| borrowRateMode | Indicates whether the borrowRate is stable or variable |
| collateralAmountNative | The amount of the asset in native units, normalized with the asset's decimals |
| collateralAmountUSD | The amount of the asset in US dollars |
| collateralAssetId | The smart contract address of the asset. This is the asset that was used as backing for the borrowing activity of liquidatee |
| collateralAssetSymbol | The human readable, abbreviated name of the asset |
| liquidatee | The EOA being liquidated because they are under-collateralized on their borrowed amount |
| liquidator | The EOA that is triggering the liquidation on liquidatee |
| logIndex | |
| market | |
| marketId | |
| principalAmountNative | The amount of the asset in native units, normalized with the asset's decimals |
| principalAmountUSD | The amount of the asset in US dollars |
| principalAssetId | The smart contract address of the asset. This is the asset that was borrowed by the liquidatee |
| principalAssetSymbol | The human readable, abbreviated name of the asset |
| profitUSD | The amount in US dollars that the liquidator earned from triggering a liquidation |
| timestamp | Indicates the datetime or epoch milliseconds of when the event took place |
| totalFee | The fee for the FlashLoan |
| transactionHash | The unique identifier of the transaction indicating that the transaction was validated and added to the block |
DEX - Trades
| Field | Description |
|---|---|
| exchange | The name of the exchange. |
| timestamp | Timestamp when Amberdata received the data. |
| timestampNanoseconds | The nanosecond part of the timestamp where applicable. |
| isBuy | Indicates the direction of the trade: - true means buy the base, sell the quote- false means sell the base, buy the quote |
| price | The actual price at which the asset was traded (including slippage, but not fees) |
| volume | The total amount of that asset that was traded. |
| tradeId | The exchange provided id of the trade. |
| logIndex | The index of the log within the transaction which included this trade event. |
| pairAddress | The address of the pair. |
| amountInBase | The amount of the Base asset accepted in the trade. |
| amountInQuote | The amount of the Quote asset accepted in the trade. |
| amountOutBase | The amount of the Base asset returned in the trade. |
| amountOutQuote | The amount of the Quote asset accepted in the trade. |
| fromAddress | The address which started the trade, ie the sender |
| toAddress | The recipient of the trade, ie the receiver |
DEX - Liquidity
| Field | Description |
|---|---|
| exchangeName | The name of the exchange. |
| exchangeId | The address of the exchange/pool. |
| pair | The common name for the pair - name is not unique, use pairAddress instead. |
| pairNormalized | The internal name for the pair - name is not unique, use pairAddress instead. |
| pairAddress | The address of the pair. |
| baseAddress | The address of the first underlying assert behind the pair. |
| quoteAddress | The address of the last underlying assert behind the pair. |
| address | The address of the asset for which this liquidity events is for (would be one of base or quote address). |
| timestamp | The timestamp associated with this record. |
| transactionHash | The hash of the transaction which included this liquidity event. |
| transactionIndex | The index of the transaction which included this liquidity event. |
| logIndex | The index of the log within the transaction which included this liquidity event. |
| amount | The new amount of the underlying asset after this liquidity event. |
| liquidityPrice | The new price of the underlying asset after this liquidity event. |
| timestampNanoseconds | The nanosecond part of the timestamp where applicable. |
FAQs
Why is delivery via S3 important?
- With data in S3, our customers can bulk download historical data in an analytics friendly format. This allows them to dig deep into the data and perform their own proprietary research and test trading strategies without being limited by our REST API throughput.
How does a customer get access to these datasets?
- Customers will need to have their own AWS credentials in which we will provision for S3 access. If you are interested in downloading data via S3, please contact your Account Executive.
Why Parquet format instead of a GZIP compressed JSON file?
- Parquet is a columnar storage format for structured data that is optimized for querying and analysis. In Parquet format, data is stored in columns rather than rows, allowing for more efficient compression and encoding of data. This can result in significant performance improvements for analytical workloads that involve large datasets and complex queries. Parquet is widely used in big data environments for data warehousing, analytics, and machine learning applications and can be easily integrated into existing data pipelines.
How do I download a parquet sample file and open it to see which fields are returned?
- If you only want to see the fields, simply download the sample parquet file, load it as a pandas dataframe in Python and use dataframe.dtypes, that'll give you a quick output of the field types. Here is the code available for you to try out:
#Import the pandas library
import pandas as pd
# Replace 'your_parquet_file.parquet' with the path to your Parquet file
parquet_file = 'your_parquet_file.parquet'
# Load the Parquet file as a pandas DataFrame
df = pd.read_parquet(parquet_file)
# Display the data types of the DataFrame
print(df.dtypes)
- Now if you wanted to actually read the parquet data, once you've downloaded the sample parquet file, you can run the following Python code:
#Import the pandas library
import pandas as pd
# Replace 'your_parquet_file.parquet' with the path to your Parquet file
parquet_file = 'your_parquet_file.parquet'
pd.read_parquet(parquet_file, engine='pyarrow')
Updated 6 months ago