RESTWebsocketsamberLensAWS S3SnowflakeGoogle BigQueryAsset Reference & Classification
Changelog
Back to All

Recommended Best Practice: Use S3 Object Paths Instead of Parsing File Names

29 days ago by Amberdata Support

Summary

We are evaluating a potential update to how S3 files are named. However, before making this change, we need to ensure that no customers are relying on parsing file names. Instead, we strongly recommend using the S3 object path to retrieve metadata such as year/month/day/exchange/instrument. This approach will help future-proof your workflows, improve reliability, and ensure continued compatibility with any updates we make.

Current File Name Format:

s3://amberdata-marketdata-daily/futures/funding-rates/year=2024/month=12/day=02/exchange=okex/instrument=1INCH-USD-SWAP/2024-12-02.okex.1INCH-USD-SWAP.00.parquet

Proposed File Name Format:

s3://amberdata-marketdata-daily/futures/funding-rates/year=2024/month=12/day=02/exchange=okex/instrument=1INCH-USD-SWAP/part-00000-6aba62c6-1034-46c5-b9af-ba8abae78986.c000.snappy.parquet

The key difference:

  • Today’s file names contain human-readable metadata (e.g., date, exchange, instrument).
  • The proposed format replaces this with a system-generated ID, requiring customers to extract metadata from the S3 object path instead.

Why This Matters

✅ Future-Proof Your Workflow

  • If you rely on parsing file names, your workflow could break if we implement changes in the future.
  • Object paths offer a more structured and stable way to retrieve metadata.

🚀 More Efficient & Reliable Data Retrieval

  • Object paths remain consistent over time, whereas file names may change.
  • This reduces the risk of unexpected failures due to naming conventions.

🔍 Simpler, More Scalable Data Processing

  • The object path already contains all relevant metadata (year/month/day/exchange/instrument).
  • Avoid unnecessary complexity in data parsing by leveraging structured paths instead of file names.

What You Need to Do

  • If you are already using the S3 object path to retrieve metadata, no action is required.
  • If you are currently parsing file names, we strongly recommend updating your workflow to extract metadata from the object path instead.

We appreciate your cooperation as we assess this potential update. If you have any concerns or need guidance on making this transition, please reach out to our support team at [email protected].