Updating Data in Real Time

Datamule now updates XBRL, and all SEC XML tables (Insider transactions, etc) in real time.

This is how it works:

  1. Datamule detects a new SEC filing.
  2. AWS Lambda worker downloads filing.
  3. Lambda worker processes filing. This may create many small files, one for each table.
  4. XBRL or XML files uploaded to S3.
  5. At 5:00 AM UTC, files are combined into one master dataset file.

This is how it is exposed to users:

datasets.download("simple_xbrl", filename="simple_xbrl_delta.parquet", from_storage="delta")

Downloads just the delta files from S3, then combines them into one parquet file.

datasets.download("simple_xbrl", filename="simple_xbrl_delta.parquet", from_storage="daily")

Downloads just the daily updated dataset.

datasets.download("simple_xbrl", filename="simple_xbrl_delta.parquet")

Downloads just delta and daily files as one parquet file.

Note on S3 Costs

This approach creates a lot of small files in S3. There are ballpark 500k SEC filings per day. Many of these filings are only a few kb, under S3's minimum cost limit. Many filings end up being multiple tables, of which each table gets one S3 PUT. This means ballpark, 50 million PUTS per year.

This is a case where using S3 Express for deltas, due to cheaper PUT costs, is actually cheaper than S3. Especially when combined with a TTL of one month.