Updating Data in Real Time

Datamule now updates XBRL, and all SEC XML tables (Insider transactions, etc) in real time.

This is how it works:

  1. Datamule detects a new SEC filing.
  2. AWS Lambda worker downloads filing.
  3. Lambda worker processes filing. This may create many small files, one for each table.
  4. XBRL or XML files uploaded to S3.
  5. At 06:00 UTC Tuesday–Saturday, files are combined into one master dataset file.

For daily downloads, we recommend pulling after 09:00 UTC.

This is how it is exposed to users:

See the dataset documentation for download and query options, and Transfer to your Cloud for keeping your own copy updated.

from datamulehub import datasets

datasets.download("simple_xbrl", filename="simple_xbrl_delta.parquet", from_storage="delta")

Downloads just the delta files from S3, then combines them into one parquet file.

from datamulehub import datasets

datasets.download("simple_xbrl", filename="simple_xbrl_daily.parquet", from_storage="daily")

Downloads just the daily updated dataset.

from datamulehub import datasets

datasets.download("simple_xbrl", filename="simple_xbrl.parquet")

Downloads just delta and daily files as one parquet file.

Note on S3 Costs

This approach creates a lot of small files in S3. There are ballpark 500k SEC filings per day. Many of these filings are only a few kb, under S3's minimum cost limit. Many filings end up being multiple tables, of which each table gets one S3 PUT. This means ballpark, 50 million PUTS per year.

This is a case where using S3 Express for deltas, due to cheaper PUT costs, is actually cheaper than S3. Especially when combined with a TTL of one month.