Standarding SEC SGML Header Metadata
SEC Filings are submitted in standardized generalized markup language (SGML). They contain metadata about the filing at the top. This metadata is traditionally represented in nested form, often stored in json files.
More information on SGML.
Why this is important
Filings are generally correct. If a filing claims it is a 10-K, it almost surely is. However, SEC endpoints claims of what type a filing is are often wrong.
Downloading every metadata file in a few hours
2:16 AM - 7:31 AM
from datamulehub import sec_filings_archive
for year in range(1993,2027):
sec_filings_archive.download_tar(document_type='metadata',filing_date=(f'{year}-01-01',f'{year}-12-31'), max_workers=1000, output_dir=f"metadata/{year}")Conversion to tabular form and standardization of key values
I gave GPT 6.1 Sol on Codex instructions to standardize the data, then checked in a few times. Since I had run out of OpenAI credits (generously given through the startups program), I bought a plus sub. Sol 6.1 did well - I upgraded to Pro. Well done, OAI.
Data
Code to standardize is opensourced on GitHub.