Exploring Document Parsing Router Architecture

There are about 200,000 10-Ks. Assume 100 pages per root form. Commercial document parsers cost ~$0.01/page. So parsing every 10-K root form would cost ~$200,000.

That's a bit expensive. Datamule will soon be releasing an algorithmic document parser that can parse SEC html and most pdfs well. Parsing every 10-K should cost around a thousandth of a cent. Parsing the entire corpus should cost maybe a cent. The entire corpus will be released in parquet format, preserving hierarchy and tables, for easy download.

This works well because SEC filings have specific constraints that make them easier to parse. These constraints often exist in documents outside the SEC corpus. The parser has been engineered to be able to take different rulesets for different documents. This should extend its usefulness to a wide range of documents besides the SEC corpus. An earlier parser has already been adopted for non SEC documents via its mapping dict functionality.

However, this will also fail on more complex documents. A possible solution is a document router. This will be important as Datamule expand beyond the SEC corpus.

Document Router

Take in document -> send to cheapest parser capable of producing good output.

How to actually do this

If you can classify correctness, then a workflow could be:

  1. Get document
  2. Throw document through algorithmic parser (instant)
  3. Classify (instant)
  4. If bad: route to LLM parser (slow)

Or:

  1. Get document
  2. Throw document through algorithmic parser (instant)
  3. Classify (instant)
  4. Classifer returns bad, with suggestion to tweak algorithmic parser's rules.
  5. Fine tuned LLM outputs new rules (~200ms)
  6. Throw document through algorithmic parser (instant)
  7. Classify (instant)
  8. If bad: route to LLM parser (slow)

For this workflow, the classifier has three states: {good, bad, tweak}

The algorithmic parser

Processes GB/s on a single thread. Takes a modular set of rules to allow it to generalize for different documents.

Classifier

Questions to explore:

  1. Can we identify if parsing was good/bad?
  2. Can we identify if parsing would likely be good if algorithm tweaked?
  3. Can (1) and (2) be done via algorithm or does it need an LLM.

LLM Parser

Questions to explore:

  1. Is it more efficient to have a LLM output the entire document via token, or to allow it access to e.g. underlying HTML or PDF stream with the ability to use tool calls so that it only needs to output offsets that are then mapped to the final file.
  2. Can we fit a decent LLM parser on on-demand compute that spins up quickly?
  3. Can we use an inference provider, with our own sandbox?
  4. Can we parallelize decode work?

Hosting this efficiently for public use via API

Algorithmic Parser

Algorithmic Classifier

Can we fit sufficient intelligence with a few GBs?

On demand only works if we can make it fast enough. Comparing against commercial parsers, it seems that median wait time is around 7s. So let's take as given, that we need to load the model and parse the document within that time.

If weights can only be loaded at ~1-5GB/s that limits model weights to around 7GB. That should fit on a L4 GPU, which costs ~$0.67/hour.

So the question becomes: "Can we fit enough intelligence on a L4 GPU to classify and parse documents".

If so, we can use on demand inference, which scales well.

We can't fit sufficient intelligence on a few GBs.

If we can't do this, then we have to use a larger model. Which means the weights alone will take too much time to load to use on demand compute. This means to provide a useful service, we will have to run models all the time. Which is a bit more expensive, at low use.

Expectations

I think sufficient intelligence for a classifier model should fit on a few GBs. I think sufficient intelligence for document parsing will not. I expect the final router to resemble: