I need to increase processing speed so that backfilling daily histories takes hours, not days or weeks.
At this time, I've taken up a suggestion to build my own URL parsing engine in Rust (using the url and publicsuffix crates) for Python. I've been successful so far. The Rust program builds successfully and I am able to import the new library into python and run it. Results looks reasonable so far. I have a bit more pipeline work to do. I'll share the results when I'm done.