ScrapeGen — Tool for generating Python scrapers

OK, so I was kind of bored last week so thought of coming up something anyway, even if it is useless. So, I allocated a few hours and came up with ScrapeGen.

What is ScrapeGen?

It is a simple Python-based command-line tool that generates python web scrapers based on rules and details entered in a YAML file. When it runs, it generates a new file. Rules turned to separate functions which then are called to main parsing method. View the Demo:

Why is needed?

Such kind of tool could be good for companies who write so many parses and hardcode the rules within the Py files. Imagine the rule is changed and you gotta write it again. What if you just change the rule in YAML and the scraper starts working? Sounds cool, right?

The tool is available on Github

Originally published at http://blog.adnansiddiqi.me on November 13, 2019.