Web Scraping tools for deal flow

January 17, 2025

by a searcher from Marquette University in Louisville, KY, USA

Has anyone found a web scraping tool to help automate data harvesting from sources such as the Secretary of State, PVA, etc.?

It would be useful to utilize a tool to dynamically build a data set that could then be enriched with location verification, website url, linked in url, business owner & family members etc. before connecting it with automated outreach tools. I'd love comments if anyone has successfully achieved this. Much appreciated!

0 Likes

13 Replies

72 Views

Replies

Reply by a professional
from INSEAD in Singapore

Depending on how technical you are, there are decent self-build solutions in Python via Selenium or ScraPy. A caveat here though - it can get very complex very quickly depending on the technical defences in the source website. If hiring someone, go for a fixed price contract, not time+. For something off the shelf you have hundreds of options, with varying levels of UI simplicity and technical prowess. We're seeing more LLM-/agentic ai-focused scrapers being launched, e.g., firecrawl.dev which I came across this week. Happy to chat if you want to explore more

Reply by a searcher
from University of Illinois at Urbana in Austin, TX, USA

Have you tried Perplexity or Claude? They can do the searching, but I also had Claude build a Python script to do this using RI as an example: "I will create a Python script that can help you build a comprehensive list of businesses from the Rhode Island Secretary of State database. This script will collect publicly available information on business locations, website URLs, LinkedIn URLs, and business owner/manager details." Code is pretty long but email me if you'd like the code; redacted

+11 more replies.

Join the discussion