Web Scrape Real Estate using a Proxy, Beautiful Soup, and Python | response = client.get(url)

Some sites are very strict with restrictions on requests which make it very difficult or not viable to scrape large lists of results, especially if you are getting error 403 or 405. If you are web scraping at scale then you will at some point need to use a proxy.

A paid proxy is one option to allow you to make requests, using requests, and get what you need.

🌍 https://www.scraperapi.com/?fp_ref=dr-pi 🌍 https://pypi.org/project/scraperapi-sdk/

Promo Code: SCRAPE1462937

pip install scraperapi-sdk And then just 3 lines of code:

✏️from scraper_api import ScraperAPIClient ✏️client = ScraperAPIClient("4e3d10c2650ed52c85a41c7xxxxxxx") # use your own api key (register for trial) ✏️response = client.get(url)

You still need to be sensible, use the correct user agent, headers etc.

After trialling a workaround using a bash script to change my VPN server to the next one in a list of 92 I decided to trial "Scraper API" as well.

Here you can see how I modified the code, and used my trial quota to get some results from a property listings site.

I tried other such services from other companies but found this one to be the best. I am NOT being paid to say this by the way!

Whilst these are not cheap, one paid job a month should pay for this.

I hope this has been informative and I'd like to thank you for viewing! 👍 Thumbs up yeah? 👍

This will eventually run on my Raspberry Pi 4 and if you want to see the code, it's on GitHub. It's still work in progress, but works 100%. https://github.com/RGGH/BeautifulSoup

Thanks to Code Monkey King for his tips on proxies, and his top videos.

See you around yeah? Dr P.

#webscraping #python #proxy ... https://www.youtube.com/watch?v=ILnkt1ASpGg