Scrapers 101: What are they, how do you use them?

Question

Been at real estate for seven months and just stumbled upon the subject of "scrapers". What are they? Are they useful?

Derrick Strope · Answer

I was bored the other morning and wrote a shell script using curl/wget to pull all the contact information from all of the rental properties on craigslist.

Unfortunately they didn't like that I pulled down several hundred pages of content in a couple seconds and banned the IP of the server I was using.   Will have to make it more discreet next time ;)

Ariel O. · Answer

The vast majority I have encountered are actually just Ubot software coded for specific tasks. In any event, CL is (so I've heard) particularly notorious in their efforts to prevent scraping, compared to almost anyone else really.

Pete Slaga · Answer

HI David,

A pretty good scraper is import.io .  I have used it for VRBO listings and Zillow listings.  There is a free version and the company does offer several webinars a week on how to use it for various topics.

Hope this helps,

Pete

David Rupp · Answer

I'll send you a DM to discuss what I am looking for.

Ari Schaffel · Answer

@David Rupp what type of data were you looking for?

David Rupp · Answer

Unfortunately, I did not.

Derrick Strope · Answer

@Ariel Ozick  it's just a matter of tweaking my code slightly, it was just an experiment I did the other morning.

Most of the scrapers out there are enormously bloated pieces of software and I wouldn't install one on any of my computers, due to the fact that the program in the first place is to accomplish an 'unethical' practice which violates the terms of service of any website - so it's not too far of a stretch to believe the writers of these software packages are collecting valuable data from you.

Ariel O. · Answer

You should also check out Kimono, http://www.kimonolabs.com . I find the app a lot more intuitive then Import.io.

If you're a programmer then learn how to use cURL and read up on navigating the XML/DOM tree in your language of choice.

@Derrick Strope There are whole forums out there of people discussing ways to get data off of craigslist. Google is your friend. But they also offer a fairly robust and usable RSS feed, too.

David Rupp · Answer

Hey @Pete Slaga thanks for the link. I wasn't able to get the posters contact information, but maybe I am missing something. I'll have to checkout one of the webinars to learn how to use it.

Thanks,David

David Rupp · Answer

Does anyone have a good scraper they use for zillow? I am not really finding anything out there that seems worth the money.

David

Scrapers 101: What are they, how do you use them?

Most Popular Reply