All
Members
Companies
Blog
Forums
Podcast
Webinars
    User Log in  /  Sign up
  • Forums
    Newest Posts Trending Discussions Followed Forums Real Estate News & Current Events General Landlording & Rental Properties Buying & Selling Real Estate Deal Analysis See All
  • Education

    Read

    BiggerPockets Blog BPInsights: Expert Analysis Coronavirus Content & Resources Guides Glossary Reviews Member Blogs

    Watch

    Webinars Video Library Financial Independence Blueprint Intro to Real Estate: Rentals

    Listen

    BiggerPockets Real Estate Podcast BiggerPockets Money Podcast BiggerPockets Business Podcast Real Estate Rookie Podcast Daily Podcast (Audio Blog)

    Topics

    Business Operations Finance Finding Deals Property Management Property Types Strategy
  • Network

    Recommended Vendors

    Real Estate Agents Mortgage Lenders Companies Hard Money Lenders Contractors Investment Companies

    Search

    Members Events Jobs
  • Tools

    Calculators

    Rental Property Fix and Flip BRRRR Rehab Estimator
    Wholesaling Mortgage Payment 70% Rule Airbnb

    Services

    BPInsights: Property Insights Tenant Screening Property Management Lease Agreement Packages

    New Feature

    BPInsights (beta)

    Quickly analyze a property address or ZIP Code to compare your rent in your neighborhood.

    Analyze a property
  • Find Deals
    Real Estate Listings Find Foreclosures External Link Ads, Jobs, and Other
  • Bookstore

    Real Estate Books

    Profit Like The Pros Bidding to Buy See all books

    Featured Book

    BiggerPockets Wealth Magazine book cover
    BiggerPockets Wealth Magazine

    Written by financial journalists and data scientists, get 60+ pages of newsworthy content, expert-driven advice, and data-backed research written in a clear way to help you navigate your tough investment decisions in an ever-changing financial climate! Subscribe today and get the Oct/Nov issue delivered to your door!

    Get the Magazine
  • Pricing
Log In Sign up
User
Quick search links
Podcast Hard Money Lenders Books Washington
ForumsArrowReal Estate Technology, Social Media & BloggingArrowReal Estate APIs and Data Science
  • Newest Posts
    • Newest Posts
    • Unanswered Discussions
  • Trending
    • Top Discussions
    • Trending Discussions
  • Browse Forums
Search Nova
Topic: 1031 Exchanges Topic: Akron Real Estate Forum Topic: Alabama Real Estate Q&A Discussion Forum Topic: Alaska Real Estate Q&A Discussion Forum Topic: Alberta Real Estate Forum Topic: Albuquerque Topic: Allentown Real Estate Forum Topic: Anchorage Real Estate Forum Topic: Ann Arbor Topic: Arizona Real Estate Q&A Discussion Forum Topic: Arkansas Real Estate Q&A Discussion Forum Topic: Arlington Real Estate Forum Topic: Asheville NC Real Estate Forum Topic: Ask About A Real Estate Company Topic: Athens Real Estate Forum Topic: Atlanta Real Estate Forum Topic: Aurora Real Estate Forum Topic: Aurora Real Estate Forum Topic: Austin Real Estate Forum Topic: Bakersfield Real Estate Forum Topic: Baltimore Real Estate Forum Topic: Bankers, Lenders, and Mortgage Brokers Topic: Baton Rouge Real Estate Forum Topic: Beaverton Real Estate Forum Topic: BiggerPockets Business Podcast Topic: BiggerPockets Exclusive PRO Area Topic: BiggerPockets Money Podcast Topic: BiggerPockets Real Estate Investing Summit Topic: BiggerPockets Summer Reading List Topic: BiggerPockets Webinar Discussion Forum Topic: Billings Real Estate Forum Topic: Birmingham Real Estate Forum Topic: Bismarck Real Estate Forum Topic: Boise Real Estate Forum Topic: Boston Real Estate Forum Topic: Bridgeport Real Estate Forum Topic: British Columbia Real Estate Forum Topic: BRRRR - Buy, Rehab, Rent, Refinance, Repeat Topic: Buying & Selling Real Estate Discussion Topic: California Real Estate Q&A Discussion Forum Topic: Cambridge Real Estate Forum Topic: Canadian Real Estate Topic: Cape Cod Real Estate Forum Topic: Cape Coral Real Estate Forum Topic: Casper Real Estate Forum Topic: Cedar Rapids Real Estate Forum Topic: Chandler Real Estate Forum Topic: Charleston Real Estate Forum Topic: Charleston Real Estate Forum Topic: Charlotte Real Estate Forum Topic: Chattanooga Real Estate Forum Topic: Chesapeake Real Estate Forum Topic: Cheyenne Real Estate Forum Topic: Chicago Real Estate Forum Topic: Chittenden County Real Estate Forum Topic: Cincinnati Real Estate Forum Topic: Cleveland Real Estate Forum Topic: Coastal NC Real Estate Forum Topic: Colorado Real Estate Q&A Discussion Forum Topic: Colorado Springs Real Estate Forum Topic: Columbia Real Estate Forum Topic: Columbia Real Estate Forum Topic: Columbus Real Estate Forum Topic: Columbus Real Estate Forum Topic: Commercial Real Estate Investing Forum Topic: Completed Deal Diaries Topic: Connecticut Real Estate Q&A Discussion Forum Topic: Construction & Remodeling Diaries Topic: Contractors Topic: Conway Real Estate Forum Topic: Coronavirus Conversation Topic: Coronavirus Government Assistance Programs Topic: Creative Real Estate Financing Forum Topic: Currently Under Contract Topic: Dallas Real Estate Forum Topic: Davenport Real Estate Forum Topic: Dayton Real Estate Forum Topic: Delaware Real Estate Q&A Discussion Forum Topic: Denver Real Estate Forum Topic: Des Moines Real Estate Forum Topic: Detroit Real Estate Forum Topic: Do it Yourself Topic: El Paso Real Estate Forum Topic: Eugene Real Estate Forum Topic: Evansville Real Estate Forum Topic: FAQ Forum Topic: Fargo Topic: Fayetteville Real Estate Forum Topic: First Time Home Buyer Topic: Florida Real Estate Q&A Discussion Forum Topic: Foreigners Buying in the USA Topic: Fort Collins Real Estate Forum Topic: Fort Lauderdale Real Estate Forum Topic: Fort Smith Real Estate Forum Topic: Fort Wayne Real Estate Forum Topic: Fort Worth Real Estate Forum Topic: Fresno Real Estate Forum Topic: General Foreclosure & Pre-Foreclosure Forums Topic: General Landlording & Rental Properties Topic: General Real Estate Investing Topic: Georgia Real Estate Q&A Discussion Forum Topic: Germantown Real Estate Forum Topic: Get Foreclosure Help - Stop Foreclosure Forum Topic: Gilbert Real Estate Forum Topic: Glendale Real Estate Forum Topic: Goals, Business Plans & Entities Topic: Grand Rapids Real Estate Forum Topic: Green Bay Topic: Greensboro Real Estate Forum Topic: Hammond Real Estate Forum Topic: Hartford Real Estate Forum Topic: Hawaii Real Estate Q&A Discussion Forum Topic: Henderson Real Estate Forum Topic: Hialeah Real Estate Forum Topic: Hilo Real Estate Forum Topic: Home Owner Association (HOA) Issues & Problems Forum Topic: Honolulu Real Estate Forum Topic: Housing News & Real Estate Market Discussions Topic: Houston Real Estate Forum Topic: HUD, VA, and Tax Sales Topic: Huntington Real Estate Forum Topic: Huntsville Real Estate Forum Topic: Idaho Falls Real Estate Forum Topic: Idaho Real Estate Q&A Discussion Forum Topic: Illinois Real Estate Q&A Discussion Forum Topic: Indianapolis Real Estate Forum Topic: Indiana Real Estate Q&A Discussion Forum Topic: Innovative Strategies Topic: Interest Rate Drops Topic: International Real Estate Topic: Investor Deal Diaries Topic: Investor Psychology Topic: Iowa Real Estate Q&A Discussion Forum Topic: Jackson Real Estate Forum Topic: Jacksonville Real Estate Forum Topic: Jersey City Real Estate Forum Topic: Job Loss/Company Closings Topic: Joliet Real Estate Forum Topic: Jonesboro Real Estate Forum Topic: Kailua Real Estate Forum Topic: Kānèohe Real Estate Forum Topic: Kansas City Real Estate Forum Topic: Kansas City Real Estate Forum Topic: Kansas Real Estate Q&A Discussion Forum Topic: Kentucky Real Estate Q&A Discussion Forum Topic: Knoxville Real Estate Forum Topic: Lakewood Real Estate Forum Topic: Landlord Problem Solving Topic: Las Cruces Real Estate Forum Topic: Las Vegas Real Estate Forum Topic: Lewiston Real Estate Forum Topic: Lexington-Fayette Real Estate Forum Topic: Lincoln Real Estate Forum Topic: Little Rock Real Estate Forum Topic: Local Real Estate Networking Topic: Long Beach Real Estate Forum Topic: Long Island Real Estate Forum Topic: Los Angeles County Real Estate Forum Topic: Louisiana Real Estate Q&A Discussion Forum Topic: Louisville Real Estate Forum Topic: Madison Real Estate Forum Topic: Maine Real Estate Q&A Discussion Forum Topic: Managing Your Property Topic: Manchester Real Estate Forum Topic: Manitoba Topic: Marketing Your Rental Topic: Marketplace Topic: Maryland Real Estate Q&A Discussion Forum Topic: Massachusetts Real Estate Q&A Discussion Forum Topic: Meads Real Estate Forum Topic: Memphis Real Estate Forum Topic: Meridian Real Estate Forum Topic: Mesa Real Estate Forum Topic: Miami Topic: Michigan Real Estate Q&A Discussion Forum Topic: Military Investing Topic: Milwaukee Real Estate Forum Topic: Minneapolis Real Estate Forum Topic: Minnesota Real Estate Q&A Discussion Forum Topic: Mississippi Real Estate Q&A Discussion Forum Topic: Missoula Real Estate Forum Topic: Missouri Real Estate Q&A Discussion Forum Topic: Mobile (Alabama) Real Estate Forum Topic: Mobile Homes & Mobile Home Park Investing Topic: Modesto/Merced Real Estate Forum Topic: Montana Real Estate Q&A Discussion Forum Topic: Montgomery Real Estate Forum Topic: Mortgage Payment Suspension Topic: Mt. Pleasant Real Estate Forum Topic: Multi-Family and Apartment Investing Forums Topic: Nampa Real Estate Forum Topic: Naperville Real Estate Forum Topic: Nashville Real Estate Forum Topic: Nebraska Real Estate Q&A Discussion Forum Topic: Nevada Real Estate Q&A Discussion Forum Topic: Newark Real Estate Forum Topic: New Brunswick Real Estate Forum Topic: Newfoundland and Labrador Real Estate Forum Topic: New Hampshire Real Estate Q&A Discussion Forum Topic: New Haven Real Estate Forum Topic: New Jersey Real Estate Q&A Discussion Forum Topic: New Member Introductions Topic: New Mexico Real Estate Q&A Discussion Forum Topic: New Orleans Real Estate Forum Topic: New York City Real Estate Forum Topic: New York Real Estate Q&A Discussion Forum Topic: Norfolk Real Estate Forum Topic: North Carolina Real Estate Q&A Discussion Forum Topic: North Dakota Real Estate Q&A Discussion Forum Topic: Northwest Territories Real Estate Forum Topic: Norwalk Real Estate Forum Topic: Nova Scotia Real Estate Forum Topic: Nunavut Real Estate Forum Topic: Oakland Real Estate Forum Topic: Off-Topic Topic: Ogden Topic: Ohio Real Estate Q&A Discussion Forum Topic: Oklahoma City Real Estate Forum Topic: Oklahoma Real Estate Q&A Discussion Forum Topic: Omaha Real Estate Forum Topic: Ontario Real Estate Forum Topic: Orange County Real Estate Forum Topic: Oregon Real Estate Q&A Discussion Forum Topic: Orlando Real Estate Forum Topic: Overland Park Real Estate Forum Topic: Pearl City Real Estate Forum Topic: Pennsylvania Real Estate Q&A Discussion Forum Topic: Peoria Real Estate Forum Topic: Personal Finance Forum Topic: Philadelphia Real Estate Forum Topic: Phoenix Real Estate Forum Topic: Pittsburgh Real Estate Forum Topic: Pocatello Real Estate Forum Topic: Portland Real Estate Forum Topic: Portland Real Estate Forum Topic: Prince Edward Island Real Estate Forum Topic: Private Lending & Conventional Mortgage Advice Topic: Property Insurance Questions & Discussions Topic: Providence Real Estate Forum Topic: Provo Real Estate Forum Topic: Quad Cities Real Estate Forum Topic: Quebec Real Estate Forum Topic: Questions About BiggerPockets and Official Site Announcements Topic: Raleigh Real Estate Forum Topic: Rapid City Real Estate Forum Topic: Real Estate Agent Forums Topic: Real Estate Crowdfunding Forums Topic: Real Estate Deal Analysis and Advice Topic: Real Estate Development & New Home Construction Topic: Real Estate Events and Happenings Topic: Real Estate Guru, Book & Course Reviews and Discussions Topic: Real Estate Investor Marketing Help & Advice Topic: Real Estate News & Current Events Topic: Real Estate Not-So-Successful Stories Topic: Real Estate Quizzes Topic: Real Estate Rookie Podcast Topic: Real Estate Success Stories Topic: Real Estate Technology, Social Media & Blogging Topic: Real Estate Wholesaling Questions & Answers Topic: Rehabbing and House Flipping Topic: Reno Real Estate Forum Topic: Renters Discussion Topic: Rent to Own a.k.a. Lease Purchase, Lease Options Topic: REOs Topic: Rhode Island Real Estate Q&A Discussion Forum Topic: Richmond Real Estate Forum Topic: Rio Grande Valley Real Estate Forum Topic: Riverside Real Estate Forum Topic: Rochester Real Estate Forum Topic: Rockford Real Estate Forum Topic: Sacramento Real Estate Forum Topic: Saint Paul Real Estate Forum Topic: Salem Real Estate Forum Topic: Salt Lake City Real Estate Forum Topic: San Antonio Real Estate Forum Topic: San Diego Real Estate Forum Topic: Sandy Springs Real Estate Forum Topic: San Francisco Real Estate Forum Topic: San Jose Real Estate Forum Topic: San Luis Obispo Real Estate Forum Topic: Santa Fe Real Estate Forum Topic: Saskatchewan Real Estate Forum Topic: Savannah Real Estate Forum Topic: Scottsdale Real Estate Forum Topic: Seattle Real Estate Forum Topic: Short Sales Questions Topic: Short-Term and Vacation Rental Discussion Topic: Shreveport Real Estate Forum Topic: Sioux City Real Estate Forum Topic: Sioux Falls Real Estate Forum Topic: South Bend Real Estate Forum Topic: South Carolina Real Estate Q&A Discussion Forum Topic: South Dakota Real Estate Q&A Discussion Forum Topic: Southeast NC Real Estate Forum Topic: Spokane Real Estate Forum Topic: Springdale Real Estate Forum Topic: Springfield Real Estate Forum Topic: Stamford Real Estate Forum Topic: Starting Out Topic: St. Louis Real Estate Forum Topic: Stockton Real Estate Forum Topic: St. Petersburg Real Estate Forum Topic: STR Strategies Under Travel Restrictions Topic: Surprise Real Estate Forum Topic: Tacoma Real Estate Forum Topic: Tallahassee Real Estate Forum Topic: Tampa Real Estate Forum Topic: Tax, Legal Issues, Contracts, Self-Directed IRA Topic: Tax Liens, Notes, Paper, & Cash Flows Discussion Topic: Tempe Real Estate Forum Topic: Tenant Screening Topic: Tennessee Real Estate Q&A Discussion Forum Topic: Texas Real Estate Q&A Discussion Forum Topic: Thornton Real Estate Forum Topic: Toledo Real Estate Forum Topic: Topeka Real Estate Forum Topic: Tucson Real Estate Forum Topic: Tulsa Real Estate Forum Topic: Tuscaloosa Real Estate Forum Topic: Upstate New York Real Estate Forum Topic: Utah Real Estate Q&A Discussion Forum Topic: Vancouver Real Estate Forum Topic: Vermont Real Estate Q&A Discussion Forum Topic: Virginia Beach Real Estate Forum Topic: Virginia Real Estate Q&A Discussion Forum Topic: Waipahu Real Estate Forum Topic: Waipahu Real Estate Forum Topic: Warren Real Estate Forum Topic: Warwick Real Estate Forum Topic: Washington D.C. Real Estate Q&A Discussion Forum Topic: Washington Real Estate Q&A Discussion Forum Topic: Waterbury Real Estate Forum Topic: Westchester County Real Estate Forum Topic: Western NC Real Estate Forum Topic: West Gulfport Real Estate Forum Topic: West Virginia Real Estate Q&A Discussion Forum Topic: Wichita Real Estate Forum Topic: Wilkes Barre Scranton Real Estate Forum Topic: Wilmington Real Estate Forum Topic: Wisconsin Real Estate Q&A Discussion Forum Topic: Worcester Real Estate Forum Topic: Wyoming Real Estate Q&A Discussion Forum Topic: Yukon Real Estate Forum
Create post

Real Estate APIs and Data Science

42 Replies

  • Share on Facebook
  • Share on Twitter
  • Share on LinkedIn
Rotate Log in or sign up to reply
user profile avatar
  • Posts 5
  • Votes 7

Sebastian Garcia
from New York, NY

posted almost 3 years ago

Hello everyone!

Quick bit about me- I'm a data scientist working in NYC and literally just started looking into REI a week ago. Mostly because of the dream I have of retiring early with a nice cash flow, but also partly because of the undoubtedly cool stuff I can do with the data.

I'm aware there are probably a good amount of tools/sites already available for determining hot markets in the area, where to invest in hopes of appreciation, predicting house value, etc. But it would be a dope project to build some machine learning model tuned to exactly what I want, like interactions between price and crime and population movement and income shifts and their impact on cash flow, not just property value. And so on and so forth.

I'm going with the Zillow API for now but it seems there's a pretty low rate limit. But it's a good start.

If anyone can point me to less-known real estate open databases/APIs that supply additional information, as well as any smaller data sources you find valuable, please let me know! I'd love to share any findings here.

And if anyone's looked into or done this kind of stuff- data science for real estate- I'm very curious about your experiences and I'd love to hear about them!

Thanks!

Rotate Log in or sign up to reply
user profile avatar
  • Posts 1.3K
  • Votes 680

Trevor Ewen
Rental Property Investor from Weehawken, NJ

replied almost 3 years ago

@Sebastian Garcia

Played around with this a bit myself. I am a software engineer. I also live in NYC.

My take is that disorganized and unfiltered data sources are some of the most compelling. Some ideas:

* County auction sites

* Insurance auction sites

* Educational data

* For sale by owner lists (if there is such a thing)

* Listings from smaller real-estate firms

Just a few ideas to get you going. One of the most interesting parts about real-estate is how decentralized it is. The idea of a single source for anything is almost an anathema. I think the smart people recognize that and bring together disparate parties to create win-win situations.

Rotate Log in or sign up to reply
user profile avatar
  • Posts 5
  • Votes 7

Sebastian Garcia
from New York, NY

replied almost 3 years ago

@Trevor Ewen

Thanks for the input! I was wondering why these types of discussions were rare here and it makes sense- there's no centralized source to make it easier and and there's probably hundreds of thousands of local sources that need to be pieced together.

That's definitely a good list to get started, at least for the tiny piece of Jersey I'm looking into. I'm sure as I get more into it I'll have a much better understanding on what types of data is best. Nothing beats domain knowledge.

Thanks again!

Rotate Log in or sign up to reply
user profile avatar
  • Posts 81
  • Votes 49

Nick Hakim
Analyst from New York, NY

replied almost 3 years ago

@Sebastian Garcia Awesome ideas! I would recommend looking into some of the following sites available to real estate professionals in the North East:

  1. Propertyshark
  2. Lavamap
  3. Corelogic

Most of these sites aggregate data from state/county websites, so you may find some valuable data sources there.

Rotate Log in or sign up to reply
user profile avatar
  • Posts 4
  • Votes 4

Cesar Ramirez
from weehawken, Nj

replied over 2 years ago

I'm a developer as well. I would love to collaborate on something like this.

Rotate Log in or sign up to reply
user profile avatar
  • Posts 10
  • Votes 2

Erik Chan
from Burlingame, CA

replied over 2 years ago

Same here, I am also a developer based in San Francisco. Would be interested in discussing some ideas

Rotate Log in or sign up to reply
user profile avatar
  • Posts 31
  • Votes 14

Diana Smith
from San Francisco, California

replied over 2 years ago

Craigslist

Rotate Log in or sign up to reply
user profile avatar
  • Posts 11
  • Votes 3

Tony Zuanich
from Orange, California

replied over 2 years ago

here is a API the might be useful:

http://www.ziplabs.us/documentation

Rotate Log in or sign up to reply
user profile avatar
  • Posts 11
  • Votes 3

Tony Zuanich
from Orange, California

replied over 2 years ago

Here is a site that provides data:

https://www.zaplabs.com/

Rotate Log in or sign up to reply
user profile avatar
  • Posts 11
  • Votes 3

Tony Zuanich
from Orange, California

replied over 2 years ago

Here is another one the gives a free trial:

https://www.housecanary.com/real-estate-products-technology

Rotate Log in or sign up to reply
user profile avatar
  • Posts 13
  • Votes 8

Kevin Smith

replied over 2 years ago

I worked with the Zillow API a little bit and having results populate in Excel; but then I was interested in Craiglist results to find the "For Sale by Owner" listings (web scraping; there can be some issues with that if done wrong though so was hoping for an open API there).  

That'd be cool to find the For Sale by Owner, load the info in Google Sheets, then you can use App Script for more organization and prep.

  

Then there are paid services that I haven't checked into yet: https://retsrabbit.com/pricing

Rotate Log in or sign up to reply
user profile avatar
  • Posts 78
  • Votes 56

Gavin D.
from South Carolina

replied over 2 years ago

So, data is very, very, very, readily available and cheap.... and that's why you do NOT want that data... Cause everyone has it.  The data you DO want is the stuff that makes it on to those lists.. just before it lands on those lists.... for example..
properties that have tax payments due, and not yet paid, but not yet late... like 2 months from tax sale. Some of those people will pay their taxes before its too late, but most of the people who are 60 days fro tax sale and owe 1-3k in taxes.... aren't gonna get there.  What you need is a developer...conveniently, like me.   about 60% of my week is spend scraping data... Real Estate , healthcare, industrial, and EVERY client that I have came to me as their second or third choice.... cause thats what happens when the first guy or the second guy could not get it done.

Rotate Log in or sign up to reply
user profile avatar
  • Posts 78
  • Votes 56

Gavin D.
from South Carolina

replied over 2 years ago

in anycase, Sebatian, Ive been a full time dev for over 10 years now, and there is one thing I can promise you about data...  If it was "INTENDED" to be available to you, ... Its worthless.. for numerous reasons, 1. its likely tampered with.. Zillow is a great example, dont believe me, look at a house in public mode, than switch to owner mode...2.  the companies who have useful , relevant, and profitable data do 1 of 2 things with it... use it within their own business model, or sell it for a premium.   So where do you get good data?  If its properties and realestate you are into... learn to scrape the county Auditor for 'EVERY' property... then, add some additional depth by scraping the county clerk's site for deeds, now, bounce that data down again against GIS and geolocation to get a good grasp of not only population density, but amenities , nightlife, dining etc...  And lastly (well not lastly... just lastly for now)  phone append and email append the names and addresses to get methods of contact...If you really want to get crazy from that point, compare length of phone line ownership, and age of deed on the property to EPA minor environmental disaster maps and you can sort out who is old and probably has caner, pretty quickly.  Want the best data that anyone on this site has ever seen... scrub all of the previous against obituary listing to look for surviving spouses...

Rotate Log in or sign up to reply
user profile avatar
  • Posts 1
  • Votes 0

Bert La

replied over 2 years ago

@Sebastian Garcia @Erik Chan  

Hi Guys, 

I am interested in working with on a project like this as well. I have professional experience in both Real Estate and as a Software Engineer. Lets get together and discuss!

Also, I have a plug for an up-to-date standardized and unadulterated set of data.

PM Me if you are interested in working on something!

Rotate Log in or sign up to reply
user profile avatar
  • Posts 20
  • Votes 13

Dave Nixon

replied over 2 years ago

I'm also a sw engineer just starting to look at real estate.

I expect obvious things are all ready out there like taking the median asking rent for a given property class like a 3 bed 2 bath SFR in anytown from craigslist and dividing it by the median asking price for such SFRs on zillow. Is that sort of thing already available someplace? Maybe even for free?

Rotate Log in or sign up to reply
user profile avatar
  • Posts 1
  • Votes 1

John Kelsey
from Mckinney, TX

replied over 2 years ago

Is selenium an option for getting data from some of these sites? I know there's a risk of IP getting blacklisted.... but if we stagger the search using a random time generator and spread the work amongst several of us - we could build a pretty nice data set, no?

Rotate Log in or sign up to reply
user profile avatar
  • Posts 22
  • Votes 14

Sean O'Connor
Rental Property Investor from Atlanta, GA

replied over 2 years ago
Originally posted by @John Kelsey :

Is selenium an option for getting data from some of these sites? I know there's a risk of IP getting blacklisted.... but if we stagger the search using a random time generator and spread the work amongst several of us - we could build a pretty nice data set, no?

Another option is to use an open source VPN to change your IP address every time the IP gets blacklisted. 

Has anyone been working on this? I would very much like to work on this project. I currently work as a cyber security consultant and would like to help in any way I can. Please reply or PM me so we can discuss further.

Rotate Log in or sign up to reply
user profile avatar
  • Posts 38
  • Votes 3

Alex Stepanov
Investor from San Jose, California

replied over 2 years ago

After digging into it for some time, I think it's safe to say that the choices are slim to none - there are some relatively cheap (sometimes free) data sources that are very poor quality, and the good quality data is prohibitively expensive (House Canary charges $1 per API call - WOW). 

So far, there seems to be a few types of data aggregators that make their datasets available:

*) Folks who are trying to promote "whitelabel clones" (e.g. Zillow, which comes with a host of restrictions expressly prohibiting "enriching other datasets" etc)

*) Those who will process and sanitize MLS for you (which is a huge task in itself) but you have to have MLS creds, meaning you have to deal with the zoo of MLS providers all by yourself - or pay something like $1200/mo for nationwide (US + Canada) listing feed

*) Folks who charge per call (CoreLogic, House Canary etc.) and who have high quality sets but they don't want you to mine them so they set all sorts of call rates and excessive price per call to ward off "gold diggers"

It is a mistake to say that the data "is abundant" and "if it is available, it must be worthless". It is definitely NOT abundant (and I am not talking about "websites" like Zillow, I am talking about raw datasets) and it is definitely valuable - if you can afford it of course :(

Rotate Log in or sign up to reply
user profile avatar
  • Posts 8
  • Votes 5

Christian Hubbs
from Pittsburgh, PA

replied over 2 years ago

Has anybody played with the Quandl API? It promises a lot, but it looks like some of the documentation is out of date.

Rotate Log in or sign up to reply
user profile avatar
Check Rosette Top Subjects:
Rentals and Residential
  • Posts 235
  • Votes 82

Kevin Zolea
Rental Property Investor from Parlin, NJ

replied over 2 years ago

Great ideas here! I have been working as an Environmental Scientist for the past 6 months and have been learning how to use R and Python to analyze a lot of data. The first thing that came to my mind was how can I use this towards real estate.. What language do you guys prefer to do web scrapping?

Rotate Log in or sign up to reply
user profile avatar
  • Posts 35
  • Votes 72

Khenkis K.
from Lenox, GA

replied over 2 years ago

Hey hey!!! Now this is my kind of talk!

@Kevin Zolea I think you're already on the right track with R and Python. Essentially, any language you're comfortable with that has a good crawler/scraping framework is the way to go, in my opinion. However, I would say play on your strengths and so you definitely want to stick with Python. GIS is finally starting to ramp up and specialities (including your own) appear to be incorporating the technology more into their domain. ESRI is one of the top players in the GIS scene and guess what? Python just so happens to be their accepted scripting langugage.

Rotate Log in or sign up to reply
user profile avatar
  • Posts 2
  • Votes 0

Dev Kumar
from Phoenix, Arizona

replied almost 2 years ago

I myself am a data geek and am building a Property Management Software with my partner. Any one has any idea about APIs that i can use to get lease templates by state and also tenant background checks. 

Thank You

Rotate Log in or sign up to reply
user profile avatar
  • Posts 1
  • Votes 0

Minwei Xu

replied almost 2 years ago

I'm also a developer and would love to join such a project. Do you guys have any private repo so that I can contribute perhaps?

Also, this website offers free access to MLS https://www.reso.org/mls-data-access/ but needs to put company name in the application form, so I'm a bit hesitant. Please let me know if anyone made any progress.

Rotate Log in or sign up to reply
user profile avatar
  • Posts 7
  • Votes 0

Eddie Godshalk
from Santa Clara, CA

replied almost 2 years ago

I am thinking of pivoting from a SaaS application to APIs, so that RE professionals and investors can access that better and more current market data than what CoStar, the MLS, and others provide. More info on SaaS application and data we deliver is at bit.ly/2PWRMXB

But before I build the APIs, I need some information.

Who has the market share for market data for API’s? And what is their pricing matrix? What features and data are most in demand?

Any idea where I might find some better information and facts? Then once built, looking for smart way to get this new market data to the masses.

Appreciate your feedback. Thanks Eddie…

Rotate Log in or sign up to reply
  • 1
  • 2
Related Resources Real Estate Technology, Social Media & Blogging
5 Extreme Methods I Use to Save Time in My Workday
Gather Your Dream Team: Must-Have Real Estate Team Members
The (Brutal) Truth About Why You Can’t Find a Partner
LLC vs. S Corporation: Which Is Better for Real Estate?
6 Real Estate Side Hustles to Make Extra Money & Gain Experience
Resources Read, see, and learn more!
Link Real Estate Investment Calculators
Link BiggerPockets Blog
Link Path to Purchase
Link Mortgage Loans
Link Find a Contractor
Link Real Estate Agents
Link Hard Money Lenders
Link Real Estate Listings

Top Contributors

Christian Szpilfogel
Christian Szpilfogel
Ottawa Ontario, Canada
7.38
Score
Jonathan Swartz
Jonathan Swartz
Dallas, TX
7.38
Score
Bill Goodland
Bill Goodland
Allentown PA, United States
4.88
Score
Aaron Bihl
Aaron Bihl
San Antonio, TX
4.17
Score
Eric Fernwood
Eric Fernwood
Las Vegas, NV
4.17
Score

Real Estate Technology, Social Media & Blogging Trending Discussions

  • Monday.Com? Work Flow Software for Rehabs, Sales, Deal Flow
    3 Replies
  • Task/Process Management? Looking for something to work with teams
    1 Replies
Log in Sign up

Log in

Forgot password?

If you signed up for BiggerPockets via Facebook, you can log in with just one click!

Log in with Facebook

Or
btn_google_dark_normal_ios Created with Sketch. Continue with Google

Let's get started

We just need a few details to get you set up and ready to go!

Use your real name

Use at least 8 characters. Using a phrase of random words (like: paper Dog team blue) is secure and easy to remember.

By signing up, you indicate that you agree to the BiggerPockets Terms & Conditions.

Or
btn_google_dark_normal_ios Created with Sketch. Continue with Google

Why create an account?

Receive a free digital download of The Ultimate Beginner's Guide to Real Estate Investing.

Connect with 1,000,000+ real estate investors!

Find local real estate meetups and events in your area.

Start analyzing real estate properties, we do the math for you.

It's free!

Explore

  • Membership
  • Community
  • Education
  • Marketplace
  • Tools
  • FilePlace
  • REI Resources
  • Perks
  • Glossary
  • Reviews
  • iOS App
  • Android App

Company

  • About Us
  • Press
  • Advertising
  • Careers
  • Stats
  • Contact Us

Important

  • Editorial Guidelines
  • Terms of Use
  • Rules
  • Privacy
  • FAQ

Social

  • Facebook
  • Twitter
  • YouTube
  • Instagram
© 2004-2021 BiggerPockets, LLC. All Rights Reserved.