How to create your own MLS database for FREE (without access to the MLS)

Real Estate Technology
11yFollow

How to create your own MLS database for FREE (without access to the MLS)

Cory Land
Information Technology · West Chester, PA · Member since 2014 · 40 posts · 20 votes
This is geared for anyone who is tech savy and maybe even a software engineer. This whole thing took me about 40 hours to write. I am not expecting anyone to actually do this but I am interested to see the feedback I get - techies or not. I also didn't create this to market and sell but to be able to run statistics and analytics for my own real estate investing purposes. We are in a new age and finding a competitive edge using technology is extremely important and necessary. Disclaimer: access to MLS through your licensed real estate agent will be a much better resource but this at least let's you get to play with the data. This does not update when houses are sold which sucks but I have not found any way around this 1) sign up for access to Redfin.com 2) create general search for area your market (I do city of Philly and all surrounding suburbs from $0-$500k) 3) find link for RSS feed for your saved search 4) write program to absorb the RSS feed every minute or so 5) parse MLS data and store on database 6) create front end to view data 7) optional: create notification service for your specific search criteria - mine also generates comps using zillow and google map images automatically (both sites have RESTful api which handle MLS data nicely)
8Reply
229 views

Most Popular Reply

Jay Thompson
Real Estate Consultant · Aransas Pass, TX · Member since 2014 · 21 posts · 28 votes
11y
Originally posted by @Gretchen Roberts:

So this is really interesting, and makes me wonder what is on MLS that you can't get on Zillow and Trulia and Redfin exactly? People talk about having or needing access to the MLS, which makes me think that these programs are only able to pull some of the feed instead of all of it.

There is actually quite a bit of difference between what's on your typical MLS and what you will find on publicly accessible sites such as Zillow and Trulia. Even what Redfin (a brokerage site -- Zillow and Trulia are not brokerages) displays is different than the MLS. Sites like Zillow and IDX based sites like Redfin all display a smaller subset of data than an MLS has. There were about 50 data fields that had to be completed to enter a home on my MLS and public sites typically display half of that, at best.

Some MLS only data is private\confidential -- like gate codes, owner contact info and showing instructions. Other is just more granular data of what you'll see displayed on public sites. For example, within the MLS I could search for 3 bedroom 2 bath homes that have a block fence, have had the roof replaced since 2007 and have a bidet in the master bath and laundry hookups in the garage. And I can filter that by areas with home owner association fees of less than $X.

Then there is the whole localized MLS rules thing. Some areas prohibit the public display of sales data. So in restrictive non-disclosure states, sometimes the only way to get data on past sales prices, type, financing etc is via the MLS.

5Reply
See this reply in the discussion

27 Replies

Jump to latestLatest
  • Gretchen Roberts
    Investor · Knoxville, TN · Member since 2014 · 54 posts · 18 votes
    11y

    So this is really interesting, and makes me wonder what is on MLS that you can't get on Zillow and Trulia and Redfin exactly? People talk about having or needing access to the MLS, which makes me think that these programs are only able to pull some of the feed instead of all of it.

    1Reply
  • Cory Land
    OP
    Information Technology · West Chester, PA · Member since 2014 · 40 posts · 20 votes
    11y

    There are differences and that's why I still use the MLS through my realtor. Some of the listings that an agent adds can choose a status to which will hide them from the consumer facing sites. They do this to list to other agents for some reason. Also, I think MLS can have sales completed that were never added for the purpose to be marketed.

    One of the reasons I made this application is to streamline the process for leads through last step 7 which can show google maps and zillow comps. Right now my realtor sends us the listings and then I would have to take those extra steps manually. 

    Another reason is the legacy software TREND which is used in my area for PA realtors to send listings through email is old and clunky. I am able to take the normal MLS data and present it in a way which works better for me.

    0Reply
  • Kyle Critchnau
    Owasso, OK · Member since 2014 · 113 posts · 43 votes
    11y

    Software developer here. This looks like it would be a pretty easy thing to do, what would be even better is if you could figure out what the parameters were and set up several searches and criteria in one program. 

    1Reply
  • Jay Thompson
    Real Estate Consultant · Aransas Pass, TX · Member since 2014 · 21 posts · 28 votes
    11y
    Originally posted by @Gretchen Roberts:

    So this is really interesting, and makes me wonder what is on MLS that you can't get on Zillow and Trulia and Redfin exactly? People talk about having or needing access to the MLS, which makes me think that these programs are only able to pull some of the feed instead of all of it.

    There is actually quite a bit of difference between what's on your typical MLS and what you will find on publicly accessible sites such as Zillow and Trulia. Even what Redfin (a brokerage site -- Zillow and Trulia are not brokerages) displays is different than the MLS. Sites like Zillow and IDX based sites like Redfin all display a smaller subset of data than an MLS has. There were about 50 data fields that had to be completed to enter a home on my MLS and public sites typically display half of that, at best.

    Some MLS only data is private\confidential -- like gate codes, owner contact info and showing instructions. Other is just more granular data of what you'll see displayed on public sites. For example, within the MLS I could search for 3 bedroom 2 bath homes that have a block fence, have had the roof replaced since 2007 and have a bidet in the master bath and laundry hookups in the garage. And I can filter that by areas with home owner association fees of less than $X.

    Then there is the whole localized MLS rules thing. Some areas prohibit the public display of sales data. So in restrictive non-disclosure states, sometimes the only way to get data on past sales prices, type, financing etc is via the MLS.

    5Reply
  • Joseph Cox
    REALTOR® · Altamonte Springs, FL · Member since 2013 · 71 posts · 32 votes
    11y
    Originally posted by @Gretchen Roberts:

    So this is really interesting, and makes me wonder what is on MLS that you can't get on Zillow and Trulia and Redfin exactly? People talk about having or needing access to the MLS, which makes me think that these programs are only able to pull some of the feed instead of all of it.

    Accuracy plays a large factor. Zillow and Trulia are 3rd party syndicated sites. So the information they have is often wrong or outdated. I sometimes use the "Trulia for Agents" app to Check-In properties I show, but there's been several from the MLS that aren't on Trulia, just in my area.

    Realtor.com isn't syndicated, and is updated every 15 minutes. If I didn't have access to my MLS, I would use that website, or if I knew a Realtor, I would use his/her "public MLS portal" which is not the agent's website. The agent or office's website proper is a basic IDX site with a lead capture system behind it. (The added benefits of using a Realtor's portal, is that you don't get bombarded with ads, and the Realtor is able to analyze your viewing to help you find what you're looking for. Yes, he or she can enter your details into their system, but that will happen as soon as you contact an agent or click on the "More Info" or "Find an Agent" button on any RE site)

    2Reply
  • Walt Payne
    Real Estate Investor · Sebastian, FL · Member since 2014 · 812 posts · 432 votes
    11y

    @Cory Land  One problem is Redfin is not nationwide at this point. Zillow sucks as a source, no accuracy whatsoever.

    0Reply
  • Cory Land
    OP
    Information Technology · West Chester, PA · Member since 2014 · 40 posts · 20 votes
    11y
    Originally posted by @Kyle Critchnau:

    Software developer here. This looks like it would be a pretty easy thing to do, what would be even better is if you could figure out what the parameters were and set up several searches and criteria in one program. 

     Yeah you can have multiple saved searches on Redfin and my application does handle multiple RSS feeds. I currently do not have a need for that as I can get any house added to Redfin with my target area through one large search. (It covers like 6 counties.)

    1Reply
  • Kyle Critchnau
    Owasso, OK · Member since 2014 · 113 posts · 43 votes
    11y

    Oh ok. Too bad they don't cover my area :/

    0Reply
  • Cory Land
    OP
    Information Technology · West Chester, PA · Member since 2014 · 40 posts · 20 votes
    11y
    Originally posted by @Walt Payne:

    @Cory Land  One problem is Redfin is not nationwide at this point. Zillow sucks as a source, no accuracy whatsoever.

     Yeah I did notice that about Redfin but I guess I am lucky with the area which is my market. I agree Zillow accuracy is not the best. 

    Essentially I generate a report daily that will have all houses added to Redfin in the last 24 hours. This will have the stats of the house, multiple google map images, and zillow comps. If I get a list of 10 houses I can look at the report in 5 minutes or less and see if I need to contact my Realtor to look into a specific property. 

    0Reply
  • Linda Quinton
    Real Estate Investor · Adelaide, South Australia · Member since 2014 · 36 posts · 25 votes
    11y

    I would love this!  Don't have the technical knowledge unfortunately.

    0Reply
  • Mark Ferguson
    Flipper/Rehabber · Greeley, CO · Member since 2013 · 2k+ posts · 1k+ votes
    11y

    Realtor.com does use syndicated data. Many offices are not allowing zillow, trulia or realtor.com to get that syndicated data because those sites are then charging agents if they want to be featured as the agent for their own listing. There are many houses on MLS that are not on zillow or realtor.com

    3Reply
  • Marc Boyd
    Investor · Chicago, IL · Member since 2014 · 9 posts · 2 votes
    11y

    Are you using any visualization tools like Tableau?  Also, are you using any benchmarks like price per square foot or are you creating your own benchmarks?

    0Reply
  • Cory Land
    OP
    Information Technology · West Chester, PA · Member since 2014 · 40 posts · 20 votes
    11y
    Originally posted by @Marc Boyd:

    Are you using any visualization tools like Tableau?  Also, are you using any benchmarks like price per square foot or are you creating your own benchmarks?

     I am not familiar with Tableau but will definitely check it out. 

    In regards to benchmarks it is hard to determine what we can define as a benchmark. Currently, I am able to get average prices in a certain area based on recent listings (I am not able to get completed sales) in respect to the sqft, BR, BA, etc. I can also determine if I am comparing against other flips in the area with the formula (if last sale date > current date - 2 years AND if 70% * listing price > last sale price).

    Every market is going to be different and in the city of Philly every other block is different. Benchmarks are a powerful tool and it seems like the only national standard is the Case-Shiller Home Price Index.

    I am hoping this can be the beginning of something great and I am absolutely open to ideas that can enhance this.

    1Reply
  • Marc Boyd
    Investor · Chicago, IL · Member since 2014 · 9 posts · 2 votes
    11y

    I moved to Chicago from Philly almost a year ago, so I'm familiar with the Philly area.  I  believe part of the value of the system you've created is that you can start to create neighborhood-level stats which beat national indices like Case-Shiller.  

    For example, the Northern Liberties neighborhood is an up & coming neighborhood that many folks find desirable which has a mix of new & old construction.  It's probably safe to say that rents & housing prices have increased in the neighborhood over the past decade.  To your point about every block being different...in a neighborhood like Northern Liberties, I would argue that the price per square foot (price/SF) in an apartment built in the last 10yrs is a worthwhile metric to know...especially when it comes to creating a set of rules to programmatically evaluate a new property on the market.  Depending on how you capture & store your data, I could see a the following series of questions being asked:

    -what's price/SF in Philly?

    -what's price/SF in the Northern Liberties neighborhood?

    -what's price/SF in Northern Liberties for "new construction" (ex: defined as age<10yrs)?

    Being able to answer the questions above will not only give you a set of reliable comps (which has value) but will also allow you to see how that metric trends over time.  One could argue that Zillow provide a similar (and more robust) service.  However, I have found Zillow has credibility issues when talking with agents.  Generating your own metrics from comps in your data set could lend yourself credibility when it comes to bidding on a property.  

    I created a hedonic price model of houses in the Durham, NC area in for an Econometrics class in college.   Since then I've considered building a similar system to what you've described to do econometric modeling and the above is one of many potential use cases for such a system.  Feel free to PM me if you'd like to share some more ideas. 

    Also, did you create your own data model or find one online?

    1Reply
  • Tessa Parish
    Real Estate Agent · Cupertino, CA · Member since 2016 · 1 post · 0 votes
    10y

    My only complain about our local MLS is that (as most are) they have a monopoly on their service and are therefore not very motivated to service the public as they should if they "could" lose clients for bad service. They recently update (don't know why?) all their upgrades are hardly used and their system has not been fine every since. Sometimes, a property will not come up in a search that should. I have to log out and re-enter and then it will come up. It is NOT reliable! Would love to fire them because I'm done complaining. Is anyone out there interested in starting a new one? lol offer real service and make a killing!

    0Reply
  • Prasad J.
    Rental Property Investor · Washington/Texas · Member since 2016 · 104 posts · 12 votes
    10y

    @Cory Land the stuff you have developed is pretty interesting. Can u share the code, better yet GitHub it? Would love to enhance it and share it back of course

    2Reply
  • Andrew Nuttall
    Investor · Malvern, PA · Member since 2016 · 8 posts · 1 vote
    9y
    So I started working on a data parsing script to import data from MLS searches my realtor sends me. I figured I'd do a quick search to see if anyone had done it yet and imagine my surprise when this post popped up on google and the author lives just down the road (just moved from WC to Frazer/Malvern). I like your approach and was going to look into Zillow and google for additional local data as well. Once I get a bit farther on the MLS part we should get together to compare notes.
    0Reply
  • Sandra Willford
    Contractor · Toledo, OH · Member since 2017 · 13 posts · 7 votes
    9y
    Originally posted by @Gretchen Roberts:

    So this is really interesting, and makes me wonder what is on MLS that you can't get on Zillow and Trulia and Redfin exactly? People talk about having or needing access to the MLS, which makes me think that these programs are only able to pull some of the feed instead of all of it.

    The biggest difference is that many, if not most local MLS boards intentionally delay the feeds that go out to the Big 3 and other non-MLS member sites, whereas our local MLS for example, updates their feed every three hours. This is done mainly to give the member broker's websites priority.

    There is typically more data available through the MLS that you can't access via the other websites. Mainly the "sold" data. This data could take weeks or months to show up on these sites, whereas it's available within days if you are an MLS member with access.

    It should also be noted that representing MLS data without being an approved vendor through any given MLS is a copyright and trademark infringement.

    0Reply
  • Joseph P McGranaghan
    Nashua, NH · Member since 2017 · 14 posts · 10 votes
    8y

    Poking around, I found a way to get sold listings from realtor.com.

    I believe there are ways from zillow and redfin also, but might require more effort and parsing.

    This is fairly straight forward though with a curl call like:

    curl 'https://www.realtor.com/search_result.json' -H 'content-type: application/json' -H 'accept: application/json, text/javascript, */*; q=0.01' --data-binary '{"search_criteria":"Nashua_NH","city":"Nashua","county":"Hillsborough","discovery_mode":false,"state":"NH","postal":null,"sort":null,"position":null,"facets":{"beds_min":"3","beds_max":"","baths_min":null,"baths_max":null,"price_min":null,"price_max":null,"prop_type":"","sqft_min":null,"sqft_max":null,"acre_min":null,"acre_max":null,"lot_unit":null,"age_max":null,"age_min":null,"radius":null,"pets":null,"days_on_market":null,"open_house":null,"show_listings":null,"pending":null,"foreclosure":null,"new_construction":null,"multi_search":{},"include_pending_contingency":false,"features_hash":[],"prop_types":[""]},"search_controller":"Search::RecentlySoldController","neighborhood":null,"street":null,"searchType":"city","school":null,"pos":"42.763275,-71.579493,42.790491,-71.412467,13","types":["property"],"searchFacetsToDTM":["beds_min"],"searchFeaturesToDTM":[],"page_size":80,"viewport_height":216,"pin_height":25}' --compressed

    Pastebin link: (Above call here also)

    Pastebin link: JSON Output

    That's stuctured JSON data. I don't see MLS number in the results, but the address is there,

    so one way or the other you could join to other data sets.

    3Reply
  • Wes Wang
    Member since 2018 · 3 posts · 0 votes
    8y

    Thanks @Joseph P McGranaghan.  I'll try the realtor code.  Do you know if Redfin's RSS feed still work?

    0Reply
  • Jeremy Ferguson
    Chapel Hill, NC · Member since 2014 · 116 posts · 56 votes
    8y

    Very interesting, I might look into this. I have a script now that scans local county tax records and then grabs all the listings that meet my criteria. Then I dump them into a database.

    Collecting the data above should be pretty straightforward but designing some sort of front-end/map view might be a little time consuming.

    0Reply
  • Michael J.
    Rental Property Investor · Louisville, KY · Member since 2008 · 342 posts · 123 votes
    8y

    @Jeremy Ferguson, that is interesting I am planning on doing something similar with the counties in my area. I am a few notches below an armature coder though. Do you have any suggestions on what type of language to use for something like that where it will pull specific data from a web site? I would love to pull sales prices on houses and figure out a way to use it for comps a little easier than pumping in addresses on the tax record site.

    0Reply
  • Daniel Kong
    Rental Property Investor · Honolulu, HI · Member since 2018 · 335 posts · 251 votes
    8y

    @Cory Land Cool stuff man! Software Developer here too. What technology stack are you using for your stuff? I've built my own little scrapers using node and npm recently. A friend of mines said that for 1200 you can become a broker and have access to the entire MLS for the country. Was thinking of doing something like that to get access to more data. Would love to chat a little more with you about this.

    0Reply
  • Tony Zuanich
    Orange, CA · Member since 2015 · 11 posts · 3 votes
    7y

    Here is a redfin URL that will give all the on market homes based on search, in a CSV file:

    https://www.redfin.com/stingray/api/gis-csv?al=3&m...

    0Reply
  • Mike Mart
    Developer · TX · Member since 2019 · 1 post · 0 votes
    7y

    For some reason the link didn't work but I found the original documentation in reference to the link if people are still interested in using what Tony posted.

    https://www.pydoc.io/pypi/crawl_redfin-0.0.3/autoapi/urlencoder/index.html

    0Reply
Join the conversationCreate a free account to reply, vote on answers and follow this thread.
Log InJoin for Free