For anyone trying to access the site over the past 4 hours, you noticed that we were completely offline. We had a massive crash to the servers and are still working on a long-term fix, but we're at least up temporarily.
I'm going to try and explain why we've been having problems with minor and MAJOR crashes over the past few months, as I think you all deserve to know.
1 - Every time you login to the site the application creates what is called a session. These sessions tell BP who you are and allow you to participate. This data is supposed to be purged at a regular interval, but it seems that one of our developers failed to ensure that this took place.
Our sessions table in the BP database is hundreds of gigabytes in size. Tonight's crash occured because the database grew so large that we simply ran out of space on our database hard drive array. We've managed to clear out some free space and are dumping the old data to make room.
2 - We've been crashing fairly regularly in the mornings (typically around 6:30am EST). This problem was rectified a few days ago when we changed how we run backups of the site. The reason for these crashes is that in creating a backup for the site, the huge amount of space (partially due to #1 above) that was being zipped up and then written to our backup server took a huge load on our main BiggerPockets Application server. This problem looks like it is resolved.
3 - We've also been crashing in the afternoons fairly regularly. We believe that these crashes are caused by our site's search engine, called Sphinx. For the techies out there, it seems that when our search indices are rotated, Apache craps out. We need to then kill and start or restart Apache to get the site back up. The resolution to this problem, we believe, is to replace the search engine. We've found replacements and hope to have them integrated within the week. If this doesn't solve the problem, there are likely other issues that we're missing and will then have to diagnose them.
Please understand that every time we go offline, I get a page and the act immediately to get us back up and running. I've had dozens of days that I've been woken up at 3 or 4 in the morning and have had to do this. I've also had my life interrupted on many many occasions otherwise.
The reason I'm sharing this with all of you is that I want you to understand what we're dealing with.
BiggerPockets is an upstart company, and while we're helping many of you fill your pockets, we're struggling to do the same for ourselves. Given these constraints, we can't hire the best programmers because we simply can't afford them. This is an ongoing battle that I face daily. It is extremely frustrating.
Bottom line . . . . please be patient while we resolve these issues. We're not going to just let them defeat us. Also, if you're making or saving money as a result of BP, please consider donating to help us out.
Sincerely,
Josh
