My heroku postgres production DB was emptied out. How to diagnose?

Ruby on Rails
1

About an hour ago, I noticed that the production DB for one of our applications has an emptied out database. All of the tables have no records.

The last push to production was about an hour before that, but I browsed around following that push and the app was running fine.

This is odd, and I have a backup I can copy over (although we will lose some records). But I really want to know what happened. Could this be related to heroku’s problems this morning with Amazon / EC2 / S3? It seems odd that a symptom would be a database that has been reset, rather than an outage.

I don’t know where to look to see when / how the database was altered. Any ideas?

phil

2

I’ve seen this happen before where a developer using the heroku-config plugin synced the production configuration (environment variables) to their local machine and then ran the tests. The configuration included a DATABASE_URL variable that, when present, will override whatever you have in your database.yml for the test database. So the developer was running tests against the production database. These tests included database_cleaner which truncates all of the tables…

Could it have been something like that?

A good first step to findin gout would be to look through your logs. Do you have PaperTrail or some other logging plugin enabled? Look for TRUNCATE or DELETE calls. Also, don’t hesistate to get heroku support involved if you can’t find something obvious. They may be able to help pinpoint the issue and may also be able to help you restore more data than you had available in your backup.

3 Likes
3

Derek, I think that is what happened;

I exported heroku config to my .env file (I only needed some S3 keys) and I deleted the lines I didn’t need, but I left by accident the DATABASE_URL line in the .env file

And, I ran tests before I deployed. What a sad thing. I’m amazed that you diagnosed this.

upcase comes through again.

thank you.

4

@pdbradley: Just FYI this sparked a pretty extensive discussion, and we’ll be making it impossible to make this mistake in Rails 5.0.

1 Like
5

Maybe unrelated but I noticed that RSpec-rails now adds this line to rails_helper:

abort("The Rails environment is running in production mode!") if Rails.env.production?

Might help prevent future occurrences as well. Sad that it doesn’t come in time to help you @pdbradley :frowning:

On an even more unrelated side note, might be a good time to consider PRing that code over to Suspenders.

1 Like
6

@sgrif do you mean this particular incident of mine sparked a discussion? Or one was underway previously.

If it was mine, it is a small comfort that I have become a rails contributor as a kind of crash test dummy.

:smile:

1 Like
7

This particular incident. :slight_smile:

2 Likes
8 
abort("The Rails environment is running in production mode!") if Rails.env.production?

@pedrosmmoreira Would that line have stopped this issue? Because I think I was in test mode, but using the production database as defined by DATABASE_URL.

9

It depends :slight_smile: I assumed you have separate env file for testing and the error was caused by loading the production .env file. This can happen since rails_helper memoizes the env, which allows it to be set externally but can cause issues. I’ve encountered situations where running the test suite via rake sets the RAILS_ENV to development, which is correct, but that will load your actual .env file instead of an environment specific one.