If you’re using Rails 4.1, you can store these items in environment variables and then reference them in secrets.yml.
In any other version of Rails (this also works in 4.1), the best way to do it is to have the credentials stored in a .env file, and then have the dotenv and dotenv-rails gems in your development and test environments. Those gems load the configured variables in your local file just for your application, which is great if you hack on more than one app. Environment variables can be useful for other parts of your configuration that aren’t passwords as well.
Heroku has a great config part of the toolbelt for pushing up and pulling down environment variables.
The most important part of whatever solution you use here is to NEVER CHECK THE ACTUAL PASSWORDS IN TO VERSION CONTROL. Your git history should never contain passwords or other sensitive credentials. If you accidentally do check passwords or other credentials into git, if you do a little Googling you can find instructions for how to comb through your history and remove the passwords. The procedure does completely rewrite your commit history in the process, so any collaborators of yours will have to force-pull to get their repositories back in sync.