I’m curious if anyone has any suggestions or experiences on how to solve this.
I’m going to be sending a large amount of event data to Keen.io - about 100 events a second - using the server-side API. That part is pretty straightforward. I also want to log/store the events (which are small JSON packets) in a secondary data store so I can audit the data that was sent and make sure it is accurate. Reliability of the secondary data store is important.
The site is running on Heroku, so I don’t have access to a durable filesystem.
I’ve been thinking about saving them in a database and periodically flushing it to S3, sending it to a custom Splunk source via their API, dropping the JSON into DynamoDB, and all manner of other things.
Any other suggestions or idea?