Let me know if this is too specific.
I have an AbstractConnection class which handles updating my ActiveRecord models with data retrieved from external APIs. I then have a class inheriting from AbstractConnection which handles all of the specifics of the API I’m using (as there will be others in the future) and returns the attributes in arrays/hashes.
One operation involves the loading of 10,000+ records, and the API limits the number or records per request, and provides a “next” URL which offsets the query so you can load all the records in a series of requests. I would prefer not to load an unknowably-huge number of records into memory in my API-specific method, so I’d like to be able to update each batch of records as soon as I get them. However, I worry about exposing too much of the API-specific implementation to the AbstractConnection.
Anyone handled something like this before? Thanks!