UX Patterns when interacting with an external API via a background job

Ruby on Rails
1

When interacting with an external API, as a best practice, you should do it via background jobs.

I fully agree with that advice.

However, how do you proceed to build the front-end given that you’ve made a previously synchronous request, asynchronous?

With email, you can fire and forget.

But, what about the scenario where the result is required to be used in the next request?

I was recently working through a problem where I had to interact with an external API and the result of the request determined what to do on the next screen. I ended up solving the problem with polling. And I can think of other ways for e.g. Server Sent Events, Web Sockets, etc. Are these the typical ways of solving the problem?

Here was the solution without background jobs:

  • User clicks a button to perform some action via POST
  • Request is sent to the server and Rails routing determines the correct controller/action to invoke
  • Within that action I send a blocking request to the external API which creates some resource and returns it to me
  • Assuming everything went well I now have that resource
  • Since all the information is now available to me I know where to redirect and hence which page to show

Here was the solution with background jobs:

  • User clicks a button to perform some action via POST
  • Request is sent to the server and Rails routing determines the correct controller/action to invoke
  • Within that action I tell a background job to perform an external API request which creates the resource I need
  • Since that previous action was non-blocking I need to continue without knowing what will occur, so I immediately render a view
  • The view then shows a spinner so the user knows something is happening and they have to wait and there is some JavaScript code that polls my server to determine when the resource becomes available
  • When the resource does become available the view is updated

Now, I only described the happy path but obviously in a production app we have to deal with errors as well. So in summary,

How would you structure such an app so that it still has a great UX in both success and failure of the background job?

Here’s the app. And here’s the code I wrote that works for me.

PS I made a short video to explain my problem a bit better.

PPS Thanks @cpytel. I got to ask my question.

1 Like
2

What you’ve described feels like a solid flow to me.

One thing we’ve done recently is move an app from polling to using pusher (pusher.com). It’s a nice win because the user gets a faster response (no need to wait for the next poll) and lowers the load on our server.

As for how to handle error, I suppose it depends on what the user’s options are at that point. Can they retry? Skip that action? I’d try to make it clear to the user what they should do when things went wrong (if anything, maybe it’s unrecoverable and they need to email you?).

1 Like
3

Thanks @benorenstein.

In the hours after asking this question I came across this article on Heroku that attacked the same problem and described a similar approach.

So, +2 from respected sources (Ben + Heroku) must mean it’s highly probable that I’m doing it right.

And a follow on question: Where would you write that code? I just made a non-resourceful action and polled against it. I know you guys advocate sticking to the 7 default resourceful actions. Is this one of those times when you can break this rule or should I be thinking harder to map it onto a new type of resource.

4

I try to follow REST whenever possible, but if this is something that really doesn’t make sense to represent that way, just stick to what you’re doing and don’t worry about it. :smile:

1 Like
5

@dwaynecrooks Hey, I watched your video, nicely done! I think the UX is really nice. You have indeed followed the right patterns, but the only thing, as Ben alluded to, is that your server load could end up quite high when using long polling to check for a response from the delayed job. So, the answer is to use Web Sockets for browsers that support it and fall back to long polling as a last resort. One alternative to Pusher that I’ve played with in the past is socket.io, which falls back to long polling automatically for you. I’m sure Pusher does the same but can’t find any reference to that in their docs at the moment.

6

I’m told that’s true.

7

Thanks @aaronmcadam. Yeah right now it’s polling the server every second. That’s not cool. I am aware of both Pusher (wow, they did a redesign) and socket.io (and they did too, nice) but didn’t want to bring in those dependencies if there was another way.

PS For those interested here’s a tutorial I was working on a while back on building a chat application with Pusher and Rails. And here’s a Tic-tac-toe game I wrote with CoffeeScript and socket.io.