Activejob tasks get stuck after a while

In my Rails app I am using ActiveJob to run some tasks in background. The task uses a simple ActiveModel based model (no db call) to send a chat message to a xmpp group chat room (using xmpp4r)

There is a cronjob that makes a REST call to this job every 5 minutes. After running fine for a few hours (this time varies) the jobs are just enqueued but not picked up and executed anymore. When I restart apache/passenger or unicorn things start working again.

I have tried both sucker_punch as well as delayed_job with both apache passenger as well as Nginx/unicorn with similar results.

Any ideas what could be going wrong here?