Job scheduler via Azure Service Bus

There are many kinds of jobs that applications can run regularly. For instance:

• renewing subscriptions for paying users on the first of every month
• downloading new data from partner systems every X minutes
• sending reminder or close subscriptions for high-debt users
• jobs related to well-known events like opening/closing stock exchange, closing fiscal year, public holidays, black Friday and so on.

What tools can be used? Usually, cron, Windows Task Scheduler, Quartz.NET are used or Task.Delay after all.

Problem

Mostly, job schedulers are doing their job fine unless you run multiple instances of the same application for scalability and/or availability purpose. Just imagine you have 5 instances of the application. cron, Windows Task Scheduler and Task.Delay will fail in such setup. However, Quartz.NET will work if database backplane is enabled.

Solution

I would like to show how a cloud solution can help, specifically Azure Service Bus.

Azure Service Bus is a fully managed enterprise integration message broker. Service Bus is most commonly used to decouple applications and services from each other, and is a reliable and secure platform for asynchronous data and state transfer — Microsoft

It has a really useful feature - scheduled messages at a low price. $0,05 per million operations should be enough. Azure Service Bus should work well - Stack Overflow proof. So our jobs will be distributed evenly across multiple instances and if some instance dies the others will get their jobs to run.

To implement the scheduler I will use C# and Microsoft.Azure.ServiceBus package. However, it is easy to implement in your programming language since .NET, Java, Node.js, PHP, Python, Ruby are supported and REST API fills that gap for other languages.

public class ServiceBusJobScheduler
{
    private readonly string _connectionString;

    public ServiceBusJobScheduler(string connectionString)
    {
        _connectionString = connectionString;
    }

    public async Task Run(
        string queueName,
        Func<Message> init,
        Func<Message, Task<Message>> job,
        CancellationToken cancellation
    )
    {
        var queueClient = new QueueClient(_connectionString, queueName);
        var created = await EnsureQueueCreated(queueName);
        if (created)
            await queueClient.SendAsync(init());
        queueClient.RegisterMessageHandler(
            handler: async (message, _) =>
            {
                var nextMessage = await job(message);
                if (nextMessage != null)
                    await queueClient.SendAsync(nextMessage);
            },
            exceptionReceivedHandler: _ => Task.CompletedTask
        );
        await Wait(cancellation);
        await queueClient.CloseAsync();
    }

    ...
}

A new queue queueName must be created per a job. init callback is called the first time only to schedule very first run. job function is our job itself, it receives a scheduled message and returns the next scheduled message. In the scheduled message you can store a state for the job between runs. cancellation is the way to stop the job. EnsureQueueCreated creates queueName queue if it is not there. Wait just waits until cancellation is requested. The complete source code is on Gaev.Blog.AzureServiceBusTaskScheduler.

That’s how a job looks like.

var scheduler = new ServiceBusJobScheduler(ConnectionString);
await scheduler.Run(
    queueName: "TakeABreak",
    init: () => new Message
    {
        ScheduledEnqueueTimeUtc = DateTime.UtcNow
    },
    job: async (message) =>
    {
        var scheduledFor = message.ScheduledEnqueueTimeUtc.ToLocalTime();
        Console.WriteLine($"Take a break! It is {scheduledFor:T}.");
        await Task.Delay(100);
        return new Message
        {
            ScheduledEnqueueTimeUtc = CronExpression
                .Parse("*/30 8-18 * * MON-FRI")
                .GetNextOccurrence(DateTime.UtcNow, TimeZoneInfo.Local)
                .Value
        };
    },
    cancellation.Token
);

Let me introduce you TakeABreak job. It will remind me to take a break from the computer at */30 8-18 * * MON-FRI. I’m using cron expression to compute the next date via Cronos. That’s it and we are done.

Take a look at how tasks are distributed across multiple instances.

By the way, you can implement almost the same via Azure Queues but it has a limit for scheduling time. Messages cannot be scheduled more than 7 days ahead.

Pitfalls

Watch out for exceptions! By default, Service Bus retries 10 times then moves the message into the dead-letter queue.

Watch out for long-running tasks! By default, Service Bus waits 5 minutes then returns the message to the queue so others can process is again.

What job scheduler works well for you? Let me and readers know in comments :)