Understanding TPL Dataflow – Conceptual Overview

TPL Dataflow is a fairly recent and awesome addition to the .NET framework. It provides developers a high level approach to dealing with asynchronous programming. Asynchronous programming can be daunting, especially when synchronizing threads and protecting shared memory. TPL Dataflow abstracts away a lot of that tedious and error prone code.

This series provides an in depth look into TPL Dataflow to see how it can benefit you as a developer. In this video I’ll cover the conceptual overview and describe the core concepts of TPL Dataflow. Enjoy.


Taskmatics Scheduler and Windows Azure Scheduler: Redux

There has been a lot of coverage lately about the Windows Azure Scheduler offering currently in preview. After getting familiar with the product, I thought it might be a good idea to talk about what it brings to the table and how it differs from Taskmatics Scheduler, which is currently in beta. We’ll also talk a little bit about how you can use the two platforms together to get robust automation for your environment.

What is Windows Azure Scheduler?

Windows Azure Scheduler is a service that exposes a REST API that allows you to schedule jobs that will be run either in a web or worker role hosted within Azure or on a remote server outside the cloud. These jobs could be run once or recurring in nature, and the API provides you with ways to group these jobs into a job collection, which is a logical grouping of jobs that share settings such as the region from which the job should be triggered. Currently, Windows Azure Scheduler offers two distinct ways of executing jobs scheduled:

  • The simplest way to execute jobs is by providing an HTTP or HTTPS endpoint address which the service will call at the scheduled time(s). This could be any HTTP endpoint that will execute the job when it receives a request from the scheduler. While simple, this method is more suited to small jobs with a duration of 30 seconds or less. The scheduler currently records the responses of the HTTP/HTTPS calls as whether or not the job succeeds, and the default timeout for the call is 30 seconds, which means that longer jobs will record a failure if this method is used.
  • The second method for job execution is for the scheduled job to post a message to an Azure Storage Queue. This will require you to set up and configure Azure Storage Queues on your account but it provides a way to trigger job execution by watching a queue from your own process. How the message is processed is determined completely by whatever application is listening to the queue, and the success or failure of the scheduled job is simply whether the queue receives the posted message successfully.

The Windows Azure Scheduler API also has methods to track and recall job status for all jobs and get success/failure history for one or more jobs as well, though again note that the success or failure not of the job execution itself, but simply the posting of the message to the endpoint or queue. Being that the service is currently in preview, I’m sure that even more functionality will be added over time. There are a few shortcomings of the new scheduler offering:

  • While jobs scheduled are run in the cloud and are therefore reliable, the endpoints or queue subscribers are not necessarily cloud based and the reliability and scale of these applications which will actually perform the work is still a burden placed on the developer. Running web/worker roles to host endpoints that can process messages can get costly since they’ll need to be listening (and thus active) 24/7.
  • Reporting and statistics for the jobs is currently very basic and doesn’t provide a way to extend the collected information such that it can be reported on.
  • Job execution cannot be controlled once the job has started running. Jobs cannot be paused/terminated.

What is Taskmatics Scheduler?

Taskmatics Scheduler is a task scheduling platform that combines a fully extensible .NET API for creating and executing jobs with an administration site that can manage and provide reporting on the jobs created with the API. Where Windows Azure Scheduler focuses on ways to trigger job execution, Taskmatics Scheduler covers the triggering of jobs, but also manages the distribution and execution of those jobs and provides health monitoring and detailed reporting during execution and beyond. Taskmatics Scheduler has the edge on Azure Scheduler when your automation needs extend beyond just a few small nightly tasks because it can handle distribution of the job load internally, without relying on the developer to ensure load balancing and availability of resources to run the desired job. The true power of Taskmatics Scheduler, however, is the extensibility it provides for developers. There are basically three main components to the job scheduling API:

  • Triggers define how job execution will be started. Like Windows Azure Scheduler, a calendar based trigger is provided out of the box, but you can create custom triggers that can fire based on domain specific criteria.
  • Tasks are the central unit of work in the system. They represent the work to be automated, and are executed by one or more triggers. Tasks can be paired with custom configuration to allow reusability and contain built in controls that allows users to pause or terminate running tasks.
  • Event Listeners define code that will execute when one or more target events are raised by a task during execution. Custom event listeners can be created that can be used for real time error notifications or integration with line of business applications such as Enterprise ERP and CRM systems.

Taskmatics Scheduler also provides a web based administration console where customized tasks can be loaded, scheduled and managed. The console also provides detailed reports and execution history for all tasks. If your job automation landscape is fairly complex or involves many long running tasks, Taskmatics Scheduler might be a better fit than using the Windows Azure Scheduler service.

The Best of Both Worlds

On one hand you have a cloud based scheduler that can reliably fire off messages, and on the other you have a fully customizable system that is designed to distribute and execute jobs. Can they be used together? The short answer is yes. If you want to benefit from scheduling jobs in the cloud environment, you can create a custom trigger for Taskmatics Scheduler that will listen on a given HTTP address and port as a receiver for the cloud based scheduler. Another option is a custom trigger that subscribes to the Azure Storage Queue that gets messages posted to it that can fire off one or more tasks within Taskmatics Scheduler. If you are drawn to the potential of Windows Azure Scheduler as a reliable, cloud based scheduling tool, I encourage you to put Taskmatics Scheduler to the test for that same reliability and much more.