Why use Taskmatics Scheduler

Why use Taskmatics Scheduler?

Taskmatics is preparing for its first major release of its flagship application, Taskmatics Scheduler.  If you’re a .Net developer you should be excited about this application.  To understand why, it helps to understand the motivation behind creating the system in the first place.  We believe that the same reasons that compelled us to create Taskmatics Scheduler will be the same set of criteria that drive .Net developers to adopting it.

The Problem

As .Net developers, most of us have been involved with the development of Enterprise class systems.  These are generally large, often complex applications that:

-        Have encapsulated business logic in code – via services, assemblies, etc…  e.g., the ‘Customer’ object that encapsulates all rules for managing a customer

-        Encapsulates complex business processes involving multiple objects, e.g. onboarding/ingesting product, creating an order, etc…

-        Uses one, or multiple data stores for persistence

-        Have one or multiple UI layers, e.g. an administrative app, consumer facing app, etc…

All of these components work together to form our respective ‘systems’;  However, problems begin to arise when we need to support batch or offline operations utilizing the business rules already built into these systems.  For example:

-        At regular intervals we need to ingest new product into our catalog

-        At specified times we need to check with a 3rd party to see if there are new orders to add to our system

-        ETL activities (import/export data)

-        Rebuild indices, aggregate data

Thus, the need for a centralized job management solution in this type of environment is crucial.

A Pseudo Solution

Like many .Net developers, we turned to Windows Task Scheduler for a solution to our job management needs.  Windows Task Scheduler provides an ability to start batch or .exe files, has a multitude of scheduling options, and is available on any flavor of Windows servers’ installations.

Issues with approach

Windows Task Scheduler views each task under its management as an independent entity.  This approach has some benefits, but also some severe drawbacks that quickly become management headaches as the underlying job infrastructure evolves.  Notably:

-        No common framework, job infrastructure quickly becomes the wild west

-        No centralized logging solution

-        No remote management

-        Very difficult/clumsy mechanism for using shared files (assemblies)

-        No extensibility

-        Single server solution

-        Laborious to update/maintain existing jobs

(Note:  We evaluated SQL Server, Quartz, and ActiveBatch as well.  Please stay tuned for pending discussions regarding these products shortcomings and why we were compelled to create Taskmatics Scheduler)

Finally, a Solution!

We concluded that there was not a suitable task management system available that addressed the needs of an enterprise .Net developer, so we decided to write our own.  From our experience, we were certain that a task management solution needed to have at least the following:

-        Job isolation (a poor performing job can NOT bring down the entire job infrastructure)

-        Remote management

-        Extensibility

-        Common Framework

-        Common Logging

-        Ability to update jobs while jobs are running

-        Ability to leverage common code (assemblies)

-        Reporting

-        Resource utilization by Job

-        Security (access and authorization)

-        Ability to scale out to multiple job servers

-        High Availability configuration

We knew that there weren’t any applications available in the marketplace that supported our desired feature set, so we decided that the only way to get what we needed was to create our own.  So, we did just that.

Our initial version of the application was created for our own internal use.  It was a bare bones application that lacked an administrative console and had a very crude configuration system.  However, our internal adoption and reliance on that crude system convinced us of the needs for this feature set, so after 2 years of evolution and internal use we re-developed the application with a goal of releasing it to the public.

And thus, Taskmatics Scheduler was born!  Taskmatics Scheduler represents all of the knowledge learned from our experience with the initial system and, most importantly, addresses the areas that we found lacking (configuration, installation).  The result is a full featured task management system that is a must have for any .Net developer.

Dials and Switches – Building Configurable Tasks

As a developer, it’s rare to see an application that doesn’t make use of external configuration settings. Whether it’s to set the connection string to a database or to store custom keys with the credentials to a service being consumed, configuration settings are a ubiquitous tool for developers. It’s not a mystery why this is the case as there are many advantages to using configuration settings when developing. Here are some of the key values that configuration settings can add to an application.

Reconfiguration without Recompilation

Many times an application will need to make use of values that can change after deployment. A prime example of this would be the connection string for data storage. Many times these connection parameters differ based on the environment to which the application is being deployed. Hard coding these values into the program itself introduces the need to recompiling and deploying the code each time any of these values are changed. This leads to poor maintainability of the code base and an inability to easily deploy applications across multiple environments. External configuration allows us to centralize and maintain all of the parameters of an application independent from the build and deployment process.

Reusability of code

Another frequent use for configuration settings can be seen in applications that share common behavior while differing only in the data they work with. For example, if I’m writing an application that zips up a directory and sends the zipped contents to a backup drive, I wouldn’t want to duplicate the application each time that a new folder needed to be backed up. External configuration settings allow for reuse of common logic while providing a way to introduce non-static data elements.

Basic Data Validation

Configuration not only allows you to specify parameters for use within your applications, but they also define a general syntax that allows for rules to be created that control the validity of the values entered for those parameters. The ability to restrict data types, specify string patterns and control a host of other criteria over the values being set increases the integrity of the data and the application itself because developers know what to expect when referencing these parameters in the code.

Configuration Options in Taskmatics Scheduler

There are a couple of ways to specify configuration values in the scheduler. Standard .NET configuration files are supported and are managed in the file system. In addition, the scheduler API allows developers to create their own custom parameters objects that can be used to describe input or output values for their custom scheduler components. Furthermore, the parameters objects are dynamically translated into a custom form in the administration website where the values can be set when creating or modifying scheduler components. The two methodologies can be used in conjunction with one another, so you can use both of them if you so desire.

.NET Configuration files

Tasks for the Taskmatics Scheduler are written in .NET, therefore they include support for .NET configuration files. When developing a custom scheduler component, simply adding an app.config to your project will allow you to access that configuration from your code. Here are some key things to know about using .NET configuration files when you create scheduler components:

  • Using .NET configuration files does not preclude you from being able to use a custom parameters object as the two mechanisms can be used in conjunction. You may want to use .NET configuration to define database connection strings while using a custom parameters object for specifying service address Uri values.
  • In the scheduler system, updating a configuration file requires that you locate the folder where the file is stored and update the file at that location. This can be done from the administration website by updating that file from the built in file management screens. It can also be done from the file system directly, but production deployment folders often have restricted permissions making this strategy less than ideal.

Custom Parameters Object

The scheduler API allows developers to create their own objects that can define the input or output parameters of the components they create. These objects are nothing more than classes that are linked to the components that they are creating. Consider a sample configuration for a task that copies files from a source folder to a destination folder:

public class CopyTaskParameters
{
    public string SourceFolderPath { get; set; }
    public string DestinationFolderPath { get; set; }
}

 

This class gives us a basic parameters object for copying files between two folders, but the custom API allows us to also specify some basic validation criteria of these properties using well known attributes from the DataAnnotations .NET library. Adding that to our example we have something like:

public class CopyTaskParameters
{
    [Required]
    [Display(Name = "Source Copy Path", Description = "The folder that files will be copied from.")]
    public string SourceFolderPath { get; set; }

    [Required]
    [Display(Name = "Destination Copy Path", Description = "The folder that files will be copied to.")]
    [RegularExpression(@"\w{1,100}")]
    public string DestinationFolderPath { get; set; }
}

 

As you can see from the above, it’s fairly trivial to describe the parameters object and even set some basic validation. As a last step, we need to link this object to the task that we’re developing so that the scheduler can associate the task with the configuration object when we are administering the task later on.

[InputParameters(typeof(CopyTaskParameters))]
public class CopyTask : Taskmatics.Scheduler.Core.TaskBase
{
    protected override void Execute()
    {
        var copyParameters = (CopyTaskParameters)Context.Parameters;

        //your task implementation goes here
    }
}

 

In the code above, we’re linking the CopyTaskParameters object to the CopyTask by using the InputParameters attribute from the scheduler API. When scheduler components are linked to custom parameters objects, the administration website will use the custom parameters object to display an entry form for users to configure those parameters directly from the website. It will also provide validation feedback to the user to ensure that the data being entered is what’s expected by the parameters object. Once the task is initiated by the system, it will create an instance of the CopyTaskParameters object and pass it in as part of the Context object that’s accessible from the task. Here’s a shot of what our configuration object will look like when we’re configuring our task from the website:

copy-task-sample-config-ui

Here are some key points to keep in mind when using custom parameter objects in the scheduler system:

  • Like .NET configuration files, custom parameter objects allow you to avoid hardcoding data elements that may change frequently or that don’t warrant a recompile of the code.
  • Custom parameter objects are integrated with the administration website to make managing the configuration for custom tasks easy. It abstracts the need to know the .NET configuration file schema so that non developers can tweak configuration when desired.
  • Currently developers can only use scalar and enumeration properties in a custom parameters object. List and complex object support are to be added in future releases.

Summing It Up

Regardless of the complexity of application you write, using external configuration settings will go a long way towards making your code more versatile and facilitate the maintenance of your application post deployment. Taskmatics Scheduler allows you to use the methods you already know and love, and also introduces a new, simple way to manage external settings that integrates seamlessly with the administration website. For a fully working sample that contains the code used in the article, head over to our GitHub repository, and for more information and more advanced configuration scenarios, check out the online documentation.