Monday, 7 June 2010

C#: Per-process CPU Usage and Network IO Tracking using PerformanceCounter

I have been tacking per-process CPU percentage and Network IO data bytes in my recent project using System.Diagnostics.PerformanceCounter class. I felt the online materials I found does not explain the usage of the PerformanceCounter very well.

Some basics about PerformanceCounter.
1. A performance category has many performance counters underneath. A performance counter may be single-instanced or contain multiple running instances. The structure is like this:
PerformanceCategory
    -- PerformanceCounter
        -- Instances
To get all the built-in categories:
PerformanceCategory.GetCategories()

To get all the counters under the category that has exactly one running instance:
PerformanceCategory.GetCounters()

2. The constructor to create a performance counter is:
PerformanceCounter(string categoryName, string counterName, string instanceName.

To monitor the IO (file or network) data bytes sent/received by FireFox:
PerformanceCounter counter = new PerformanceCounter("Process", "IO Data Bytes/sec", "firefox");

To monitor the CPU percentage used by FireFox:
PerformanceCounter counter = new PerformanceCounter("Process", "% Processor Time", "firefox");

3. To take a sample and get the calculated value:
counter.NextValue();
Before calling this method, make sure you wait for at least one second.
counters.Add(new PerformanceCounter("Process", "IO Data Bytes/sec", "firefox"));
counters.Add(new PerformanceCounter("Process", "% Processor Time", "firefox"));

System.Threading.Thread.Sleep(1000);

foreach(PerformanceCounter counter in counters)
        Console.WriteLine(string.Format("{0}:{1}}, counter.CounterName, counter.NextValue());

Some lessons I have learned:
1. The calculated value of the first sample is always 0. E.g. the very first time you call NextValue(), the returned value is always 0.

2. If an application has more than one running instances, even though from the Task Manager, these running instances all have the same process name, E.g. "TestProcess", their instance names are not the same. The instance started first has its name as "TestProcess", the second "TestProcess#1", the third "TestProcess#2"...the nth, "TestProcess#n". So make sure the right instance name gets passed into the performance counter constructor.

3. If any of the running instances of the same application exits, the instance names will be changed. If your code doesn't update the instance name accordingly, an "Instance not found" error will occur.

E.g. If "TestProcess" of the 2 running instances, "TestProcess" and "TestProcess#1", stopps running, "TestProcess#1" will be renamed to "TestProcess". If the counter still has the name "TestProcess#1", it will throw an exception when NextValue() is called. The instance name can be updated by simply calling counter.InstanceName = "new instance name".

4. In my case, I am only interested in getting per-Process statistics. Unfortunately, the PerformanceCounter does not operate against the process id. To get around this limitation, I have created my own wrapper class called ProcessPerformanceCounter.
public class ProcessPerformanceCounter
{
    ...
    public ProcessPerformanceCounter(string counterName, int processId) {...}
    public string CounterName {get{...}}
    public string InstanceName {get{...} set{...}}
    public NextValue(){...}
}

Based on the process id, I can get the process name. I can then get all running processes under the same name. From there, I can get the correct instance name for that particular process:
private string GetInstanceName(int processId)
{
        string instanceName = GetProcessNameById(processId);
        bool found = false;
        if (!string.IsNullOrEmpty(instanceName))
        {
                Process[] processes = Process.GetProcessesByName(instanceName);
                if (processes.Length > 0)
                {
                        int i = 0;
                        foreach (Process p in processes)
                        {
                                instanceName = FormatInstanceName(p.ProcessName, i);
                                if (PerformanceCounterCategory.CounterExists("ID Process","Process"))
                                {
                                        PerformanceCounter counter = new PerformanceCounter("Process", "ID Process", instanceName);

                                        if (processId == counter.RawValue)
                                        {
                                                found = true;
                                                break;
                                        }
                                }
                                i++;
                        }
                }
        }

        if (!found)
                instanceName = string.Empty;

        return instanceName;
} 
 
private string FormatInstanceName(string processName, int count)
{
        string instanceName = string.Empty;
        if (count == 0)
                instanceName = processName;
        else
                instanceName = string.Format("{0}#{1}", processName, count);
        return instanceName;
} 
 
5. When calling counter.NextValue(), do not forget to handle the instance name changed scenario:
public float NextValue()
{
        float nextValue = 0;
        try
        {
                nextValue = counter.NextValue();
        }
        catch (Exception ex)
        {
                if (InstanceNotExistError(ex))
                {
                        // adjust instance name based on process id
                        instanceName = GetInstanceName(processId);
                        counter.InstanceName = instanceName;
                        nextValue = counter.NextValue();
                }
                else
                        throw;
        }

        return nextValue;
} 

Tuesday, 1 June 2010

C#: Policy based actions in C# using the Action delegate

I think it is not just me that, as a C# developer, sometimes I want that AOP feature so badly. Yes, a lot of AOP features can be simulated, but it's not that easy and mostly comes with a penalty of performance.

For example, I want to have some custom exception handling policy. It would be neat if I could simply decorate the method with an attribute as below.

[ExceptionHandlePolicy]
public void Method(){...}

To achieve this in C#:
  1. An attribute class implementing the IMessageSink is to be defined;
  2. The class, to which this attribute is going to be applied, must inherit from ContextBoundObject or MarshalByRefObject. 
Ehmmmm...all these are not easy changes to the existing code base, and the change to hook into the message sink will degrade the overall performance for sure.

The other way around, is to use the Action delegate from the .Net Framework to help to build a custom exception handling policy class.

public class RetryPolicy
{
    ...
    private Action _action;
    ...
 
    public RetryPolicy(Action action)
    {
        _action = action;
    }

    public void Execute()
    {
        int retryCount = 0;
        if(_action != null)
        {
            while(retryCount < MAX_RETRY)
            {
                try
                {
                    _action();
                    retryCount = MAX_RETRY;
                }
                catch(Exception ex)
                {
                   if(ShouldRetry(ex) && retryCount < MAX_RETRY)
                        i++;
                   else
                        throw;
                }
            }
        }
    }
    ....
}

To apply my custom retry policy, I can modify the existing data access layer slightly.

public static string GetConfigurationValue(string dbConnString, int configId)
{
    string configValue = string.Empty;
    RetryPolicy policy = new RetryPolicy( () =>
    {
        using(TestDAL dal = new TestDAL(dbConnString))
        {
            configValue = dal.GetConfigurationByValue(configId);
        }
    });

    policy.Execute();
    return configValue;
}

Thursday, 1 April 2010

SQL Server 2005: How to report time-series data in vanilla SQL

Let's assume I have a table recording some user session data like the following.



Now, I want to report the number of concurrent sessions at a 5 minute interval. Suppose the report time window is from 7 am - 7 pm. How to build a time series table?


The SQL above gives a time series table like: 07:00 07:05 07:10 07:15 ... 20:00

Define the report table as this



Let me populate the report table using the query as below.



I can see at what time on which day of the month the system has the max concurrent session.

Friday, 19 February 2010

A Custom BizTalk File Adapter - BizTalk 2006 R2

The native FILE adapter released with BizTalk eats all zero-byte (empty) files without triggering any associated processes. This behaviour, according to Microsoft, is by design. Though you can argue that it is not consistent how empty files are treated by different adapters, e.g. FTP adapter can transfer empty files with no problem.

This behaviour has caused us grief after an upgrade of a third-party system. The system now creates an empty trigger file first without locking it. Then it grabs the file again and writes some data to it. So you see the problem here, the BizTalk FILE adapter may grab the file AFTER the system creates the empty file but BEFORE the system attempts to write to it.

What we want here is to have a custom file adapter that will ignore zero-byte files and leave them untouched.

After reading this good MSDN article, I got some idea and am ready to give it a go.

I go to the AdaptersDevelopment folder which is part of the BizTalk 2006 SDK.


Open the File Adapter folder.


Open the project file under the Runtime folder. I found DotNetFileReceiverEndpoint.cs is the file I'm interested in. The DotNetFile adapter is implemented as a Timer thread which checks the configured folder at the configured polling interval. To make it to ignore zero-bytes files, one change is required in the PickupFilesAndSubmit() method. I've made some other changes to suit my specific needs but the change highlighted below is the most important one.


I then open the Design Time folder, and edit the ReceiveLocation.xsd. This file defines the configuration properties that will display for the custom file adapter. I don't want to have some property to be mandatory so I change it to optional in the schema file.


Compile the Runtime and Design time projects. GAC the Microsoft.Samples.BizTalk.Adapter.Common.dll under C:\Program Files\Microsoft BizTalk Server 2006\SDK\Samples\AdaptersDevelopment\BaseAdapter\v1.0.2\bin\Debug.

Go back to the File Adapter folder, there are 2 .reg files. If you want to use the provided configuration UI, open StaticAdapterManagement.reg and make sure everything is correct or update the file paths to wherever the compiled DLLs are saved. Then simply double click DynamicAdapterManagement.reg to register the new custom file adapter. If you want to customize the property configuration UI, use DynamicAdapterManagement.reg instead.


Start up BizTalk Admin console, add the new adapter. The newly registered "Static DotNetFile" will appear in the Adapter dropdown list.



OK, the custom file adapter is all ready to be used! And it's not hard at all to re-brand the file adapter if you need to.