Hello world!

Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!

Father Forgets

My wife sent me this heart touching snippet from a story written by W. Livingston Larned. In this crazy complex world of deadlines, stress and worries, I have many times done exactly what I should not have done - Forgotten that my son is just 2 years old. It brought tears to my eyes and I hope to never always remember that they are just children which is why they keep on doing what they are supposed to do. We as adults need to just spend a few moments in understanding them better.

Listen, son; I am saying this as you lie asleep, one little paw crumpled under your cheek and the blond curls stickily wet on your damp forehead. I have stolen into your room alone. Just a few minutes ago, as I sat reading my paper in the library, a stifling wave of remorse swept over me. Guiltily I came to your bedside.

There are things I was thinking, son: I had been cross to you. I scolded you as you were dressing for school because you gave your face merely a dab with a towel. I took you to task for not cleaning your shoes. I called out angrily when you threw some of your things on the floor.

At breakfast I found fault, too. You spilled things. You gulped down your food. You put your elbows on the table. You spread butter too thick on your bread. And as you started off to play and I made for my train, you turned and waved a hand and called, "Goodbye, Daddy!" and I frowned, and said in reply, "Hold your shoulders back!"

Then it began all over again in the late afternoon. As I came Up the road, I spied you, down on your knees, playing marbles. There were holes in your stockings. I humiliated you before your boyfriends by marching you ahead of me to the house. Stockings were expensive – and if you had to buy them you would be more careful! Imagine that, son, from a father!

Do you remember, later, when I was reading in the library, how you came in timidly, with a sort of hurt look in your eyes? When I glanced up over my paper, impatient at the interruption, you hesitated at the door. "What is it you want?" I snapped.

You said nothing, but ran across in one tempestuous plunge, and threw your arms around my neck and kissed me, and your small arms tightened with an affection that God had set blooming in your heart and which even neglect could not wither. And then you were gone, pattering up the stairs.

Well, son, it was shortly afterwards that my paper slipped from my hands and a terrible sickening fear came over me. What has habit been doing to me? The habit of finding fault, of reprimanding – this was my reward to you for being a boy. It was not that I did not love you; it was that I expected too much of youth. I was measuring you by the yardstick of my own years.

And there was so much that was good and fine and true in your character. The little heart of you was as big as the dawn itself over the wide hills. This was shown by your spontaneous impulse to rush in and kiss me good night. Nothing else matters tonight, son. I have come to your bedside in the darkness, and I have knelt there, ashamed!

It is a feeble atonement; I know you would not understand these things if I told them to you during your waking hours. But tomorrow I will be a real daddy! I will chum with you, and suffer when you suffer, and laugh when you laugh. I will bite my tongue when impatient words come. I will keep saying as if it were a ritual: "He is nothing buy a boy – a little boy!"

I am afraid I have visualized you as a man. Yet as I see you now, son, crumpled and weary in your cot, I see that you are still a baby. Yesterday you were in your mother’s arms, your head on her shoulder. I have asked too much, too much.

W. Livingston Larned.

Boiling the IT Frog

I was reading an article in ComputerWorld  which really caught my attention. Here is an excerpt that I found so true across various companies I have worked. This is part of an interview between Kathleen Melymuka and the Harwell Thrasher, author of the book "Boiling the IT Frog".

Why do most projects fail? In my experience, there are six primary reasons. First, you’re doing the wrong project — it’s not what the business really needs. Second, you’re missing prerequisites in the proposal. You start and then realize you have to beef up the infrastructure, for example, so you’re in trouble right from the beginning. Third, you’re going for home runs instead of base hits; for example, a global rollout of a major system instead of chipping away at it a little at a time. Fourth, the project’s duration is greater than the job tenure of the sponsoring executive. If the project isn’t completed before he or she leaves, you may be in deep trouble with the successor. Fifth, you gather requirements instead of negotiating them. If you go around asking all the stakeholders what they want and your project goal is the sum of all those things, you will undoubtedly have contradictions — for example, a simple system that must do everything. Sixth, there’s not enough contingency planning. You’ve got to anticipate things that can go wrong and take them into account in your plan.

Boiling the IT Frog

TechTalk: Bulk Update into SQL from C# App

In my previous article, I explained using a sample C# app on how to bulk insert data into SQL Server.

There is an inherent problem. Bulk Inserts work fine as long as there are no constraint violations. What if the primary key data already exists? Or, how do we update using BulkCopy?

These are some of the issues I ran into while doing some actual work.

I will soon post a solution to this problem…

TechTalk: Bulk Insert into SQL from C# App

 

One of the common problems I have seen is to bulk upload data to a SQL Server database. If you have the flexibility to directly run your code in SQL, you have a ton of options. But let’s say that you have to massage the data before you throw it in to the database, then you have to really know your SQL (well to do it in SQL).

Let’s say you have to read data from an RSS feed, parse it and then load it into SQL. Let’s assume further that this feed updates every 2 hours. It would be a trivial task to write a C# app that reads and parses the feed. One crude way to upload this data would be to do a single row insert for each data element. This would be terribly inefficient. The other option would be to use .Net framework’s SqlBulkCopy class.

The basic template would be something like

private void WriteToDatabase()
{
    // get your connection string
    string connString = "";
    // connect to SQL
    using (SqlConnection connection =
            new SqlConnection(connString))
    {
        // make sure to enable triggers
        // more on triggers in next post
        SqlBulkCopy bulkCopy =
            new SqlBulkCopy
            (
            connection,
            SqlBulkCopyOptions.TableLock |
            SqlBulkCopyOptions.FireTriggers |
            SqlBulkCopyOptions.UseInternalTransaction,
            null
            );

        // set the destination table name
        bulkCopy.DestinationTableName = this.tableName;
        connection.Open();

        // write the data in the "dataTable"
        bulkCopy.WriteToServer(dataTable);
        connection.Close();
    }
    // reset
    this.dataTable.Clear();
    this.recordCount = 0;
}

The above code snippet shows you the API usage. But before you actually do that, you need to follow a couple of steps to setup your data table.

First, let’s look at a simple record structure (as reflected in C# class):

using System;
using System.Data;
using System.Configuration;

/// <summary>
/// Summary description for MyRecord
/// </summary>
public class MyRecord
{
    public int TestInt;
    public string TestString;

	public MyRecord()
	{
	}

    public MyRecord(int myInt, string myString)
    {
        this.TestInt = myInt;
        this.TestString = myString;
    }
}

Now, let’s start dissecting the class that we will use to upload the data:

using System;
using System.Data;
using System.Collections.Generic;

using System.Data.SqlClient;
using System.Configuration;
using System.IO;

namespace SqlExamples.FileLoader
{
/// <summary>
/// Summary description for BulkUploadToSql
/// </summary>
public class BulkUploadToSql
{
    private List<MyRecord> internalStore;

    protected string tableName;
    protected DataTable dataTable = new DataTable();
    protected int recordCount;
    protected int commitBatchSize;

Note that we have an internal List data structure as well as the DataTable. This is redundant and you can avoid using the internalStore if your application does not need to massage the data before it’s sent to SQL.

I then define 2 private constructors. The reason is that we want to use the factory pattern to return our object to the caller.

    private BulkUploadToSql(
        string tableName,
        int commitBatchSize)
    {
        internalStore = new List<MyRecord>();

        this.tableName = tableName;
        this.dataTable = new DataTable(tableName);
        this.recordCount = 0;
        this.commitBatchSize = commitBatchSize;

        // add columns to this data table
        InitializeStructures();
    }

    private BulkUploadToSql() :
        this("MyTableName", 1000) {}

Note that we set the commit batch size. This is a very important factor that needs to be fine tuned for your database. What this defines is the number of records that we would send in one shot to the database.

The next step is to Initialize the data table with columns that reflect the actual table structure.

private void InitializeStructures()
{
    this.dataTable.Columns.Add("TI", typeof(Int32));
    this.dataTable.Columns.Add("TS", typeof(string));
}

I then provided a factory method to load data into my internal structure from a data source. In the example code below, I use a Stream, but this can be any data source from where you wish to populate your data.

public static BulkUploadToSql Load(Stream dataSource)
{
    // create a new object to return
    BulkUploadToSql o = new BulkUploadToSql();

    // replace the code below
    // with your custom logic 
    for (int cnt = 0; cnt < 10000; cnt++)
    {
        MyRecord rec =
            new MyRecord
            (
            cnt,
            string.Format("string{0}", cnt)
            );
        o.internalStore.Add(rec);
    }

    return o;
}

This would make sure that our class is properly initialized and loaded with data. Once the caller has a valid object, they can now "Flush" the data as shown below:

public void Flush()
{
    // transfer data to the datatable
    foreach (MyRecord rec in this.internalStore)
    {
        this.PopulateDataTable(rec);
        if (this.recordCount >= this.commitBatchSize)
            this.WriteToDatabase();
    }
    // write remaining records to the DB
    if (this.recordCount > 0)
        this.WriteToDatabase();
}

private void PopulateDataTable(MyRecord record)
{
    DataRow row;
    // populate the values
    // using your custom logic
    row = this.dataTable.NewRow();

    row[0] = record.TestInt;
    row[1] = record.TestString;

    // add it to the base for final addition to the DB
    this.dataTable.Rows.Add(row);
    this.recordCount++;
}

In the example above, the call to Flush() actually massages the data (and at the same time loads it into the actual data table). As I mentioned before, you can actually skip this step if your application does not require massaging.

As a example of an app that uses this class:

using System;
using System.Collections.Generic;
using System.Text;
using SqlExamples.FileLoader;
using System.IO;

namespace DemoApp
{
class Program
{
    static void Main(string[] args)
    {
        using (Stream s =
            new StreamReader(@"C:\TestData.txt"))
        {
            BulkUploadToSql myData =
                BulkUploadToSql.Load(s);
            myData.Flush();
        }
    }
}
}

As always, this is JUST demo code to explain a concept. This is NOT production quality code and please make sure to follow the coding guidelines in your team.

Happy coding….

TechTalk: Handling global web service unhandled exceptions

One of the most tiresome (but important) things when developing web services is handling un-handled exceptions. A good design principle forces you to catch and cast relevant exceptions raised by your web methods into more meaningful SOAP exceptions. But exceptions will occur.

It is quite tedious to wrap each web method in a try/catch loop. This dictates a need for a common framework to handle unhandled web services exceptions. In this blog post, I will guide you through a step by step process for building one.

Summary:

1. Extend SoapExtension class and override the ProcessMessage() method.
2. In ProcessMessage(), add special handler code for message stage of SoapMessageStage.AfterSerialize.
3. Modify the web service’s web.config file and add a <soapExtensionTypes> node to <webServices> section.

Details:

Ok, now let’s dig deeper into the code.

Step 1:

The first step in the whole process is to extend the SoapExtension class override the ProcessMessage() method.

using System.IO;
using System.Web.Services;
using System.Web.Services.Protocols;
using System.Xml;
using System;

public class TelspaceSoapExtension : SoapExtension
{
    private Stream originalStream;
    private Stream updatedStream;

    public override object GetInitializer(Type serviceType)
    {
        return null;
    }

    public override object GetInitializer
    (LogicalMethodInfo methodInfo, SoapExtensionAttribute attribute)
    {
        return null;
    }

    public override void Initialize(object initializer)
    {
    }

    public override Stream ChainStream(Stream stream)
    {
        originalStream = stream;
        updatedStream = new MemoryStream();
        return updatedStream;
    }

    private void Transfer(Stream inStream, Stream outStream)
    {
        StreamReader sr = new StreamReader(inStream);
        StreamWriter sw = new StreamWriter(outStream);
        sw.Write(sr.ReadToEnd());
        sw.Flush();
    }

    public override void ProcessMessage(SoapMessage soapMessage)
    {
        switch (soapMessage.Stage)
        {
            case SoapMessageStage.BeforeDeserialize:
                Transfer(originalStream, updatedStream);
                updatedStream.Position = 0;
                break;

            case SoapMessageStage.AfterSerialize:
                if ((soapMessage.Exception != null))
                {
                    ExceptionProcessor processor = new ExceptionProcessor();

                    string details;

                    // handle our exception, and get the SOAP <detail> string
                    details = processor.HandleWebServiceException(soapMessage);

                    // read the entire SOAP message stream into a string
                    updatedStream.Position = 0;
                    TextReader tr = new StreamReader(updatedStream);

                    // insert our exception details into the string
                    string s = tr.ReadToEnd();
                    s = s.Replace("<detail />", details);

                    // overwrite the stream with our modified string
                    updatedStream = new MemoryStream();
                    TextWriter tw = new StreamWriter(updatedStream);
                    tw.Write(s);
                    tw.Flush();
                }

                updatedStream.Position = 0;
                Transfer(updatedStream, originalStream);
                break;
        }
    }

}

Step 2:

The next step is to handle the exception and get out meaningful details from the exception. For this purpose, let’s dig deeper into the ExceptionProcessor class. This class has one public method: HandleWebServiceException(System.Web.Services.Protocols.SoapMessage sm). This method is called from our ProcessMessage()case SoapMessageStage.AfterSerialize.

Here is the code:

using System;
using System.Collections.Generic;
using System.Text;
using System.Xml;
using System.Web.Services.Protocols;
using System.Runtime.InteropServices;
using System.Diagnostics;
using System.Reflection;
using SASMTPLib;
using System.IO;

class ExceptionProcessor
{
    private string currentExceptionDetails;
    private string currentExceptionTypeName;

    private const string RootExceptionName = "System.Web.HttpUnhandledException";
    private const string RootWsExceptionName = "System.Web.Services.Protocols.SoapException";

    public string HandleWebServiceException(System.Web.Services.Protocols.SoapMessage sm)
    {
        HandleException(sm.Exception);
        XmlDocument doc = new XmlDocument();
        XmlNode detailNode = doc.CreateNode(XmlNodeType.Element, SoapException.DetailElementName.Name, SoapException.DetailElementName.Namespace);
        XmlNode typeNode = doc.CreateNode(XmlNodeType.Element, "ExceptionType", SoapException.DetailElementName.Namespace);
        typeNode.InnerText = currentExceptionTypeName;
        detailNode.AppendChild(typeNode);

        XmlNode messageNode = doc.CreateNode(XmlNodeType.Element, "ExceptionMessage", SoapException.DetailElementName.Namespace);
        messageNode.InnerText = sm.Exception.Message;
        detailNode.AppendChild(messageNode);

        XmlNode infoNode = doc.CreateNode(XmlNodeType.Element, "ExceptionInfo", SoapException.DetailElementName.Namespace);
        infoNode.InnerText = currentExceptionDetails;
        detailNode.AppendChild(infoNode);
        return detailNode.OuterXml.ToString();
    }

    private void HandleException(Exception ex)
    {
        try
        {
            currentExceptionDetails = NormalizeException(ex);
            currentExceptionTypeName = ex.GetType().FullName;

            // ignore root exceptions
            if (((currentExceptionTypeName == RootExceptionName) || (currentExceptionTypeName == RootWsExceptionName)))
            {
                if (ex.InnerException != null)
                {
                    currentExceptionTypeName = ex.InnerException.GetType().FullName;
                }
            }
        }
        catch (Exception e)
        {
            currentExceptionDetails = string.Format("Error \’{0}\’ while generating exception string", e.Message );
        }

        ExceptionToEmail();
    }

    private string NormalizeException(Exception ex)
    {
        StringBuilder sb = new StringBuilder();

        if (ex.InnerException != null)
        {
            if (((ex.GetType().ToString() == RootExceptionName) || (ex.GetType().ToString() == RootWsExceptionName)))
            {
                return NormalizeException(ex.InnerException);
            }
            else
            {
                sb.Append(NormalizeException(ex.InnerException));
                sb.Append(Environment.NewLine);
                sb.Append("(Outer Exception)");
                sb.Append(Environment.NewLine);
            }
        }

        // get exception-specific information
        sb.Append("Exception Type:\t\t");
        try
        {
            sb.Append(ex.GetType().FullName);
        }
        catch (Exception e)
        {
            sb.Append(e.Message);
        }
        sb.Append(Environment.NewLine);

        sb.Append("Exception Message:\t\t");
        try
        {
            sb.Append(ex.Message);
        }
        catch (Exception e)
        {
            sb.Append(e.Message);
        }
        sb.Append(Environment.NewLine);

        sb.Append("Exception Target Method:\t");
        try
        {
            sb.Append(ex.TargetSite.Name);
        }
        catch (Exception e)
        {
            sb.Append(e.Message);
        }
        sb.Append(Environment.NewLine);

        sb.Append("Stack Trace:\t\t");
        sb.Append(Environment.NewLine);
        try
        {
            sb.Append(ex.StackTrace);
        }
        catch (Exception e)
        {
            sb.Append(e.Message);
        }
        sb.Append(Environment.NewLine);

        return sb.ToString();
    }

    private void ExceptionToEmail()
    {
        // Send e-mail code
    }

}

Step 3:

The final step is to modify web.config of the web services to include the following section (under <system.web> node)

<webServices>
  <soapExtensionTypes>
    <add type="TelspaceSoapExtension, ExceptionHandler" priority="1" group="High"/>
  </soapExtensionTypes>
</webServices>

You CAN change your blog post date…

One of the biggest pain points that I have seen for Windows Live Spaces users is the in-ability to change a blog post date (once posted).

Well, for those of you (including me) who have been bitten by this problem more than once, cry no more!!! Windows Live Writer team recently launched their flag ship product (read about it here) that has a "mechanism" to change the blog post date. Here it goes… (don’t get scared by the number of steps I have listed below. I have broken the process down in very minute steps).

1. Install Windows Live Writer from here and set it up for Windows Live Spaces and make it point to your space (e.g. Nikhil’s space)
2. Click on "Open" (from either File->Open Post… OR the Open button on the tool bar.
3. Select your Space from the list on the left side. There will be a brief wait while the Writer is getting blog posts from your space. You will then be shown your latest 5 blog entries. Let’s say you want to edit post #2 which you had posted a week back.
4. Double click on the post. The post opens in edit mode in a new window. There are 2 ways to reach to the "Change date" section – either press "F2" or navigate to the bottom of the page and click onthe "Properties" line.
5. You will see a Date/Time: edit field. Check the checkbox next to the date and update the date/time to what ever you desire. Make sure you do not specify a date greater than NOW.
6. Click on "Publish" at the top on the tool bar.

YOU ARE DONE. It’s that’s simple. Enjoy!!!