Tuesday, 24 November 2009

HSBC Payment Intergration using .Net

The HSBC payment gateway is a pain in the arse to deal with the error messages are very poor, and their support line is next to useless.

The other massive ommision is the lack of any kind of .net intergration support. Java, C, and COM are supported, but no .net or PHP code.

Without being to daunted we dived in and managed to get everything working from our ASP.Net app using the COM dll.

Weeks passed, and the web site was finished and we pushed everything onto the new web server. At this point it all stopped working.

After much scratching of heads, and shouting, we figured out the 32 bit COM dll can not be called from the 64 bit ASP.Net app. Arrrrhhh.

After looking at a number of alternatives (registering using WOW, creating a service for it etc), we eventually managed to make it work from .net.

The solutions not ideal, but it gives you a working 100% managed solution.



Step 1. Use the following Java code to get build the key

import java.io.Console;
import java.lang.*;
import java.util.*;
import com.clearcommerce.CpiTools.security.HashGenerator;
import com.clearcommerce.CpiTools.security.SecCrypto;
import javax.xml.bind.annotation.adapters.HexBinaryAdapter;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.Vector;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;

public class Extract {

/**
* @param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
try
{
String encryptedKey = "";
if (args.length == 1)
encryptedKey = args[0];

HexBinaryAdapter hb = new HexBinaryAdapter();
SecCrypto sc = new SecCrypto();

byte abyte0[] = sc.decryptToBinary(encryptedKey);
System.out.println("New Secret Base64 Encoded : " + new String(Base64Coder.encode(abyte0)));
System.out.println("New Secret Hex Encoded : " + hb.marshal(abyte0));
return;
}
catch(Exception ex)
{
System.out.println("Error:" + ex.getMessage());
}
}
}


Step 2. add the following code to your ASP.Ner app.

using System;
using System.Collections.Generic;
using System.Text;

namespace HsbcIntergration
{
internal static class CpiHashing
{

private static readonly byte[] _secret = new byte[] { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };

public static string ComputeHash(List inputList)
{
return ComputeHash(inputList, _secret);
}

public static string ComputeHash(List inputList, byte[] secretData)
{
List orderedDataToHash = new List(inputList);
orderedDataToHash.Sort(StringComparer.Ordinal);

StringBuilder sb = new StringBuilder();
foreach (string s in orderedDataToHash)
sb.Append(s);

List dataToHash = new List();
dataToHash.AddRange(Encoding.ASCII.GetBytes(sb.ToString()));
dataToHash.AddRange(secretData);

System.Security.Cryptography.HMAC sha = System.Security.Cryptography.HMACSHA1.Create();
sha.Key = secretData;
return Convert.ToBase64String(sha.ComputeHash(dataToHash.ToArray(), 0, dataToHash.Count));
}
}
}

Monday, 12 January 2009

Take Control of your WSDL

Introduction

Its good practice to design the interface to a web service by applying the same standards that would normally be applied to an XML Schema. It
should be versioned, extensible and be well structured allowing for good reuse.

Background

Within a Web Service all the messages are described using an XML Schema (XSD), so it makes sense to design our interface in terms of an XML Schema, and then embed this into our web service. Liquid XML Studio is an ideal editor for this, as it’s free and easy to use.

The following example shows the messages we would like our simple web service to exchange. Our simple Web Service will provide information about a given product, when provided with a product code, we will also need to provide login details to the service.

The information returned describes the product, pricing, and stock levels.

So we would like to build a web service with a web method that takes a ProductDetailsRequestType and returns a ProductDetailsResponseType.

So that’s what we’re after, now lets look at how to achieve this.

Using the standard .Net Framework Serialisation.

We could create a class for each entity in the schema, give them the appropriate properties and let the standard .Net framework [WebMethod] serialisation do the rest. This is an acceptable approach for small projects, but if you want to pass complex objects or objects from 3rd party
standards (HL7, FpML, ebXML, etc) this starts to become overwhelming.

Another approach I've seen used a lot is just to pass the XML data via a string parameter. This has a number of drawbacks, the XML will all be escaped so it expands (< becomes &lt; etc). Also the web service description (WSDL) now tells you nothing useful about the web service - it takes in a string and returns a string. Not a great deal of help for implementers. Also your web service provides no validation, your web service method will get called regardless of the content of the string - it doesn't even have to be XML.

A better approach is to use an XmlElement as the in/out parameters to your web service. This solves some of the problems, the XML is not escaped, your method only gets called if the XML is valid (well formed), but implementers still have no idea what the web service data is supposed to look like, and you only get minimal validation performed on your XML.

It is however possible to apply the techniques to allow an XmlElement to be passed via a web service and have it fully described in the WSDL. But more about this later.

The XML Data Binding Approach

A really easy way to deal with XML within your application
is XML Data Binding. This involves the automatic creation of a class for every
entity in your schema. These classes are bound to the structure of the schema; so
using our example we would end up with an object called 'Credentials', with
properties called 'Username' and 'Password'. This removes any notion of dealing
with XML from the application, and means the developer just has to deal with
objects, properties and collections, all strongly typed, and enforcing the structure
described in the XML Schema.

These objects can be generated using a Data Binding tool, Liquid
XML Studio has an XML Data Binding tool which will create .Net code for 1.1,
2.0, 3.5 & Silverlight both in C# and VB.Net.

Once we have generated our Data Bound class library, we can
pull together our web service. The first thing to do is declare our web method.

namespace ProductEnquiryWebService
{
[WebService(Namespace = "http://www.liquid-technologies.com/ProductDetailsSample")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[ToolboxItem(false)]
public class Service1 : System.Web.Services.WebService
{
[WebMethod(Description = "Gets the details about a given product.")]
public ProductDetailsResponse RequestProductDetails(ProductDetailsRequest productRequest)
{
ProductDetailsResponse response = new ProductDetailsResponse();
response.ProductDetails.Name = "Widget";
response.ProductDetails.Description =
"The important widget that goes between the flange and the spindle";
response.ProductDetails.Price = 25.36;
response.ProductDetails.StockLevel = 15000;
return response;
}
}
}

Note the web service uses the targetNamespace from the XML Schema we defined at the start of the example. The web method takes in and returns the Data Bound classes generated by Liquid XML Studio.

If we left things here then the .Net framework would use its default serializer on our ProductDetailsRequest and ProductDetailsResponse objects. This works using reflection and would end up serializing a large amount of unwanted properties. Instead we are going to let these objects serialise themselves. In order to do this we must tell the framework about them.

[XmlSchemaProvider("MySchema")]
partial class ProductDetailsRequest
{
// This is the method named by the XmlSchemaProviderAttribute applied to the type.
public static XmlQualifiedName MySchema(XmlSchemaSet xs)
{
// This method is called by the framework to get the schema for this type.
// We return an existing schema from disk.
if (xs.Schemas("http://www.liquid-technologies.com/ProductDetailsSample").Count == 0)
{
using (FileStream fs = new FileStream(@"ProductEnquiry.xsd", FileMode.Open))
{
XmlSchema s = XmlSchema.Read(fs, null);
xs.Add(s);
}
}
return new XmlQualifiedName("ProductDetailsQueryType",
"http://www.liquid-technologies.com/ProductDetailsSample");
}
}

In order for the framework to be able to correctly describe the web service we first need to tell it about the schema. This is done by adding the XmlSchemaProvider attribute to all classes that are in or out parameters in the web service. The XmlSchemaProvider attribute tells the framework that information about the schema can be obtained by calling the “MySchema”
method.

Now when the framework is trying to describe the web service it will call the MySchema method. The framework expects our implementation of MySchema to add all the schemas it needs to know about into the schema set collection, so if you are working with elements from external schemas (HL7, FpML etc) then you can add these to the schemas collection as well.

The return parameter from MySchemas is the fully qualified name of the type within our XSD that describes the parameter. Note this must be a complexType, the .Net framework will not allow you to specify an element.

Customising the Serialization Performed by XmlElement

As discussed earlier, it is possible to pass XmlElements as parameters in your web service, and have them fully described in the WSDL.

In order to do this a class wrappering an XmlElement must be passed instead of the XmlElement, this class must implement the IXmlSerializable interface, and have its
own XmlSchemaProvider attribute.

Because the 'MySchema' method is static, one wrapper class must be created foreach type of object being passed, as the return value tells the .Net framework which complexType in the schema this argument represents.

A further improvement on this would be to add validation to the element prior to serialisation, thus ensuring that the XML was actually valid before reading or writing it.

[XmlSchemaProvider("MySchema")]
public class MyXmlElement : IXmlSerializable
{
private XmlElement _element = null;

public MyXmlElement() {} // requires a parameterless constructor
public MyXmlElement(XmlElement xmlElemnt) { _element = xmlElemnt; }

// This is the method named by the XmlSchemaProviderAttribute applied to the type.
public static XmlQualifiedName MySchema(XmlSchemaSet xs)
{
// This method is called by the framework to get the schema for this type.
// We return an existing schema from disk.
if (xs.Schemas("http://www.liquid-technologies.com/ProductDetailsSample").Count == 0)
{
using (FileStream fs = new FileStream(@"ProductEnquiry.xsd", FileMode.Open))
{
XmlSchema s = XmlSchema.Read(fs, null);
xs.Add(s);
}
}
return new XmlQualifiedName("ProductDetailsRequestType", "http://www.liquid-technologies.com/ProductDetailsSample");
}


#region IXmlSerializable Members
public System.Xml.Schema.XmlSchema GetSchema()
{
throw new NotImplementedException();
}

public void ReadXml(System.Xml.XmlReader reader)
{
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(reader);
_element = xmlDoc.DocumentElement;
}

public void WriteXml(System.Xml.XmlWriter writer)
{
_element.WriteTo(writer);
}
#endregion
}

[WebMethod(Description = "Showing the technique with XmlElement.")]
public MyXmlElement Test()
{
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(@"SampleData.xml");
return new MyXmlElement(xmlDoc.DocumentElement);
}

Testing your Web Service

Because your web service contains complex parameters it is not
possible to call it via the simple Web Service Explorer.

WS_IE.png

So in order to call it you need to either build yourself a test harness or use a test tool. Liquid XML Studio contains a handy tool for calling web services. This will automatically generate the request envelope and message, allowing you to modify can call it.

CallWebService_640x529.png

Download Sample Request
Download Sample Response

Examining the WSDL

Because we have taken control of the serialisation process the WSDL describing the web service contains the contents of the schemas that you added in via the MySchema method.

Download the WSDL - 4.62 KB

It is also possible to explore this graphically using the Web service Call Composer in Liquid XML Studio

Conclusion

If you are building web services that require a hierarchy of objects to be sent or received, especially if those messages contain sections from existing schemas (formal standards or internal data structures) or contain complex constructs (inheritance, substitution groups etc), then you need more control over the WSDL that defines your web service.

XML Data Binding not only simplifies the reading and writing of your XML data, providing a framework of strongly typed objects in which to work, but can provide information to the .Net Web Service Framework that allows it to fully describe even the most complex data
structures accurately within the WSDL.

Using this approach makes it possible to include parts of complex standards within your web service and still have them properly described in the WSDL, enabling other tools to properly understand and use the interface.

Resources

Tuesday, 6 January 2009

XQuery implementation for .Net

I've been looking for an XQuery implementation for .Net. It seems Microsoft put an implementation together in the .Net 2 Beta framework, but this never made it into the full version, and has never been seen since. The beta version is still available here, but its license is to restrictive to use in commercial software.

Saxon seems to be the recommended route, but this is written in java, and cross compiled (along with the whole java class library) to .Net compliant byte code. From accounts that I've seen this works fine, but does massively increase the footprint of your application (compiled code is in the region of 25MB), it also adds an element of risk, as you are at the mercy of an open source cross compiler.

As XQuery is an integral part of the XSLT 2.0 standard, not having a standard implementation makes life difficult for developers, hopefully there will be .net implantation available soon…

If anyone has any updates on this please let me know.

Monday, 5 January 2009

XML Editors

We've been doing some work with FpML (Financial Products Markup Language) which is an XML standard for describing trades between investment banks, swaps, futures etc. Its a complicated standard in the form of an XSD (XML Schema). So producing XML that is valid against the FpML schema is not entirely straight forward.

The best approach we found was to prototype the XML messages by hand before writing the code that would do this automatically from our front office system. This was reasonably straight forward where we had examples to copy, but suitable example only covered a few of our use cases, so we had to start designing some of the XML from scratch.

We started off trying to use notepad to create the XML, but this was a non-starter, tiny errors would take an age to track down and it's not what anyone would call a fully featured editor. We downloaded and started using Microsoft's XML Notepad, but it was awkward to use, and didn't seem to have any knowledge of the schema, so the validation was very limited.

Eventually one of the team found Liquid XML Editor, this is a free text based editor that is schema aware, and even shipped with the FpML standards so when we opened our XML file it automatically loaded the schema, and told us about our errors.



As well as validation, the editor also uses the schema as a source for the intellisense and documentation, which is pretty powerful given the complexities of the FpML standard. We ended up playing with a number of XML Editors, and this was the only one that could cope with the way the complexTypes are defined within the FpML standard.



The intellisense and validation helped a lot when trying to build up our XML prototypes, but what was still lacking was a feel for the structure of the FpML entities. We tried looking at the FpML schemas, but these were all but un-readable in code form. We eventually stumbled on a feature within the XML schema editor, that allows you to create HTML documentation from an XML Schema, we later discovered this documentation was already online see FpML online documentation. This gave us a good feel for the shape of our target XML document. We then used Liquid XML studio's XML editor to finish prototyping our XML documents.

Our final problem was the target system could not cope with the XML documents we produced. Initially we figured our XML was wrong, but after running our XML through a number of validation applications, and digging through the XSD standard we concluded the error was in the target system. After a lot of hassle with there support team, we eventually figured out a solution which involved adding in a few additional elements. This effectively made our XML non-FpML compliant, but it fixed the problem we set out to solve. We left a load of comments in the code and XML to explain the issues, but I've seen this kind of thing to many times where a standard is 'interpreted' by a number of 3rd parties and the resulting inconsistencies end up causing issues for years to come. Good tools, especially in the XML validation area should start to reduce this problem in future but its unlikely to resolve it altogether.

We are now looking at the feasibility of setting up an XML firewall on all our service clients. Basically anything that comes in must have an associated schema and it must validate against it. If it fails validation, then it doesn't get anywhere near our system. This seems like a great idea in principle, but I recon we'll end up bouncing so much traffic, we'll be forced to take it down…time will tell.

Friday, 19 December 2008

Building XPath Queries

I've just spent that last week building a translator, allowing us to convert a new customers catalogue data into a form our sales system can cope with. Luckily the source data was XML, so it was already in a form I could use, but there was no schema, and the structure was a bit flaky in places.

I started off trying to read through the data, but the structure was just to messy, in the end I figured out I could probably do most of the grunt work with XPath. This approach seemed promising, and I got good results initially, however my lack of XPath experience quickly started to let me down. Changing your XPath expression in code (C#) is possible, but typically means a re-compile - especially when you have anonymous methods, which I use a lot for sorting collections.

So I went in search of a good XPath tutorial, and I was somewhat disappointed with what I found. There are a few - but they are a bit lacking, the W3Schools XPath Article one was the most complete, and an XPath article on Google Knol had some nice examples. If anyone comes across a really good XPath tutorial let me know!

What I did find while I was looking was a free XPath Viewer, it highlights the selected nodes as you type your XPath query. This made the learning process so much quicker, and allowed me to debug a lot of my XPath queries before they got as far as the code. The Liquid XML Studio XPath viewer also added itself to Visual Studio 2008 which made my life even easier.



Liquid XML Studio XPath Viewer