As someone who has a lot of experience with developing data translation and integration software and solutions, I have come to be all too familiar with the phrase “The devil is in the details”. The issue is far more complex than one might think when designing an integration solution.
The issue is that when designing and implementing a solution one must be able to predict both the normal state for a solution to function, and also try and guess all possible paths or states that can occur.
Worst case, we build some kind of mechanism to alert when an exception occurs. The idea is for the solution to be a black box. Chances are, you spend more time ironing out the edge cases than you do with the core of the implementation. We cannot always account for all possibilities, which leads to the proverbial “punt” case.
For example, what happens when inbound data passes syntax and semantic validation, but still creates an error condition because the source system generated a data case not handled by either check correctly?
Hence an exception.
Problem is – what happens when a solution is running for weeks, months, even years and then an exception occurs?
I think about the times I acquire new technology – like a video camera, or a set-top media box. Like most people, I spend most of my time up front learning how to use the device and reading the manual (yes, I am old school and still read manuals). Then I configure and tweak and get the thing just how I like it. After that it might as well be a black box, like the integration solution previously mentioned. I’ve moved on to the next thing. If something goes wrong, I have to go relearn all over again just to solve a problem (don’t ever lose your WEP key and router password for your home wireless network).
When designing integration solutions, best practices borrowed from the area of software development come into play here.
1. Documentation: depending on the tool and the approach, leave comments behind as if you are going to turn the solution over to someone else to manage (chances are you probably will). Documentation can be done in many ways. In addition to an overview document that references specifications, such as when I am working in Liaison Delta, I like to write as many comments in a translation map as possible. I am assuming someone else will be in here soon.
I recently stumbled over this dandy of a blog related to commenting code, enjoy this gem: http://improvingsoftware.com/2011/06/27/5-best-practices-for-commenting-your-code/
2. Organization: Organize the solution in a way that is clear. If a standardized process does not exist in your organization, or you are deviating from it – make sure it is clear. Use self describing variables, filenames, configurations, etc. Assume someone else is going to have to get in here at some point – be nice – give them a roadmap.
3. Plan for the Unknown: You may not own this implementation forever, so when in doubt, defensive “coding” techniques are in order. This is where setting up meaningful logging, alerting, and exception handling can go a long way.
4. Test, Test, Test: The more you can afford to test a solution up front, the less likely it will come back to you down the road. For Contivo, we have hundreds of maps that run continuously all day every day for every single build we produce. If a map stops working, we’ll know about it, and fix it, long before a customer ever does.
Of course more can be done than just what I described here but keep in mind: The less you do now, the more you will pay for it down the road.