Contact Support
Part 4: The failed promise of graphical mapping
In the first post of this blog series, Re-thinking Integration, I explained how I have spent a great deal of time examining my assumptions about what middleware needs to deliver.
At the forefront of my original assumptions was my belief that the majority of mapping could be done graphically with only a small fraction requiring scripting.
The reality that I’ve personally observed in my customer’s integration projects tells a very different story. The amount of code used greatly exceeded the amount of work that was being done graphically. In fact, it was looking more like 90% scripting and 10% graphical mapping.
It was quite humbling to be honest. Here we were trying to deliver graphical mapping tools when customers relied heavily on scripting.
While I recognize the error in this misconception, it is one that is shared across our industry.
The entire middleware industry is built up on the promise that integration can be achieved “without programming”, that it can all be done with graphical point-and-click mapping.
It’s a nice and enticing marketing message. Unfortunately, it does not appear to be true.
Why do graphical mappers fail to deliver on their promise?
The over promise of what GUIs can deliver is not a phenomena that is unique to our industry. Since the 1970’s there have been countless claims of products that would eliminate the need for programmers.
Remember the HTML editor Front Page from Microsoft? On the surface it appeared simple, but behind the scenes it was really just hiding all of the ugly HTML code it was generating.
That is often the case with graphical solutions to technical problems. They appear to offer huge benefits and cost savings to the end user – and do offer the means of a temporary solution to relatively simple problems – however, this ease comes at the expense of true control over your solution.
If you talk to any web developer and ask what they would do if they had to expand a website designed with FrontPage – or any other drag-and-drop editor – nearly all of them would say they’d throw away the code and start from scratch.
Graphical tools often make it less visible to see what is really going on ‘under the hood’.
Can’t we build a better graphical mapper?
Though I now know better, I initially and perhaps instinctually stuck with my original assumptions: that we needed to make a better graphical mapper. I resolved to make a graphical mapper that was more flexible and avoided the need to resort to writing code.
However, with every prototype, I continued to face the same problems that every graphical mapper runs into:
- It’s difficult to comprehend the order in which data is transformed. The order of execution is not intuitive or transparent.
- They work for simple problems (such as scenarios often displayed in sales demos) but quickly become overwhelming with the complexities of real-world integration problems.
- They don’t handle conditional business logic that occurs frequently in real-world interfaces well.
- They tend to be awkward at handling issues relating to repeating parts of the data being mapped.
- It’s often hard to find the functionality you are looking for nested deep down in a GUI which makes them time consuming to learn and slows down experienced users.
It seems that every general purpose middleware product that I have ever looked at is following the same unimaginative design. They all seemed very cumbersome.
As a programmer, I appreciate code. I like the flexibility it gives. Why not embrace it and love it? Instead of trying to make tools to eliminate code, make tools that allow customers to take control and overcome the challenges they face with it.
Why not take a completely different approach to middleware?
The question I was asking myself had changed from “How can we achieve integration without code?” to “How can we make code based integration work better?”.
I’ve never seen a graphical mapper that does a better job dealing with real-world integration logic than what can be achieved by using plain code.
Instead of trying to replace code, we need to acknowledge that it is the most efficient way to implement transformation logic. The focus needs to be placed on making tools that suck the cost out of writing and maintaining that code.
Why not make a new unconventional GUI that makes scripting easy rather than trying to replace it?
That’s exactly what we’ve done. While Iguana 5.0 still contains our graphical point-and-click mapping for simple integration problems, it features this new innovative approach to middleware by making scripting easy.
Iguana 5.0 will be on display at HIMSS next week at our booth #3621, so if you’re going to be there, please stop by and see it for yourself.
I urge you to stay tuned for the next and final post in this blog series, where I will focus on the specifics of what Iguana 5.0 delivers.
Eliot Muir
President and CEO, iNTERFACEWARE™
Remember to subscribe to our blog or follow us on twitter (twitter.com/interfaceware) for product updates, future resources and discussions.
Yep – I’d say that’s right on the money. Mapping is the trivial part of integration. It’s all the data munching that kills you.
I’ve been on teams using Cloverleaf where we tried to use the graphical mapper part of it and it just proved very inflexible and hard to actually use. We found it easier to fall back to TCL scripting.
Be curious to see what you guys have come up with. When will a public beta be available for 5.0?
Agreed. Graphical mapping is a simple, efficient approach to integrating quickly. However, it often falls short when faced with the real world requirement of integrating correctly.
The lion’s share of effort in an integration project deal with obstacles that only coding can solve. Examples of such include: message filtering, data manipulation, data transformation, validation and referencing lookup tables.
Even more complicated scenarios require coding business logic to regulate how and when a message is processed or created. For example, some lab systems require batching of lab orders in order to share specimens between different tests (since you don’t want to draw 10 blood samples for 10 different lab tests). The business logic will need to delay processing until the full set of orders were placed (either by waiting for specified timeout or checking a flag), then arrange all order data by specimen type, then create the batch orders as required by the lab system. This is not a trivial solution that can be point and clicked through.
That said, middleware should be positioned as a framework for integration projects and not the be all, end all solution. A thorough and intuitive library of solutions would save the developers the trouble of reinventing the wheel for every little function (and allow for customer contributions back into the framework). Coupled with the rapid prototyping available from the graphical mapper, this framework will allow the customer to focus on core issues and meaningful development.
An integration company, with its depth and variety of experiences in the field, is uniquely positioned to provide such a framework. It sounds like you have been one of the first to realize this, and I am very interested to see what’s in store with Iguana 5.0.
I tend to agree with you. Most of the fine tuning needs scripting and a tool that would make this easy would be the right direction to go in.
BTW , I am from India and would like to know if Interfaceware has a cloud implementation with the ability to pay per use (SaaS) or buy a ‘block’ of interface activities. What I am looking for is as follows: I would go online to your cloud app, create interfaces between two client apps using their IPs and be billed per connection. This is because I come across clients who wish to pay only for a few interfaces (sometimes as less as one)and the cost of an interface engine turns out to be prohibitive for these folks. Using cloud services would allow me to service many such clients (assuming HIPAA rules are in place as well).
My Business Plan: Low interface costs + high volumes = A tidy profit
Ed – sorry for the late reply – we’ve been flat out at HIMSS – it’s been a great show – the positive response to the translator has been overwhelming. I think overall I have only had 2 people that weren’t too keen on the translator concept. They both had “VP” in their title and neither of them had ever actually implemented a single HL7 interface. I think we’re on to a winner here
Even non programmers are loving the translator interface since they can visually see what the logic is doing. To be quite honest this was the group of people I was most concerned would be intimidated by a script based mapper but they seem to like it – I even spent time with some recruiters that were curious about what the heck it was that the HL7 programmers they recruit actually do…quite fun.
I’ll send you a link to gain access to the beta – please keep in mind it is in beta – we expect to be in beta with Iguana 5.0 for at least the next 4-6 weeks – caveat empor! Known problems do exist. Still really fun to play with though.
Bryce – Yes the blood test scenario is a really common one. One customer I spoke to today had a great term for it – he described the problem as having to deal with a bunch of “dehydrated” messages with incomplete information and having to collect the data from them together before sending out a single “rehydrated” transaction.
I loved the freeze dried food analogy.
That will be a pretty easy thing to do with the Iguana Translator technology. One way one could implement it
would be something along these lines:
LLP Listener –> Translator Filter –> LLP Client
So in the translator filter it’s dead easy to parse the HL7 and do adhoc SQL queries within it and push data down into a SQL
database. If you haven’t got a complete message the the message should be filtered and not sent on out the LLP client.
Once you get all the data you need then that code can pull out the relevant data from the database to make the “rehydrated” message which can then be pumped downstream.
The translator environment makes this whole process very visual. If you happen to have some sample test data (no confidential patient data please) I’d love to throw an example together for you.
Because the translator comes with it’s own source code repository we’re already shipping a number of non-trivial examples of interfaces. The plan is to just keep on increasing that so we build up a big library of real example code.
I’ll be doing more blogging getting into the nitty gritty of how the translator is designed the overall philosophy behind how the libraries are structured etc.
Hello Eliot: Good to hear from you. Let’s connect in the next few weeks and discuss. I have some feedback and would be very happy to share with you.
If you’re a programmer, you’ll hate the cumbersome GUIs. If you’re an analyst (and not a programmer) you’ll like the GUIs, and you’ll hate having to ask the programmers to write snippets of code where necessary.
Do graphical mappers solve everything? Absolutely not – but not having graphical mappers would reduce a message broker to a mere programmers toolkit.
The only exposure I’ve had to graphical mapping is BizTalk, which I found cumbersome.
I found it a pain to jump from functoid to functoid trying to troubleshoot problems.
Since moving to Mirth I’ve been much happier. Scripts are defined discretely, I can find and fix problems in about a quarter of the time.
Scripting is the only answer when you have a large quantity of interfaces to handle.
My personal experience is that GUI is very limiting in integration work. You can drag and drop to create mapping relationships but in most cases, the logic is more complex than that which demands a scripting language. So I agree with Anthony, scripting is really the only way to go to both handle the complexity and the volume of work that needs to be done when a large number of interfaces are involved.
I think that the ‘graphical mapper’ approach in integration work is very much like the promises made for CASE tools in the 80′s, i.e. many vendors tried to get us to believe that this was the way that all software would be developed, no programmers required. This hasn’t really been the case, but the many of the approaches have adapted to the areas which they are good at. I look at the GUI tools for modern RAD tools, and see them supporting the areas were they are strongest, and traditional code approaches are less efficient (e.g. GUI development).
I think that once integration suppliers get the idea of using the GUI to support the coding process, and heavily link the two, that we’ll get some good advances in our toolkits.
Having worked with HL7 for many years, I’ve been fully sold on the idea of not leaving the development and management of interfaces to non-coders for quite some time. Demonstrations of graphical mappers and translators may help to convince IT Managers that the product underneath will solve their headaches and allow them to produce HL7 interfaces in 5 minutes flat, but the truth is that this simply isn’t the case, and neither should it be. These interfaces need to be robust, stable and reliable, attributes which aren’t realised by graphical mapping. By employing coders to do what they do best, and giving them the best tools to code with, facilities will get strong interfaces in short time periods – in other words, an efficient and manageable solution.
In terms of what you have seen so far has the translator met up with your expectations?
After receiving a demo regarding the Iguana 5 beta, I decided to create a new channel from scratch. Despite having no experience with Lua, I was able to complete the channel and begin inserting records in the database in just a couple of hours. With previous versions of Iguana, it could take days to create the the mapping and then debug and test the Python snippets that were needed to keep track of the relationships between all the items.
With Iguana 5, not only is it much easier to keep track of the relationships between repeating segments, but testing and (more importantly) debugging is much easier with the real-time preview that includes both the data contained in the message and where it is mapped in the table. At any point in time you can see the values that will be going into the database tables for any sample message you may have.
Although this blog post focuses on how scripting is better than the graphical mapper, scripting in Iguana 5 includes the most useful portions of the graphical mapper allowing you to bring up segments and fields based on their names (or even their values) rather than having to know their position. More importantly viewing and filtering the available columns, segments, etc. as you type makes mapping and transforming data much faster and easier to follow than the various mapping and Python scripting windows found throughout the previous versions.
That’s good to hear. I think that’s one of most complete implementations so far with the translator that I have heard of. It should a relief to any managers reading this that it’s not hard to pick up the use of Lua which is the scripting language used in the translator. Lua is a wonderful little language. I’ll have more to say about that later. Incidentally for anyone reading this we are able to provide Iguana 5.0 if you are interested. We’re in private beta at the moment with the goal of releasing a private production ready release within the next couple of weeks based on feedback from private beta users. Co-ordinating an official 5.0 release will take us more time – I think we’re going to have to scale up on our support and sales infra-structure to cope with the demand. The team and I have been flat out since HIMSS doing demos…