FREE DM Review Site Registration!
Sign-up today and access DM Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all DM Review content

FREE access to web seminars, resource portals, our white paper library and more!

   

Is there any ETL tool that takes input from a message stream like JMS or MQ Message Queues?

Question: Is there any ETL tool that takes input from a message stream like JMS or MQ Message Queues? The idea is there are messages coming in that need to be parsed and put into a database. Is this possible?

Evan Levy's Answer: There are a few details that you didn't include as part of your question that I'll include in order to responds.

I'll assume that you've got a system or application that's depositing XML messages into a message queue. You want to be able to transform or convert these XML messages into more manageable delimited values so you can load them using an ETL tool. It also sounds like you'd prefer to load these values into your tables in a trickle fashion (one or a few at a time).

The simplest approach (based on my assumptions) would be to build a simple script-based program (AWK, Perl or even C) that listens to the message queue, converts the XML messages into a comma delimited record that you then feed to the ETL tool. Most DBMS vendors provide a streaming (or record-at-a-time) data loading utility.


Evan Levy is a partner and co-founder of Baseline Consulting Group, a multivendor systems integration and consulting firm. As the partner in charge of Baseline’s largest practice, Levy leads both executives and practitioners in delivering technology solutions that help business users make better decisions. He has led strategic technology implementations at commercial and public sector organizations and advises vendors on their product development and delivery strategies. Levy has been published in a wide array of industry magazines and has lectured on a range of technology delivery experiences at leading conferences and vendor events. He has been a featured speaker at the Marcus Evans Analytical CRM symposium, DCI’s Data Warehousing conference, the CRM Association, DAMA International, the AMA and the Data Warehousing Institute. His current work involves delivering and lecturing extensively on the topic of data integration. You can contact him at evanlevy@baseline-consulting.com.

For more information on related topics, visit the following channels:



Industry Vendors