- High Availability
- Abstraction
High Availability can be achieved with queues because I can have multiple TDI instances reading from a queue and performing actions as dictated by messages in the queue. I could have a queue called "actionQueue" which contains messages with actions embedded; a queue called "responseQueue" which could contain the status return code for each action and a queue called "errorQueue" for all the nasty things that may happen to us. I could have many TDI instances pick up a message, perform the action, respond and I would sleep soundly in bed each night safe in the knowledge that the way in which MQ queues operate would mean that I would not get duplication of effort and all messages would be processed even if a TDI instance were to crash.
Abstraction, of course, means that I can send a message to another TDI Server or Assembly Line without having any real need of understanding what or where that process is. I can merely put a message on a queue (in a format previously agreed) and proceed with my normal processing safe in the knowledge that my message will be delivered - eventually.
Environment Setup
If you want to follow this article, you will need:
- IBM Tivoli Directory Integrator v7
- Apache ActiveMQ 5.3.2
Installing ActiveMQ is really very straightforward. Grab it from the Apache website and unzip it somewhere on your file system - I chose c:\apache-activemq-5.3.2 as my location on a Windows 2003 Server host.
As I like my Windows Services, I installed ActiveMQ as a service by running the c:\apacher-activemq-5.3.2\bin\win32\InstallService.bat batch file. After a few minutes, I could see ActiveMQ listed as a service though not, at this stage running. I started the service!
The activemq-all-5.3.2.jar file (found at c:\apache-activemq-5.3.2) should be made available to TDI. A simple way of achieving this is to drop the file into the TDI\V7.0\jars\3rdparty directory .
We're now ready to create an Assembly Line which will write messages to an ActiveMQ queue!
MQ Connector
The first step we should take when we start the TDI GUI and create an Assembly Line is to drop in a JMS Connector (in Add mode) and populate the form as follows:
The JMS Driver Script needs to be populated with the following code so that the connector understands how to communicate with the ActiveMQ instance:
var brokerURL = env.get("jms.broker").trim();
var connectionFactory = new org.apache.activemq.ActiveMQConnectionFactory();
connectionFactory.setBrokerURL(brokerURL);
ret.queueConnectionFactory = connectionFactory;
ret.topicConnectionFactory = connectionFactory;
var connectionFactory = new org.apache.activemq.ActiveMQConnectionFactory();
connectionFactory.setBrokerURL(brokerURL);
ret.queueConnectionFactory = connectionFactory;
ret.topicConnectionFactory = connectionFactory;
I used the Simple XML Parser on this connector to parse my output in XML format:
As I decided I'd write to an errorQueue, I updated my Output Map to include an errorCode attribute and an errorMessage attribute:
NOTE: Although not shown, my work entry was populated by an iterator which read through an input file of error codes and messages!
Running the Assembly Line, which is always the exciting bit, resulted in my ActiveMQ errorQueue queue being populated with 9 messages. Queues can be "browsed" with a standard browser by pointing the browser at http://localhost:8161/demo/queueBrowse/{queue-name}:
9 messages. As expected.
The Consumer
Now that I have populated my queue, I need "something" to consume these messages. A JMS Connector in iterator mode can be used to act as a mechanism to constantly poll the queue for new messages:
This will certainly read the messages and with an appropriately configured Parser and Input Map I will end up with a work entry containing the errorCode and errorMessage attributes I expect. At this point, I can take decisions on what I want to do with these "errors". I could merely write them to a file. I could send an email to someone. I could even send an alert to the world via Twitter (see previous article). And I can base these decisions on the contents of my message. For example, if my errorMessage contained the word Tweet, I could send the message to Twitter, right?
The End Result
For simplicity's sake, I wrote each errorCode and errorMessage to a flat file (along with a timestamp) and the result was this:
dateTime|errorCode|errorMessage
"2010-08-02 21:02:18"|"1"|"Msg1"
"2010-08-02 21:02:18"|"2"|"Msg2"
"2010-08-02 21:02:18"|"3"|"Msg3"
"2010-08-02 21:02:18"|"4"|"Msg4"
"2010-08-02 21:02:18"|"5"|"Msg5"
"2010-08-02 21:02:18"|"6"|"eMail me a message 6"
"2010-08-02 21:02:18"|"7"|"Tweet me a message 7"
"2010-08-02 21:02:18"|"8"|"Tweet me a message 8"
"2010-08-02 21:02:18"|"9"|"Msg9"
Within my IF branch, I include an AssemblyLine Function call to my Twitter Status Update AL (from my previous tutorial) and saw messages 7 and 8 appear on my Twitter timeline.
Conclusion
ActiveMQ is a very simple product to deploy and get working. Of course, it would be prudent to secure the queues. It would also be prudent to introduce some serious error handling in the Assembly Lines - for example, reading a message from a queue removes it from the queue immediately which may or may not be desirable before the message has actually been processed!
Hopefully, however, this will give you a taster of how TDI can interact with MQ. Good luck "queueing".
