ITIM Messaging & Synchronisation

WebSphere gurus all the world over will probably understand what a messaging cluster is; what tranlogs are; and how all this "stuff" works.

For systems integrators who don't necessarily specialise in any one technology, these things may seem more like a dark art and certainly the information made available in the myriad of documents produced by the various authors of WebSphere technical books doesn't seem to shed enough light on what is going on.

Yesterday, I had a misbehaving IBM Tivoli Identity Manager (ITIM) instance which saw all transactions sitting in a pending state with not a single transaction being flushed through to completion.

The setup was: ITIM v5.0 running in a WebSphere v6.1.0.9 clustered environment with two physical servers. Everything about the deployment was fairly vanilla (though the amount of data going through the system is quite large).

The problem started on Sunday night during the automatic regeneration of LTPA keys - it seems. The Deployment Manager lost the ability to control the Node Agents/Clusters which forced the following sequence of events:

• Forced shutdown of the Application Servers and Node Agents
• Removal of Global Security
• Manual synchronisation of the nodes on the physical servers
• Startup of the Node Agents
• Reconfiguration of Global Security
• Resynchronisation of the nodes (via the Deployment Manager)
• Startup of the Messaging Cluster
• Startup of the Application Cluster

Everything looked OK, until transactions started to appear in ITIM and they still sat pending!

The logs showed that the Messaging Cluster would start, then stop, then start, then stop, then start, then stop.....

Communications channels between the local queues and the shared queues weren't as they ought to have been and the root cause seems to be an inconsistency between transactions that ITIM expected to be pending; transactions the Messaging Cluster thought it had; transactions stored in the "physical" storage of DB2 for the messaging cluster and the tranlog.

NOTE: The recovery procedure for this is probably not something anyone should undertake but it was a final straw after 12 hours of trying various tactics which could have saved my pending transactions (which proved futile).

• Stop the clusters
• Stop the Deployment Manager
• Kill any rogue WebSphere processes (of which there were a few)
• Stop the database manager supporting ITIMDB
• Delete the various tranlog files for the two application server instances
• Reboot the two servers (which probably wasn't required, but these were Windows 2003 server instances which seemed to be hanging on to various ports needlessly and I wanted a clean environment to start everything up again)
• DANGER: delete from itiml000.sib000 in the ITIM Database (as well as itiml001 and itims000 and for sib001 as well)
• Start up ITIM

At this stage, all was well. Apart from my physical well being.

CAUTION: The above approach was a very brutal way of clearing out everything to a point where the system was operational once again. I never want to have to repeat this process and having a clean ITIM instance which is well tuned and well looked after is very much more preferable than having to perform this kind of recovery.

Reputation On The Net

I've commented on more than one occasion on the power of the internet and, in particular, the power of the various social networking sites that have sprung up in recent years. It's not enough to have an account on Facebook any more. We have to have a Twitter account feeding Facebook. We also want our random thoughts automatically updating our professional networking pages on LinkedIn. We use Plaxo to amalgamate the email addresses stored in various repositories. We have our .TEL domains, we're Buzzed and we blog.

We must spend a lot of time generating unstructured information for the consumption of the masses.

But what about the information that is generated ABOUT us BY others? It's one thing to look after one's online reputation by ensuring that we portray ourselves as professional in our tweets and blogs. But what if the baddies out their are destroying our reputation in forums that we have no control over?

I've spent a number of years applying my brain cells to the task of working out how to control access to systems and how to manage the credentials within systems. We in the industry call it Identity & Access Management as these tasks are frequently joined "at the hip". Maybe my brain has had enough of IdAM activity as most of my recent conversations seem to be related to Cloud based services, reputation management and how social networking can be an invaluable business tool.

Here's a question for you? How frequently to you actively search for information related about YOU on the net? What dangerous comments could be lurking out there? What do people really think of you?

Sometimes it may be best to not know. But, sometimes you can be presented with a very nice surprise.

Take, for example, Vintage1951's recent blog entry! In it, I've been called experienced and thoughtful! In an industry where reputation is everything, I can think of almost no higher accolade (at least, no higher accolade that won't sound over-the-top and smug!)

It's nice to see that Vintage1951 takes a similarly thoughtful approach to his reputation by connecting with only those people he has the utmost respect for. I feel privileged to be amongst them.

On a final note - have a browse around Vintage1951's blog. There are some excellent articles in there!

Enterprise Compliance For Business Managers

I spent a number of years acting as a Line Manager for a team of Identity & Access Management specialists at a time when there was increasing focus on compliance controls within the organisation. Certifying that my employees only had the systems' access rights that they required for their job was a quarterly event to look forward to.

Being told that Joe Bloggs can access System X with a role of Developer might seem like sufficient information for me to testify that the this is appropriate, but I would have to have made an assumption that developer access on System X has been correctly configured by whomever configured the system. What would the impact be if the administrator of System X had inadvertently given superuser access to those users in the developer group?

Traditionally, we have merely accepted that assigning users to groups is enough to satisfy our legislative requirements. But this is not the case! We also need to attest that the lower level permissions have been configured appropriately.

So what if we could find a way to deliver the lower level permissions to the Line Manager during his quarterly certification? What if I could see that superuser access had been inadvertently granted to my set of developers on System X?

This raises another problem, though. Lower level permissions on various systems take the form of data values which may be meaningless in their own right and therefore not in a format that would allow a Line Manager to understand them. As an example, informing a Line Manager that their employee has access to a system which means they have been assigned the value of S to the attribute ACCESS_RIGHTS. What does S mean? Standard? Super? Something-Else?

So what if we could provide the details in a manner which is more meaningful for the Line Manager?

I've spent some time in the recent past working on a mechanism to augment these lower-level permissions to provisioning data in such a format, thus enhancing the attestation experience of access certifiers but also as a means of validating that systems have been configured properly (and, as a consequence, the ability to perform lower-level Segregation of Duties checking).

The result of this work has been documented by Alan Harrison and is now available on the Pirean website. The work shows how low-level attributes held in RACF can be presented in a meaningful way through the IBM Tivoli Identity Manager interface. The principles, however, can be applied to any credential storage mechanism.

How I wish I had had that ability when I was a Line Manager!

ITIM Provisioning with SPML

I had an interesting day last week playing with IBM Tivoli Identity Manager and IBM Tivoli Directory Integrator. The brief was to send provisioning requests from ITIM via ITDI as SPML requests to a target credential store. Sounds easy, doesn't it?

SPML parsing support has been available within ITDI since v6.1.1 (Fix Pack 1) and dropping the openspml2-toolkit.jar into ITDI's JAR directory should've been enough for me to get motoring with the requirement. And there is a certain amount of truth in that statement in that it was easy to put together a Proof Of Concept showing an ITDI Assembly Line sending SPML formatted provisioning requests to a SPML listener.

That was fine until I elevated the Assembly Line into ITIM. Sending the AL to an ITDI RMI Dispatcher as part of an ITIM provisioning request has been a feature of the ITIM setup since its v4.6 days and there hasn't been any reason to suspect that ALs invoked via the dispatcher mechanism would behave any differently than ALs invoked from either the command line or through the ITDI GUI.

But there is...

Unfortunately, I had created an Assembly Line that was made up the following:

• Parser Function (to parse my work entry into SPML format)
• HTTP Client Connector (to send my SPML formatted string to a SPML listener)
• Parse Function (to parse the response to my connector above)

There's nothing wrong in what I did. I could've attached the SPML parser to the HTTP Client Connector, but I didn't. I wanted to build up my AL slowly and show what was happening during the SPML creation process.

The AL worked fine through the GUI. But it did not work when "dispatched" as it throw a nullPointerException during the attempt to call the parser. My RMI Dispatcher had all the required JARs at its disposal and the environment variables were identical to my GUI environment. I even tried to use the CSV parser rather than SPML (thinking that maybe a parser introduced as part of a Fix Pack may not have full support) but I got the same result.

If only I had attached the parser to my HTTP Client Connector because guess what? You guessed it. That worked! It seems that (in my environment) parser functions cannot be instantiated when the AL is invoked by the RMI Dispatcher. NOTE: I did upgrade my RMI Dispatcher code to the very latest version during my investigations.

So a note to all - Parser Functions may not behave as you expect when dispatched!

No Matter How Diligent You Are...

Today, I got an email from a service provider I have only rarely used stating that they had been the victim of a successful hacking attempt. They reassured me that they didn't store any credit card details but I should change my password as soon as possible.

This is the second time this has happened in the past year and both sites are fairly insignificant in the grand scheme of things. They aren't, for example, financial services sites so I shouldn't panic unduly.

I have no changed my password on the offending website so all's well, right?

Wrong!

Whoever hacked this site may now have my name, address, email address and a password they know I have used in the past. Matching this password and email address may grant them access to other services I use online. That's not an unreasonable assumption on the part of the hacker, of course. How many people reuse their "favourite" password on each service they sign up to?

I think of myself as fairly diligent, though. I do have a "favourite" password. In fact, I have 3/4 "favourite" passwords! But NONE of them are used for my mission critical services: banking, being one!

However, I should now consider that one of my "favourite" passwords has been compromised and should be decommissioned. That now means I should take a look through my password safe and change the password on those services that make use of this now redundant password. What a pain!

The personal damage has been contained somewhat. My mission critical services are still safe. (In fact, I have no idea what the password is for these services and have no intention of ever "learning" what I have them set to!) If only everyone else behaved as responsibly. In fact, if only I behaved even more responsibly by having a separate password for every service I use!

My tips for the day, then, are:

• Even if you have a high-quality password, always consider it vulnerable as you have no idea how your service provider stores your password
• Try to create a different password for each of the services that you use (and avoid prefixing or suffixing your "favourite" password with the service details - passw0rdebay and passw0rdfacebook are stupid passwords)
• If you still feel confident reusing your "favourite" password, avoid the temptation to use it for your banking services
• Re-evaluate what a mission-critical service is. Banking is definitely mission-critical, but your reputation is also critical and having your Facebook or Twitter accounts compromised because of a slack password policy isn't clever either
• Invest in a password management tool - even something as simple as Password Safe ought to do the trick and it is free!

Y2K Memories

I remember what I was doing on January 31st, 1999. I'm sure a lot of people do but I bet not many were doing what I was doing because I was being sick!

I was supposed to be working that night. Sitting patiently waiting for the world to end as the Y2K bug bit hard. I was a mainframe developer back in those days but I worked on a fantastic banking platform that had been Y2K compliant since its inception in the early 1980s. I was supposed to sit in work as an insurance policy. As it turned out, there were no issues. Not one. Nothing. Nada. Zero. Zilch. But I missed even that because I was driven home by my manager because of my illness.

I was reminded of this event because my O2 Connection Manager software for mobile broadband stopped working this week. On attempting to start it up, it would immediately die with no real explanation as to why it was being so fussy about doing some work on my behalf.

Of course, this week is the first week of 2010. Setting my system clock back to some date/time in 2009 enabled the O2 Connection Manager software to initialise successfully. In effect, this is the same as the Y2K bug, except it is ten years late!

Clearing out all files under "Documents and Settings/All Users/Applicatoin Data/O2CM-CE/O2 Connection Manager" resolved the issue (after I had reset my system date, of course).

So it would seem that back in the 1980s, we had a clearer idea of how to build software fit for purpose and with a view that it would still be around 20 years later. These days, code is hacked into existence and wobbles at the first sign of any kind of change. How depressing.

Identity & Access Management Predictions For 2010

I should point out that I am not psychic. I haven't found a way to see into the future just yet. (If I had managed to do that, I'm quite sure I wouldn't be writing this article!)

Identity and Access Management has promised much in recent years and in the case of Identity Management, the promise has yet to be realised in a lot of deployments. I hear potential customers making claims that there has never been a successful identity management project and any organisation intent on attempting to realise the benefits of such a project are doomed.

Thankfully, I have been involved in many successful deployments that have realised some, if not all, of the anticipated benefits. The problems I've seen have typically been political issues rather than technical issues. Does that sound familiar?

It would seem to me that there is a disconnect between what technology can offer, what businesses can afford and the political will-power to ensure that an IAM programme will succeed. Which brings me on to my predictions...

1. Ding-Ding - Round 2
The early adopters of IDM technology went through the pain and heartache of spending big on new technology in an effort to leverage their legacy technology in the "always connected" world. Their 18 month programmes a number of years ago are probably starting to provide some benefit around about now and the political collateral required to leverage their infrastructure will be in place because it has become too darn expensive to rip out all that kit that was deployed all those years ago. In other words, the initial hype surrounding the technology that was followed by disillusionment is now starting to pay for itself.

The rotation of staff around the various enterprises that exist will ensure that every enterprise now has "someone" in their team who has been involved in a successful IDM deployment. These people will become crucial in pushing their new employers down the path of embracing IDM as a workable solution.

2. Risk
I'm on thin ice with this one but the days of locking down everything because a manual said it could be locked down are disappearing. We used to live in a world which had adopted the 80/20 rule. An 80% delivery rate on a project was usually enough to get businesses working effectively and the remaining 20% was usually too expensive and made a mockery of the original business case. I see those days returning. For example, a two-factor authentication system for high-net worth banking users or treasury departments may be a great idea bearing in mind the risk of a security breach for either user but such a system may not be necessary for the thousands of people who only have a few coppers in their deposit account.

The same rule can be applied within the enterprise as well. Do we want to lock-down our enterprise systems to the point where they become difficult to use? Do we want our users fed-up with the tedium of trying to do their job with a system that seems hell-bent on preventing them to do so?

IT Security professionals will finally find the word pragmatism in their dictionary and understand that they are there to help rather than hinder.

3. Personal Ownership
For many, the notion of an Identity Management System may seem crazy. Surely it is up to the individual to manage their identity properly rather than delegate such responsibility to a "system". 2010 will see IT users taking ownership of their identities (and not just those binary-speaking geeks we all like to poke fun at). Real people performing real duties in the real world will start to take more care of their online persona. Facebook and Twitter have become vital tools  - they are no longer being used to merely jabber on about what was on television the previous night!

Most people are sensitive about how others perceive them. Now is the time to protect our online personae. It is time to manage our own identities.

4. Compliance
Enterprises need to demonstrate that they have control over their processes. In a nutshell, that seems to be what Sarbanes Oxley is all about. How an enterprise demonstrates their control, however, is up to the enterprise. Quill and Parchment record keeping may actually suffice.

There are tools available which can help an enterprise keep control over its systems. Identity Management systems typically look after the provisioning aspect of a system and can certainly be beneficial in achieving compliance. But what about those systems that aren't managed by such a clever tool? Log file scraping and database dumps can provide an auditor with the necessary data to determine how an application is being managed but unless she is super-human, she will need an analysis tool to help her make sense of the information.

Compliance has always been a tricky topic because there are legacy bespoke systems which contain data that nobody else on Earth could possibly understand. How do you build a tool capable of analysing information from every possible application without major customisation and significant up-front consultancy fee hell. How can "SOX IN A BOX" be achieved?

This year should see the major vendors of IDM solutions attempt to address this area.

5. The Cloud
I've written about "The Cloud" before and 2009 has already seen a quickening in pace of Cloud Services and IDM solutions specifically for The Cloud. I can see one or two niche players operating in the "IDM proxy" world being gobbled up by the big boys.

Until now, enterprises have attempted to manage access to The Cloud from within their perimeter. 2010 will see the start of a mirror-imaging of this approach, ie The Cloud will start to manage access within the enterprise.

Conclusion
The above five predictions are safe bets, to be honest. All of these things are already happening so I guess my predictions aren't really predictions. Maybe they are "realisations"? This year will be the year that the IT user base will become more aware of the above.