The default initialisation sequence for a TDI Assembly Line's connectors is to initialise on Assembly Line startup. That might not sound too unreasonable but let's consider the following scenario.
Two connectors, in iterator mode, connecting to Active Directory instances are instantiated on Assembly Line startup. Connector A trundles retrieves data from its Active Directory container and passes its work entry to a myriad of other connectors and scripts which perform some logic on the data before committing it to another repository.
10 minutes later, Connector A gets to the end of its cycle and control passes to Connector B which similarly retrieves data from its Active Directory container and logic and commitment are performed.
All is well and good. Unless, of course, Connector A has a lot of data to process and more time is taken to do so.
If Connector A takes longer than 15 minutes, Connector B will keel over unceremoniously. Why? Well, it is all to do with the MaxConnIdleTime setting within Active Directory which has a default setting of 900 seconds (or 15 minutes). Microsoft's website gives an explanation for this setting as:
"The maximum time in seconds that the client can be idle before the LDAP server closes the connection. If a connection is idle for more than this time, the LDAP server returns an LDAP disconnect notification."
The TDI documentation states that when a connector in Iterator mode initializes, however, "the Prolog – Before Selection Hook is processed, and the Connector performs the entry selection; this triggers a data source specific call, like performing an SQL SELECT or an LDAP search."
So if the LDAP search was invoked at initialisation time, how can the MaxConnIdleTime have any impact on our processing? The answer, my friend, is that search requests are paged. On initialization, the LDAP search is indeed invoked, but only the first page of data is returned - 500 entries, for example. Only when Connector A has finishing iterating through its data will control pass to Connector B at which point Connector B can request the next page of data. At this point, however, the MaxConnIdleTime has been surpassed and Active Directory refuses to play ball.
To get around the problem, though, is really quite straightforward. Instead of initialising the connector when the Assembly Line starts, initialise it when it is required. Clicking on the little more button and selecting only when used for the initializate parameter as such:
Next time you write an Assembly Line, have a think about how connectors should be initialised. It might save you from a whole world of pain!
NOTE: Just because a connector in your test environment can process data inside 15 minutes, doesn't mean you will get the same result in your production environment ;-)
