WTF IT Stories #3: The 30 Minute Endless Loop

March 07, 2007

My favorite employer (see the previous two stories) had a wealth of interesting characters who besides making dumb decisions also had a penchant for making absurd quotes. This post tells the stories of the ones I remember hearing or being told of.

"Our email system delivers 80% of the emails; that's the industry standard."

Our email system was notorious for not delivering emails even to people in the company. Even though we used the popular Microsoft Exchange mail server our NO (network operations) team had little understanding in configuring it properly. One of the leaders made this comment when challenged about this.

"Email is not to be used for communications."

This quote came from the head of NO who informed all the employees and our entire field staff not to use email to communicate with our customers. He sent this message via email. I always wondered how many people actually got it. Some time later we discovered that he was blocking our web applications from sending password reminders to customers via email, saying he wanted to avoid sending spyware to them.

"Open a ticket with BEA and find out why Java is deleting users from Active Directory."

During a rollout of a new customer portal, 5000 newly registered (but preexisting customers) users suddenly vanished from our Active Directory/ADAM database. Permanently lost, each of the users had to be contacted manually to get them to re-register. During the fiasco the head of operations told the project manager to file a ticket with BEA (we used BEA Weblogic Portal) about why their Java implementation was deleting users from AD. We tried to explain that there were no delete() calls anywhere in the code and it was impossible that the JVM could make calls on its own. We did however imagine the hoots at BEA Support when they got this message.

It turned out later (and not admitted by anyone at NO) that the head of operations had been poking around in the live AD database that day and used a buggy version of a Microsoft tool (imagine that!) which had a bug where it would delete groups of entries for no apparent reason.

"Clearly the software is changing between mouse clicks."

Our field staff web application was throwing all sorts of random exceptions, mostly data related. Since the database was not at fault (see WTF #1) the Java code and coders were clearly at fault. When a programmer pointed out that data exceptions appeared at random places, the head of operations made this interesting comment.

"You can look at anything to find the problem except the network which is not at fault."

Our field staff was complaining about terrible performance of all of our web applications for quite some time. A strike team was put together and met daily for months to determine the cause. This quote was part of their directive at the beginning, anything could be looked at other than the network, which was perfect (this was said by the head of NO who had designed the network). The team had eliminated all possibilities (other than the network) after about a month but continue to meet daily. Eventually they suggested that each field office (which were independent and had to pay for all access) would have to buy a T1 line to improve the access. After this was rolled out the problems appeared to vanish.

Later it was learned that someone at NO had found that a half-duplex 10BaseT card was being used at the center of the network pathway to the outside world. All of the traffic had been moving through a pinhole. When it was replaced with the correctly configured 100-BaseT card all the problems went away. Of course by then everyone had paid for the T1 so it appeared "that had fixed the problem".

The network was never acknowledged as a problem.

"The Architect of the AS/400 won a Nobel prize for its design in 1947."

The head of the AS/400 team was known for interesting, head-scratching statements. This one came in a presentation to all the director level managers on the benefits of continuing to use the AS/400 as the database of record. We checked and the architect was something like 10 when he "won" the prize.

"The AS/400 can process an endless loop in 30 minutes."

Same guy responding to a comment in a meeting where the Java team mentioned finding an endless loop in some piece of code. I can't even begin to imagine where this came from.

"The AS/400 is self-tuning. It happens below the hardware level."

The "DBA" for the AS/400 team was explaining why he didn't need to pay attention to what was happening to the database on the system during our battle over performance issues. We imagined that some special carpet on the floor did the work.

"The [system] allowed us to cut development time from 7 months to 4 weeks".

Our CIO liked to write articles for vendors we had purchased software systems from, bragging how effective his choices were. This system was barely used in a minor feature just to make him happy; otherwise it sucked so bad no one would touch it. One day the vendor sent a team to interview the developers on how we were using it to create such an impressive result. We told them we never used it outside of the little feature. They went away quietly.

There were more, but these will have to do for now.