The Codist

What could be worse than a computer virus infection? How about a mother-of-all-virus-checkers infection? Note that this story is true despite its unbelievable details.

The day known as Black Wednesday started innocently enough, 600 employee's PCs humming away, running Mcafee version 7 which although occasionally irritating was not causing much trouble. The company had had no virus infections since I started working there so our internal security seemed to be working. Then at 12 noon our dear beloved Network Operations group (please note the sarcasm intended) without any warning turned on the Windows XP automatic update and simultaneously auto-updated Mcafee to version 8 for all non-production computers and servers.

The whole company ground to a halt.

Every PC tried to collect and install around 70 updates to Windows (requiring multiple restarts) and install the new version of Mcafee (with all of its various parts) all at the same time. The network was overwhelmed, even our external web presence stopped working (our internal and external network traffic shared the same bottlenecks, don't ask). The lunch people were unable to supply lunch (their POS PC froze). Customers couldn't be served, our field staff was unable to create orders, basically the whole company went dormant for the rest of the day.

The rest of the Java team and I went home that evening assuming things would work themselves out by the next day. We were wrong; the hell of the coming months started the next day.

Our Java team used the usual Java tools such as Eclipse, IntelliJ, Weblogic Workshop, DBVisualizer, and Weblogic. In Java, every piece of code (and often associated data) is packaged in JAR, WAR and EAR files, which are specialized versions of ZIP files. Everything you do in Java involves reading and writing these files, both when running Java applications and building applications. In all cases these files are read by the Sun Java runtime, and executable code is limited to valid and secure Java bytecode, not native code.

The Network Operations folks had turned on every feature in the new Mcafee, including the dreadful "Uncompress compressed files" setting. All of our Java development was stopped virtually dead in its tracks. Every access of a JAR, WAR or EAR file now resulted in the computer to freeze as Mcafee opened the file, uncompressed its contents, scanned each file inside, and then only returned control to the application once it was complete. During this time the CPU was entirely utilized by Mcafee, which was set to high priority. Launching your IDE usually took a few seconds, now it took 20 minutes. Some application builds took hours (especially with the weblogic.jar file, which was enormous) instead of minutes. Just typing in your IDE was type type type, wait 2 minutes, type type type. It turned your PC into a single-tasking computer circa 1979 running the full Windows XP.

At first we though it was simply a configuration error, and reported it as such. Network Operations said it was working as intended. We had to blink, this made no sense. Every Java developer was getting about 1 hour of work done per day at best. Being engineers we figured out what feature of Mcafee was causing the trouble. We couldn't turn it off, it was locked. We complained to our management, but Network Operations was more highly regarded in the company pecking order, and they simply said we were whining and the company's safety versus virus attacks (even though we had never had one before) was more important. We argued that native code viruses hiding in Java JAR files couldn't be executed by the Sun runtime and that there were no such viruses.

So they said (to everyone) we know of many Java viruses in the wild. Given that we had done an exhaustive search and found nothing, this was unbelievable so we demanded a meeting. The head of the NO department said (to a roomful of Java engineers) that he had a list of 500 Java viruses and that was the basis of why this was being done. No matter what we said he said the same thing over and over.

After the meeting I challenged him to turn over the list to us but he passed it off saying the head of Security was assembling the list. This guy never returned any messages (what could do say, he really worked for NO). Later one of the NO employees said that they had typed Java into a virus checker company search form and found 500 hits. Really. We looked and the only actual viruses were in the ancient Microsoft java runtime, which only ran in IE, and only for applets. Pointing this out to everyone made no difference. The NO had spoken.

This went on for about 3 months (I had just started a project that should have take a couple weeks and it dragged out over the whole 3 months) before we finally got our manager to suggest that we hire an "expert" to come in and suggest some kind of improvement. NO agreed but only if we paid for it, and they were allowed to pick the vendor. This was approved and so they hired an "expert" who turned out to be the vendor whose main customer was the NO group. His "analysis" said to turn off the compressed file option, but only for those employees who would run their PCs as ordinary users and give up their administration rights. Say what?

For those who don't do development on Windows XP, everyone runs as admin since ordinary users don't have enough rights to make any changes in their environment. It can be (painfully) made to work for simple Word users, but for programmers it's impossible. So now we had paid for the "expert" and it would look like we were just whiners who didn't want to get any work done if we didn't acquiesce to this. So we had no choice. First they wanted to test the choice so a few Java developers (and the QA team) lost their admin privileges.

Let me also add that this was not just affecting the Java team, though much worse. Our DBAs used DBVisualizer which is a Java application, and even they had trouble getting anything done. It didn't help that people like our "Enterprise Architect" said he didn't understand why we were whining, since it didn't make his computer any slower (he only used Word and Powerpoint). Some ordinary users reported virus infections as their computers were sluggish and stalled at times (no one ever told them about the changes) but were told it was nothing.

Upper management didn't seem to care or understand why the Java team had so much trouble getting any work done. After all, the production servers still ran as fast as usual (still running Mcafee version 7, yes our production servers ran virus checkers!) and their PCs seemed OK. The web systems folks were also in pain, as builds took all day (and often they turned off the virus checker, despite warnings of firing). Many of our test and QA systems were built with multiple copies of VMWare, thus they ran terrible slow as each virtualized system ran a copy of Mcafee 8.

Anyway the admin rights turned out to be a disaster as well, as we could no longer install software, even a Java runtime upgrade was impossible. The QA team would not even install our C++ applications to test them. Every install required filing a ticket, waiting for someone to get around and do it. The support staff was overworked now as well.

It didn't matter, viruses were everywhere, security was more important!

The "test period" went on for months as well, nearly a year later not everyone was even "upgraded" to the new option, which itself was terrible. However by that time I was gone, and a flood of other senior people left as well. Eventually the departure of so many valuable people was enough that upper management finally told the NO team to start caring.

Of course I no longer cared.

In this entire time not a single virus was ever caught anywhere in the entire software development area. I never heard of any computer actually spreading a virus anywhere in the company. And of course our production servers still ran Mcafee 7, so the argument that Mcafee 8 was necessary was itself silly.

The real irony of this whole story is that it's the same company in WTF #1:It's Not The Database Stupid.