Programmers Are from Mars, Project Managers Are Sometimes from Hell
One of the common characters in most software development shops is the "Project Manager", whose job is to oversee the various phases of a project, from requirements gathering to hopefully shipping the result on time and on budget. Sadly few of the projects managers I have worked with had any actually clue on how to do any of these things.
In many shops the project manager is not a former developer but more likely someone who moved up from other positions. At my favorite employer (well documented in several sad but entertaining posts here) three of the main project managers were (1) a former forest ranger (2) a former Maître d' (3) a customer. These three were given a large amounts of authority and despite a long history of failure to deliver continued to gather glowing reviews from upper management. Perhaps this was an indication of the woes of upper company management, but you would think that delivering projects months and even years late and costing the company reams of money due to poor decision making would be a cause for some kind of change. The actual criteria for success seemed to be attention to the company's ever-changing formal methodology, not any actual project deliveries.
It never hurt to blame the developers, the customers, the consultants, the vendors and probably the man on the moon. Sometimes these are valid complaints but like the boy who cried wolf after a while it begins to sound like the problem is the project manager. The PM's had virtually absolute authority over all project details including technology decisions.
So what is a good project manager? First of all, I prefer the term "Project Facilitator". Larger projects especially need a lot of communication between all the developers, customers, operations, QA and management with a stake in the project. Someone needs to manage that communication, to ensure that everyone knows what they need to know and when, and track project details, schedules, estimates and progress.
Often upper management seems to prefer the concept "Project Czar", an omnipotent person controlling all aspects of the project, and thus guaranteeing success. In practice this type of dictatorship rarely works unless the PM is truly talented and experienced in all areas of the project. It can happen but none of the PM's I have worked with or for ever had that level of competence. The perfect ideal of a political leader is the benign dictator but that never works out either.
In the end someone does have to be in charge, democracies are great ideas but usually give way to republics (representatives of the people are given authority to act in their stead). Where do you draw the line in a software project? I've always preferred having people with actual experience make decisions in their areas (eg architects making technology decisions, business managers managing customer interactions, etc). Sometimes you can find people who are capable in multiple areas and they can take on multiple roles (I've done this a number of times).
If you talk with project managers, the most common thing you hear is "it doesn't matter that I know nothing about the technologies involved in the project, managing people is what's important". Usually you only hear that if they really don't know anything about the technology. Sometimes they don't even know how to manage people, especially the occasionally complicated programmer. In the business of developing software projects you really need to know and understand both people and technology. If you make technology decisions without a clue, the best team in the world won't be able to produce; likewise if you know the technology so well you could have invented it, but build and manage the team without a clue, you will still fail. I've seen developers-turned-managers fail as often as tech-clueless-managers.
Like any speciality in our programming world, the project manager (or again as I prefer project facilitator) cannot simply be promoted from some random position. Would you hire a programmer to write code with no coding background? Prospective project managers really need to be exposed to all facets of projects, from analysis of customer needs to technology selection to architecture (at least at some level) to some basic understanding of what programmers do. Once they have been involved in these various activities for some time, only then they should be allowed to facilitate entire projects.
I have been in places where there were no project managers as well, and with no central decision maker. This rarely works either unless the teams are truly talented. If the projects wind up disorganized and fail to be delivered then it should become obvious that you really need someone to manage them. I think companies often start off with no managers and then wind up going to far to dictatorial management.
The best projects managers I have worked with (not many but they stood out) were organized, understood the technologies or at least spent time learning them, communicated well with everyone, and especially spent more time listening than talking. The whole point is to make the project itself move from inception to delivery smoothly with a minimum of trouble. Even in a dysfunctional organization, if someone can actually deliver success despite the challenges while avoiding needless conflict with everyone involved, then you have found the ideal PM. One of the less experienced PM's at my favorite company (in this case a former Marine) did an awesome job in every project he managed but refused to play the political games the other PM's played or sugarcoat complex issues so he remained low man on the PM totem pole. I would work on any project he managed again but I had zero respect for the others.
Software development projects are usually hard, there is no denying that. If you use good technologies, assemble a good team of engineers (development and QA), use a flexible methodology, work with helpful and involved customers, and then have a solid project manager facilitate the process, whatever you are doing is likely to succeed.
One can dream, anyway.