After writing yesterday on good programmers, I came across an article that attempted to aexplain the ideal phone screen. Having been on both ends of the process, I find such detail-oriented questioning pointless in finding good programmers.
The most important trait of a good programmer is their ability to think. It sounds trite, but over the years of my career it's the one thing that separates the barely conscious programmer from the truly productive and successful.
I have known people who passed the Java certification exam on the first try, and yet when faced with a real project, produced nothing but absolute crap every time. Programming these days is not about knowing facts (which you can always look up) but how to take a real problem or request, and create a solution that works. How do you find that in a phone screen about facts? You don't.
Sure you can force the person to write some trivial code problem, which only proves that they can write trivial code problems. That's only relevant if you are hiring people to write methods. One airline I interviewed at proudly pointed out that their architects never wrote code, their project leaders only wrote API's and their programmers only filled in the API's. OK for them it might make sense to hire trivial coders, but I ran out of the building as quickly as possible.
So how do you find programmers that can think? For one thing you need to ask about projects that they did on their own, or at least lead. How did they discover requirements or communicate with the customer? How did they go about designing or prototyping a solution? How did they organize the team or at least manage their schedule. Did they interact with the customer regularly? How did they chose the technologies (large and small) to implement a solution? What sucked during the project and how did they overcome the problem? How did they interact with other people involved who were not programmers? What about tools that they used and why they used them?
The point of such a narrative is to figure out if they know how to think about solving the whole problem, but just the programming part. Most programming jobs are not one guy in a garage, there is more to the job that isn't just the code itself. Yes, its nice to hire people that know the coding but what good is it if they can't function in the environment? A couple of jobs ago my only technical interview was two top programmers taking turns telling me how crappy the environment in the company was, and seeing how I reacted. Later on I discovered it was actually worse than they said (hard to believe) but I was able to make a difference despite the challenges (and yes it required lots of new technologies to me).
So thinking and dealing with complex environments (political, technical and customer) are important assets but what about EJB's? Or Weblogic? Or AOP? Or whatever? Well if the discussion about projects is detailed enough you can tell if they know how to think about the coding aspect as well. After all if your code sucks the project probably failed anyway. I can remember the development details of virtually everything I have ever written, but most of the technologies have long since faded away, if you ask me details about JMS I have no idea anymore; but I can talk about the project where I used them in great detail. It's the thinking that matters, everything else can be looked up. I need to know how to develop a project irregardless of the technology since everything changes over time anyway.
The trick is to find a way to see how a person would work on a real project, without giving them one. Maybe it's not easy, but I am sure that just asking coding details will get you a coding monkey but not a good programmer employee.