Why can't we hit dates?

My boss asked a few months back why it is so hard to hit dates with big software development projects. It's almost a rhetorical question, and I think at a high level he, and most managers, know many of the answers.

But - here goes: In software development we're asked to provide a completion date with most of the work still unknown. Only when a PM or a project team is estimating a project they have done several times, and have absolute certainty of the scope, can they provide an accurate estimate of effort and schedule. With software development, every project is an unknown. The only way to hit a date to is to set it and then accept whatever deliverables can be done by that date. This does not achieve the goal of meeting what stakeholders want by that date but rather achieves a purely date-driven goal.

Not to get on the Agile bandwagon too much, but the idea of getting the stakeholders to look at completed software at regular intervals and declare the project done enough when they have the functionality they want is a pretty good one. Gathering all requirements up front helps the planning process but by no means guarantees completely accurate effort estimates or duration estimates unless the work very closely mirrors other projects and holds no major risks.

My boss expressed a desire to create a model where, in many cases, he could "turn the resource knob" and offer faster completion at higher costs when necessary. There is a point at which you cannot add more resources and go faster. It is important to note that more and more research shows that offshoring does not mitigate this - offshoring or outsourcing will hit business requirements road blocks, quality issues, or eventually the same law of diminishing returns. This is not new or unique to my current employer- this is a global condition that IS managers and PMs beat their heads over consistently.

Different situation - When I left a job as head of a custom solutions project group, my team and I could estimate video production projects within 5%. Publications projects could be a little more variable, but were still pretty accurate. Software projects were complete unknowns. Even when there were some basic similarities in content or subject matter, the programming side was unpredictable. The point here is that I had a team that developed estimates for a variety of project types on a daily basis - estimates that, if these projects went forward, would impact the team's performance bonuses each quarter. And despite that incentive and expertise, software projects remained incredibly difficult to estimate accurately.

Having had the need to do a lot of reading and research on software project estimating, the best piece I have seen on this topic recently is Joel Spolsky's piece, which you can find here: http://www.joelonsoftware.com/items/2007/10/26.html

Read it - it's good stuff.