QoL and SPI: battle of the ROI’s

Jason Della Rocca has an article on the Escapist about investing in game developers and SPI (software process improvement.) As a game developer, I have no problem with the first part of this argument :) but the second is a minefield. SPI is too big a subject to fully cover in a single blog entry or article, but I will jot down a few thoughts raised by this issue.

Jason’s article is a pretty good summary of Steve McConnell’s lecture at GDC 2005. In it, Steve talked about the business concept of ROI (return on investment) and investing in software development “best practices.” The article claims 1000% ROI for SPI. It’s an impressive sales pitch. But does that claim stand up to scrutiny? Steve has a reputation for documenting his sources so it’s easy to track: “Benefits of CMM Based Software Process Improvement: Initial Results.” Pittsburgh: Software Engineering Institute, Document CMU/SEI-94-TR-13, August 1994.

However, reading that paper, it quickly becomes apparent that the authors did not intend their research to be sold this way. Firstly, they caveat all their findings:

• the results were not meant to be representative
• the results were to provide an upper limit for what might be possible in a supportive environment
• if improvements in one area created costs in another, those costs were ignored
• only set-up costs were accounted for, running costs were not included
• improvements attributable to factors outside of the CMM-based changes were not accounted for
• the ROI was (and could only possibly be) a guess

Then they further caveat the 1000% case (Schlumberger):

• the data points were extremely variable
• some data points were real cost data multiplied by further guesswork
• other data points used more subjective criteria
• the interpretation heavily influenced the result

With all these caveats it’s a wonder they felt confident enough to come up with any figure at all. But then, as they state in their introduction: “We urge anyone planning to make use of data contained in this report to read Section 3.1, which includes information essential to appropriate interpretation of the data.”

In contrast, both Steve in his lecture, and Jason in his write up, characterise these findings as “no-brainer” investments. I beg to differ. Statements like these might even impede progress: one of the more commonly reported features in reviews of SPI efforts is the failure to manage expectations.

Jason’s article does correctly identify that most of the ROI comes in the form of savings, not profits. In order to profit from these savings we’d first need to change our business model. Without that change, all the returns on these investments will go to waste. I followed up on the Schlumberger case and they say this: “it is not enough to spend time improving software development; the improvement activities must involve the other parts of the business with which software interacts, namely marketing, hardware development, sales, manufacturing, etc.”

I believe investment in process can produce returns; modest returns. But we can only benefit from them by first changing the way we do business. However, if we did all that, it still would not make any difference to developer QoL. The reason our QoL sucks is that we give our developers too much to do. And if we improved our processes to the point that we got done in half the time, we would just get twice as much to do.

We know this because it happens every time we do improve our processes: with the widespread adoption of middleware we have already (as an industry) made the most profitable of the improvements identified by the SEI study.

The way to improve QoL is to spend money on improving QoL. SPI is for tweaking performance when the fundamental business issues are sorted out. That is what the people who do SPI tell us. That is what we already know.

PS. There is an interesting response from Jamie Fristrom: he feels somewhat insulted by Jason’s comments and suggests that he is offering a silver bullet solution. (This is true by definition for anybody offering 1000% improvement in production in less than 10 years.)

PPS. In his first entry on the subject Jason rejected the criticism that: since games production involved more than software, modifying software processes would not help. He suggests that software production represents a large chunk of the production time and therefore there is a lot of fat to cut. However in reponse the Jamie’s open letter, he appears to change his stance on that and accepts that improvements in software process alone are likely to be minor. He also states that he still pushing for a revolution rather than an evolution. I believe this is the right position to take since process improvement efforts often fail due to the perception that the returns are not worth the costs.