Managing Product Quality: A Story of Neglect and Oblivion

One of my constant struggles in managing software products has been in establishing and keeping tabs on the correspondence between the amount of work completed on a project and the quality of the software produced.

Oddly enough the main claim on time that would otherwise be spent on “designing quality” is spent on developing features, which, unlikely quality, are easily counted and plotted against costs. Shipping features often gets you off the hook quickly because you can easily explain a feature. Quality, on the other hand, eludes a snappy description.

All this is fine until one must negotiate something out of the product. Negotiating on a feature is usually a minor inconvenience, but when negotiating on quality the conversations get tougher and less pleasant. Customers leave, teams are demoralized, and product management is dismayed.

Why then do we still rally behind the feature list every single time? Here is why and what to do to rebalance the equation… (or skip to TL;DR section below)

Every product I’ve ever been involved with has had a day of reckoning dealing with quality.

It is natural for the engineering team’s priorities to be skewed toward feature-building while the product is in it’s early stages of development and toward scalability, robustness, maintainability – the ‘-ilities’ –  in the later stages. There is nothing wrong with that, per se, especially since the priorities don’t appear in a vacuum — product managers are responsible for setting them.

At the end of the day it’s an issue of timing. As product matures it goes through rapid feature implementation to “oh, shit” refactoring to more rounds of maturation followed by incremental feature enhancements followed by maintenance.

In the early stages, when management, investors and customers come calling, a longer feature list is equated with progress. There just isn’t any other type of progress to show. So the product managers are naturally seduced into thinking about their product as a list of features.

The seduction is easily undone when suddenly you learn that feature A cannot be used by more than 10 people concurrently. “More features, more progress” turns into “more features, more problems”. It is anticipating this critical juncture that sets apart good product managers, and it’s a skill that comes with experience watching different software products mature.

Here are the warning signs to help you diagnose that the turning point is nigh.

  1. Poor or non-existent QA process
  2. No one knows how the product performs, or everyone has a hunch about how the product performs
  3. All requirements are expressed as “product must do X”
  4. -ilities are rarely mentioned in engineering meetings

Our tools – Agile in particular – makes quality hard to talk about and hard to track. 

One of the items above is “all requirements are expressed as ‘product must do X'”. If this is happening then it’s clearly a product manager’s faux pas. How could she not see that there is more to the product than a list of features?

Having done Agile to the various degree of fidelity over the last 4 years, I can say that our methodologies are at least partially to blame. Pardon the pun, but we are the product of our tools, or “when all you have is a hammer, everything looks like a thumb”.

There are only so many things that can be written down using the Agile formula for stories: “As a user, I would like to do X so I can do Y”. Try writing down a story that says that feature X needs to work when there are 100 people trying to X simultaneously. The -ilities simply fall by the wayside because you can’t write down a story if it doesn’t have a user.

Well, there is always the acceptance criteria. My answer is that if you, as a product manager, have to waterproof the acceptance criteria for software quality you have already lost the war. It’s just far too difficult to write every story with a list of throughput, scalability, robustness and latency parameters tucked to the end.

Of course it’s a function of how strictly one chooses to adhere to the Agile method. But the point is this isn’t a criticism of any specific method. Just the way feature descriptions are easier for users and investors to grok, they are easier for product managers to convey to engineering.

Listening to engineering’s pains is perhaps the biggest thing you can do to move the needle on software quality.

We, the product managers, can be a snobbish lot indeed. It’s too easy to fall under the impression that we know better because we talk to customers. Turns out there is more to product success than the needs of the customer, especially when it comes to the highly technical software product.

So the most obvious population to talk to next is to our own engineers. Why, they have been there all along telling us what’s wrong with the product and how it won’t scale or stand up to the load. While the customers worry about dashboards and idiot lights, engineers worry about the engines. And you product will need a robust engine if you actually want to go places.

Engineering insights are also invaluable because they surface the risks. Customers would never think of implementation risks, they just don’t operate in that dimension. The product manager may not be technical enough to appreciate these risks either, and yet being oblivious to them is a recipe for failure.

Take off your customer rep hat, grab an engineer, and ask them what they worry about when they worry about the product.

TL;DR

  1. There comes a point in every product’s lifetime when the feature list is no longer sufficient to demonstrate progress – at some point product quality (i.e. the -ilities) must catch up. There are warning signs when this point may be drawing near. Watch for them!
  2. Our tools can be our curse. Fitting everything into the same mold is nice and tidy, until it’s not. Be mindful of the range of requirements that are naturally expressed with the tools at hand. It may be limited.
  3. Getting out of the building to talk to customers is still a must, but do talk to your engineering team as well. They will have insight into the product a customer could never bring. These conversations are critical throughout product lifetime to balance between feature building and quality.
Advertisements

3 responses to “Managing Product Quality: A Story of Neglect and Oblivion

  1. Wow. Great read, and you are spot on. I have long worried about “quality” and I am shocked when I submit requirements (we do hardware) that list things like out of box failures, MTTR, MTBI, and MTBF and the engineering team sneers.

    Or detailed requirements about how to recover from a SW crash, because you know someone’s valuable sample is in our system in an indeterminate state and needs to be recovered without damage. I long ago learned that if you don’t specify it up front, it is a mad scramble at the back end.

  2. Pingback: Suppressing the Excellence | ReStreaming

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s