Events / Login / Register

Quality Metrics

(this section includes contributions from InsideSpin member: Adam White)

The customer service functions tend to be the main source of direct and mostly independent input as to what is going on with customers and prospects. Putting in place a metric gathering system, whether formalized or not, is one of the best ways of gathering requirements for future versions of a product. In fact, the customer service group itself tends to be an often overlooked area to obtain valuable product feedback from (this is covered in other sections, including Product Management) so most companies miss a chance to fine tune their product investments in ways that would directly impact the success of new sales situations (pre-sales) and the satisfaction of existing customers (post-sales).

Tracking the number of calls that come in, the time taken to close them, which area of the product is most called about, etc are just a few of the valuable sources of input the Company should pick up on when planning next releases. This information is centered in the customer service functions and is usually easily mined for useful data.

As a business leader, you should be aware of the overall quality status of your Company's products, You should not have any hesitance engaging a customer in a direct discussion about overall product quality. Few customers ever get a chance to speak to the business leader (CEO), most would be very willing to work with the team to help solve even the most critical problems if they knew the CEO was overlooking the Company's commitment to getting issues resolved. It's invaluable establishing a reputation for quality customer services, it's often unrecoverable to get rid of a bad one. Let's look in more detail at a variety of quality metrics and how they can be fed back into the business to create continual improvement.

Basic Metrics

There are a variety of basic quality metrics the team can collect without implementing much in the way of a formal tracking system. It is better to have a proper tracking system, recording events as early in the lifecycle of a product as possible (ideally during the testing of the first development cycles). Like all quantitative metrics, you need to decide what you will do with the knowledge. Gathering data for the sake of it does not make sense, gathering data that will help drive core business decisions does.

Some basic metrics to consider include:

As with sales metrics, we can also factor in product component. With this information each sub-team can take decisive action. The verification (test) team can focus it's efforts proportionately on specific problem components, development can see what they may need to re-work/refactor a weak area of code and Product Management can prioritize appropriately as well as develop and communicate a compelling story to customers about what they are doing about their specific issues.

You'll need a consistent way to refer to product components across teams so everyone is on the same page. The systems that record customer calls must have the same component list as the systems the other teams use. The support team has to have as part of their process the desire to categorize calls effectively. Also the people fixing the issue have to dig in, see if the original classification is correct and remember to change it if they find an error (this is the hard stuff to make happen effectively).

With basic metrics in place, you can look for areas where investments can be made to improve product quality and then reassess how the metrics change at a later point in time. If the metrics do not change, you might have a gap for product management to examine between how requirements are specified and how they get implemented. Some obvious areas that influence product quality include:

One often used strategy is to have the team knock off the top 5 issues as recorded by call volume each release cycle. Over time, you will hit the critical issues and although there is always a new definition of critical, the "real" critical issues will have been addressed (you will need to actually have a reasonable volume of users to take this approach though). Don't forget to count prospects and customers -- some teams assume prospects don't count, but in many ways, evaluating users count more to the long term success of the business than do the customers that already paid. Tracking problem areas evaluators come across as a separate analysis over customers is an excellent way to make sure you focus on the growth of your business properly.

Link to Product Management

As mentioned at the start, it is not uncommon for Customer Services to be completely overlooked when it comes to providing input to the product planning process. Yet, if you were to take the extreme action of only fixing the issues defined by Customer Services, you would probably find that overall customer satisfaction levels would grow higher very fast. If you consider that support issues typically reflect product issues, the CS team is the hub of activity. They are witness to issues from first installs all the way through to extended and mature product usage. The CS team also becomes acutely aware of all the workarounds needed for the product to perform as advertised, all important data hard to glean from anywhere else in the organization. Even customer surveys rarely raise to the forefront the issues the CS team sees.

Given the cross-functional issues at stake, the business leader (CEO) should make sure the CS team has a role to play in providing input into the product management process. The product manager should spend time working with the CS team as an integral part of release cycle management. In fact, an excellent approach is to have each product manager spend one week per year (minimum) working the CS job functions -- putting them on the front lines, hearing issues first hand. Great product managers will have no issue with this and will likely seek out this kind of direct customer input.

If formalization works well in your organization, you may want to allocate a certain percentage of all release cycle investment to priorities raised by the CS team. When treated as an end-customer, the CS team's level of job satisfaction goes higher when they see their issues treated with similar priority as paying customers. Keep in mind the CS team needs its own features and functions to properly handle customer issues. Although most CS teams end up developing their own tools for support purposes, tools that come with the product have wider benefit and better contribute to the valued IP of the Company.

Keeping CS happy will positively impact customers given one is a reflection of the other. if you offer this respect for the CS team, you should also create an accountability whereby the CS team is responsible for identifying problems to product management that would create positive changes in the support metrics described above. It should work both ways after all -- just like asking a prospect to become a customer if you agree to implement a missing feature.


If you have the data, you should communicate what is learned in the form of reports.

Daily reports provide insight into spikes that might be occurring related to patterns of product usage (e.g. certain features are used on Monday while others are used more commonly on a Friday). Reporting call metrics by feature highlights this.

Weekly reports summarize overall load and activity through a 7-day cycle. Ideally some history is also provided so the reader can eye-ball any trends. Perhaps each weekly report includes the data from the past 4 weeks and perhaps the data from that same week one year ago. This all should reflect the maturity of your business, clearly this level of historical data is not relevant if the product has been in the market for but a few months.

Monthly reports provide a more macro view as to load and activity. Perhaps these reports also highlight important points in time during the month such as new product releases, marketing campaigns (don't forget to track evaluators), etc. Similar to the weekly reports, providing some monthly trending is important. Perhaps should the previous 3 months of data along with the same month the previous year.

Quarterly reports roll up the overall perspective seen from the customers viewpoints. Aside from the trending details, overall analysis of the metrics should be shown as well as any corrective actions taken to influence the metrics. Highlight periods where call volume changed unrelated to customer growth and explain what might have happened.

Annual reports provide an excellent way for the CS team to line up with overall business objectives for the year. An annual report would typically contrast actual results with predictive results from the operational plan (e.g. the op plan should have had some performance assumptions at the core of how the plan was built). Variances should be highlighted with suggested actions for the following year -- perhaps more people need to be hired, infrastructure needs to be improved or more likely, product changes need to be made to reduce critical areas of weak functionality.

Reports should be communicated in a variety of forms:

it's not uncommon for most of the reports to go unnoticed. The challenge when setting up a system to gather metrics is to obtain the commitment to manage the operation according to what trends show (combined with experience and intuition). Reward the team for positive trends, incent the team to correct negative trends, be open about whatever trends do occur. Make sure the customer is properly represented in the loop of communication and decision making.

Team Incentives

Many organizations fail to incent the customer services team. Often treated as a back-room activity, often dealing only with negative issues, the CS team is rarely seen as something that can affect the success of the Company. This is clearly not the case, in fact, a technology company known for excellence in customer services can use that reputation to overcome many other weaknesses. As such, the CS team should be properly recognized for its successes.

Some simple things you can do include:

Set some achievable goals, bind the team properly with product management and watch the positive results come in.

Provide feedback to improve overall site quality:

(please be specific (good or bad)):