Quality Metrics
(this section includes contributions from InsideSpin member: Adam White)
The customer service functions tend to be the main source of direct and mostly independent input as to what is going on with customers and prospects. Putting in place a metric gathering system, whether formalized or not, is one of the best ways of gathering requirements for future versions of a product. In fact, the customer service group itself tends to be an often overlooked area to obtain valuable product feedback from (this is covered in other sections, including Product Management) so most companies miss a chance to fine tune their product investments in ways that would directly impact the success of new sales situations (pre-sales) and the satisfaction of existing customers (post-sales).
Tracking the number of calls that come in, the time taken to close them, which area of the product is most called about, etc are just a few of the valuable sources of input the Company should pick up on when planning next releases. This information is centered in the customer service functions and is usually easily mined for useful data.
As a business leader, you should be aware of the overall quality status of your Company's products, You should not have any hesitance engaging a customer in a direct discussion about overall product quality. Few customers ever get a chance to speak to the business leader (CEO), most would be very willing to work with the team to help solve even the most critical problems if they knew the CEO was overlooking the Company's commitment to getting issues resolved. It's invaluable establishing a reputation for quality customer services, it's often unrecoverable to get rid of a bad one. Let's look in more detail at a variety of quality metrics and how they can be fed back into the business to create continual improvement.
Basic Metrics
There are a variety of basic quality metrics the team can collect without implementing much in the way of a formal tracking system. It is better to have a proper tracking system, recording events as early in the lifecycle of a product as possible (ideally during the testing of the first development cycles). Like all quantitative metrics, you need to decide what you will do with the knowledge. Gathering data for the sake of it does not make sense, gathering data that will help drive core business decisions does.
Some basic metrics to consider include:
- Counting number of calls received -- a clearly basic metric but one that can give some indication of changes in overall product quality. You will likely need to divide the raw number of calls by the number of active customers or something equivalent to obtain a normalized ratio that you can use.
Our usual challenge emerges though, what do the trends mean?- if the number of calls per customer is going up, it could indicate declining quality (bad) but it could also indicate increased product usage (good).
- if the number of calls per customer is doing down, it could indicate increased quality (good) but it could also mean declining product usage (bad).
- How long it takes to close calls (time2close) -- overall customer satisfaction often hinges on this metric. Although people often demand near-instant service, they do reasonably expect resolutions to reported problems in reasonable time frames. Calls that average 20 days to close probably indicate failure in many parts of the organization, calls that average 24-48 hours to close probably provide the kind of service expected. The required average does depend on product complexity, target market, user sophistication, etc. The key is to choose a target metric for each product and measure the Company on variances.
- Priority classifications -- organizing calls by some sort of priority classification scheme helps refine the way in which the metrics can be used. Separating the measures such as close times by priority classification tends to help isolate where focus should occur. If done this way, it is not uncommon to never get to the lower priority issues despite best intent. The best advice here is to be careful what you commit to so customers are not expecting every single documentation error to be corrected or small user interface issue to be addressed with equal priority to core feature bugs.
- Defect ratio - If we take the number of calls one step further and look at the number of calls that get turned into product defects we can start to look at the overall quality of the development process as a whole. This will provide another level of granularity. A high number of calls with low defects might indicate that there is process problem or something easily fixed with a documentation or knowledge base enhancement. A 1:1 call to defect ratio might mean that it's time to release a hot fix/patch to the general public or that there are serious quality problems. You need to set your own expectations for base ratios (minimal quality goals) and establish incentives for all the teams involved to improve them
As with sales metrics, we can also factor in product component. With this information each sub-team can take decisive action. The verification (test) team can focus it's efforts proportionately on specific problem components, development can see what they may need to re-work/refactor a weak area of code and Product Management can prioritize appropriately as well as develop and communicate a compelling story to customers about what they are doing about their specific issues.
You'll need a consistent way to refer to product components across teams so everyone is on the same page. The systems that record customer calls must have the same component list as the systems the other teams use. The support team has to have as part of their process the desire to categorize calls effectively. Also the people fixing the issue have to dig in, see if the original classification is correct and remember to change it if they find an error (this is the hard stuff to make happen effectively).
With basic metrics in place, you can look for areas where investments can be made to improve product quality and then reassess how the metrics change at a later point in time. If the metrics do not change, you might have a gap for product management to examine between how requirements are specified and how they get implemented. Some obvious areas that influence product quality include:
- improving product documentation (the self-help variety). Does call volume go down? Do call types shift into different product areas?
- improving on-line resources such as knowledge base articles on common problems. Do calls that really should be something the customer can handle reduce dramatically?
- improving product packaging and first experience usability. Does the number of calls from prospects go down (as ratio of total number of evaluating prospects)?
One often used strategy is to have the team knock off the top 5 issues as recorded by call volume each release cycle. Over time, you will hit the critical issues and although there is always a new definition of critical, the "real" critical issues will have been addressed (you will need to actually have a reasonable volume of users to take this approach though). Don't forget to count prospects and customers -- some teams assume prospects don't count, but in many ways, evaluating users count more to the long term success of the business than do the customers that already paid. Tracking problem areas evaluators come across as a separate analysis over customers is an excellent way to make sure you focus on the growth of your business properly.
Link to Product Management
As mentioned at the start, it is not uncommon for Customer Services to be completely overlooked when it comes to providing input to the product planning process. Yet, if you were to take the extreme action of only fixing the issues defined by Customer Services, you would probably find that overall customer satisfaction levels would grow higher very fast. If you consider that support issues typically reflect product issues, the CS team is the hub of activity. They are witness to issues from first installs all the way through to extended and mature product usage. The CS team also becomes acutely aware of all the workarounds needed for the product to perform as advertised, all important data hard to glean from anywhere else in the organization. Even customer surveys rarely raise to the forefront the issues the CS team sees.
Given the cross-functional issues at stake, the business leader (CEO) should make sure the CS team has a role to play in providing input into the product management process. The product manager should spend time working with the CS team as an integral part of release cycle management. In fact, an excellent approach is to have each product manager spend one week per year (minimum) working the CS job functions -- putting them on the front lines, hearing issues first hand. Great product managers will have no issue with this and will likely seek out this kind of direct customer input.
If formalization works well in your organization, you may want to allocate a certain percentage of all release cycle investment to priorities raised by the CS team. When treated as an end-customer, the CS team's level of job satisfaction goes higher when they see their issues treated with similar priority as paying customers. Keep in mind the CS team needs its own features and functions to properly handle customer issues. Although most CS teams end up developing their own tools for support purposes, tools that come with the product have wider benefit and better contribute to the valued IP of the Company.
Keeping CS happy will positively impact customers given one is a reflection of the other. if you offer this respect for the CS team, you should also create an accountability whereby the CS team is responsible for identifying problems to product management that would create positive changes in the support metrics described above. It should work both ways after all -- just like asking a prospect to become a customer if you agree to implement a missing feature.
Reports
If you have the data, you should communicate what is learned in the form of reports.
Daily reports provide insight into spikes that might be occurring related to patterns of product usage (e.g. certain features are used on Monday while others are used more commonly on a Friday). Reporting call metrics by feature highlights this.
Weekly reports summarize overall load and activity through a 7-day cycle. Ideally some history is also provided so the reader can eye-ball any trends. Perhaps each weekly report includes the data from the past 4 weeks and perhaps the data from that same week one year ago. This all should reflect the maturity of your business, clearly this level of historical data is not relevant if the product has been in the market for but a few months.
Monthly reports provide a more macro view as to load and activity. Perhaps these reports also highlight important points in time during the month such as new product releases, marketing campaigns (don't forget to track evaluators), etc. Similar to the weekly reports, providing some monthly trending is important. Perhaps should the previous 3 months of data along with the same month the previous year.
Quarterly reports roll up the overall perspective seen from the customers viewpoints. Aside from the trending details, overall analysis of the metrics should be shown as well as any corrective actions taken to influence the metrics. Highlight periods where call volume changed unrelated to customer growth and explain what might have happened.
Annual reports provide an excellent way for the CS team to line up with overall business objectives for the year. An annual report would typically contrast actual results with predictive results from the operational plan (e.g. the op plan should have had some performance assumptions at the core of how the plan was built). Variances should be highlighted with suggested actions for the following year -- perhaps more people need to be hired, infrastructure needs to be improved or more likely, product changes need to be made to reduce critical areas of weak functionality.
Reports should be communicated in a variety of forms:
- by email to the appropriate circulation list
- archived for anyone to look at the history
- printed for a proper record for those who may want to examine the data off line (don't forget to mark as confidential)
- summarized in PowerPoint so not everyone has to read all the detail
it's not uncommon for most of the reports to go unnoticed. The challenge when setting up a system to gather metrics is to obtain the commitment to manage the operation according to what trends show (combined with experience and intuition). Reward the team for positive trends, incent the team to correct negative trends, be open about whatever trends do occur. Make sure the customer is properly represented in the loop of communication and decision making.
Team Incentives
Many organizations fail to incent the customer services team. Often treated as a back-room activity, often dealing only with negative issues, the CS team is rarely seen as something that can affect the success of the Company. This is clearly not the case, in fact, a technology company known for excellence in customer services can use that reputation to overcome many other weaknesses. As such, the CS team should be properly recognized for its successes.
Some simple things you can do include:
- cash rewards -- the CS team is often one of the lower paid parts of the Company so any additional financial rewards goes a lot farther than it might for a group such as sales.
- meals and outings -- making it easy for the team to enjoy the job by providing food and drink is an excellent form of short term reward. You could even provide a regular lunch outside the scope of any reward system.
- weekend activities -- sending the top performers away for a paid weekend with their spouse or family is also an excellent high impact reward.
- record the achievers on a special plaque or bulletin board. Don't hesitate or think hokey to have the employee of the month kind of recognition system. It all works.
Set some achievable goals, bind the team properly with product management and watch the positive results come in.