• Starting in late development stages, once a system is built enough so that it can be used by real users.
• They are also often run on existing system when planning major system upgrades, and process re-engineering.

What this is: Metrics tests provide precise micro- and macro-level measurements of user performance:

• For example, if a CSR (Customer Service Rep) uses 6 windows and takes 10 minutes to solve a typical customer problem,
• What test findings can be applied to reduce that to 3 windows and 5 minutes with the same level of customer satisfaction?
• In this case, the high-level business objectives would be:
• To increase (double?) the CSR's productivity.
• To provide shorter calls, which increase customer satisfaction

Average turnaround time: A metrics test takes about 50% longer than a usability test of the same size and scope.

• That is, if a qualitative usability test takes a usability specialist one week for all activities, a comparably sized metrics test is likely to take 7.5 days of time.

How it works: Productivity and performance improvements often come from these types of activities, which is a larger process than one metrics test:

• Use or create measurable performance goals
• Identify the most frequent and important tasks, with their steps and sub-steps
• Test or simulate the system (with one user at a time) under normal and peak loads, in the real context if possible (that is, real callers or using scripts based on real calls)
• Measure outcomes, durations, errors, learning rates, etc.
• Identify improvement opportunities
• Design improvements:
  • If possible, re-engineer the work flow and process.
  • If that's not possible, sum the improvement opportunities.
• Simulate and test the new design
• Identify and sum the strongest improvements
• Calculate their cost/benefit across the system life cycle
• Prioritize, implement and re-test (as needed to maximize productivity)

Key knowledge questions: A typical mistake is to focus on process-based metrics. Metrics is more than monitoring actions. It has three sides, and all should be included:

• There is an often overlooked external side:
  • Compare this system's performance and satisfaction to other systems.
  • Are there other, better ways to achieve the same goals?
  • If so, how much better are they?
  • Which system is the best or right solution?
• If this is the right system, then measure system performance, efficiency, effectiveness, outcomes, costs and benefits of improvements, etc.
• And if this is the right system, then measure user satisfaction using a standard set of questions across all tests, to clarify what is and is not achieved from the users' viewpoint.

How to get it right: Contact us.

Top