Chapter 10: Usability Measurements

Usability Definition (ISO 9241-11)

Definition

  • Usability is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO 9241-11 standard).

Causes

  • Not specified in notes.

Goals / Objectives

  • Provide a standardized framework to evaluate how well users can interact with a system.

Importance

  • Forms the foundation for all usability evaluation and measurement efforts.
  • Guides design and assessment across diverse contexts and user groups.

Benefits

  • Offers a clear, three-dimensional structure (effectiveness, efficiency, satisfaction) for evaluating user experience.

Procedures

  • Not specified in notes as a process; presented as a definition.

Advantages & Disadvantages

  • Not specified in notes.

Impact / Effect

  • Shapes how usability is understood, measured, and improved in both academic and industry settings.

Examples

  • Not provided as a specific example, but the three components are elaborated in subsequent subtopics.

Effectiveness

Definition

  • The accuracy and completeness with which users achieve specified goals.

Causes

  • Not specified in notes.

Goals / Objectives

  • Measure whether users can correctly and fully complete tasks.

Importance

  • If users cannot accomplish their goals, other aspects of usability become irrelevant.

Benefits

  • Provides clear diagnostic data about task success and system reliability.

Procedures

  • Define success criteria for each task before data collection.
  • Record binary outcomes (1 = complete, 0 = failed).
  • For accuracy: count errors (e.g., spelling mistakes).
  • For completeness: calculate ratio of completed work to total required work (e.g., words transcribed ÷ total words).

Advantages & Disadvantages

  • Not explicitly contrasted in notes.

Impact / Effect

  • Directly influences user trust and system adoption.
  • Errors often map to specific UI problems needing redesign.

Examples

  • Transcribing a 2-page document into PDF:
    • Accuracy: number of spelling mistakes made.
    • Completeness: (number of words transcribed) ÷ (total words in source).
  • Task: “Find the current price for a share of Google stock” (clear success criterion).

Efficiency

Definition

  • The resources (time, effort, actions) expended in relation to the accuracy and completeness with which users achieve goals.

Causes

  • Not specified in notes.

Goals / Objectives

  • Quantify how quickly and with how much effort users complete tasks.

Importance

  • Reflects productivity and cognitive/physical workload during interaction.

Benefits

  • Enables comparison between designs, user groups, or tasks.
  • Supports optimization of workflows and interface elements.

Procedures

  • Measure task time: from when the user finishes reading the task scenario to when all actions are complete.
  • Count number of actions/steps taken per task (more steps = more effort).
  • Compare efficiency across:
    • Different product versions
    • Different user types
    • Different tasks on the same system

Advantages & Disadvantages

  • Not explicitly contrasted in notes.

Impact / Effect

  • Poor efficiency can lead to user frustration, abandonment, or errors.
  • High efficiency supports better user retention and performance.

Examples

  • Time taken on first attempt to complete a registration form.
  • Number of clicks required to find a product in an e-commerce app.

Satisfaction

Definition

  • The comfort and acceptability of use, reflecting users’ subjective experience with a system.

Causes

  • Not specified in notes.

Goals / Objectives

  • Capture users’ emotional and attitudinal responses to a system.

Importance

  • Determines long-term adoption, loyalty, and perceived quality.

Benefits

  • Reveals how users feel about control, helpfulness, and ease of learning.
  • Complements objective metrics (effectiveness, efficiency) with human-centered insights.

Procedures

  • Use standardized questionnaires:
    • Net Promoter Score (NPS): % of users who would recommend the product.
    • Customer Satisfaction Score (CSAT): % rating the product as “satisfying” or giving high ratings (e.g., stars, thumbs-up).
    • SUMI (Software Usability Measurement Inventory): 50-item standardized questionnaire measuring five sub-scales:
      • Affect
      • Efficiency
      • Helpfulness
      • Control
      • Learnability

Advantages & Disadvantages

  • Not explicitly contrasted in notes.

Impact / Effect

  • Low satisfaction can lead to abandonment even if tasks are completed successfully.
  • High satisfaction correlates with user retention and advocacy.

Examples

  • NPS: “After two hours of use, would you recommend this app to a friend?”
  • CSAT: “Rate your satisfaction with the checkout process (1–5 stars).”
  • SUMI: Internationally standardized tool for comprehensive satisfaction assessment.

Usability Metrics and Measurement Methods

Definition

  • Quantifiable indicators used to assess usability across effectiveness, efficiency, and satisfaction dimensions.

Causes

  • Need to objectively evaluate and compare system performance and user experience.

Goals / Objectives

  • Track progress between design iterations.
  • Assess competitive position.
  • Support go/no-go decisions before product launch.

Importance

  • Enables data-driven design decisions.
  • Provides evidence for usability claims in business and research contexts.

Benefits

  • Makes usability tangible and comparable.
  • Supports both formative (iterative) and summative (final) evaluation.

Procedures

  • Select appropriate metrics based on usability dimension:
    • Task time → efficiency
    • Error counts → effectiveness
    • Completion rate → effectiveness
    • Page views / clicks → web interaction tracking (e.g., Google Analytics)
    • Conversion rate → eCommerce success (binary: 1 = converted, 0 = not)
    • Questionnaires → satisfaction (NPS, CSAT, SUMI)

Advantages & Disadvantages

  • Not explicitly contrasted in notes.

Impact / Effect

  • Combined metrics (e.g., errors + time + abandonment) explain real-world outcomes like low conversion rates.
  • Enables A/B testing to validate design improvements.

Examples

  • Error example: “User entered last name in the first name field” → classified as “input error.”
  • Conversion rate: Measured at each stage (landing page → registration → checkout → purchase).
  • Task success: Binary metric (1 = success, 0 = failure) for clearly defined tasks.

Task Success and Completion Rates

Definition

  • A binary measure (1 = success, 0 = failure) indicating whether a user completed a task according to predefined criteria.

Causes

  • Not specified in notes.

Goals / Objectives

  • Provide a simple, foundational usability metric focused on goal achievement.

Importance

  • “If users cannot accomplish their goals, not much else matters.”

Benefits

  • Easy to collect, interpret, and communicate.
  • Serves as a baseline for other metrics.

Procedures

  • Define clear, objective success criteria before testing (e.g., “Find Google stock price” is clear; “Research retirement savings” is not).
  • Record outcome as 1 (success) or 0 (failure) per user per task.

Advantages & Disadvantages

  • Not explicitly contrasted in notes.

Impact / Effect

  • Poor completion rates signal critical usability flaws.
  • Informs prioritization of redesign efforts.

Examples

  • Clear task: “Find the current price for a share of Google stock.”
  • Unclear task: “Research ways to save for your retirement.” (not suitable for binary success measurement).

Error Measurement

Definition

  • Recording and categorizing unintended user actions (slips, mistakes) during task performance.

Causes

  • Interface design flaws that lead to user confusion or incorrect actions.

Goals / Objectives

  • Identify specific usability problems through diagnostic error patterns.

Importance

  • Errors provide direct insight into UI weaknesses.

Benefits

  • Helps map user behavior to interface issues.
  • Supports targeted redesign.

Procedures

  • Review recordings of user sessions.
  • Log each error with a description (e.g., “user entered last name in first name field”).
  • Classify errors into categories such as:
    • Navigation error
    • Selection error
    • Interpretation error

Advantages & Disadvantages

  • Not explicitly contrasted in notes.

Impact / Effect

  • High error rates correlate with poor learnability and user frustration.
  • Error types guide specific design fixes (e.g., relabeling, repositioning).

Examples

  • “User clicked ‘Cancel’ instead of ‘Submit’ due to poor button placement.”
  • “User failed to notice required field indicator” → error prevention issue.

Sample Size in Usability Testing

Definition

  • The number of users required to collect valid and reliable usability data, varying by study type.

Causes

  • Trade-off between resource constraints and statistical confidence.

Goals / Objectives

  • Balance practicality with data quality in usability studies.

Importance

  • Ensures findings are meaningful without wasting resources.

Benefits

  • Guides planning for both qualitative and quantitative studies.

Procedures

  • Qualitative testing (e.g., identifying usability problems): 2–5 users sufficient (per Nielsen, 2001).
  • Quantitative testing (e.g., measuring task time, error rates): requires larger samples.
    • Nielsen (2001) recommends 20 users per design for high-confidence quantitative results.

Advantages & Disadvantages

  • Not explicitly contrasted in notes.

Impact / Effect

  • Too few users in quantitative studies → unreliable metrics.
  • Appropriate sample size increases validity of conclusions and business decisions.

Examples

  • Finding major usability issues in a prototype: test with 5 users.
  • Comparing task completion time between two app versions: test with 20 users per version.