Unit 5: Testing, Validation and Customer Experience - Subjective Questions

User testing is the process of observing representative users while they interact with a product, prototype, or service to complete predefined tasks.

Importance of user testing:

• It verifies whether the proposed design addresses actual user needs.
• It identifies usability problems before full-scale development or launch.
• It reveals differences between the designer's assumptions and actual user behavior.
• It provides evidence for improving and prioritizing design decisions.
• It reduces development risk, rework, and the cost of correcting problems later.

User testing supports the iterative nature of design thinking by enabling teams to test, learn, refine, and retest their solutions.

Design validation determines whether a proposed solution solves the correct problem, satisfies user needs, and achieves the intended business or project outcomes.

Difference between design validation and user testing:

• Scope: User testing mainly examines how users interact with a design, whereas validation evaluates the overall suitability and value of the solution.
• Primary question: User testing asks, "Can users use it?" Validation asks, "Is this the right solution for users?"
• Measures: User testing may measure task completion, errors, and ease of use. Validation may also assess desirability, usefulness, feasibility, and business viability.
• Methods: User testing uses observation and task-based sessions, while validation can include interviews, field trials, analytics, surveys, and experiments.

User testing is therefore an important method within the broader process of design validation.

Neutral user testing aims to collect reliable evidence without influencing participants.

Major principles include:

• Avoid leading questions: Ask, "What do you expect this button to do?" instead of, "Is this button easy to use?"
• Use neutral language: Do not imply that one answer or behavior is preferred.
• Do not teach during the task: Allow participants to interpret and use the design independently.
• Observe before intervening: Give users time to think and act, even when they hesitate.
• Separate the person from the design: Make it clear that the product is being tested, not the participant.
• Record behavior as well as opinions: Note actions, errors, pauses, comments, and task outcomes.
• Apply consistent procedures: Give comparable participants the same instructions and scenarios.

These principles reduce moderator bias and improve the credibility of findings.

A neutral user-testing session can be conducted through the following stages:

1. Welcome and consent: Explain the session, obtain permission for recording, and protect the participant's privacy.
2. Set expectations: State that the design is being tested and that there are no right or wrong answers.
3. Collect background information: Ask brief, non-leading questions about relevant experience.
4. Present task scenarios: Provide realistic goals without revealing the exact steps required.
5. Encourage thinking aloud: Ask the participant to describe expectations, decisions, and confusion.
6. Observe neutrally: Record behavior, errors, hesitation, completion, and comments without guiding the participant.
7. Use neutral probes: Ask questions such as, "What were you expecting at this point?"
8. Conduct a debrief: Gather overall impressions and ask follow-up questions after the tasks.
9. Thank the participant: Explain how the feedback will be used and securely store the collected data.

A task scenario is a short, realistic description of a situation that gives a participant a goal to accomplish using the product without prescribing the exact procedure.

Characteristics of an effective task scenario:

• It is based on a realistic user need or context.
• It gives the participant a clear and meaningful objective.
• It does not reveal interface labels, menu names, or the sequence of actions.
• It uses language familiar to the intended user.
• It has an identifiable completion point.
• It is relevant to the research questions and product risks.
• It can be completed with the available prototype.
• It avoids unnecessary details that might confuse or lead the participant.

For example, instead of saying, "Click Filters and select tomorrow's date," a neutral scenario could ask the user to find a train departing tomorrow morning.

A task-based testing plan for an e-commerce checkout could include:

Objective: Evaluate whether users can confidently purchase a product and understand delivery, payment, and confirmation information.

Participants: Recruit representative customers with varied levels of online-shopping experience and include users of different ages and abilities.

Sample scenario: "You need to purchase a backpack for an upcoming trip. Choose a suitable product, have it delivered to your home, and complete the purchase using the provided test payment details."

Tasks:

• Find and compare suitable products.
• Add one product to the cart.
• change the quantity or remove an unwanted item.
• Enter delivery and payment information.
• Review the order and complete checkout.
• Locate the order confirmation and expected delivery date.

Measures:

• Task completion and time on task.
• Navigation errors and abandoned steps.
• Requests for assistance.
• User comments, confidence, and satisfaction.

Procedure: Use a neutral moderator, encourage thinking aloud, record observations with consent, debrief participants, group recurring issues, and prioritize improvements by severity and frequency.

The Show, Don't Tell approach means allowing users to experience and interact with a prototype instead of giving a detailed explanation of how it is intended to work.

How it is applied:

• Present the prototype with only the minimum required context.
• Give the participant a realistic goal.
• Ask the participant to demonstrate what they would do.
• Observe actions, expectations, hesitation, and errors.
• Ask neutral follow-up questions after observing behavior.

Why it is useful:

• It tests whether the design communicates its purpose independently.
• It prevents the designer's explanation from hiding usability problems.
• It reveals the difference between stated preferences and actual behavior.
• It exposes unclear labels, controls, workflows, and feedback.
• It creates more realistic evidence about learnability and navigation.

If a product works only after a designer explains it, the interface itself may require improvement.

Behavioral observation records what participants actually do, while self-reported feedback records what they say, feel, remember, or prefer.

Behavioral observation:

• Includes clicks, navigation paths, errors, pauses, and task completion.
• Provides direct evidence of interaction with the design.
• Can reveal problems that users do not notice or mention.
• Requires careful interpretation because behavior does not always explain motivation.

Self-reported feedback:

• Includes interview responses, ratings, comments, and stated preferences.
• Helps researchers understand expectations, emotions, and reasons.
• May be affected by memory, politeness, or the desire to please the moderator.
• Does not always predict actual behavior.

The methods should be combined. For example, a participant may describe a process as easy while repeatedly taking wrong turns. The observed difficulty and the participant's explanation together provide a stronger basis for design decisions.

A Feedback Capture Matrix is a structured tool used to organize evidence collected during prototype reviews and user-testing sessions.

It commonly contains four sections:

• Likes or positive observations: Features, interactions, or ideas that users valued.
• Criticisms or problems: Confusion, errors, unmet expectations, and negative reactions.
• Questions: Issues that require clarification or further research.
• Ideas: Suggestions, opportunities, and possible design improvements.

Purpose of the matrix:

• It prevents feedback from remaining as scattered notes.
• It separates observed problems from proposed solutions.
• It makes patterns across participants easier to identify.
• It supports team discussion and synthesis.
• It helps convert research evidence into actions for the next design iteration.

Entries should be concise, specific, and linked to the relevant task or evidence whenever possible.

A design team can analyze the matrix through a systematic process:

1. Consolidate evidence: Combine notes from all observers and remove exact duplicates.
2. Cluster related findings: Group comments and behaviors into themes such as navigation, trust, terminology, or accessibility.
3. Separate evidence from interpretation: Preserve what users did or said before drawing conclusions.
4. Measure frequency: Count how many participants experienced each issue.
5. Assess severity: Determine whether an issue blocks a task, causes major delay, creates minor friction, or is merely cosmetic.
6. Assess impact: Consider effects on user goals, business outcomes, trust, accessibility, and safety.
7. Evaluate effort: Estimate the complexity and cost of possible improvements.
8. Prioritize actions: Address frequent, severe, high-impact issues first, especially task blockers and accessibility barriers.
9. Assign ownership: Turn priorities into clear design or development tasks.
10. Retest: Validate revised solutions with users instead of assuming that the changes solved the problem.

This process converts raw feedback into evidence-based design decisions.

Customer experience (CX) is the customer's overall perception of an organization formed across all interactions before, during, and after using its products or services.

Usability is the degree to which specified users can use a product to achieve specified goals with effectiveness, efficiency, and satisfaction in a particular context.

Relationship:

• Usability focuses mainly on the quality of interaction with a particular product or interface.
• Customer experience includes usability but also covers branding, expectations, support, delivery, pricing, trust, emotions, and long-term relationships.
• Poor usability can damage the broader customer experience by causing frustration, errors, or abandonment.
• Good usability does not guarantee excellent CX if other touchpoints, such as customer support or delivery, fail.

Thus, usability is a critical component of customer experience, but CX has a wider scope.

A customer journey map is a visual representation of the stages, actions, interactions, thoughts, and emotions experienced by a customer while pursuing a goal with an organization.

Essential components include:

• Persona or user group: The customer whose journey is being represented.
• Goal: The outcome the customer wants to achieve.
• Stages: Broad phases such as awareness, consideration, purchase, use, support, and renewal.
• Actions: What the customer does at each stage.
• Touchpoints and channels: Interactions through websites, applications, stores, email, telephone, or staff.
• Thoughts and questions: What the customer expects or needs to understand.
• Emotions: Feelings such as confidence, uncertainty, frustration, or satisfaction.
• Pain points: Obstacles and negative experiences.
• Opportunities: Areas where the experience can be improved.
• Ownership or metrics: Teams responsible for touchpoints and indicators used to evaluate them.

A journey map gives teams a shared, end-to-end view of the experience.

The process of creating and validating a journey map includes:

1. Define the scope: Select a customer goal, journey boundaries, and relevant business objective.
2. Identify the customer segment: Create or select a research-based persona rather than assuming that all customers have the same journey.
3. Collect evidence: Use interviews, observations, support records, surveys, analytics, and contextual research.
4. List journey stages: Organize the experience chronologically, including activities before and after direct product use.
5. Map actions and touchpoints: Record what customers do and the channels they use at each stage.
6. Add thoughts and emotions: Represent expectations, uncertainty, satisfaction, and frustration using research evidence.
7. Identify pain points: Locate delays, repeated effort, missing information, inconsistent channels, and accessibility barriers.
8. Find opportunities: Propose improvements that address root causes and important customer needs.
9. Assign metrics and ownership: Define responsible teams and measures such as completion, complaints, retention, or satisfaction.
10. Validate and update: Review the map with customers and frontline staff, compare it with behavioral data, and revise it as the service changes.

A journey map should be treated as a research-backed, evolving model rather than a one-time illustration.

A user touchpoint is any interaction between a user and a product, service, employee, channel, or organization. A pain point is a specific obstacle, frustration, risk, or unmet need encountered during the journey.

Online banking touchpoints may include:

• A search-engine result or advertisement.
• The bank's website and mobile application.
• Account-registration and identity-verification screens.
• Email, SMS, or push notifications.
• An ATM, branch employee, chatbot, or support call.

Possible pain points include:

• Repeatedly entering the same personal information.
• Unclear identity-verification requirements.
• An inaccessible one-time-password process.
• Hidden transaction fees.
• Poor error messages during a transfer.
• Inconsistent information across the app and support center.
• Difficulty locating a transaction or contacting support.

Mapping touchpoints shows where interactions occur, while identifying pain points reveals where those interactions need improvement.

Pain points should be identified using multiple forms of evidence:

• Observe users completing realistic tasks.
• Conduct interviews and contextual inquiry.
• Analyze complaints, support requests, reviews, and search queries.
• Study analytics such as abandonment, repeated actions, and error rates.
• Include customers with disabilities and users in different contexts.

Analysis process:

• Group related observations into themes.
• Identify the journey stage and touchpoint involved.
• Investigate the root cause instead of treating only the visible symptom.
• Assess the frequency, severity, emotional impact, and number of affected users.
• Examine organizational dependencies and channel inconsistencies.

Conversion into opportunities:

• Reframe the issue as a user-centered question, such as, "How might we help customers recover from a failed payment without repeating the entire process?"
• Generate several possible responses.
• Prioritize ideas according to user value, impact, feasibility, and accessibility.
• Prototype the strongest ideas and validate them with users.

This approach ensures that redesign efforts address meaningful problems rather than isolated preferences.

Effective software navigation helps users understand their location, available choices, and route toward a goal.

Key principles include:

• Consistency: Menus, labels, controls, and interaction patterns should behave predictably.
• Clear information architecture: Content and functions should be grouped according to user expectations.
• Descriptive labels: Navigation terms should be familiar, specific, and unambiguous.
• Visible system status: The interface should indicate the current page, selected tab, process step, or loading state.
• Shallow, logical paths: Common tasks should not require unnecessary steps or repeated navigation.
• User control: Users should be able to go back, cancel, undo, and recover from errors.
• Recognition over recall: Important choices and recent context should remain visible.
• Accessibility: Navigation must work with keyboards, screen readers, zoom, and different input methods.

These principles reduce cognitive effort, navigation errors, task time, and user frustration.

Software usability can be evaluated through objective and subjective measures.

Effectiveness measures:

• Task-completion rate.
• Accuracy of completed tasks.
• Number and severity of errors.
• Percentage of users requiring assistance.

Efficiency measures:

• Time on task.
• Number of steps, clicks, or screens used.
• Effort required to recover from errors.
• Rate of successful completion relative to time, which may be expressed as:

Satisfaction measures:

• Post-task ease ratings.
• Standardized usability questionnaires.
• Confidence, trust, and perceived workload.
• Qualitative comments from interviews.

Additional measures:

• Learnability across repeated attempts.
• Retention after a period of non-use.
• Accessibility conformance and performance with assistive technologies.

A balanced evaluation combines these measures because speed alone does not show whether a product is accurate, understandable, satisfying, and inclusive.

Digital accessibility is the practice of designing and developing digital products so that people with visual, auditory, motor, cognitive, speech, and other disabilities can perceive, understand, navigate, interact with, and contribute to them.

Accessibility should be considered throughout development because:

• It provides equitable access to information and services.
• It improves usability for people using assistive technologies.
• It benefits users facing temporary or situational limitations, such as an injured hand or a noisy environment.
• Early consideration is more effective and less costly than retrofitting accessibility after launch.
• It reduces legal, ethical, reputational, and commercial risks.
• It often improves overall design through clearer content, better navigation, and stronger error handling.

Accessibility is not a final compliance check. It should influence research, requirements, content, design systems, implementation, testing, and maintenance.

The Web Content Accessibility Guidelines organize accessibility around four principles, commonly remembered as POUR:

• Perceivable: Information must be available in forms users can perceive. Examples include text alternatives for meaningful images, captions for videos, sufficient color contrast, and content that can be resized.
• Operable: Users must be able to operate controls and navigation. Examples include keyboard access, visible focus indicators, adequate time limits, and avoiding content that may trigger seizures.
• Understandable: Content and operation should be clear and predictable. Examples include readable language, consistent navigation, descriptive instructions, and useful error messages.
• Robust: Content must work reliably with current and future technologies, including assistive technologies. This requires semantic markup, valid relationships between labels and controls, and correctly communicated component states.

A product must address all four principles because failure in any one area can prevent a user from completing a task.

A comprehensive strategy should combine iterative research, behavioral testing, accessibility evaluation, and journey-level validation.

1. Define goals and risks

• Identify critical user journeys, business outcomes, high-risk assumptions, and accessibility requirements.
• Establish success measures for completion, efficiency, errors, satisfaction, and inclusion.

2. Recruit representative participants

• Include primary and secondary user groups.
• Recruit people with varied experience, devices, abilities, and assistive-technology use.

3. Test prototypes iteratively

• Begin with low-fidelity concepts and progress to interactive prototypes and production builds.
• Use realistic task scenarios and a neutral moderator.
• Apply the Show, Don't Tell approach and capture both behavior and comments.

4. Evaluate accessibility

• Perform automated checks, but do not rely on them alone.
• Test screen-reader output, keyboard or switch access, focus order, zoom, text scaling, contrast, touch-target size, captions, and error messages.
• Conduct sessions with users with disabilities.

5. Validate the broader experience

• Map the end-to-end customer journey across the app, notifications, support, payment, and offline touchpoints.
• Identify pain points, emotional lows, handoff failures, and inconsistent information.

6. Synthesize and prioritize

• Use a Feedback Capture Matrix to organize likes, criticisms, questions, and ideas.
• Rank issues by severity, frequency, customer impact, accessibility impact, and effort.

7. Improve and retest

• Assign actions, revise the design, and test critical changes again.
• After release, monitor analytics, crashes, abandonment, support contacts, reviews, and accessibility feedback.

Validation is complete only when evidence shows that target users can achieve important goals effectively, efficiently, satisfactorily, and accessibly across the full experience.