User testing for one of India’s largest UPI-based digital payment platforms

My Role

Outcome

Context

Existing situation

• The clients (P) and our agency’s design team had been working on some “north-star” designs for their platform
• Once they narrowed their design explorations down, they wanted to validate their approach

Problem statement

Research objectives

• User testing

  Qualitative & quantitative research to understand UPI-oriented usability challenges

  • Preferences between light and dark mode
  • Sentiment on the Minimal version
• General preferences

  Researching user preferences & sentiments for competitor apps & UPI-based services & journeys

  • User preferences on app size
  • General sentiment on “Lifestyle” section
• Brand perception of P

  Researching user awareness, consideration, usage, adoption & preferences for the P app

Key stakeholders

CXOs of P & Product Managers of different aspects of P

Team

Conducting UTs

• Design lead
• Senior UX designer (me)
• 3 UX designers
• 1 visual designer

Synthesis & presentation

• Senior UX designer (me)
• 2 UX designers
• Design director

Time

2 weeks

• Conducting UTs: 9 days
• Synthesis & presentation: 5 days

Approach

Participant recruitment

• Personas: Young (18–25), mid-aged (25–50), elderly (50+)
• Locations: Primarily tier-2 cities
• Sample size: 8~9 users per city
• Occupation: As diverse as possible
• Gender ratio: Almost 50:50

Questionnaire & task set

• Initial questionnaire: mix of qualitative & quantitative – a combination of task-based questions, and exploratory questions
• Task prioritisation: 4 core features of P were used for testing by the candidates as they were the critical user journeys

Prototypes

3 detailed prototypes were created:

1. main variant – light mode
2. main variant – dark mode
3. minimal variant

Tools & testing environment

• Screen recording of prototype – capturing interactions
• Video recording of participant – body language & easily missable cues
• Audio recording of participant & facilitator – hi-def audio for clarity
• Note taking (in detail) during interview – capturing instant answers & observations

Test protocol & testing

Duration & structure of each session

• 1 hour per candidate
• 70% of the session was hands-on with prototype – task-based questions
• 30% of the session was Q&A – exploratory questions

Facilitation guidelines (roles & responsibilities)

• Test candidate – participant
• Facilitator – Senior UX designer (me) in Team 1 & design lead in Team 2
• Scribe – 1 UX designer per session per team

Pilot test

Bangalore – 9 users

• Initial findings came through for the client – we presented as a summary
• We made adjustments to the script and refined it:
  • Removed redundant questions
  • Added nudges to facilitator for follow-up questions
  • Added optional extra questions for participant if time permitted

What went well

• The junior members of the team acclimatised well to the process & the equipment-setup
• We were able to make adjustments for the real deal in the cities of interest

What could have been done better

• We hit a minor language barrier in one of the UTs

  We immediately made an adjustment – one of our team members in the room had to know the candidate’s language from the next sessions

Main tests

• Total participants: 43
• Gender ratio (M:F) = 53:47
• Age ratio (young:mid-age:elderly) = 14:19:10

Testing teams:

• Team 1: 3 designers (Mangalore, Nashik)
• Team 2: 3 designers (Agra, Jaipur)

Observations & synthesis

Core stories

After some thematic analysis, 4 stories surfaced – these became the guideposts for strategic decision-making by the key stakeholders.

Note: the % numbers have been changed for IP protection

Task analysis

Key journeys evaluated:

1. Pay to QR
2. Pay to contact
3. Receive money via QR
4. (a) Pay electricity bill, or (b) Recharge mobile

Key metrics tracked:

1. Success ratio
2. Median of time of task
3. Mode of time on task
4. Average user rating

Dimensions:

1. Age group
2. City (not particularly relevant since all but Bangalore are Tier 2 cities)

Feature-based sentiments

For specific features we wanted to test, we captured qualitative data that led to interesting insights, captured in the form of:

1. What works
2. Opportunities to improve
   1. Recommendations from us – strategic & tactical

Light & dark theme sentiments

After synthesis of data pertaining to the app theme, detailed qualitative and quantitative insights were derived.

Eg. qualitative:

“Dark mode users pointed out that the dark mode looks flat as the colours are not prominent like in light mode, but would choose the dark mode for personal taste.”

Eg. quantitative:

“x% of users preferred light mode for consuming information citing good readability, but would like to switch between themes based on the time of day”

Note: the % numbers have been changed for IP protection

Similarly we uncovered insights for all other areas covered under our research objectives:

• Minimal version sentiments
• P brand awareness & perceptions
  • Domain & competitor awareness & perceptions
• Screen-based analysis & recommendations
  • What works
  • Opportunities to improve
    • Recommendations from us – strategic & tactical
• Feature-based insights
  • UPI Lite
  • Tap to pay & other new features
  • Raising disputes

Strategic recommendations

Immediate

• Creating a robust design system for the new designs, based on insights generated from the UTs
• Use the recommendations to tweak the design to further strengthen the experience
• Share insights with business, sales, marketing, product teams for shared understanding
• Work on sprint plans to implement the design language for all the sub screens and journeys

Long-term

• Have a continuous testing plan for the entire sprint cycle
• Identify more sub-cohorts and conduct UTs for the subsequent journeys that will be created based on the new approach
• Reach out to the same set of participants once the design is developed, for beta testing if possible

Retrospective

What went well?

Adjusted many aspects of our approach on-the-go:

• Created vernacular versions of the screens for vernacular users
• Got on the road and recruited participants
• Improved the script and note-taking process with every interview

Key learnings for future projects

• Have a native language speaker on standby
• Test and know your recording devices better

Additional links

📁 Figma file