Lead UX Researcher

ROLE

Nov - Dec 2022

TIMELINE

Usability Tests, Task Analysis, TAP

SKILLS

Microsoft Office Suite, Figma

TOOLS

PROBLEM

Since Pokémon GO’s release in 2016, the amount of active players has decreased by over half 📱

The app was monumental and hooked audiences of all ages, but why isn’t it as popular now?

OVERVIEW

Background

As part of a final capstone project for my Usability Testing class at my university, I led a team of three other designers to conduct a usability test on the Pokémon GO interface. We all pitched an interface we wanted to test, and enough people were interested in my idea so I became a team lead. As a huge Pokémon and Pokémon GO fan myself, I wanted to find out why the number of active and new users has been declining steadily over the past few years. Could there be any UX related issues?

Process

  1. Discover the main pain-points of the app

  2. Outline a usability test plan

  3. Conduct usability tests with five different participants

  4. Create a formal report of our findings

Why Pokémon GO?

GOALS & OBJECTIVES

Is the app’s UX influencing their decline? 📌

Goals

Before starting the research phase, I outlined my main goals I had for this project:

  1. To identify and discover the problems of Pokémon GO as perceived by users/participants

  2. Observe the user’s behavior and interaction with Pokémon GO

  3. To discover what user’s experienced as opportunities for improving the design

Uncover Problems

in the design

Discover Opportunities

to improve the design

Learn About Users

behavior and likes

HEURISTIC EVALUATION

Lack of error recovery & recognition 🗝

After I outlined my goals, it was time to start researching. I opened up the Pokémon GO app and began searching for any potential issues. After playing with the app a bit, I decided to conduct a heuristic evaluation based on Jakob Nielsen’s 10 usability heuristics. I wanted to discover the major pain points of the app and start identifying any interface features or processes that could have led to the app’s slow downfall.

Discoveries

Pokémon GO breaks these heuristics often:

  • Heuristic #6: Recognition Over Recall

  • Heuristic #9: Helping Users Diagnose and Recover From Errors

TEST PREPARATION

Creating tasks and gathering participants 🗒

Task List

After conducting some basic research and a heuristic analysis, I began crafting a task list for participants to follow in a live usability test. We had fifteen tasks in total, eleven in the first part of the test (where users complete tasks on a brand new account) and four in the second part of the test (on an advanced/higher level account).

Participants

I created a pre-test questionnaire where I asked things like age range, Pokémon franchise familiarity, Pokémon GO familiarity, tech familiarity, and more to understand critical details about our participants. This helped us make sure we had a good pool of participants with varying experiences. We recruited both beginner and advanced players to see if there was a difference in the way they interacted with the app.

FINDINGS

Lack of error feedback & unintuitive navigation 🔎

Once we had the task-list completed and pre-test questionnaires returned, it was time to conduct live usability testing.

Lack of GPS Error Feedback

Many of our participants had difficulty catching pokémon at the beginning of the test as their avatar was inconsistent with their current location. Both the avatars of Participant 01 (Bulbasaur) and Participant 05 (Cyndaquil) were running aimlessly around the screen instead of staying in place.

“It thinks I’m a passenger, but I’m definitely not. Not even moving” (01 Bulbasaur).

“It’s just running and there’s no pokémon spawning” (05 Cyndaquil).

Navigation Difficulty

Both of the beginner participants relied on the Poké Ball menu to complete almost all of the given tasks. The profile and the nearby buttons were visited much less and usually as the last option to complete tasks that were located there (such as favorite a postcard, check nearby raids, and check the time they caught their first pokémon).

“I wish they put more things in the Poké Ball button, as it’s the easiest way to interact with the interface” (02 Charmander).

“I think raid should be under the Poké Ball button, because I didn't really know about the nearby button so I would have just looked around on the screen to see them” (03 Squirtle).

Unintuitive Shortcuts

It is possible to heal and transfer multiple pokémon. This process bypasses the otherwise arduous task of hopping back and forth between menus, which advanced and immediate players alone were able to take advantage of.

Bulk Gift Sending

Although gifts were not included in our tasks, Participant 04 Chikorita verbalized his frustrations with the friend menu and explained how inefficient sending and opening gifts can be for players with many friends. Unlike the transfer and healing shortcuts, there is no way to send or open multiple gifts at a time.

“Opening and sending gifts is hard. I have like a hundred friends, so opening and sending gifts to a lot of people will take me over ten minutes” (Chikorita 04).

“Definitely should be more streamlined like how you can transfer twenty pokémon at once, you should be able to send twenty gifts at once” (Chikorita 04).

USABILITY SYSTEM SCALE (SUS)

A disparity between beginner and advanced players ⛔

Definition

The SUS (System Usability Scale) is a ten-item questionnaire based on the Likert Scale. Participants rank each statement from Strongly Disagree to Strongly Agree. Although it’s not always the best tool for grading interfaces, I wanted to see what our participants thought of the experience.

Scoring

Our participants scored Pokémon GO very differently. The expert players rated incredibly high; they believed the interface was intuitive, easy to learn, and didn’t cause them any trouble. However, the beginners believed the app to be a bit more difficult and realized they would need to spend much more time on it to understand it fully.

Analysis

Seeing the intense disparity between expert and beginner players made it evident the app is seen in very different ways. Our expert players rated it so highly because they had been playing them game almost daily for years, therefore they were very acquainted with its interface. Our beginner players thought it was overwhelming and weren’t eager to continue playing.

Expert player mean score: 92.5

Beginner player mean score: 55

RECOMMENDATIONS

  1. Give better error feedback

    If the GPS signal is bad, tell the users

  2. Include important features in multiple sections

    More features should be added to the “Poké Ball” menu

  3. Allow users to heal pokémon on their individual page

    Reduces the amount of taps

  4. Add a zoom out function

    Allows players to locate more raids and gyms

A few possible changes… 🛠

Recommendation #3

REFLECTING

Usability testing yields interesting results, sometimes contrary to your own beliefs 🌱

Learnings

As the team-lead, I learned how to operate and lead a team. I oversaw every decision my team made (who was moderator/facilitator for the sessions, who we chose for participants, how we conducted the heuristic evaluation, how we created the task list, etc.) and wrote the majority of the formal usability test report. This was my first time leading a team through such a rigorous project, but watching our communication and everyone’s skills grow made it immensely rewarding. Ultimately, we made a neat research project! I also realized how important usability testing is; listening and empathizing with users/participants helps to reduce designer bias.

Things I would have done differently

If we had more time, I would have loved to conduct more research prior to our live-testing sessions and create a more cohesive task list as well as interview a few more participants of varying Pokémon GO skill level.