Author: cassidst

Categories
design research

Making FAFSA work better for students and families

I led UX research and design work for the U.S. Department of Education’s Free Application for Federal Student Aid, better known as the FAFSA. My work included leading nationwide research with students and families, supporting the phased launch of the 2025–26 and 2026-27 form, and helping redesign one of the most difficult parts of the experience: inviting a parent or other contributor to complete their portion of the form.

Problem

Each year, millions of students and families complete the FAFSA to apply for grants, loans, and work-study funding for college.

The FAFSA is not a typical online form. Its requirements are shaped by legislation, tax policy, financial-aid regulations, and complicated family circumstances. Dependent students must also coordinate with a parent or other contributor, who may need to create an account, provide consent for federal tax information, complete a separate section, and sign the form.

Following the difficult launch of the redesigned 2024–25 FAFSA, FSA needed to rebuild confidence in the experience and identify issues before the next form was released to the entire country.

For the 2025–26 FAFSA, the Department used a phased beta launch. Starting in October 2024, the form was released to limited groups of students and schools before becoming generally available on November 21, 2024.

Our challenge was to build a research program that could identify problems in real time, help teams distinguish isolated incidents from systemic issues, and turn findings into changes before millions of families began using the form.

Approach

I helped develop a continuous research program that combined moderated usability testing, field research, survey monitoring, contact-center feedback, and product data.

Instead of relying only on controlled usability studies, we wanted to observe students and parents completing their real FAFSA applications in the environments where they normally seek help.

Our research included:

  • More than 20 moderated usability sessions with 31 students and parents
  • Direct observation of hundreds of families at FAFSA completion events
  • Research a schools and community-based organizations around the country
  • Daily analysis of post-transaction surveys and beta issue reports
  • Review of contact-center feedback and product data
  • Interviews and feedback from financial-aid professionals
  • Follow-up sessions after participants’ applications were processed

Participants included dependent and independent students, parents, first-generation students, English learners, veterans, adult students, foster youth, students experiencing homelessness, and families with lower incomes.

The participant mix helped us evaluate whether the FAFSA worked across a wide range of circumstances—not simply for students with straightforward family and financial situations.

FAFSA beta events gave us an opportunity to observe hundreds of students and parents completing real applications.

FAFSA “Beta Nights” across the U.S. in 2024. I led UX research efforts and a team of UX researchers to gain insights on the ground with students and families across the U.S. in preparation of the 2025-26 FAFSA launch.

Research at FAFSA events

FSA partnered with schools and community-based organizations to hold FAFSA completion events around the country.

I led a team of UX researchers (FSA folks, U.S. Digital Service, and Accenture contractors) who all attended events in cities including Atlanta, Birmingham, Dallas, Fort Lauderdale, Minneapolis, New York City, Northern Virginia, and Santa Maria, California.

At each event, researchers:

  • Observed students and parents completing the form
  • Conducted short contextual interviews
  • Documented technical and usability issues
  • Gathered feedback from counselors and financial-aid professionals
  • Submitted an end-of-day summary of emerging findings

We instructed UX researchers to “observe first, ask questions second, and help third.” These were real students completing a consequential government form, not participants working through fictional scenarios.


Students and parents complete the FAFSA form at an event at Santa Maria High School in California. 

The events revealed behaviors that were difficult to reproduce in traditional usability testing. Students frequently drove the process for their families. Some completed portions of the form through a parent’s account. Others called parents to obtain authentication codes or personal information.

We also saw how important counselors and community organizations were to successful FAFSA completion. Their support sometimes concealed confusing questions or interaction problems that would have blocked a family completing the form alone.

Observing families complete real applications helped us uncover issues that did not always emerge in a controlled research session.

What we learned

Feedback on the 2025–26 form was largely positive. Students and parents frequently said that it was faster and easier than they expected. From November 1 through November 20, satisfaction for initial submissions remained between 91 and 94.

Research also identified several areas where the experience continued to create confusion:

  • Students had difficulty inviting and matching parent contributors
  • Legal-residency and financial questions were difficult to interpret
  • Students did not always understand whether a college had been added successfully
  • Blank answers on the review page made families worry that their form was incomplete
  • Participants frequently overlooked help text and tooltips
  • Parents struggled to locate invitations on their StudentAid.gov dashboard
  • Students did not always understand their form status or submission summary
  • Participants could not easily find actions such as canceling a correction

The research produced more than 30 recommendations for improvements to the form and its surrounding experience.

Several findings led to immediate content updates and bug fixes. For example, research uncovered a critical issue in the college-search interaction that sometimes displayed or added the wrong institution. The development team addressed the issue in a code update.

Top Findings

  1. Continued issues with inviting a parent contributor, especially among non-SSN parents due to strict data matching and poor conceptual model
  2. There continues to be slight confusion on a few questions such as date of legal residence question and number of family members in college 
  3. Adding colleges and selecting the correct one sometimes didn’t work as expected and discovered a critical bug affecting the college search typeahead 

We consolidated findings from usability testing, FAFSA events, surveys, and contact-center feedback into a shared set of recommendations.

Contributor invite

Inviting a parent or other contributor was one of the most significant problems we identified.

In the existing experience, a student had to enter personal information that exactly matched the contributor’s StudentAid.gov account. This could include the contributor’s name, date of birth, address, and Social Security number.

Students frequently did not know this information. Minor differences—such as “Road” instead of “Rd.”—could prevent the invitation from matching the correct account. The problem was especially difficult for parents without Social Security numbers.

Our analysis showed the scale of the issue:

  • Approximately 5.7% of applications—about 850,000—stopped at the contributor invitation
  • Funnel data showed a 16% drop between the preceding form step and successful creation of a parent contributor
  • Up to 40% of students may have needed to pause the form to obtain a parent’s Social Security number
  • More than 10,000 contributor-invitation contact-center cases were opened during one two-week period

The invitation process placed the burden on students to know and accurately enter information belonging to someone else. It also did not match the way familiar invitation systems work on other digital products.

Contributor invite research

During the FAFSA beta, we showed early concepts to eight students and parents to gather directional feedback.

We then developed mid-fidelity prototypes and conducted 45-minute usability sessions with an additional group of dependent students and parents.

We tested several approaches:

  • The existing personal-information matching process
  • An invitation sent by email with a one-time code
  • A link that students could copy and send themselves
  • Email and text-message options
  • Codes requiring additional identity information
  • Different locations for the invitation within the form
Proposed new consumer-friendly email invite for the FAFSA form.

Personal information was a barrier

Every participant said that they—or their student—would not know all the information required by the existing invitation.

A parent’s Social Security number was the most significant barrier, but participants also raised questions about dates of birth, account email addresses, and last names.

Requiring this information could force students to leave the form before completing their own section.

Students wanted to finish their section first

All participants preferred to invite a contributor after completing and signing the student section.

The existing invitation appeared partway through the student experience. Participants saw the end of the student section as a more natural handoff between the student and parent.

Participants preferred an invitation code

Participants generally preferred a familiar invitation-code model over exact matching based on personal information.

Email felt familiar and secure. Participants also appreciated having a link or code that they could share manually if the email did not arrive.

Replacing personal-information matching with a one-time code removed the need for students to know a parent’s Social Security number.

Initial Recommendations

Based on the research, we recommended:

  • Replace personal-information matching with a one-time contributor code
  • Allow a student to initiate an invitation using an email address
  • Give students the option to copy and share the link or code manually
  • Move the invitation to the end of the student section
  • Carry the code through the authentication process when possible
  • Take contributors directly to a clear “Accept invitation” action
  • Make it clear whether the student or contributor is expected to act
  • Design states for canceled, expired, and previously accepted invitations
  • Conduct additional testing with English learners and people with disabilities

These recommendations combined usability findings with product analytics, survey feedback, contact-center data, legislative requirements, and technical considerations.

Early concepts explored different ways to invite a contributor without requiring the student to enter the contributor’s personal information.

New for 2026-27 form, an easier invite model for the FAFSA form.

Building Empathy

Research findings can easily become another presentation that product teams review and then forget.

To make the findings actionable, I helped organize a large “Building Empathy” session for the FAFSA team. Staff worked in cross-functional groups to review specific problems observed during the beta and develop recommendations for product leadership.

Topics included:

  • Contributor invitations
  • Confusing financial questions
  • College search
  • The FAFSA review page
  • Form corrections
  • Application status
  • The submission summary
  • Account creation and identity matching

The workshop gave staff from design, policy, product, engineering, operations, and communications a shared understanding of what students and parents had experienced.

Cross-functional groups used evidence from research to develop and prioritize recommendations for FAFSA leadership.

Results

Our research and design work helped FSA:

  • Observe hundreds of students and parents completing real FAFSA applications
  • Conduct more than 20 in-depth usability sessions during the 2025–26 beta
  • Establish a continuous feedback loop across research, surveys, contact-center feedback, and product data
  • Produce more than 30 recommendations for immediate and longer-term improvements
  • Identify content issues and product bugs before the national launch
  • Maintain initial-submission satisfaction scores between 91 and 94 during the later beta period
  • Establish field research as a central part of the FAFSA launch process

For the 2026–27 FAFSA, FSA implemented a new contributor invitation process using a one-time code. Students could initiate an invitation with an email address instead of having to know and accurately enter a parent’s Social Security number and other personal information.

The release also included improvements to the FAFSA review page, transactional emails, account verification, account recovery, and authenticated navigation.

The most important result was not a single interface change. We established a new way of evaluating FAFSA: observing the service in real environments, combining multiple sources of evidence, and working with product teams to address problems before they affected millions of families.

Lessons Learned

  • Field research revealed family dynamics and workarounds that were difficult to reproduce in a traditional usability session.
  • Partnering with schools and community-based organizations helped us reach students who are often left out of government research.
  • Combining qualitative research with survey, contact-center, and behavioral data made our recommendations more persuasive.
  • Daily synthesis during the beta allowed product teams to respond while research was still underway.
  • Opening research sessions and workshops to the larger product team helped create shared understanding and buy-in.
  • Research was most valuable when we translated findings into specific product and technical recommendations.
  • Even when legislation limits the solution space, observing users can reveal better ways to implement those requirements.

Categories
design research

Income-Driven Repayment

I worked as the UX design lead for the U.S. Department of Education’s overhaul of the entire income-driven repayment (IDR) application and introduction of a new, more affordable IDR plan.

Problem

The FUTURE Act was a new law that cleared the way for Federal Student Aid (FSA) to build new back-end integrations with the Internal Revenue Service (IRS) to more quickly connect to a student loan borrower’s tax information to calculate family size and adjusted gross income. This was groundbreaking for borrowers, as this new integration allowed seamless transfer of information and more quickly determined eligibility and payment amount right in the application. This was a forcing function for FSA to build a new application process, and launch a feature we call “autorecertification” meaning that an IDR borrower would only need to apply once, and we’d recertify your income each year and recalculate their payment annually without the need for any forms.

Approach

Given the legislative mandate of a new IRS integration, we needed to understand the entire student loan servicing environment (the companies FSA contracts to actually collect/process student loan payments) and what our limitations where. As with most government projects, we were working under tight deadlines.

I started with an initial discovery sprint working with others to interview student loan borrowers on the current experience with repaying loans under an IDR plan. We also conducted usability testing on the legacy application, and held several interviews with student loan servicers to unpack their back-office operations. Our main insights were:

  • Borrowers are often surprised to find there are multiple IDR plans; 50% have their servicer pick a plan
  • Paper application volumes are higher than expected, with up to 30% being processed as paper
  • Critical IDR concepts, like when to enroll and requirements to “recertify” confuse borrowers
  • IDR borrowers top reason for calling is to get a status update on their application as they wait until their servicer processes their app
  • Providing documentation of income after the application can be a cause of fallout and requires lots of back and forth with a loan servicer. Servicers spent a considerable amount of time manually calculating plans with and processing paper apps
  • IDR forgiveness is often misunderstood; many don’t understand remaining balances may be counted as income. Borrowers want more clarity on their progress toward IDR forgiveness

Initial Recommendations

These helped FSA save time and cost early on as the it was before the work was under contract with the development vendor and servicing teams.

  • Allow borrowers to save applications along the way to gather necessary income documentation (if needed)
  • Include the option for borrowers to upload income documentation in the application vs. having to mail/process it afterwords
  • Allow loan servicers to access manual uploaded income documentation within an electronic application; provide a way for borrowers to report their income/pay frequency
  • Allow borrowers to track their apps via their FSA account
  • Provide IDR borrowers a payment count and progress toward ID
    forgiveness. Provide more clarity around IDR forgiveness to reduce false hope around early forgiveness
  • Remove the cosign flow (aligns with new IDR regulations)
Provide borrowers an easy and understandable application Educate prospective IDR borrowers about important IDR concepts Build trust with borrowers that FSA is handling personal and financial information securely Empower borrowers to pick a plan that suits their needs Consider how we're communicating new program changes --what's our communication strategy? What happens post submission? Build an experience that helps borrowers understand new IDR rules (consent, taxes, recertification) Design an an experience that accommodates borrower's life situations (married, multiple loans, no income) that doesn't compromise UX Provide delighters throughout the application to reduce users' anxiety of the process (illustations, animations, toast notifications) Aim for the same time it takes to complete the current IDR app
Early draft of experience goals, our driving design tenants as we shifted into early prototyping.
A user flow of the different audiences for IDR
Mapping the initial experience.
Sticky notes that define the success criteria with the product team.
Defining success criteria with the product team.

Results

After rounds of usability testing, we felt confident in several new aspects of the application and flow.

Early concept showing a borrower an initial summary page.
Early concept showing a borrower an initial summary page.
early wireframes of the IDR app
Early concepts of the IDR application, showing the integration with the IRS.
Early concept showing a repayment plan selection page. We tested this version against a horizontal card option.
Early concept showing a repayment plan selection page. We tested this version against a horizontal card option.

High-Fidelity Mockups

A simple splash page serves as an orientation on what a borrower will do in the form, reducing the chances their expectations are misaligned as they begin.
A simple splash page serves as an orientation on what a borrower will do in the form, reducing the chances their expectations are misaligned as they begin.
A loading page indicates the application is securely connecting to gather a borrower's tax information. Borrowers really appreciated this feedback and transparency in the application.
A loading page indicates the application is securely connecting to gather a borrower’s tax information. Borrowers really appreciated this feedback and transparency in the application.
The select a repayment plan page on the IDR app.
The IDR application shows a borrower the exact amount they will pay under the plan and also does several “out year” calculations to estimate the total paid. IDR plans are difficult since each year you may pay something different deepening on what your income was. We added status pills that call out the cheapest monthly payment and the plan with the lowest total paid over time.
The final page of the IDR application
A borrower can be done in as little as 8 minutes (average time to complete).
A borrower's dashboard with an action required IDR app
A borrower with a “action required” status on their application.
An IDR app in the "in review" step
A borrower can easily track their application status at any time and get email updates on when it’s processed. This feature’s goal was to significantly reduce the times a borrower called a customer service rep for a status check.
processed IDR application
An application that was successfully processed.

Autorecertification

In addition to launching a new application, business process, and a new user experience, we also launched a new “autorecertification” process. This process allowed us to “recertify” a borrower’s income without them needing to do anything. We would simply email them and let them know if it was happening. If their income changed since their last tax return, they could simply do a recertification on their own and override the automatic one. We knew we needed to:

  • Provide proper notification to a borrower about payments that are changing and give them time to adjust and recertify on their own
  • Reduce the burden on borrowers that don’t need to take any action (one win was getting approval to place the monthly payment amount in an email so a borrower doesn’t actually have to log in to view anything)
  • Make the process feel natural and keep the borrower in control
Early concepts on the customer journey and back-stage actions relating to autorecertification.
Early concepts on the customer journey and back-stage actions relating to autorecertification.
Mapping the email and service design for borrowers in an IDR plan.
Mapping the email and service design for borrowers in an IDR plan.
Draft email for a borrower who successfully “autorecertifies”

Results

  • Developed and implemented an entirely new income-driven repayment plan and application which reduced the overall time to completion by 5 minutes.
  • Increased the overall satisfaction score from 82% to 89%.
  • Designed an “autorecertication” process where student loan borrowers don’t have to reapply each year, saving borrowers millions in lost hours filling out forms.
  • Significantly reduce paper submissions by marketing the new IDR application and ensuring loan servicer customer service reps drive callers toward the online application.
  • Helped increase IDR enrollment to more than 10 million student loan borrowers.

Lessons Learned

  • Early stakeholder interviews were critical to establish trust with the project team, as well as loan servicers back-office processing folks.
  • Allowing an initial discovery research sprint saved development costs because we could focus on drafting contract requirements with a high confidence in knowing what the system needed to support to deliver a user experience that matches user needs and goals.
  • Opening up usability testing sessions to the entire project team helped create early buy in for our research efforts.
Categories
design research

Student Loan Debt Relief

I worked as the UX design lead for the Student Loan Debt Relief project at the U.S. Department of Education’s office of Federal Student Aid.

Problem

The U.S. Department of Education wanted to implement a broad student loan forgiveness program for most student loan borrowers. Borrowers under the 120k income threshold would receive up to $10,000 in forgiveness and some borrowers who had received a Pell grant would get $20,000. We had less than 3 months to design, develop, test and launch an application for 30+ million student loan borrowers.

Approach

Our goals were to:

  • Make the form as usable and accessible as possible to maximize forgiveness opportunities for borrowers
  • Minimize burden on the public by having to document income
  • Move quick to implement an entire forgiveness program within 3 months
  • Ensure the system can handle the volume and built trust with the public

We knew we needed to develop a simple application and form but were nervous about the tremendous amount of traffic from people applying for forgiveness. It was decided the form would be available to unauthenticated users and our back-end systems would match personal identifiers in the application to a borrower’s loan records. Due to time constraints and reducing the burden on borrowers to provide income documentation, it was decided to build an “attestation” form, basically a form where you are agreeing you make under a certain income threshold. Then on the backend, a predictive model would flag certain borrowers who would need to provide additional income verification to reduce fraud.

Starting off with Legalize

As with government forms, our initial concept was a 2-page form but loaded with dense text that our team were told was necessary to include.

Realizing the form’s rather simple logic, we worked to refine the design and worked with our policy team to reduce text where we could.

Usability Testing Before Launch

We recruited dozens of student loan borrowers and conducted usability testing in English and Spanish. We learned that some borrowers felt the form was “so easy they thought they missed something.” We were able to tweak some language and helped build out a splash intercept page that we deployed to studentaid.gov.

student loan debt relief form spanish

Results

The form was launched in fall of 2022 during a “beta” period. There were 24 million submissions in the first week. But we were not done refining the user experience.

End to End Experience

In order to ensure borrowers were properly notified about progress we built additional elements into the student loan debt relief process.

First was ensuring that after each application submission, borrowers got a clear and concise message confirmation email. We tested the language of the next steps with borrowers and revised to improve comprehension and clarity.

Proper Status of Forgiveness Requests

The other element we made sure to build was proper status since we knew it would take a few weeks to process the forgiveness requests and actually have it applied to borrowers accounts. We quickly built an ability for customer service representatives to be able to check application submissions in Salesforce, and we created a plan to launch a “status tracker” for a borrower’s application on studentaid.gov.

Application details page within a borrower’s account.
The different statuses we developed.

Ability to Verify Income if Needed

One last piece of the puzzle in the entire service delivery and design of student loan debt relief was to create an easy process for borrowers who were chosen for income verification to be able to easy upload their documentation. The challenge was the initial application was not behind a log in, so we had to connect the application request to a borrower’s account so they could securely upload income documentation.

Lessons learned

  • While there wasn’t a lot of time for upfront discovery research we were able to leverage past usability testing on studentaid.gov forms and flows and leverage existing design patterns that had been properly vetted with users.
  • Cross team collaboration was critical in the projects success, despite the Supreme Court striking down the program
  • Ensuring the application was user tested before launch was initially difficult as folks were nervous about the application leaking to the public, but we ensured testers knew the sensitivity around the form
  • Forms don’t need to be complex or even muti-step to be successful. Sometimes people just appreciate how easy something is and this can increase trust and confidence in the system.
  • We got some much needed good praise about our efforts!
Categories
design

Policy Infographics

Helping to make sense of complex student loan policies.

Saving on a Valuable Education (SAVE) Repayment Plan

Student loan borrowers as part of the SAVE plan may be eligible for early forgiveness of their loans. In this graphic, I designed a quick way for student loan borrowers to understand how the policy may affect their early forgiveness amount.

Early forgiveness amounts under the SAVE plan
Early forgiveness amounts under the SAVE plan
SAVE Repayment Amounts based on Income and Family Size

In this infographic, I developed a way for student loan borrowers to quickly determine where they fall in terms of monthly payment. Under the income-driven repayment plan, you pay you can now buy back certain months in your payment history to make them qualifying payments for PSLF. Specifically, you can buy back months that don’t count as qualifying payments because you were in an ineligible deferment or forbearance status.

Explanation of the SAVE repayment plan and how much you would pay per month

Public Service Loan Forgiveness (PSLF)

Under the 10-year public service forgiveness program, a student loan borrower can buy back certain months in their payment history to make them qualifying payments for PSLF. Specifically, you can buy back months that don’t count as qualifying payments because you were in an ineligible deferment or forbearance status. In order to help illustrate that, I created this graphic for the information page on studentaid.gov.

image of the PSLF buyback concept

Categories
research

Federal Student Aid Top-Task Testing on StudentAid.gov

view studentaid.gov

Problem

Federal Student Aid, an office of the Department of Education, manages StudentAid.gov, the main website that supports Title IX higher education funding in the country. One of the most visited federal government sites, StudentAid.gov handles more than 20 million FAFSA (Free Application for Federal Student Aid) submissions per year, and supports other relevant tasks like a personalized dashboard with student’s aid summary, digital, self-paced counseling for student loans, and applications for income-driven repayment plans.

Approach

This was the first top-task usability test to be conducted on the production site. Various individual products and features on StudentAid.gov have been usability tested independently, either during design sprints or after release. This was an opportunity to take a holistic view of the site and incorporate more content-based pages.

Our goals were to:

  • Evaluate 10 common tasks and scenarios with users
  • Uncover any navigation pain points and unexpected issues
  • Test on both desktop and mobile devices
  • Evaluate the site’s content in context with top tasks on content pages

We tested with a diversity of students, parents, and student loan borrowers in repayment with different incomes, ethnicities, and geographic location across the U.S.

Results

Key Takeaways

Mobile navigation: The “Log In” and “Create Account” links are swapped with several logged-in tasks/links once a user logs in. Many participants didn’t notice this is where key authenticated pages were located since the menu’s hierarchy focuses more on broad topics and content pages.
  • Most users don’t associate Federal Student Aid with the FAFSA. The term FAFSA has much stronger household recognition than Federal Student Aid, despite Federal Student Aid’s efforts to consolidate most of its program sites into one website.
  • The site’s main mega-menu on desktops has room for improvement. A lot of users misunderstood categories such as “Manage Loans” which users thought was more personalized to them.
  • When users log in, there is a utility menu that appears. Many users didn’t notice it since it was not prominent in the main navigation area and on mobile, it appears below other sections within the navigation.
  • Long content pages meant users often hunted and pecked for information, resulting in longer than expected time on task metrics. Very few users used the site search, despite its prominence.
  • The mobile website’s navigation proved difficult for users. Many felt the menu was overwhelming and they couldn’t easy find the logged-in pages, such as Dashboard, My Aid, My Docs, and Settings.

Key Recommendations

On mobile, the site’s logged-in menu prioritizes Help Center and Contact Us over a user’s main pages like Dashboard and Aid Summary.
  • Continue mobile testing as part of a core user research approach
  • Redesign the in-page navigation for content pages to better support anchor/jump links within the page and to provide mobile users a better experience
  • Improve the mobile website’s navigation and consider how the logged-in links interact with the main site’s navigation. Consider a robust, secondary navigation on logged-in pages.
  • Conduct a card sort on the site’s main topics to evaluate the site’s information architecture and navigation schemes. Consider condensing the amount of links in the site’s main navigation.

Lessons learned

  • Remote testing with mobile devices proved easier than anticipated. Using Microsoft Teams, we were able to easily have participants share their phone screens
  • Top-task testing, outside of feature-based testing, is a helpful activity and should be part of any evaluate user research approach. Developing a cadence for this type of testing insures a regular interval for continuous improvement discovery
Categories
design research

JMU’s Health Policy Initiative

view health policy site
This website project was part of a grant with a team of five faculty members at James Madison University with a goal of educating the community on health policy. The goal of the initiative is to educate the community about JMU’s health-related civic engagement in the community and the world.

The Problem

As the lead project designer, I worked with various stakeholders to conduct information gathering sessions. While a relatively new initiative from James Madison University — the Health Policy Collaborative had little visibility and lacked a dedicated website.

Approach

I conducted research which included meeting with faculty, nurse practitioners, state policy experts, and local politicians. These meetings and interviews helped get a sense of what the newly formed group, the Health Policy Collaborative, would want to showcase on the website and how the general public would receive the information.

We learned users’s top tasks and goals as well as clear goals from our stakeholders.

Results

After identifying target audiences and developing user personas, I explored an information organization activity, attempting to piece together the various themes and information needs of the site.

Working inside the JMU content management system was limiting, but we were able to do certain things on the site, like pull in outside news sources. I led an effort to include an interview I conducted with a health policy expert on Medicaid expansion in Virginia.

This relatively short project meant the site’s needs were going to change in the future. My goal for this group was to create them a foundation website, that presented their information clearly and professionally.

This meant creating a information architecture framework that could grow, and not impact the discoverability factor for users.

Considering JMU’s site wasn’t responsive at the time, performance and usability on a phone or tablet still remains an issue.

Lessons learned

  • Logistics of scheduling interviews with stakeholders can prolong the time necessary to get information critical for a website.
  • Content management systems (CMSs) often have limitations and it’s important to understand the technology behind it in order to not waste time designing something you know will not work.
view health policy site
Categories
blog

Questionnaires: how to ask questions that give you true user insights

For the purposes of this example, I use my work on Marriott.com’s booking process. This survey questionnaire was designed for research on Marriott’s digital products.

With any opportunity you have to engage with users, customers, or the general public, it’s important to remember that you are asking for someone’s time. Asking purposeful, relevant questions that lead you to a better understanding of your users is important. Leave the questions that don’t help you learn more behind. A method I use is to write out your questions, then justify to yourself why you want to ask it.

Survey questionnaires help to refine user research goals and questions. The qualitative and quantitative feedback collected will help to assist in the redesign of your product (in my case, the Marriott website and mobile app.) For the purposes of this questionnaire, we are focusing on closed response questions, and looking to gather and collect categorical, behavioral, and attitudinal information from users.

Questions

Introduction

Thank you for agreeing to participate in our survey! Your feedback helps us create more useful and intuitive products. In this survey, we ask a variety of questions based on your hotel booking preferences, perception with rewards programs, and digital booking apps. Your participation is voluntary.

1.  How do you most commonly book a hotel room? (check all that apply)

  • Marriott.com
  • Marriott smartphone app
  • Another hotel company’s website or app
  • Travel site (Expedia, Priceline, Hotels.com, etc.)
  • Travel agent or travel company (online or by phone)
  • Other [type other answer]

Why ask this question? This questions is an easy introductory questions that helps us gather data about how users most commonly book hotels. Marriott would like to increase digital bookings by 10 percent, and it’s important to understand this user behavior as a benchmark metric.

2. When picking a hotel what top things are most important to you? (check all that apply)

  • Distance from location (city center, airport, etc.)
  • Price
  • Loyalty program (points earned or ability to use points)
  • Amenities offered (fitness center, pool, restaurants)
  • Hotel services (airport shuttle, bike rentals, etc.)
  • Other [type other answer]

Why ask this question? Understanding user attitudes toward hotel choices is key to understanding what matters to users when making a selection. This may help refine what is advertised about a specific brand of hotel, or reinforce the need for price filtering and more upfront information about the particular hotel. It may also help to identify new organization schemes for filtering or advertising hotels.

3. If you are a Marriott Bonvoy rewards member, how satisfied are you with the rewards program overall?

  • Very satisfied
  • Somewhat satisfied
  • Neutral/no opinion
  • Somewhat dissatisfied
  • Very dissatisfied
  • N/A – I’m not a rewards member

Why ask this? This simple question helps to understand who is a member of the rewards program and who isn’t. It also helps to set a benchmark for satisfaction levels and gives us better insights on how well the rewards program is performing for users. This is a question we could ask after some point in time to see if changes to the website or app not only increase rewards members, but also increase satisfaction with the program overall.

4. Please rate the various benefits associated with a hotel rewards program. On a scale of 1 to 5, with 1 being the least important and 5 being the most important, rate the following:


12345 
Ability to use rewards to book free travel




Complimentary services (free wi-fi, drinks, food, etc.)




Dedicated customer service




Exclusive rates




Free room upgrades




Mobile check-in or keyless entry to room




Third-party offers and discounts on rental cars




Credit card offering that gives you rewards points to use to book travel




Other: [type answer]

Why ask this? This question helps to understand user attitudes toward a hotel rewards program. Part of Marriott’s business goals are to increase new rewards members and that starts with understanding what people value. This helps to discover certain reward benefits that may be better to advertise to entice new users to join the program. Also, insights from this may help Marriott’s business strategy and customer experience strategy to refine the rewards program to better suit its customers.

5. If you use a smartphone, have you used a hotel rewards/hotel booking app in the past year?

  • Yes, I use a hotel app on my phone regularly
  • Yes, I have a hotel app, but don’t use it
  • No, I don’t have a hotel app on my phone
  • N/A – I don’t use a smartphone

Why ask this? Answers to this question help to understand user behaviors when it comes to mobile apps for hotels. Onboarding someone for an app experience requires that the app is making its value proposition clear to users. An app that is downloaded, but not used, can indicate users don’t perceive the app helpful for them to accomplish their goals and tasks. Part of the redesign effort involves understanding what should a hotel app offer that a mobile site doesn’t?

6. On average, how many times a year do you stay in a hotel?

  • More than once a week
  • Once a week
  • Once or twice a month
  • Once or twice a year
  • N/A – haven’t stayed in a hotel this year

Why ask this? This is a behavioral question that helps us to understand common customer habits when it comes to hotel stays. More than once a week may indicate a business traveler, whereas once a year may be a more leisure traveler. Part of the redesign goals are to increase digital bookings, and this question helps to separate out repeat stay users and potentially new users or infrequent users. Certain things may matter more to frequent travelers vs. infrequent and using this question we can show the data applied with this categorization.

7. Have you booked a hotel and airfare together in the past 2 years? If so, how have you booked this travel? (check all that apply)

  • Marriott.com website
  • Marriott mobile app
  • Other hotel website
  • Third-party travel site (Expedia, Priceline, etc.)
  • Through a travel agent or travel company
  • Airline website
  • Other
  • N/A – haven’t booked airfare and hotel together

Why ask this? This behavioral question helps to uncover user patterns relating to airfare and hotel coupling. First, it helps to identify the audience that has done this in the past two years, and additionally filters out those that have booked on one of the Marriott’s digital properties. It can help set a benchmark and identify users that have interacted with booking airfare and hotels with Marriott.

8. What types of online activities have you participated in the last 30 days?

  • Web searches
  • Shopping 
  • Buy or reserve travel
  • Social media
  • Email or IM/chat
  • News/weather/sports/blogs
  • Online banking / financial services
  • Lookup restaurants or menus
  • Order food for pickup/delivery
  • None of the above 
  • Other: __________________________________

Why ask this? This question helps to identify a user’s most common activities online. It can help to uncover technical skill when it comes to the web, and helps to categorize users when looking at the data from other questions. 

9. Into what age range do you fall?

  • 18-21
  • 22-37
  • 38-53
  • 54-72
  • 73-90

Why ask this? Categorization questions like this one help to understand who the audience taking the survey is. Ideally, our recruitment for this survey matches the typical user base for Marriott’s digital products, and we can better understand in what age groups our audiences fall into. This can be helpful in the redesign process since we have a better understanding of who we are designing for.

10. What country do you live in?

  • –select dropdown– (list of all countries)

Why ask this? Marriott is a global brand and this categorization question helps to learn more about Marriott’s audience. People outside the U.S. may have different values and its important to approach the redesign with information that we are designing for a global audience. This question also helps if we want to categorize particular respondents answers based on the country they live in.

11. Do you have any additional thoughts to share about hotel bookings, rewards programs, or the Marriott digital app or website? (optional)

Why ask this? This optional, open-ended question is nice to include at the end of a survey in case there are thoughts users would like to share about things not covered on the survey. This space is there for users to describe additional thoughts or opinions about booking hotels, rewards programs, or specific comments about Marriott’s digital products.

Conclusion

Keep your questionnaire short and purposeful and don’t be afraid to pre-test your survey with colleagues or friends you trust. Working out the kinks in the design of the questionnaire (i.e. do the questions make sense) before launching your survey is critical.

Happy surveying!

Categories
blog research

What conducting nano usability testing for Marriott.com is like

Usability testing is one of the most useful tools to uncover how well a website is supporting users’ main goals and tasks. Testing can occur anytime during a development of a new system, but it also can be used to discover how well a system is performing; this is often called benchmark usability testing. For the purposes of this testing, we conducted a nano usability test with three users of Marriot.com using the production live website.

Our nano usability test is a great first start to evaluating how well Marriott’s online booking system is working for users. Testing helps to uncover usage patterns, challenges encountered by users, as well as additional research questions. The purpose of testing is to not walk away knowing how to fix every issue with the system. Rather, it’s a first step to understanding where improvements might be made within the online booking system.

To kick things off, we generated three major tasks and recruited three participants.

Task completion

Overall, participants performed well. Only participant 2 had trouble understanding the larger concept of Marriott.com offering different hotel brands within the same booking experience. Users commented that the process felt “easy, and relatively quick” given two users knew a destination and dates they desired. Two out of three commented that they most likely would first compare prices with another travel website like Kayak or Priceline before purchasing. 

TaskParticipant 1Participant 2Participant 3
Select destination, date and detailsCompletedCompletedCompleted with difficulty
Select hotelCompletedCompletedCompleted
Select roomCompletedCompletedCompleted with difficulty

Participant 1: Had very little trouble picking a destination and dates from the main homepage. On the results page, they noted they liked how different hotel chains within the Marriott brand were displayed, but they couldn’t find a way to sort by price, a filtering option they often like to use. On the room selection page for the Park Central Hotel in San Francisco, they said they liked the different price options, and the prepay and save would be a feature they’d use. Often, they would pick the cheapest room option.

Participant 2: This user was using a smaller laptop and didn’t initially see the homepage call to action booking feature. They selected “Find and Reserve” which dropped down the same selections to pick a destination and date. Once they searched for Austin, Texas, on the results page, the user was briefly confused as why there were multiple hotel brands displayed. “I guess Marriott must own them,” the user questioned. Another thing the user noted on, was why the the results list showed hotels not available on the dates selected. They noticed a checkbox that filters to hotels available, but was confused why that wasn’t the default option. They selected a downtown Austin hotel and selected a prepay room; although commented on how long the title of the room was.

Participant 3: This user didn’t have a destination in mind, so they selected the Deals and Packages page, and selected the Hawaii vacations link. The system sent the user to the vacations by marriott site, a separate site with a different booking experience. On this page, featured hotels were listed but with no price. The user selected the first option, then was prompted to input a date and departure city. Once the list of results was returned, the user selected the first option for the king room. While they successfully booked a room, albeit through the vacations website, they did ask “Am I on the right site? Did I do this right?”

Main findings: user challenges

  • Finding an option to sort by price was difficult
    • Participant 1 had difficulty finding an option to sort the list of hotels by price, which doesn’t appear to be an option within the results page. Sorting by price is a common organization scheme that most travel sites offer. On the results page, the default scheme is to sort by distance.
  • Deciding on a room type took special care, since the room’s titles are often long
    • While choice is often good for users, too much can overwhelm them. In this case, selecting a room at a particular hotel was somewhat difficult. This was due to the fact that hotel rooms have long titles, with a strong font that makes it difficult to read. This can increase users cognitive load and easily lead to frustrations. One room for a Courtyard Marriott was titled, “Cool Weekend Getaway, 10 percent off 2 night stay, prepay in full, non-refundable if cancelled less than 1 day before arrival, no changes, based upon availability” in a large, bold font.
  • The Marriott vacations experience is completely separate from the main Marriott.com booking experience
    • One user that didn’t have a destination in mind selected the deals section of the website. After browsing about Hawaii, they were taken to the Marriott Vacation website, a separate experience that had a different look and feel. This separate site may confused users and there wasn’t any indication that you were leaving the main Marriott site.
  • The hotel results list often included hotels not available or coming soon
    • Even though a user selected dates for a trip, the results page often displayed hotels that were not available or “opening soon.” The option to display only available hotels was not selected by default. Showing all hotels in the list may be helpful for someone that is browsing, but could be frustrating to someone that knows the dates they want to travel. More results in the list means more items users have to scan through, which can increase cognitive load.

Additional user research questions

Our test stopped at the payment section and was narrowly focused only on the booking process. Since we were running a quick nano usability test, we didn’t have time to prepare fake information to use in the payment step. However, this is a critical step that deserves to be tested in the future. In addition, further research questions may surround:

  • What additional challenges are encountered by user attempting common tasks on Marriott.com? Tasks could include:
    • Changing or canceling a reservation
    • Signing up for loyalty program/credit card
    • Contacting customer service
  • What substantial usability impediments exist? (items that may influence a user to abandon the website or booking process)
  • What are additional customer insights that may help Marriott’s digital strategy?

One of the most important next steps is to ensure Marriott’s website has clearly identified and measurable goals. Understanding Marriott’s goals and business strategies will help determine research activities to further improve Marriott’s digital products.

Conclusion

Conducting user research helps designers conceptualize systems and processes that match how users think. Designing an experience that just plain works well doesn’t happen with luck. Through proven research methods and user engagement, organizations can build products with users, rather than “for” users. Involving users in prototyping, development, and testing, is a necessary step that helps give a product a higher chance of success.

As demonstrated with the nano usability test, research doesn’t have to be a long, drawn-out process. With only three users, we were able to quickly identify areas of the booking process to explore. This quick testing method gave us insights that we didn’t have before. Testing often allows a continuous check-in on how a design is performing and gives organizations insights into how to improve their products. Research is a great way to ensure designs match user expectations. Any form of research, big or small, assists with improving the user experience for users and helps to align digital products with the goals of an organization.

Categories
blog

Basics of usability testing and how to plan for it

Usability testing is one of the most useful tools to uncover how a website works for users. With effective planning, recruitment, and execution, any UX team or individual will be able to uncover a site’s main pain points and formulate recommendations that will help to improve the overall user experience in very little time.

A method of usability testing that can be useful to engage with in the beginning of a project is simply discount usability testing. These tests are planned with a small amount of users, since we are not looking for statistical significance in terms of usability issues. Instead, you are going to plan a test and recruit users to try and identify qualitative information and observe how users interact with our website while conducting a number of tasks. To accomplish this, we don’t need to test with more than seven users (generally).

Here’s how I would approach it.

Number of participants

For our tests, we only need to recruit eight testers — one being an alternate in the event of a cancellation. We are aiming for seven users for a number of reasons. According to Jakob Nielsen, an expert on usability and user-centered design, after testing with more than five users, you start to see the same issues and don’t learn additional insights. Our limited time frame only gives us a chance to run one round of usability testing. With that, we need to maximize our time with our testers, and recruiting eight testers (one alternate) is our best course of action.

Testing with only a few participants

Testing with less than 10 users may appear insignificant to draw major conclusions around a website’s usability issues. However, industry research tells us that we can uncover nearly 75% of the major usability issues in five sessions or less (Nielsen 1993). Again, we aren’t looking for statistical significance since our method of usability testing is not conducted in a true academic sense. Instead, we are attempting to identify what about a product is difficult to use. We will use direct observation, and encourage our testers to “think aloud” as they perform the tasks set for them. This will help us uncover the major usability issues that are contributing to a website’s main issues.

In addition, the severity of issues uncovered during usability tests are also related to the first few participants in the rounds of testing. In other words, usability issues and their severity and frequency are significantly correlated, which implies that severe issues are commonly discovered during the first few tests (Virzi 1992). This means we can get more value out of only testing with seven users since it will save us considerable amount of time, while still giving us strong usability insights.

The testing plan

Our testing plan is designed to accommodate the development team’s agile software development process. We will be efficient with our time, but take real consideration when we are drawing conclusions, findings, and recommendations in order to be smart about any changes we propose to the website. While the development team is working in week-long sprints, the UX group will need three weeks to complete our work.

Week 1 – Plan & Recruit users -Begin recruiting eight users for testing
-Develop facilitator script, key tasks and questions for participants, and collaborate with the the development and business teams to make sure our tasks are reflecting our site’s main tasks
-Begin to schedule usability sessions with participants
-Brainstorm a possible honorarium we can provide testers (gift card, cash, etc.)
Week 2 – Test-Conduct usability test sessions in person at our usability lab (UX group facilitator will lead sessions)Invite development team or interested parties to observe remotely
-After each session there will be a debrief with anyone that observed, and we will discuss top user pain points, user emotional reactions (nonverbal body language) and any other issues
-Sessions will be limited to two per day
Week 3 – Analysis & Recommendations-Analyze usability findings and synthesize major themes 
-Develop recommendations that address major findings
-Depending on severity of issues uncovered, work with development team to fold changes into weekly sprints thereafter.

In addition to the first round of usability testing, I would recommend building this type of iterative testing into your software development process. Since we can observe significant usability issues with only a few tests, conducting more regular analysis will help to proactively identify usability issues that otherwise could hurt our sales and frustrate our users.

This usability testing plan allows you to work efficiently and will not cost us much money. While we often have a limited timeframe, our tests are planned to maximize our testing resources. Considering no other issues come up, the a UX group should be able to plan, test, debrief, and recommend development changes to the site just in time to not upset your scrum master.

Categories
research

Brightline App Mobile Usability Testing

Brightline is a new high speed train service in Florida. We were tasked with testing the booking app and related top tasks. Using moderated in-person usability testing, we recruited nine participants and generated a findings and analysis report based on our analysis.
view UX brief
view test script

The Problem

We wanted to conduct usability testing to identify potential improvements for the booking app.

Our research goals were to:

  • Observe user patterns and behaviors as they attempt a set of common tasks using the mobile application
  • Identify how easily users accomplish common tasks
  • Understand users’ experiences and expectations as they work with the app
  • Gather user attitudes and perceptions as they relate to this type of booking app

Approach

Our formative usability study followed the “think aloud” protocol by asking test participants to describe their thoughts as they attempted to complete three separate tasks. We recruited nine participants for this study. We used a moderated in-person testing method.

Video screenshot of conducting in-person mobile usability testing.
Video screenshot of conducting in-person mobile usability testing.

Results

  • Each video was observed, which helped surface patterns and themes based on how users navigate the app and attempt the tasks.
  • From the initial observations and findings, themes were sorted and given prioritization based on frequency of the issue and other qualitative feedback. Post-test questions also help to influence the findings.
  • We then developed recommendations that address several of the findings.
Main takeaways:
  • The booking process is clear and the app gives users a straightforward entry point to begin booking a trip.
  • The app’s overall performance and technical issues can easily frustrate users.
  • Aspects of the seat selection task in the booking process has room for improvement.

Lessons learned

  • Usability testing on mobile apps can be difficult to see how a user is interacting with the app. Having a way to record the phone screen or video record the phone is important so you can later view how the user is interacting with the app.
  • If possible, make users comfortable and allow them to use their phone. If they are not comfortable downloading an app, have a backup phone ready for them to use. People are often the most familiar with their own devices.
  • Probing users to “think aloud” is an important reminder when conducting in-person moderated usability testing.
view UX brief
view test script