Category: blog

For the purposes of this example, I use my work on Marriott.com’s booking process. This survey questionnaire was designed for research on Marriott’s digital products.

With any opportunity you have to engage with users, customers, or the general public, it’s important to remember that you are asking for someone’s time. Asking purposeful, relevant questions that lead you to a better understanding of your users is important. Leave the questions that don’t help you learn more behind. A method I use is to write out your questions, then justify to yourself why you want to ask it.

Survey questionnaires help to refine user research goals and questions. The qualitative and quantitative feedback collected will help to assist in the redesign of your product (in my case, the Marriott website and mobile app.) For the purposes of this questionnaire, we are focusing on closed response questions, and looking to gather and collect categorical, behavioral, and attitudinal information from users.

Questions

Introduction

Thank you for agreeing to participate in our survey! Your feedback helps us create more useful and intuitive products. In this survey, we ask a variety of questions based on your hotel booking preferences, perception with rewards programs, and digital booking apps. Your participation is voluntary.

1.  How do you most commonly book a hotel room? (check all that apply)

• Marriott.com
• Marriott smartphone app
• Another hotel company’s website or app
• Travel site (Expedia, Priceline, Hotels.com, etc.)
• Travel agent or travel company (online or by phone)
• Other [type other answer]

Why ask this question? This questions is an easy introductory questions that helps us gather data about how users most commonly book hotels. Marriott would like to increase digital bookings by 10 percent, and it’s important to understand this user behavior as a benchmark metric.

2. When picking a hotel what top things are most important to you? (check all that apply)

• Distance from location (city center, airport, etc.)
• Price
• Loyalty program (points earned or ability to use points)
• Amenities offered (fitness center, pool, restaurants)
• Hotel services (airport shuttle, bike rentals, etc.)
• Other [type other answer]

Why ask this question? Understanding user attitudes toward hotel choices is key to understanding what matters to users when making a selection. This may help refine what is advertised about a specific brand of hotel, or reinforce the need for price filtering and more upfront information about the particular hotel. It may also help to identify new organization schemes for filtering or advertising hotels.

3. If you are a Marriott Bonvoy rewards member, how satisfied are you with the rewards program overall?

• Very satisfied
• Somewhat satisfied
• Neutral/no opinion
• Somewhat dissatisfied
• Very dissatisfied
• N/A – I’m not a rewards member

Why ask this? This simple question helps to understand who is a member of the rewards program and who isn’t. It also helps to set a benchmark for satisfaction levels and gives us better insights on how well the rewards program is performing for users. This is a question we could ask after some point in time to see if changes to the website or app not only increase rewards members, but also increase satisfaction with the program overall.

4. Please rate the various benefits associated with a hotel rewards program. On a scale of 1 to 5, with 1 being the least important and 5 being the most important, rate the following:

	1	2	3	4	5
Ability to use rewards to book free travel
Complimentary services (free wi-fi, drinks, food, etc.)
Dedicated customer service
Exclusive rates
Free room upgrades
Mobile check-in or keyless entry to room
Third-party offers and discounts on rental cars
Credit card offering that gives you rewards points to use to book travel

Other: [type answer]

Why ask this? This question helps to understand user attitudes toward a hotel rewards program. Part of Marriott’s business goals are to increase new rewards members and that starts with understanding what people value. This helps to discover certain reward benefits that may be better to advertise to entice new users to join the program. Also, insights from this may help Marriott’s business strategy and customer experience strategy to refine the rewards program to better suit its customers.

5. If you use a smartphone, have you used a hotel rewards/hotel booking app in the past year?

• Yes, I use a hotel app on my phone regularly
• Yes, I have a hotel app, but don’t use it
• No, I don’t have a hotel app on my phone
• N/A – I don’t use a smartphone

Why ask this? Answers to this question help to understand user behaviors when it comes to mobile apps for hotels. Onboarding someone for an app experience requires that the app is making its value proposition clear to users. An app that is downloaded, but not used, can indicate users don’t perceive the app helpful for them to accomplish their goals and tasks. Part of the redesign effort involves understanding what should a hotel app offer that a mobile site doesn’t?

6. On average, how many times a year do you stay in a hotel?

• More than once a week
• Once a week
• Once or twice a month
• Once or twice a year
• N/A – haven’t stayed in a hotel this year

Why ask this? This is a behavioral question that helps us to understand common customer habits when it comes to hotel stays. More than once a week may indicate a business traveler, whereas once a year may be a more leisure traveler. Part of the redesign goals are to increase digital bookings, and this question helps to separate out repeat stay users and potentially new users or infrequent users. Certain things may matter more to frequent travelers vs. infrequent and using this question we can show the data applied with this categorization.

7. Have you booked a hotel and airfare together in the past 2 years? If so, how have you booked this travel? (check all that apply)

• Marriott.com website
• Marriott mobile app
• Other hotel website
• Third-party travel site (Expedia, Priceline, etc.)
• Through a travel agent or travel company
• Airline website
• Other
• N/A – haven’t booked airfare and hotel together

Why ask this? This behavioral question helps to uncover user patterns relating to airfare and hotel coupling. First, it helps to identify the audience that has done this in the past two years, and additionally filters out those that have booked on one of the Marriott’s digital properties. It can help set a benchmark and identify users that have interacted with booking airfare and hotels with Marriott.

8. What types of online activities have you participated in the last 30 days?

• Web searches
• Shopping 
• Buy or reserve travel
• Social media
• Email or IM/chat
• News/weather/sports/blogs
• Online banking / financial services
• Lookup restaurants or menus
• Order food for pickup/delivery
• None of the above 
• Other: __________________________________

Why ask this? This question helps to identify a user’s most common activities online. It can help to uncover technical skill when it comes to the web, and helps to categorize users when looking at the data from other questions. 

9. Into what age range do you fall?

• 18-21
• 22-37
• 38-53
• 54-72
• 73-90

Why ask this? Categorization questions like this one help to understand who the audience taking the survey is. Ideally, our recruitment for this survey matches the typical user base for Marriott’s digital products, and we can better understand in what age groups our audiences fall into. This can be helpful in the redesign process since we have a better understanding of who we are designing for.

10. What country do you live in?

• –select dropdown– (list of all countries)

Why ask this? Marriott is a global brand and this categorization question helps to learn more about Marriott’s audience. People outside the U.S. may have different values and its important to approach the redesign with information that we are designing for a global audience. This question also helps if we want to categorize particular respondents answers based on the country they live in.

11. Do you have any additional thoughts to share about hotel bookings, rewards programs, or the Marriott digital app or website? (optional)

Why ask this? This optional, open-ended question is nice to include at the end of a survey in case there are thoughts users would like to share about things not covered on the survey. This space is there for users to describe additional thoughts or opinions about booking hotels, rewards programs, or specific comments about Marriott’s digital products.

Conclusion

Keep your questionnaire short and purposeful and don’t be afraid to pre-test your survey with colleagues or friends you trust. Working out the kinks in the design of the questionnaire (i.e. do the questions make sense) before launching your survey is critical.

Happy surveying!

Usability testing is one of the most useful tools to uncover how a website works for users. With effective planning, recruitment, and execution, any UX team or individual will be able to uncover a site’s main pain points and formulate recommendations that will help to improve the overall user experience in very little time.

A method of usability testing that can be useful to engage with in the beginning of a project is simply discount usability testing. These tests are planned with a small amount of users, since we are not looking for statistical significance in terms of usability issues. Instead, you are going to plan a test and recruit users to try and identify qualitative information and observe how users interact with our website while conducting a number of tasks. To accomplish this, we don’t need to test with more than seven users (generally).

Here’s how I would approach it.

Number of participants

For our tests, we only need to recruit eight testers — one being an alternate in the event of a cancellation. We are aiming for seven users for a number of reasons. According to Jakob Nielsen, an expert on usability and user-centered design, after testing with more than five users, you start to see the same issues and don’t learn additional insights. Our limited time frame only gives us a chance to run one round of usability testing. With that, we need to maximize our time with our testers, and recruiting eight testers (one alternate) is our best course of action.

Testing with only a few participants

Testing with less than 10 users may appear insignificant to draw major conclusions around a website’s usability issues. However, industry research tells us that we can uncover nearly 75% of the major usability issues in five sessions or less (Nielsen 1993). Again, we aren’t looking for statistical significance since our method of usability testing is not conducted in a true academic sense. Instead, we are attempting to identify what about a product is difficult to use. We will use direct observation, and encourage our testers to “think aloud” as they perform the tasks set for them. This will help us uncover the major usability issues that are contributing to a website’s main issues.

In addition, the severity of issues uncovered during usability tests are also related to the first few participants in the rounds of testing. In other words, usability issues and their severity and frequency are significantly correlated, which implies that severe issues are commonly discovered during the first few tests (Virzi 1992). This means we can get more value out of only testing with seven users since it will save us considerable amount of time, while still giving us strong usability insights.

The testing plan

Our testing plan is designed to accommodate the development team’s agile software development process. We will be efficient with our time, but take real consideration when we are drawing conclusions, findings, and recommendations in order to be smart about any changes we propose to the website. While the development team is working in week-long sprints, the UX group will need three weeks to complete our work.

Week 1 – Plan & Recruit users	-Begin recruiting eight users for testing -Develop facilitator script, key tasks and questions for participants, and collaborate with the the development and business teams to make sure our tasks are reflecting our site’s main tasks -Begin to schedule usability sessions with participants -Brainstorm a possible honorarium we can provide testers (gift card, cash, etc.)
Week 2 – Test	-Conduct usability test sessions in person at our usability lab (UX group facilitator will lead sessions)Invite development team or interested parties to observe remotely -After each session there will be a debrief with anyone that observed, and we will discuss top user pain points, user emotional reactions (nonverbal body language) and any other issues -Sessions will be limited to two per day
Week 3 – Analysis & Recommendations	-Analyze usability findings and synthesize major themes  -Develop recommendations that address major findings -Depending on severity of issues uncovered, work with development team to fold changes into weekly sprints thereafter.

In addition to the first round of usability testing, I would recommend building this type of iterative testing into your software development process. Since we can observe significant usability issues with only a few tests, conducting more regular analysis will help to proactively identify usability issues that otherwise could hurt our sales and frustrate our users.

This usability testing plan allows you to work efficiently and will not cost us much money. While we often have a limited timeframe, our tests are planned to maximize our testing resources. Considering no other issues come up, the a UX group should be able to plan, test, debrief, and recommend development changes to the site just in time to not upset your scrum master.

What is accessibility and universal design?

Accessibility is the means in which something can be used by everyone. It is important that your organization’s websites and mobile applications are accessible so you are not excluding people from being able to use them. It also makes smart business sense.

Universal design is another term that is often thought about when conceptualizing designs for systems. Universal design is the idea that we strive to build technologies that include features that benefit everyone, not just for people with disabilities.

Luckily, we have guiding principles when it comes to helping us build accessible technologies. The Web Content Accessibility Guidelines (WCAG), developed by the W3C organization, supports this effort with the WCAG 2.0 guidelines. These cover a wide range of recommendations for making web content more accessible. In addition to these standards, federal laws, including the Americans with Disabilities Act and Section 508 of the Rehabilitation Act deal with accessibility. Section 508 standards, which are now based on WCAG 2.0 standards, mandate that any technology procured or built by the federal government be accessible.¹

Why should we care about accessibility?

Making websites and mobile platforms accessible has numerous benefits for people with disabilities, average consumers, and our business. Among these benefits include building an expanded customer base, reducing development expenses, and preventing lawsuits.

By ensuring offerings are accessible, you are opening up your business to the maximum amount of customers and not leaving any group of people out. In addition, being proactive about building accessible technologies will save costs since it’s always more expensive to add features and fixes after initial development.

There is another reason why it’s so important — it potential puts you in legal trouble. The U.S. Department of Justice is beginning to consider websites under the public accommodation aspect of the ADA law, meaning they’d have to achieve WCAG 2.0 AA standards.² While not finalized, it’s clear that soon this may be a requirement for all company websites.

Accessibility standards and guidelines

You can easily remember key aspects of WCAG 2.0 simply by following the POUR acronym — which stands for perceivable, operable, understandable, and robust.³ These categories and related guidelines are often the standard in guiding a website or application in ensuring high levels of accessibility are met. There are different levels of conformance: A-AAA. We should strive to achieve at least AA for all our platforms.

Perceivable

• Provide text alternatives for any non-text content (includes use of alt tags for images)
• Page structures should follow logical orders  (proper use of h1-h6 HTML page headings)
• Ensure proper color contrast (WCAG AA – check with webaim.org/resources/contrastchecker)

Operable

• Make all functionality available from a keyboard (system can’t rely on mouse)
• Provide users enough time to read and use content (give users control)
• Do not design content in a way that is known to cause seizures
• Provide ways to help users navigate, find content, and determine where they are (skip navigation, clear explicit links, corrected coded tables)

Understandable

• Make text content readable and understandable
• Make web pages appear and operate in predictable ways
• Help users avoid and correct mistakes

Robust

• Maximize compatibility with assistive technologies (screen readers)

Info from W3C: https://www.w3.org/TR/WCAG20

Methods to ensure accessibility

Making platforms accessible will not happen overnight. However, you need to make a concerted effort to build in testing and validation procedures into development processes. 

Here is an example of processes to adopt:

• Accessibility testing and code checking tools – development teams should start to build in accessibility code checking into development processes. Two ways to achieve this:
  • Google Chrome’s built-in tool Audit tools
  • FireEyes Plugin (Firefox and Google Chrome)
  • ANDI validation tool by the Social Security Administration
• Keyboard-only testing – Developers or QAs can build in keyboard only testing. Set aside the mouse, and try to navigate the website using only with a keyboard. Iterate and test often.
• Screen reader testing – Use open source screen reader software or research purchasing JAWs licenses for development teams to test systems using a screen reader.
• Accessibility testing – Conduct accessibility testing with users with disabilities to test scenario-based tasks using users’ personal devices.
• Research more on how to integrate accessibility into teams: accessibility.digital.gov

References

¹ https://section508.gov/manage/laws-and-policies

² https://www.ada.gov/regs2016/sanprm.html

³ https://www.w3.org/TR/WCAG20/

One of the hardest parts of UX research is being confident in research planning. You can easily get good at front-line skills like interviewing and prototype testing, but being able to plan for different projects can be hard. No one UX project will be like the other, so don’t expect it to be the same process over and over, as in “we did a survey once so we must do surveys each time.”

Being able to articulate the what and why behind intentional user research will help you communicate to your organization the most efficient ways to get good information from users. It will also help squash the whole “Let’s do a survey!” suggestion from a product owner or developer when you know that won’t help.

Being able to articulate the what and why behind intentional user research will help you communicate to your organization the most efficient ways to get good information from users.

Below are five example scenarios with two UX research activities that could directly apply to the situation. In a real project, you may have to do some stakeholder interviews and learn more about the problem you are trying to solve before making any concrete recommendations concerning research methods.

1. The University of Washington is planning to redesign the U of W library website and wants to ensure that students and faculty can get the best use out of all of the services offered.

Usability testing — Benchmark usability testing is a great way to kick off this project. In order to fully understand how to approach a redesign project, we would first need to understand what are the current limitations and challenges with the site as it is now. This would involve recruiting users, coming up with common tasks to test, and analyzing the results.

Analytics review — a secondary research method is more quantitative in nature, and involves analyzing web traffic on the current site. Analyzing data from web traffic can help answer questions like:

• How are users getting to your site?
• What are users doing on your site?
• How often are users interacting with a particular feature/service?
• How often do users abandon their task before completing it?

An analytics review is a great way to discover a baseline for a site’s user engagement and other key indicators.

2. The photo printing company Shutterfly is looking to understand more about how, when and why people create, print, display, and share their photos … they are interested in directions for evolving their business to increase their customer base and find new potential revenue streams.

Survey — A great way to understand customer attitudes and behaviors is a survey. A well constructed survey can incorporate closed and open questions, and give organizations baseline data for what, how, and when customers do something. As in this case, Shutterfly is interested in the habits of customers creating, printing, and displaying photos. Particular insights from a survey may help to refine additional research questions, or can be used to test an organization’s assumptions about users. The survey results may help the business generate new concepts or ideas for products.

Focus groups — Surveys aren’t the best method for testing “hypothetical” scenarios, as if Shutterfly wants to gauge the interest in a new product idea. Instead, focus groups can be better suited to test new concepts and ideas with users. Selecting this research method would depend on what is learned in a survey. Additionally, it would depend on if Shutterfly wanted feedback on a new idea. For smaller testing of services and features, usability testing may be more appropriate. However, a focus group can be helpful for generating ideas during early product development.

3. Apparel company J.Crew is growing their ‘omnichannel’ retail functionality to better support consumers’ changing shopping expectations. Now they are trying to prioritize between two particular service features: Buy online, pick up in store — the credit card is charged at the time of online order OR hold online, pick up in store — the credit card is charged in the store when the customer picks up the item.

Focus group — Since J.Crew is considering two options, a focus group with representative customers may prove the most valuable in helping to decide which concept to go with. Focus groups, while they require a highly skilled moderator, are helpful to have an organized discussion about how customers might use a product or service. Customers in the focus group may also be able to speak to other companies and their experiences with online ordering and in-person pickup.

Literature review — J.Crew isn’t the first company to offer some sort of online shopping/store pickup experience. Likely, there is research on this buying habit and a literature review may be helpful to gain particular insights that would help J.Crew to make a decision. In a literature review, a researcher would search and identify previous academic or industry research conducted within the context of the business scenario.

*While research methods may be helpful to pick between two options for online ordering, implementing a solution requires special attention, as simple things like a poorly designed ordering/payment service could cause user frustration and abandonment. Low-fidentify, formative usability testing is essential in the actual design and implementation of either decision.

4. The New York City subway system is going to overhaul their payment and ticket system for greater efficiency and customer satisfaction. They are looking for a recommendation on what the revised experience should be.

Field studies — In discovering how customers currently interact and use payment systems within the subway system, a field study would be helpful for researchers and designers working for this project. In a field study, you can observe customers in their natural setting, and with consent, ask questions about payment systems. A field study isn’t a means to test new ideas, but rather gain insights into customer behaviors and attitudes, environment settings, and particular pain points in the payment systems/services.

Low-fidelity prototype testing — Simple low-fidelity wireframes or even sketches can be tested with users. This would be helpful during the ideation stage of the project, where we could test new concepts for both the payment system and service offered. This is a low cost method of testing early ideas in a project, and can be done after initial foundational research is conducted through field studies, observations and interviews.

5. Grocery delivery service Instacart is trying to expand, but they aren’t getting either the account signups they expected or the number of orders from those signups needed to support their business costs. Instacart wants to know why they aren’t getting more customers and orders.

Usability testing — It’s unclear if the issues with customer signups is due to usability issues with the site, so I’d recommend testing the current site to see how well it performs with users attempting common tasks. If there are usability issues, these can be addressed and analyzed to see if it improves signups and orders. If the site is performs well during usability testing, there may be larger user experience issues that would require more intensive, exploratory research like user interviews.

Analytics review — Looking at the website traffic and metrics will help to identify what pages users are leaving the site from. It could also be helpful to understand how users are coming to the site, which could identify how well any promotional or marketing campaigns are working.

User research is paramount to the success of an organization’s digital products. Building products without having done the background and foundation research about user needs, goals, and challenges, is akin to building a house without a blueprint. Research helps inform design decisions and gives decision makers confidence that their products will work for users.

While many user experience (UX) issues can be identified by an expert design review, or a heuristic analysis, often deeper issues with a system’s workflow, content, usability, and other factors are uncovered by involving real users.

Basics: what is user research?

User research is a process of understanding your users through a variety of activities that help to inform the design of a solution that matches user goals and needs. User research helps to support the design and development of systems that are successful — systems that are intuitive, usable, and even delightful. Research activities are always informed by research questions — what do we want to learn and why? 

Here are a few examples of user research activities:

User interviews

One-on-one interviews with actual users (or new users) of a product where we discover qualitative information, high-level themes and direct user insights

Usability testing

Tests the current products (or redesigned ones) against a set of common tasks with typical users. Each task is measured for success or failure; helps to uncover usability issues and possible user frustrations.

Card sorting

An activity that asks real users to group items together in categories; helps to inform a site or app’s information architecture (navigation, taxonomies, labeling).

Benefits of user research

Arin Bhowmick, the VP of design at IBM, describes user research as “help[ing] us to understand how people go about performing tasks and achieving goals that are important to them. It gives us context and perspective and puts us in a position to respond with useful, simplified, and productive design solutions.” Research not only helps set up a project redesign for success, it may even save money in the long term. Investing the time to conduct user research can saves hours of development time on potential features that aren’t needed, or save countless hours refactoring code for a product that isn’t working for users. Having the research insights help to ensure we are building products that will work for users.

How would user research help your organization?

Firstly, user research allows you to set a benchmark for your current product offerings. Research helps answer questions like “How well is the current site or app supporting users’ needs and goals?” User research not only helps understand the current status of your organization’s products, but can help uncover new site or app offerings and features that may greatly support your existing users or new users.

Next steps: How to begin user research

Any research is better than no research. Here are just a few easy things to start your user research journey and which could be used to kick-off a new project.

• Benchmark usability testing — involves recruiting a set of users to do usability testing of a website, app, or service. This will be helpful to identify how well any products are currently performing against a set of common tasks.
• Stakeholder interviews — part of design research, stakeholder interviews are held with key people from the project team or organization. These help to identify any business requirements or insights from the organization that are helpful when approaching the redesign. They also help to minimize 11th hour changes by upper management. Engage them in the process early.
• User interviews — these help to learn about user attitudes, behaviors, and usage patterns relating to behavioral and attitudinal factors. Learning customer insights directly from the source will help you identify new features or areas of your product to refine. Once you begin a design phase, you may rely on additional research activities like card sorting, concept testing or further usability testing.