CTFL-UT Questions and Answers

Usability labs provide a controlled environment that offers several advantages for observing and recording usability test sessions. Specifically, they allow for easy observation of participants (ii), standardized conditions for each session (iv), and facilitate audio and video recording for detailed analysis (v). However, the context is not natural (i), because the lab is an artificial setting rather than the user’s actual environment. Also, setting up a professional usability lab is not inexpensive (iii); it typically involves significant costs for equipment, space, and staffing. Therefore, options ii, iv, and v are true, while i and iii are false.

In usability evaluations, a distinction is made between actual usability problems (where a user struggles to complete a task or is confused by the interface) and suggestions or ideas that users provide based on their preferences or perceived improvements. When a participant offers a new feature idea (such as a quick search box), this is classified as a “good idea” or “feature suggestion,” not necessarily a usability problem. It may inform future design enhancements but does not indicate a failure in usability for existing functionality.

The Web Content Accessibility Guidelines (WCAG) provide standards to ensure websites are accessible to people with disabilities. They specifically address:

i. Text resizability (SC 1.4.4),

iii. Text alternatives for non-text content (SC 1.1.1),

iv. Keyboard accessibility (SC 2.1.1),

v. Device independence, which includes not relying solely on mouse input.

Visually appealing design (ii) is not an accessibility requirement and is thus not a WCAG focus. The correct answer is D: i, iii, iv & v are true; ii is false.

Formative usability evaluation is conducted during the development process to identify usability problems and improve the product iteratively. It is diagnostic and improvement-focused. Summative evaluation, on the other hand, is done after development to assess the final product’s usability, measuring how well it meets defined usability goals. Therefore, the correct distinction is that formative evaluation focuses on improvement, and summative evaluation assesses the outcome. This distinction aligns with widely accepted models such as those defined by ISO 9241-210 and usability.gov.

A best practice in usability reporting (based on ISO/IEC 25062:2006 – Common Industry Format for usability test reports) is to include a clear description of the evaluation method used. This includes how the test was designed, how participants were selected, what tasks were performed, and under what conditions the test was conducted. This ensures the results are credible and reproducible. The provided structure omits this essential information. While the report length is not excessive and positive findings may or may not be present, the key missing component is the method description.

A usability evaluation is specifically designed to assess how real users interact with a software product. It focuses on effectiveness, efficiency, and user satisfaction in completing tasks. This may involve usability testing, expert reviews, or heuristic evaluations. The goal is to identify any usability problems and understand user behavior during real use.

Option A (Accessibility evaluation) targets inclusivity for users with disabilities, not general user interaction. Option B (Usability maturity assessment) evaluates the organization’s usability practices, and Option C (Risk assessment) evaluates potential project or system risks. Only option D directly involves observing and measuring user interaction.

A pilot usability test session is a critical step prior to conducting the actual usability test. Its purpose is to ensure that the test setup, task design, and moderator instructions are clear and function as expected. According to ISO 9241-210 and the Nielsen Norman Group's best practices, pilot testing helps detect unforeseen issues and fine-tune the process. Options A and B are incorrect because testing conditions must reflect realistic scenarios, and thorough preparation is essential to gather meaningful usability data. Option D is also incorrect; a usability test typically consists of multiple sessions with different users to gather sufficient data for analysis. The pilot session ensures everything works smoothly before involving real participants.