Icon Usability Testing


As a part of my design research at IDC IIT Bombay, I worked on an eye-tracking interface for ALS patients. Creating the visuals in the form of icons for the interface led to the birth of a side project that involved concretising a methodology for testing the usability of icons in our enviornment.


We are constantly bombarded with signs in the form of pictures, icons and symbols around us. Icons are usually the most simplisitic, torn-down versions of their meanings with varying degrees of abstraction added. Even then, it is rare that these icons undergo any form of testing to check for their comprehensibility by the users who consume them. We devised a methodology involving two levels of testing to test the comprehensibility of icons in order to come up with the most suitable one. 


The International Organization of Standardization (ISO) has prescribed certain testing guidelines in their standardized Comprehension Test (ISO 9186-1:2014-Graphical symbols — Test methods — Part 1: Method for testing comprehensibility​). We got in touch with Prof. Keiichi Koyama (MD, iDesign INC from Tokyo) who specialises in signage design of public facilities, such as airports, railway stations, and environmental design fields.

His paper on symbols for priority facilities and seating was one of the first references for our study. Graphical symbols intended to indicate priority facilities and priority seating for those with limited mobility were tested in Japan, Australia and the United Kingdom using the ISO 9186-1:2007 comprehension test. Mr. Keiichi was extremely kind to provide us with the raw data for this study along with pictures of its implementation in Japan. 


Priority symbols implemented in Japan


The Comprehension test provides a measure of the extent to which a variant of a graphical symbol communicates its intended message through an open ended evaluation. The intention is to encourage the development of graphical symbols which are correctly understood by users when no supplementary (i.e. explanatory) text is presented. The responses collected from the users are to be marked under one of the following categories: Correct, Wrong, Opposite of the intended meaning, Don't know, No response. The standard recommends that >66% correct responses and <5% critical confusions qualify the icon to be used in the environment without any supplementary text. However, after some detailed research we realised that this methodology has certain loopholes and drawbacks in its implementation.

Broad drawbacks of the ISO 9186-1:2014
• An open-ended response method might not be the best first step to gauge comprehensibility due to it being open to subjectivity.
• A single level of testing leaves out a gradual way of narrowing down into the most likely icon to be tested for a certain degree of comprehension.
• The single icon being tested is at the mercy of the graphic designer's understanding of the meaning, his/her biases and past exposure.
• The test does not take into account users' past experiences and hence, there's no way of normalising them.
• Tossing an open-ended response into strict categories like 'Correct' or 'Incorrect' leads to a lot of assumptions being formed on behalf of the user.


In order to standardise a process to test comprehensibility of icons, we came up with a two-step methodology explained in detail below. There's also a concise inforgraphic at the end of this section explaining the process with an example. 

Preliminary Test: Appropriateness Ranking

Main Test: Comprehensibility Test (ISO 9186­-1:2014­ Graphical symbols — Test methods — Part 1: Method for testing comprehensibility)

Aim: To zero down on the most suitable icon representative of the intended meaning, also known as the ‘referent’.


For every referent, we bring together a set of icons that map all possibilities of its representation. This set of icons undergoes two levels of testing: the preliminary test, i.e. Appropriateness Ranking and the main test i.e. Comprehensibility Test prescribed by the International Organization for Standardization. The user group selected for this testing has to be representative of the core audience. The end product involves coming up with the most suitable icon for every referent.

 Appropriateness Ranking:

The Appropriateness Ranking Test works as a preliminary test for the set of icons in which it picks out the most appropriate icon (pertaining to a single referent) from a set. To begin with, the user is made to understand the context of the usage of icons provided to him/her. He/she is provided with the set of icons with the meaning that they stand for. The user is then asked to categorize the icons under 3 categories: Most Appropriate, Moderately Appropriate, Least Appropriate based on how aptly they think the icon is able to convey the said meaning. This classification is done keeping in mind the context of every icon. The icon with the highest frequency in the ‘Most Appropriate’ category goes ahead as an input to the Comprehension Test.

Comprehension Test (ISO 9186-1:2014):

To implement the Comprehension Test, the testing guidelines prescribed by the International Organization of Standardization (ISO 9186-1:2014-Graphical symbols — Test methods — Part 1: Method for testing comprehensibility) have to be followed.

 ISO 9186-1:2014 specifies a method for testing the comprehensibility of graphical symbols. It provides a measure of the extent to which a variant of a graphical symbol communicates its intended message through an open ended evaluation. (The purpose is to ensure that graphical symbols and signs using graphical symbols are readily understood.) The intention is to encourage the development of graphical symbols which are correctly understood by users when no supplementary (i.e. explanatory) text is presented. When such a graphical symbol cannot be developed, it might be necessary to present a graphical symbol together with supplementary text explaining its meaning in the language of the intended users.

The Comprehension Test procedure provides for screen-based testing. ISO 9186-1:2014 requires a certain size and resolution for such symbols, with the intent to ensure clear legibility of the symbols by participants. In implementing the testing ISO 9186-1:2007 requires respondents to be told the context in which the symbol being tested could be seen. The Standard also provides for a photographic image to be used to show the context and this was the method chosen for the present testing.

In this test, the user is provided with context and informed about where the icon may appear in his/her environment. He/she is then handed over a booklet with various icons for different referents. Next to every referent, is a blank space for the user to fill in his/her answer.

The evaluation guidelines suggest that the scoring of the responses and analysis of the data will be done by experts in the field. The primary purpose of the comprehension testing is to be sure that each symbol meets acceptance criteria. Comprehension rates (percent correct interpretations) must be determined for each symbol. The ISO 9186 standard recommends the general acceptance criteria of 66% correct responses with a maximum of 5% critical confusions.


comprehension testing

Here's an infographic explaining the methodology with an example:


The testing methodology was used to narrow down the icon set for the eye-tracking interface we made for ALS patients. Multi-level testing helps increase comprehension levels for icons being tested. Some problems like normalising the users' past experiences that remain unsolved need to be approached by setting a common metric for exposure besides just collecting a user's demographics.

Other projects

NH7 Weekender AppMobile App Design

Asian Paints Book DesignPublication Design

Icon Usability TestingDesign Research

Eye Tracking Interface for ALS PatientsHuman Computer Interaction

Project MudraVisual Design

© Apoorva Savant 2018