EVALUATING USER EXPERIENCES

Evaluating user experience is essential to inform certain design and development decisions of products and services. However, it is equally essential to understand that the evaluation itself is a serious business due to the subjective nature of user experience. This is a major reason behind defining, preparing and setting up the scene that produces the carefully made evaluation protocol and eventually leads to its efficient implementation.

Often a need for a deep interpretation of design artefacts can appear in various forms and stages. The artefact can be an existing service that requires a major reform, or the first gist of an idea behind a new product waiting to have a healthy “partum”. The need can entail a ‘black & white’ choice between attributed user values or can seek for a clear relationship between certain attributes, consequences and values. That very nature behind the interpretation of design is the reason why there are different techniques out there to understand the experiential aspects of products or services.

Zooming into a particular focus on the need to evaluate initial ideas -that later on are constructed as ‘concepts’ in the design process-, interview-based techniques come in handy. Those techniques heavily rely on implicit and explicit user insights, meanings, attributions, human values, emotions and aim to trigger their visual or verbal imagination or reasonings. In that sense, we can say that interview-based techniques like AXE (Anticipated eXperience Evaluation), Contextual Laddering and Sentence completion are useful to get a better hold of hedonistic qualities like stimuli, feelings or perceptions. Those methods are also purposive though to complement usability methods when they are limited, especially in the initial concept phase.

AXE focuses on visual stimuli to make evaluations. It aims to facilitate metaphorical thinking. As van den Hende (1) argues, the way AXE is exercised has to stimulate imagination about future ideas; ‘the concept’. It does that via the usage of image pairs, which brings the black and white contrast to the user so that the preference is clearly stated. In an AXE method in use, you can picture a researcher working on a new wellness app showing two still images of nature and ask the participant to indicate which image is closely associated with the need of a user to feel better using a smart application. Or the same researcher can use 2 different images of a low-fidelity prototype and tries to inquire which type of scale user associated two different images from the prototypes. Imagining the participant has indicated a scale of rest-unrest, the researcher can deepen the understanding by asking the participant what made her feel rest or unrest.

While AXE is really useful in both evaluative (user perceptions) and collective fashion (suggestions for improvements) for an early product or service concept like Gegner and Runonen (2) states, it may come across with complications when the concept being evaluated is very complex or when the comparisons to be made are not enough in contrast. After all, user experiences, with all their sensual, emotional, spatial, social or cultural aspects engraved in abstract levels of meanings, stimuli or perceptions, are not straightforward at all. It may also present major limitations for people who don’t possess visual literacy or who have visual impairment.

Contextual laddering can be operable when it comes to the level of abstraction. It is a technique combining qualitative and quantitative approaches. Implementation of the technique, -the actual evaluation- is qualitative while quantitative approach is fairly dominant in analysis. This brings a balanced bridge to the analysis itself for understanding attributes that offer value and meaningful user experiences, as Abeele and Zaman (3) referred to. The central question of contextual laddering is WHY. WHY being in center helps to detect the dominant attributes and then tie them to consequences and value chains. The laddering method is often called to base itself to the means-end theory, that un-discloses the core underlying values as motivational factors of desiring the ‘ends’ of a product or a service. A matchmaking between the stimuli to chains of beliefs are expected to be happening in the unconscious of users, therefore the question WHY is constantly present in contextual ladder method to reveal these connections.

Although contextual laddering can fill the abstract holes of other techniques like AXE and is fully compatible with merging the evaluation with usability, utilisation of it under hands of non-experts or slightly different natures of analysis and evaluation can be concerning in terms of reliability of outcome. However, comparing 2 complex concepts like 2 different applications or web sites, for instance, can make use of contextual laddering bearing in mind the efforts needed to apply technique and procedures.

Unlike laddering, the implementation of Sentence Completion technique does not come with huge hassles. The emphasis on meanings is very prominent in this technique. It starts by asking the users to complete a sentence. The beginning of a sentence is designed to trigger the experiential aspects of a product. You can recognise this technique if you come across with a ‘fill-in-the-blanks’ style of inquiry, such as “when I use this service, I feel myself…”. Products and services are said to carry meanings, as such that the actual use or images behind a possible interaction with them are related to fantasising or expected use of it, which include memory associations, identities, beliefs or values, as Desmet and Hakkert (4) considers. Therefore, the idea behind this technique is to identify symbolic meanings. The actual implementation of it is often fun, while the analysis might not be as easy as a thorough quantitative study or might not make a rich use of a clear (although also difficult) procedure for the analysis because the data gathered will be highly qualitative in nature.

At this point, I believe it would be noteful to remind that as UX pursues a range of subjective and hedonic qualities, those techniques, along with many others proposed or still in the making are promising to support the idea that those subjective qualities play a stronger role in UX than the traditional usability (5). Yet I still insist that the interplay between the pragmatic nature of traditional usability assessments and subjective essence of rather fresh user experience evaluation techniques would be a more effective and complete lens through understanding experiences.

References:

(1) van den Hende, E. (2010) Really New Stories The Effect of Early Concept Narratives on Consumer Understanding and Attitudes. Delft, The Netherlands: Delft University of Technology.

(2) Gegner, L. and Runonen M. (2012) For what it is worth Anticipated eXperience Evaluation

(3) Abeele, V. Zaman, B. (2009) Laddering the User Experience!

(4) Desmet, P., & Hekkert, P. (2007). Framework for product experience. International Journal of Design, 1(1), 57–66.

(5) Mäkelä, A., Fulton Suri, J. (2001) Supporting Users’ Creativity: Design to Induce Pleasurable Experiences.