Proceedings of HF 2002, Nov. 25-27, 2002, Melbourne, Australia

Each sold separately: ethnography as a tool for integrating  online and off line use of educational toys

Keywords: Customer Experience, Rapid Ethnography, Usability, Education, Children

Abstract 

This paper describes the contributions of rapid ethnography to the design of a web-enabled educational toy, specifically installation and registration.  While HCI practitioners increasingly advocate rapid ethnography for early discovery and requirements phases, we discuss the role played by ethnography that spans open ended observation, story telling, conceptual design and iterative prototype evaluation in natural environments of use.  As the web enables new types of services, including educational toys, Experience Design professionals must use new methods to ensure that these services are truly intuitive and learnable.  Focused on everyday environments, offline and online experiences, an ethnographic approach combined with an iterative design process helped integrate knowledge of user motivation, sensitivities and tolerances throughout the design project.

1.      Introduction 

This paper describes how rapid ethnography was used to create a new online service for an already successful children’s toy maker. Ethnography was originally planned as a method of discovery research and of gathering requirements for specific steps of the process. In the end, ethnography served as a design tool, integrating multiple phases of the project, including discovery, requirements gathering and prototype testing.

Our task was to make a new type of toy interaction intuitive and fun. A successful toy maker, which we will call Educational Toy Manufacturer [ETM], decided to “Internet enable” its products.  For example, a math quiz toy can download new questions, upload children’s scores and provide parents with a view of their child’s progress. From the users’ perspective, this makes the toy more economical and longer lasting, offering content that changes with a child’s age and ability level. From the ETM’s perspective, creating a line of Internet enabled toys represented a significant change in business strategy. Customers could purchase additional content for their toys from the manufacturer’s Web site, vastly reducing the complexity of a retail supply chain. If the strategy worked, the ETM would increase its sales of upgrades – and potentially of its toys – by creating an economical system encouraging the ongoing use of its products.

Problems initially encountered by researchers focused on the discontinuity between what users thought they should be able to do and what they were actually able to do. In fact, the design agency was asked to solve problems with use of the online system that the ETM already knew to exist. The toy, the connection device and the content were each designed, produced and sold separately; each promised to be “easy” and “fun!” to use. The connection software, placed on a CD and included with the connection device, was also designed separately. The result was a set of physical objects and procedures – both online and offline – that frustrated and confused users instead of providing the promised enjoyment and educational enrichment.

The installation software for the connection device and the Web site were expected to solve many of the problems resulting from the “each sold separately” experience that consumers found so frustrating. We focused our research and set our design goals: to help reduce set up time from an average of over 2 hours to under 30 minutes and to create a registration system that would help users remember what they did long enough to go back to the website and repeat the task.

As is typical of many private sector design problems, only a small budget was allocated to initial “discovery” research. Further research had to be incorporated into the overall project proposal, with costs being absorbed by the design agency. For this reason, research methods had to be economical, and the results had to be immediately useful to the design team.

2.      Methods

Our approach draws from Cultural Anthropology training in field research methodology [Mead 1963, Malinowski 1922], a tradition of HCI contextual research and participatory design for productivity applications [Bodker and Gronbaek 1991, Wixon 1996, Simonsen and Kensing 1997, Holtzblatt and Beyer 1998, Jordan 1996, Kyng and Mathiassen 1997], and a growing literature on user-centered design for embedded and invisible technology [Norman 1988, Bergman 2000, Braiterman et al 2000, Kuniavsky 2003]. We adapted and created methods appropriate to compressed product cycles and to social activities that are “computer enabled” but, above all, voluntary and firmly within domestic rather than work spheres.

Working within the constraints of product development, we conducted collaborative rapid ethnography and customer modeling with a team that included a lead researcher, a business and brand strategist, interaction designers and visual designers. An initial sample of 6 families was selected based on the presence of children 8 to 12, interest in educational toys and willingness to allow us to enter their homes and videotape their efforts to set up the toy. Home visits lasted 2 to 3 hours, and allowed us to see how families would approach the toy and the Internet connection.

Subsequently, we followed some families’ use of the toys over a period of six months, returning periodically to document the location and use of the toys, the connection device and related materials in the household. Additional families who reported prior use of the toys were recruited for tests of the installation software, Web site registration and online acquisition of new content. In total, researchers conducted approximately 25 home visits over the course of the project; additional phone and email contact provided additional channels for data collection. Study participants were given their choice of toys or a nominal cash sum for their assistance. It is worth noting that many participants were motivated to help with the study because of their enthusiasm for the product and their desire to help their children acquire new math and spelling skills.

Initial visits began with the gift of a mathematics toy—a plastic, cylindrical device that asks multiple-choice questions and encourages children to “whack” it on its side to make their selection.  The toy’s dramatic sound effects, challenging questions and colorful design contributed to children’s receptivity: most became quickly involved and even entranced by the new toy. Along with the toy, we distributed the “Mind Link” module that permits the toy to be connected with the ETM’s Web site.  We asked parents and children to set up the online module, and our cross-functional design and research team observed and recorded on digital video their efforts which lasted up to 4 hours.

Figure 1 : ETM physical toy, sold separately from internet connector, "Mind Link."

After the initial round of 6 home visits, we modeled the behavior and the problems we observed.  This early modeling of the set up process proved to be a valuable tool in creating a new Web  application that eased the set up process for connecting the physical toy to the Web .  Because of manufacturing schedules, our Web application work was constrained by the fact that the toy had already been manufactured, so that we could not change the toy packaging or design. For this reason, subsequent home visits were increasingly focused on two aspects of the online component of toy use: the initial installation of the connection device and the registration of the toy and its owners on the Web site.

Visits in the latter two-thirds of the study included the use of design prototypes on paper and CD ROM. Our initial models of user behavior was used to develop the prototypes; users themselves were asked to generate the hypothetical use situations that were necessary for initial designs. We took several rounds of application prototypes back into parents’ homes to test and refine them in everyday environments. Our iterative design process evolved from broad to highly granular aspects of design.

Figure 2: Installation involves children and at least one parent. “Team” involvement was common for all stages of toy use, from purchase through updating of content.

3.      Results

The design project for which the Design Agency had been hired focused on the installation software and Web site where users would go to update their toys. Initial discovery research documented problems that went far beyond the registration and installation scope of this design project and included the form factor and packaging of the physical toy and “Mind Link” module. Some of these issues were directly relevant to our design work; others were significant only in that they increased the burden placed by users on the installation process and initial contact with the Web site. Here, we focus on two aspects of the results to our ethnographic study: initial installation and registration on the Web site.

The Educational Toy Manufacturer executives and product managers expected that the installation process would take about 10 minutes and would enable users to remember how to get online updates repeatedly.  Initial observations and interviews focused on users’ expectations and attitudes towards using a product that required them to install something new on their computers and go online. Parents told us that they thought the process might take “as long as 20 minutes” to complete. Most considered this to be a substantial amount of time. In two-parent households, the installation process was often delayed to permit the involvement of both mother and father during a time when children were not competing for their attention (e.g. during nap time, late at night or on weekends). In all households, installation was an activity that competed with others that would ordinarily occur in parents’ free time, which they would generally devote to personal needs. Clearly, installing the new toy was often seen as a chore – albeit one related to the academic success of their children – they were likely to put off.

Low expectations for installing the connection device were grounded in users’ previous software experience. Several users told us that their computers had “crashed,” “stalled” or “quit” as they tried to install games and other software for their children. Experience with online services and other domestic software was not encouraging: our relatively small sample included stories of problems with popular online services that resulted in repeated and total loss of data on home computers, many of which were used by stay-at-home moms for business purposes. We often marveled at how patient and willing users could be. Although clearly related to motivations that were often implicit (e.g. “computers are “good” for kids), users’ persistence and tolerance of poorly designed products seemed remarkable.

The packaging for the connection device did not help users form a realistic understanding of the time or resources necessary for installation and use. Claiming simply to be “easy and fun” to use, the packaging made promises that users had seen before and learned to disregard. To better understand just how reluctant users would be to install the new device, we documented the location and status of the product and its packaging over time. Of the 8 devices that we gave away as “gifts” to participants, 4 remained in their packages for more than 30 days. In some instances, parents went to the trouble of hiding the boxes high up in closets or cabinets so that their children would not see them and ask for immediate installation. In all but two cases, parents carefully stored and preserved the original packaging, fearing that technical troubles would require instructions or other information – such as telephone help numbers – they believed might only be available on the box.

Parents and children working together spent on average 2 hours setting up the Mind Link.  In some cases, the children took the lead on opening the box, inserting the CD and reading the instructions.  However we also saw that many parents did not want their children connecting peripherals to computer ports that were typically on the back side of the CPU towers.  Some parents resorted to asking their more technical spouses for assistance. In one instance, initial installation took over four hours and included a car trip to the grocery store to purchase batteries. In 3 of 8 cases, initial installation was not successful; in these instances, users attempted to erase software from their hard drives and begin the process again.

We also documented difficulties related to the design of and relationship between the physical objects necessary to update toys online. The Mind Link was designed in a manner similar to the toys – it was colorful and appeared “fun.”  While users liked the appearance of the object, some were also misled by it, thinking it was a toy. Furthermore, the low and oval form factor led several parents to mistake the Connector with a mouse input device.  Additionally, the device did not have an on/off switch, which caused some study participants to worry that the batteries required for use would be exhausted unless they disconnected the device from the computer after each use.

Figure 3 : The “Mind Link” internet connector: form offers few clues to function.

Users did not have a previously established cognitive model of what the device actually did, a problem that also resulted in challenges for graphically representing the Mind Link in instructions. In testing of prototypes for the installation CD and Web site, study participants referred to illustrations of the device as “that thing,” saying that it looked “like a toilet seat,” a “frog,” a “big mouse” and a “flying saucer.” These difficulties pointed to a larger problem – that of providing users with a mental model for understanding how the combination of physical objects necessary for updating their toys worked together. Without this understanding, users felt unsure of their ability to get new content for their toys and were unable to imagine what was required for successful completion of the task.

Unable to describe a process involving an object with which people were basically unfamiliar and whose function they were unable to imagine, the design team de-emphasized the role of the connector device in the overall process. “Update your toy” was the one concrete indicator of successful task completion that all users could understand; we focused on that rather than on the Mind Link itself. Previous designs had described what the device was and how it worked, information we knew to be at best unhelpful to novice users. Final success was the single most important factor in comprehension of the overall process. Once users had updated their toys, they were able to describe how this process worked; until then, we had to entice them to complete the steps without requiring them to understand how it would happen.

In response to these difficulties, the Design Agency developed an installation process organized around three basic requirements:

1.       Users had to be given concrete instructions to verify that the connection device was properly installed and that the computer was connected to a phone line and an Internet service provider. While we were unable to change product packaging or design, we knew that we had to provide a back-up method for correcting observed difficulties with setting up the device.

2.       The instructions for installation had to require little or no understanding of the relationship between physical objects and online procedures. Repeated observation and testing demonstrated that users were unable to imagine the end result of the installation process. We removed all references to outcomes other than “update your toy.”

3.        The installation of the software had to be quick and painless. Our video taped sessions showed that parents’ attention was often divided between multiple tasks. In the end, we eliminated almost half of the dialogue boxes from the original installation process and virtually all actions other than “next” or “quit.”

Based on these requirements, we were able to design an installation process that was quick and relatively stress-free. Users were very pleased to see the ETM Web site on their screen at the end of installation, the only necessary positive feedback on this process.

The next challenge was to register on the Web site so that users could obtain new content for their toys. Registration was necessary for several reasons including (1) the business model of ETM, in which content was available for a fee; (2) functionalities that included tracking of prior content, suggestions for updates and charts for parents to monitor their children’s progress, and; (3) privacy laws that required the protection of children under the age of 14 from certain types of marketing and sales techniques.

Parents had no innate interest in “registration,” as ETM did.  Instead, parents’ goal was to add new games to the toy. Particularly irksome to users in the original installation process was the requirement to register three different identities—the child, the toy and the parent.  Additionally, the terms used to describe content sets required users to understand ETM’s marketing strategy and to envision complex relationships between data sets. Terms like “Playset” and “ToyLink” were obtuse and did not help families gain a clear understanding of why or how to get started.  Worse, they led some users to believe that they were many steps away from getting what they wanted: new content for their toys.

In part, the difficulties in registering pointed to a broader set of problems: reluctance to register on any Web  site, fear of unwanted sales contacts via email, fear that giving a child’s name or personal information to a for-profit company might be harmful and a general resistance to the initial complexity of the ETM site use. As with the installation process for the connection device, the design challenges for the Web interface had as much to do with prior negative user experience as with the specific offering made by ETM.

Further complicating the interaction, ETM had previously used focus groups and other techniques that showed enthusiasm for functionalities like charting children’s progress.  Our observational research of actual product use demonstrated that surfacing additional features too early overwhelmed and confused users whose goal was simply to download more games and extend the life of the physical toy.

Based on our experience with the installation CD, we were able to develop a set of requirements for the registration process that emphasized outcomes that all users understood and perceived as valuable. These requirements included:

1.       Mapping a single, named user to toys and, later, to content. Registering toys, parents and children was too much for anyone in our sample to tolerate or understand. Instead, we chose to base all offerings on a single, personalized identity that could later be expanded to include multiple users of one or more toys.

2.       Providing visual and textual cues to tell users how far they were from getting new content for their toys. While we were able to reduce the complexity of registration to a bare minimum, we also knew that some steps would still seem unduly complicated or burdensome to some users. For them, we provided reassurances that they would get what they wanted within a predictable frame of time and effort.

3.       Grading the complexity of the interface based on the experience of the user. Initial users would be offered a single option (although others were also available). However, initial site usage had to be kept as simple as possible and focus on having a first successful experience on the site. Upon return to the site users would be invited to view progress or select from multiple types of branded content sets.

As with the installation CD, the design of the Web site generally and of the registration process specifically required us to account for problems that went beyond the scope of the design project, including customers’ prior experience with childrens’ software and ETM’s need to stage features based on first use and repeat visit scenarios. While no solution we tried was perfect for all users all the time, we were able to devise and verify a system that met minimum requirements for a successful initial experience. Iterative design methods and the use of an ethnographic approach enabled us to verify that users were able to remember what they had done and to repeat it when needed or desired. 

Figure 4: Customer models were created immediately after initial rapid ethnographic research to document the complexity of the existing installation process. These models served as both research results and an iterative design tool.

4.      Discussion

The research project resulted in a thorough mapping of the challenges of altering largely satisfactory offline behaviors onto a Web -centered service. Important limits and barriers to the integration of the ETM connector device and Web site into overall toy use included issues beyond the scope of the interface design project. Nonetheless, they were important considerations in the redesign of the installation software and the Web site. These issues included:

For installation:

·         A history of user experience with software and online services that delayed initial installation.

·         Packaging that did not clearly communicate the time and material resources necessary for installation and use.

·         An unfamiliar device whose design did not clearly convey its nature and function.

·         The combination of home computer software installation with Internet connections that were often unstable or otherwise unreliable.

For registration:

·         Difficulty in creating a cognitive model of renewable content and ongoing usage.

·         A complicated mapping process for creating identities to connect physical toys, online use histories and specific content sets.

·         The need to create mnemonic aids to help users remember how and what they previously accessed and used.

·         Sensitivities related to providing information about children on a Web site to a for-profit company.

The research and design team was surprised to learn how tolerant and patient users were of software and online services, especially given reported use histories. While this experience had an observable and negative impact on interaction with ETM’s offerings (especially in delayed initial installation and fear of providing personal information on a Web site), users’ persistence points to the importance of motivations in product design. User perception of the inherent value of information technologies in general, and of the specific educational benefit that the toys offered were key to completing tasks that were trying and often punctuated by delays, complexities and even failure.

Again, an ethnographic approach combined with an iterative design process helped integrate knowledge of user motivation, sensitivities and tolerances throughout the design project. We were able to observe these factors not only between the components of a system devised around a traditional brick-and-mortar “each sold separately” approach, but also across time and in the locations where users interacted with online and offline products and services.

5.      Conclusions

Methodologically, the results suggest some surprising advantages to the use of ethnography as a product design tool. Concept and prototype testing were integrated into a largely ethnographic study by extending home visits to include all design stages. Contrary to the expectations of the research and design team, rather than merely compressing the study into an economically and temporally feasible framework, the informal testing of design prototypes proved to be a valuable methodological enhancement. We were able not only to ask the kinds of “what if” questions that are usually limited to laboratory testing, but we were also able to observe the impact of relatively fine-grain technical modifications to design on location and over time.

Results to this study are particularly relevant to the use of ethnographic techniques in the HCI community. Often perceived as limited to “discovery” or open-ended research questions, ethnographic techniques led the design team in this project to results that were as fine grained as in laboratory testing (e.g. screen layout, wording) but were also verifiable over time and in actual usage environments, where motivation and context are key factors in product success.

One surprising result of the study was the impact of research findings on the staff of the ETM. Accustomed to designing products and business strategies around marketing data or anecdotal evidence of user behavior, the ETM went on to create its own in-house usability laboratory, to leverage the benefits of user centered design for business development.

6.      Acknowledgments 

We wish to acknowledge the staff of the Educational Toy Manufacturer, whose assistance was essential in carrying out the study. All research was conducted under the auspices of Small Pond Studios in San Francisco (USA), a user-centered, interactive design agency. Contributors to this project from Small Pond Studios include Samantha Tripodi, Brian Forst, Grace Hwang, Mike Strezo, AnnaLise Staal, Deb McDonald, and Shawn Collins.

7.      References 

·         Bergman, Eric (ed.).  2000.  Information Appliances and Beyond: Interaction Design for Conumer Products, (Academic Press: London).

·         Bodker, Susanne and Kaj Gronbaek.  1991.  Cooperative Prototyping: Users and Designers in Mutual Activity. International Journal of Man-Machine Studies.  v34, n3.  Pp. 453-478.

·         Braiterman, Jared, Sasha Verhage and Randall Choo.  2000. Designing with Users in Internet Time, Interactions, Sept-Oct, 27-34.

·         Holtzblatt, Karen and Hugh Beyer.  1998. Contextual Design, (Morgan Kaufmann: San Francisco).

·         Jordan, Brigitt.  1996.  Transforming Ethnography-- Reinventing Research.  In Groupware- Software fur die Teamarbeit der Zukunft: Grundlegende Konzepte und Fallstudien.  J Schiestl and H. Schelle, eds.  Marburg, Germany: Tectum Verlag.   Pp. 200-212.

·         Kyng, Morten and Lars Mathiassen.  1997.  Computers and Design in Context.  Cambridge, MA: MIT Press.

·         Kuniavsky, Mike.  (Fortchoming) Practical User Testing for the World Wide Web, (O’Reilly: Sebastapol, CA). 

·         Malinowski, Bronislaw. 1922. Argonauts of the Western Pacific. London: Routledge.

·         Mead, Margaret.  1963.  Sex and Temperment in Three Primitive Societies.  New York: Morrow.

·         Norman, Donald.  1998.  The Invisible Computer.  MIT Press.

·         Simonsen, Jesper and Finn Kensing. 1997.  Using Ethnography in Contextual Design.  Communications of the Association for Computing Machinery. 40 (7): 82-88.

·         Wixon, Dennis and Judith Ramey (eds).  1996. Field Methods Casebook for Software Design, (John Wiley: New York).

8.  Copyright 

Jared Braiterman, Ph.D., Patrick Larvie, Ph.D., the ESA (Ergonomics Society of Australia) and CHISIG (Special Interest Group of Computer Human Interaction).