5 stages to processing and acting on 100+ hours of ethnographic study

This post is reblogged from the Lib-Innovation blog, to tie up and follow on from the previous post on THIS blog about the Understanding Academics Project.

Understanding Academics, introduced in the last blog post, is far and away the biggest UX project we’ve attempted at York, and the processing and analysis of the data has been very different to our previous ethnographic studies. This is due to a number of factors: primarily the sheer size of the study (over 100 hours’ worth of interviews), the subject matter (in depth and open ended conversations with academics with far ranging implications for our library services), and actually the results themselves (we suspected they’d be interesting, but initial analysis showed they were SO insightful we needed to absolutely make the most of the opportunity).  

Whereas for example the first UX project we ran conformed almost exactly to the expected 4:1 ratio of processing to study – in other words for every 1 hour of ethnography it took four hours to analyse and process – the total time spent on Understanding Academics will comfortably be in excess of 400 hours, and in fact has probably exceeded that already. 

UX is an umbrella term which has come to mean a multi-stage process – first the ethnography to understand the users, then the design to change how the library works based on what you learned. In order to ensure we don’t drown in the ethnographic data from this project and never get as far as turning it into ‘proper’ UX with recommendations and changes, Michelle Blake and Vanya Gallimore came up with a 5 stage method of delivering the project. 

Two particular aspects of this I think are really useful, and not things we’ve done in our two previous UX projects: one is assigning themes to specific teams or individuals to create recommendations from, and the other is producing and publicising recommendations as soon as possible rather than waiting until the end of the whole project. 

As you can imagine the 5 stage method is very detailed but here’s a summary:

Coloured pens used in cognitive mapping (in this case with the interviewer's reminder about the order in which to use them)

      1)  Conduct and write up the ethnography. Academic Liaison Librarians (ALLs) spoke to around 4 academics from each of ‘their’ Departments, usually asking the subject to draw a cognitive map relating to their working practice, 
and then conducting a semi-structured interview based on the results. 

The ALLs then wrote up their notes from the interviews, if necessary referring to the audio (all interviews were recorded) to transcribe sections where the notes written during the process didn’t adequately capture what was said. The interviews happened over a 2 month period, with a further month to complete the writing up. 

      2)   Initial coding and analysis. A member of the Teaching and Learning Team (also based in the library) who has a PhD and experience of large research projects then conducted initial analysis of the entire body of 100+ interviews, using NVIVO software. The idea here was to look for trends and themes within the interviews. The theming was done based on the data, rather than pre-existing categories – a template was refined based on an initial body of analysis. In the end, 23 over-arching themes emerged – for example Teaching, Digital Tools and Social Media Use, Collaborations, Research, Working Spaces. This process took around 2 months. 

      3)   Assigning of themes for further analysis and recommendations. Vanya then took all of the themes and assigned them (and their related data) to members of the Relationship Management Team – this consists of the Academic Liaison and Teaching and Learning teams already mentioned, and the Research Support team. This is the stage we are at now with the project – each of us in the team have been assigned one or more theme and will be doing further analysis at various times over the next 8 to 10 months based on our other commitments. A Gantt chart has been produced of who is analysing what, and when. The preparation and assigning of themes took around 2 weeks.

      4)   Outcomes and recommendations. There are three primary aims here. To come up with a set of practical recommendations for each of the themes of the project, which are then taken forward and implemented across the library. To come up with an evidence-base synthesis of what it means to be an academic at the University of York: a summary of how academics go about research and teaching, and what their key motivations, frustrations and aspirations are. (From this we’ll also aim to create personas to help articulate life for academics at York.) And finally to provide Information Services staff with access to data and comments on several areas in order to help inform their work – for example members of the Research Support team will have access to wealth of views on how academics think about Open Access or the repository. 

These aims will be achieved with a combination of devolved analysis assigned to different groups, and top-down analysis of the everything by one individual. Due to other projects happening with the teams involved, this stage will take up to 7 months, although results will emerge sooner than that, which leads us neatly to...

      5)  Distribution and Dissemination. Although this is last on the list, we’re aiming to do it as swiftly as possible and where appropriate we’ll publicise results before the end of the project, so stages 4 and 5 will run simultaneously at times. The total duration from the first interview to the final report will be around 18 months, but we don’t want to wait that long to start making changes and to start telling people what we’ve learned. So, once an evidence-based recommendation has been fully realised, we’ll attempt to design the change and make it happen, and tell people what we’re doing - and in fact the hope is to have a lot of this work completed by Christmas (half a year or so before the Summer 2017 intended end date for the final report). 

The full methods of dissemination are yet to decided, because it’s such a massive project and has (at a minimum) three interested audiences: York’s academic community, the rest of Information Services here, and the UX Community in Libraries more widely. We know there will be a final report of some sort, but are trying to ensure people aren’t left wading through a giant tome in order to learn about what we’ve changed. We do know that we want to use face to face briefings where possible (for example to the central University Learning and Teaching Forum), and that we’ll feedback to the 100 or so academics involved in the study before we feedback to the community more widely. 

Above all, Understanding Academics has been one of the most exciting and insightful projects any of us have ever attempted in a library context. 

A UX Intern writes... Emma Gray on ethnography

This is the second post in a series about Embedding Ethnography at York. The introductory post is here; the next post will be on the same project as this one, but written by me and focusing on the logistics of organising the whole thing, rather than undertaking the ethnography.

This is Part 2, written by Emma Gray, who was our first UX Intern. At the time of her work with us she was a 2nd year undergraduate at another institution. She did an absolutely brilliant job here, and we learned a lot through her work - about our students but also about ethnography itself. Here's her take on the project. It's a great intro to what we think of at York as the Big Five ethnographic techniques in libraries. There are plenty more, but so far we've focused on these ones, in various combinations, across all our UX work.

---

UX TECHNIQUES IN A LIBRARY SETTING – INTERNSHIP REFLECTION

In August 2015, I began an intern project looking into how several ethnographic UX techniques can be applied to a library setting and used to investigate possible improvements of the library user’s experience in the University of York library. As the project took place outside of term time, there was a focus on the experience of postgraduate students, who were still using the library over the summer break. The techniques covered over the internship include cognitive mapping, love/break-up letters, interviews, behavioural observation and touchstone tours.

BEHAVIOURAL OBSERVATION

Behavioural mapping is the first technique that I tried out at the beginning of the project. Being unfamiliar with the library before the project began, it was a valuable experience to observe how students behaviour in different spaces in the library. Firstly, I made a grid to record observations over the six week period in different locations within the library, increasingly concentrating on the more busy areas. Using the AEIOU framework, I recorded action (how the students are working in the space), environments (noting atmosphere, for example noise levels), objects (which services are used, e.g. technology and printed resources) and finally users (it is useful to note who is using the space, be it students, staff or external users). Secondly, it was also useful to record pathway maps using coloured diagrams of how individual students move in the space. In the York University library, it was particularly relevant to record pathways in the entrance foyer because the result show which building of the library students head when they enter.

An example of one of Emma's pathway maps

COGNITIVE MAPS

Participants were given six minutes to draw a map of the library, and I asked participants to change the colour of their pen every two minutes so that it could be easily remembered in which order things were drawn. It was a good starting activity because it allowed students to think about how they use the library and its services as whole. This visual representation of the spaces seems to also get the students thinking about how they use the library because I asked them more detailed questions during the interview.

The idea is that students would instinctively begin by drawing areas of the library they area most familiar with. However, I found that most participants started drawing the entrance in detail, maybe because the layout of the library as a whole is quite complex. I therefore encouraged participants to begin by drawing the area of the library they are most familiar with and use the most. This gave stronger results because it showed which objects and areas are most used.

A Cognitive Map drawn by one of the participants of the study.

In order to process the results of these maps, I divided the library into each floor of each building, and for each noted number of occurrences on the maps, its identification index which reflects the percentage of time is occurs (number of occurrences /number of participants), its representative index (number of occurrences / times category is drawn), and its temporal index (3 points for first pen colour, 2 points for the second and 1 point for the third). This process was then repeated for each individual object that is drawn on the maps, which for example included desks, computer areas and other library services. The advantage of this technique is that is gives a visual representation of how the students view the library, and it can also yield quantitative data. However, it is also a very time consuming technique, so the practicality of the data very much depends on the size of the library.

A screengrab of part of Emma's coding of Cognitive Maps

INTERVIEWS

I began the interview by asking the participants to describe and elaborate on their cognitive maps. Many participants elaborated on how they use certain spaces, which provided useful context for the cognitive map. I also asked participants to explain the process of how they use library services when working through an assignment. I asked general questions about the students’ experience with various services that the library provides, and encouraged to give their opinion on how their experience could be improved.

I found that being an intern and a student myself was invaluable to the interview process because the students did not shy away from being honest about their experience and more comfortable being critical than they might have been with a member of staff. Following the advice from Bernard’s ‘Research Methods in Anthropology: Qualitative and Quantitative Approaches’, I occasionally paused after a short answer was given, and often the participant would continue talking in more detail after a few moments, which kept the conversation flowing.

The major advantage of interviews is the breadth of information they yield, especially because things that individuals bring up can be elaborated upon and questions can also be adapted for future participants (e.g. finding out whether students had similar experiences with aspects of the library mentioned by other participants). Having a few standard open-ended questions is key, but participants seem to much more at ease if the interview flows like a conversation, rather than reading off a list of questions.

The only downside is that transcribing long interviews (they usually lasted between ten and fifteen minutes) takes a long time, but is definitely more valuable than making notes at the time because you are free to focus on what the participant is saying.

LOVE / BREAK-UP LETTER

Participants were asked to write either a love or break-up letter to a specific service that the library offers, so that the students can more emotively express how they feel about this service. I encouraged students to write about they feel (either negatively or positively) passionate about.

The participants were given a maximum of ten minutes to complete their letter. I definitely found this technique the most interesting because many participants wrote very imaginative, emotive letters that gave real insight into how library services and staff are perceived, and how this affects their daily university life. I did not find that this technique had any real disadvantages, only that some students preferred to write a wish list on what could be changed in the library. This also gave insightful information, but often did not show the reasoning behind these requests.  

An example of a usefully insightful Break-up Letter... Participants write these by hand but Emma wrote them up for ease of searching / reading. 

And for balance, here's a Love Letter...

TOUCHSTONE TOURS

For touchstone tours, I asked participants to give me a guide of the library. I voice recorded the tours to be transcribed later, so that every detail of the tour could be recorded.

Before the tour began, I usually encouraged participants to give as much detail as they could about how they use the services and different spaces, and also to point out their favourite space in the library. During the tour, I tried not to interrupt the participants or ask any more questions so that they could speak freely. Occasionally, I asked questions after the tour was over if any clarification was needed, or to enquire why the tour hadn’t included a certain area of the library.

The touchstone tours were interesting because the participants were very free to take me wherever they want, and point out anything they want, meaning that the participants brought some things up which I hadn’t thought to mention in the interview. It is notable that the quality of the tour heavily depended on how comfortable the participants were speaking out loud and narrative as we went along.

CONCLUSION

All of the techniques describe above have their own advantages and disadvantages, but they also each contributed to a more multi-faceted insight into how each participant engages with library services. I found the cognitive mapping the most interesting technique because I think it was the most unfamiliar to the participants and therefore got them thinking about how they use the library in a new way. As was the case in this particular library, it also highlights any areas that are being underused. Conclusively, having an intern collect UX data for a university library is a great idea, because the students are definitely more comfortable making their feelings known than they would be with a staff member.

#UxLibs 4: Ethnography You Can Try At Home

Tweet

Post 4 about the UXLibs Conference. This one is about the actual ethnography we went there to learn about. If you're interested, there's a post about the conference organisation and format here, plus a post about the Usability keynote here, and a post about the Ethnography and Design keynotes here.

But first, the Conference badge. This genuinely was the best user experience I've ever had from a badge. The attention to detail was symptomatic of the whole approach to the conference, I felt - it was designed to be a good experience. Timetable, maps, where your accommodation is, which parallel sessions you were in, name, institution, Twitter name, the conference hashtag AND it was reversible so it could face either way and still be useful. All little things but together they made a real difference.

Badge magique!

Before attending this conference my knowledge of ethnography was limited to having read about it. I'd not done any. I knew it was useful, and I knew why I felt we needed to make use of it - but if someone had caught me by surprise with the question 'So what IS ethnography in libraries?' I would have crumbled... But not anymore!

Here are some of the techniques we learned about ethnography - in the final post next time I'll cover the process of designing a service or product off the back of what you learn.

Cognitive Mapping

I've put this one first because it's my favourite. I didn't actually get to do this one (on Team Space Grey we all went to different workshops, and mine was the Observation one), but I really loved the results of it. If you're reading this thinking "I did do this and you've misunderstood it" then correct me in a comment!

Cognitive Mapping is in essence asking your subject (student, staff or whoever) to draw a map of the library - or, ideally, of their wider learning landscape - in order to understand how they perceive the space, what they actually use, what they value and see as most important and so on. Often the subject is asked to change colour of pens every 2 minutes, for a total of 3 colours over 6 minutes, so you can later see which order the items were drawn in, an indicator of their importance.

Here's a map which Team Space Grey got one of our students to draw (thanks for sending me this Kristin!):

We got two maps overall - what was particularly interesting for us is that one of them mentioned the basement area of the library as being dark and scary, and another didn't put the basement on the map at all! This was a big part of the idea we later pitched (which I'll cover in the next post).

There are some examples from Donna Lanclos of congitive maps, here.

What we didn't really have a chance to do at the conference itself was code the findings. To quote library anthropologist Andrew Asher - one of only two people to hold that position in libraries, the other being Donna Lanclos - here's how you might go about it:

“Coding these images basically involves counting the elements drawn in order to construct two indexes: a identification index, which is the number of times that an element is drawn divided by the total number of individuals participating (i.e. the percentage of the time the element occurs), and representativeness index, which is the number of times an element is drawn divided by the number of times that category of element is drawn (e.g. the number of times a study room on the first floor is drawn divided by the number of times all study rooms are drawn) (See Colette Cauvin’s “Cognitive and cartographic representations : towards a comprehensive approach” for additional discussion). I also constructed a temporal index for each element by coding the three colors in order (1 = Blue, 2 = Green, 3 = Red) and calculating the mean value for each element (you could do more complicated things by combining the indexes if you are mathematically inclined, however, I’ve found that these three get at most questions).”

— Andrew Asher | www.andrewasher.net

Observation

Observation is exactly what it sounds like - you occupy a space in the library or wider campus and you note exactly what's going on and how people use the space. In our workshop we were encouraged to focus on the location itself, the pathways users took through it, the interactions they had (both with other people and the objects and machinery), and the tools they used.

I drew a heat-map, or rather a behavioural map, using the Paper app on my iPad - tracing the lines of every single user who came through the library over a set period of time. You quickly get a picture of how people move through the space. (Not a single person went down to the basement...)

I can't draw at the best of times, and this was done using my finger - so my handwriting which is usually terrible is now completely unusable, and the weird mass of colours near the bottom left is me repeatedly trying to get the action right to bring up the colour selection tool - but I really enjoyed making this!

The idea of course is to build an understanding of how the space is used, and then adapt the space to better suit the reality for the users. For example, at Judge Business School they moved the digital display screens so they were easily viewable from the 'desire line' - the path most students took through the library. There's more on this in slides 13 - 19 of Andy Priestner's deck here. (That presentation is also a great introduction to ethnography generally.)

Interviews

The key thing about interviews in this context (as opposed to the more traditional focus group methods libraries often use) is asking completely open ended questions. I mentioned this in my post about Donna's keynote: if you ask how a student writes an essay, you get a potentially more illuminating answer than if you ask 'what library resources do you use for an essay' or 'how do you use the library'. All of this makes the data you get messy and harder to process, but ultimately in my view more worthwhile.

Touchstone Tours

Just as the Cognitive Mapping gets the user to draw the map rather than view the library's own, Touchstone Tours ask the user to take the librarian on a tour of the library, rather than the other way around. By the user telling you how the tools, systems, building and spaces work, you get to truly understand how it feels to be a user without the in-built knowledge and understanding we have as library staff. (Our tour subjects didn't take us down to the basement because, for them, it really didn't matter.)

Love-letter / break-up letter

I'm reluctant to try and describe this one in too much detail as I wasn't in the relevant session and we didn't end up using it in my team. But it involves writing a love-letter to the library if you like it, or a break-up letter if you don't. When it works well it gets to the heart of the user experience, and helps understand the emotions Matt Reidsma talked about in his Keynote on usability.

There is more info on interview techniques, touchstone tours, and the love letters, in Georgina's workshop slides.

Anything I've forgotten, let me know!