In my professional development as a Microsoft Consultant, I’ve been focusing a lot lately the human factors that contribute to the success and adoption of a technology solution. IT Directors and CIO often ask, “How do I get people to use this new techno widget we are building?” When I was a baby consultant, my answer was clouded with my own passion and experience with Microsoft enterprise tech. I pictured this Field of Dreams vibe, “Well, this solves some problems for users, so if we build it, they will come.” But sometimes, I’d see that was not happening.
I’m a curious guy, and I’ve often found inspiration for problems in fields of study or domains outside of Information Technology. Had others studied this problem, or at least something similar? This curiosity led me to the fields of behavioral psychology and behavioral economics, reading the works of pioneers in the fields like Richard H. Thaler and Daniel Kahneman. At the heart of these fields are interesting questions and proposed answers, all around how and why people make decisions and evaluate choices. Understanding the thought processes of the people who use our technical solutions help us as solution architects present the new solution as an easy decision choice over continuing to use the old.
As I was trying to work through the thought exercise of putting a cost in dollars to NOT having a great intranet, I went searching for studies related to the wasted effort employees spend with poor or no search capability provided to them. What I found was a remarkably interesting study that explored WHY users are unhappy with enterprise search experience, and I’m excited to share what I learned.
Thing 1: There’s a lot less academic research on enterprise technology than I expected.
Ok, so the paper I found was called “Enterprise search and discovery capability: The factors and generative mechanisms for user satisfaction” by Paul H. Cleverley and Simon Burnett. It is 100% worth a full read, but it’s a long read, probably 90-120 minutes end to end. I was able to find a lot of case studies on enterprise search, but this was one of the few I could find where researchers were repeatedly examining satisfaction of a large user community over a couple of years, while following up with interviews to uncover the “Why” behind a positive or negative response to using enterprise search. I think this quote from Cleverley/Burnett highlights why we need more of these studies to understand the enterprise search problem:
Existing enterprise search studies tend to be reductionist, focusing on the ‘parts’ or single disciplines, not the interconnected transdisciplinary ‘whole’ of Enterprise Search and Discovery capability, which could be described as a system. The emergent nature of outcomes and how they change over time means in an open system, it is likely they will be poorly understood by simply studying constituent parts; it is the interaction between all the parts that may determine search task outcomes.Cleverly and Burnett
Thing 2: Understanding a complex system means looking at everything.
Cleverley/Burnett set out to understand the obstacles for effective search, and to start they aimed to identify the factors that cause them. There are a few terms that are important to understand, then we can dive into what they found!
- Factor – A factor anything that we can see and measure to have an influence on search user satisfaction.
- Cause – This is a combination of multiple factors, which in their presence (or absence) influence search user satisfaction.
Generative mechanism – The “ultimate cause”, something that can’t be observed but leads to the combination of factors (cause) that influences search satisfaction. These are dependent on the specific situation, the explanations uncovered are more tendencies than hard truths.
Thing 3: What are the commonly cited factors for search satisfaction?
Research studies often hold some background information and a review of the existing literature out there on the topic, and Cleverley/Burnett lay out what they found. Let’s explore the four buckets of factors!
There is a lot of focus in information technology literature about search features and functions being THE factor in user satisfaction. The vendors and marketers have filled the space with clever phrases like “insight engines” or “cognitive search”, big promises of the power of AI. And yes, there have been some major breakthroughs in search technology. There were a lot of tech problems around performance of enterprise search, but for the most part, a lot of the hard problems with enterprise search hardware, scaling, and load speed have been solved. A lot of the advancements are now focused on improving the relevance of search results, and better understanding a user’s intent. Other factors might include natural language processing, which enables users to search by asking questions in plain language. It isn’t that these don’t matter, it’s that only focusing on tech factors leaves out other parts of the search problem.
Information factors cover how the information users search for is structured. Do documents and web pages have meaningful titles? Do we regularly clean out duplicate content, review and remove obsolete information? Do our files have meaningful metadata that help our users refine searches?
In my consulting experience, information factors contribute to dissatisfaction with enterprise search and even intranets. I have also experienced that these factors can be some of the most challenging to make improvements on. If we ask for too much metadata and tagging, then our users may not adopt our information practices. We also need to coach users not to deposit information by just throwing it into a folder or library, but to think about how users might consume info used in the future or who the intended audience is, then publish the information with the audience in mind.
Search Service Factors
Often, Enterprise Search projects are treated as a one-off project, just set it up and never think about it again. However, a search user’s needs will change and evolve over time. Organizations that treat enterprise search as a perpetual beta, regularly review search analytics, asking for and acting on user feedback, and use these insights to fine tune the search tool experience higher levels of user satisfaction. Unfortunately, I often see overburdened IT professional not given enough time to gather insights and act on them regularly.
Information Literacy Factors
Information literacy is the skills and habits cultivated by using technology. This also includes making critical judgements about information, based on the context. This goes beyond how to use the actual search tool, and into how to think about the search before entering a query, even highlighting how Enterprise Search differs from personal Google search use.
Thing 4: What’s the root cause for unhappy search users?
Another way to think about these generative mechanisms are the decisions or actions which led to negative factors that make the users unhappy with search…