Why Search Optimization Is Impossible, and How a New Architecture Can Set You Free

Search has always been a disappointing feature.

With the rise of ChatGPT and AI chat — in addition to the rise of machine learning-based search technology — people are talking about search more than ever. Google is feeling threatened for the first time in a decade, and businesses large and small are reevaluating how customers find information on their websites and apps.

Despite this renewed interest, even typical searches remain difficult to create, maintain, and improve. What's really frustrating, though, is that everyone from the developers to the CTO agrees that search is important. However, most of the search experience remains mediocre and doesn't seem to improve the experience.

For the past decade, I've been working with developers around the world, helping them improve the search tools they build and deliver to their businesses. My goal has always been to help engineers create powerful and efficient search experiences. But over the years, I've found that one of the biggest problems is one that occurs almost before the process begins.

The scenario is simple: Engineers decide to improve search—perhaps an external team needs functionality that isn't available in the current environment. Perhaps the exponential growth in the number of users requires a new way to scale. Or maybe the features are getting old and users aren't happy -- but whatever the reason for the upgrade, it's almost inevitable that a cascade of problems will arise when engineers even touch search.

The waterfall is overwhelming, but it presents a bigger problem: misunderstanding why you're trapped. On March 30, I met withlive haystackAbout why search optimization is so hard and what exactly happens when you get stuck before you even start.

ambition meets division

Search is rarely a feature that teams create from the ground up. The engineers were at a standstill, and when their current search system (most of which they inherited) reached a breaking point, they asked me for help, decided to finally fix it, and collapsed under the weight of the almost inevitable technical debt .

A technical problem may have prompted improvements in search, but the first problem engineers faced was an organizational one: fragmentation.

A broken system will work against you

Even engineers struggling to implement a smooth implementation and thinking about seemingly simple improvements may encounter organizational issues at first. If you don't prioritize organizational challenges, the first change will be difficult, if not impossible, and each subsequent improvement will be equally difficult.

Most of the companies we work with are dealing with decentralized systems. Fragmented systems create not only chaos, but also friction and obstacles.

A consequence of the consensus importance of search is that its potential is often distributed among many different groups. I've noticed that companies rarely attribute a person or team to "owning" the search experience. This is a feature that everyone wants to provide information but no one is responsible for.

So even if an initial problem seems small and technical, project management and cross-departmental organization can quickly become a priority.

Recombination takes precedence over duplication

The instinct to make small improvements doesn't work as well as you might think here, because even small improvements can have knock-on effects of related changes and scope creep. Take these three scenarios as examples.

scene 1: Suppose you decide to make a seemingly minor improvement, such as a remap. But despite your efforts to limit this work, the problems multiply:

Do you update the index?
Are you going to reindex the same cluster?
Are you planning to put this cluster on other hardware?
Where will the material live?
How would you handle cropping without losing data?
How would you go back without losing data?
If you are upgrading the version, why not change the query?
If you're changing the query, why not change the UI so users can use the new functionality?

If you pull the thread of the search experience, a whole host of problems can arise.

scene 2: Suppose the marketing team wants search engines to capture the data they need. Again, this seems simple enough, but it can be a pain to keep track of new data without built-in support:

How do I update if you are using a browser version that does not support this metric?
Will the requirements change?
Do I need to rebuild indexes to support this new query?
Does this new query slow down population even further, making it a burning problem?
How would you define new performance intent if requirements change?
If you have new affordances, do you need to add additional functionality to the document or index?

Don't underestimate the cultural implications of such scenes. You want your marketing team to be on board, and they'll be frustrated if you can't add this metric.

scene 3: Let's say your company has grown and you need to support more users. Search is broken and you need to improve it, but scaling your current search experience is more complicated than you expected.

Divide search into two main parts: data bar and search service

The above is so common that we figured it must be an upstream issue. To find out, we looked at the history of the search industry and the history of every search experience.

We can then look back at the architecture that tends to emerge over time—layer by layer, function by function. We see the problem: sprawl.

"Architecture" is the term here, but urbanism provides a useful metaphor. Some cities, such as Washington, D.C., have intentionally designed easy-to-navigate grids, while others, such as Boston and London, have become incomprehensible intersections after centuries of chaotic growth. The buildings of the mission are more Boston-like, with streets lined with cattle tracks, leading to confusion and disorientation.

If we take the standard search architecture as it is and rethink it from the ground up, two key elements emerge: the data pipeline and the search service. By understanding these fundamental principles, we can rethink our search architecture and find solutions to the problems that make search development so frustrating.

Datapipelines create and index documents

The data pipeline includes all the work needed to create documents and the methods used to index them.

Data pipelines are most efficient when passing through buffers such as Kafka. Consumers read documents from the cache, ideally writing to multiple indexes instead of one.

An inefficient data pipeline will treat the search engine as the primary data store, which slows down the entire process and experience. The most efficient data pipelines are capable of populating search engines from a single primary data store.

What's more, data pipelines not only provide higher performance and scalability. The better your data pipeline, the easier it is for you to manage change. For example, if you need new document-level features, an efficient data pipeline will create a new version of the index so you can handle those feature changes.

Search services match user intent to search terms

At its core, the search service component takes the user's intent - as the user expresses it through the user interface - and translates it into a query language that the search engine can understand and use.

As with data pipeline components, viewing the search service as a component in itself reveals both efficient and inefficient versions.

A failed search service simply communicates search intent.

Instead, an effective search service can store intents in two types of queries—one for processing queries as-is, and one for experimentation and iteration. The first query is a control that executes in the same way as the previous queries. The second query supports iterative experiments such as A/B testing.

Ideally, you support a near-constant level of A/B testing so that you can always make data-driven improvements. But in addition to iterating incrementally, the ability to handle multiple queries is also important for larger changes.

For example, if you want to migrate your search engine, you need a search service that queries your current search engine and sends a second query to the search engine you are testing.

Build the foundation from the ground up to pave the way for improvement

Considering the two key elements of the data bar and the search service, we can define the basic expectations that the search functionality should fulfill.

Creating baselines allows us to set expectations for all involved teams and hold them accountable for what the least efficient search should deliver. But baselines also provide the basis for us to plan for further improvements and commit to long-term improvements.

The end goal is not perfect search, which is practically impossible, but search that can be easily improved—a search experience that doesn't get bogged down in a series of problems every time you try to make small improvements.

Map and create processes and dependencies

Basic search functionality relies on a series of processes, each linked to the next, and all of which must work. Many of the sequencing issues I mentioned earlier stem from incomplete baselines or dysfunctional components of the baseline.

Start: Search interface.

The search interface expresses search intent. Your application must be able to interpret user data and format the interpreted intent into a query language, which it can then send to a search engine.

The search engine must then be able to run that query through the server-hosted index, no matter where the server is located. The search engine must also be able to enrich this query with information about the user, such as the user's location and any relevant information from their previous sessions.

But building a search interface that effectively expresses intent requires building an index.

The index is essentially a preliminary calculation for anticipated future queries. For this reason, when creating an index, it is important to first understand the final state of the query—the results that a particular user will eventually see.

Starting at the end will help you organize the features you're building, making sure your index has everything it needs to get to that endpoint.

A search interface or index is only useful if it is available to users, so scalability becomes another key factor. Scalability is deceptively simple: you often need to split into pieces so that more computers can do the job than a single computer could do alone.

This baseline is clearly incomplete if the user cannot see the results. However, companies rarely need raw results, so even before they are presented, the underlying system should be able to filter and reorder results based on business needs.

From there, you can present filtered results to the user, who can then interact with the results and UI offerings through the UI.

This loop connecting the query interface to the results and back to the interface is complete, but even the basic experience needs some level of improvement.

It is inevitable that the following problems will occur:

What are users doing, what are they clicking?
Where is each click and how long does it take for the user to click?
Is every click driving a business-relevant result?

Without fundamental improvements, even initially successful searches eventually fail.

Improvement depends on disconnection

Thinking through the baseline above reveals many of the processes and components necessary for the baseline experience, but zooming out also reveals the big picture: closely related systems.

The question of whether systems are too tightly coupled has sparked countless engineering discussions. Probably the whole discussion is based on a misrepresentation of the problem. Marianne Bellotti, a legacy systems expert, states in her bookkill it with fireThe software typically leverages tight coupling early in the business lifecycle, but will benefit more from refactoring to a loosely coupled system as the business grows and grows.

There are many companies with poor search experience. When a search system is tightly coupled, the various components depend on each other, and a change in one component may require a change in another. This level of coupling limits upgradability and repairability, meaning that improvements to one component require improvements to other components, and changes that fix a single defect can create a cascade of further defects.

Suppose a company creates a user interface that, when expressing an intent as a query, translates that intent directly into the query language of a search engine. This immediacy creates a strong connection between the interface and the search engine. This process may be good enough for the company now, but if the search engine is to be changed or replaced in the future, the UI code will also need to be changed.

The bottom line is that when you design, build, or map the state of your search experience, you need to consider the realities of your opportunities for improvement. In the above scenario, replacing the search engine is still possible, but it is difficult, and if there are other priorities - and there always are - this improvement may be delayed. At some point, a system may become so tightly coupled that it can no longer be effectively improved.

Splitting search into two parts—the data pipeline and the search service—keeps the system disconnected. The ultimate goal is a system that can be iterated on, one where components can be improved without organizing the entire search infrastructure.

Work with the base instead of fighting the spread

With the right architecture, the right core functionality, and a design that prioritizes iteration, you can start improving your search over time and, perhaps more importantly, make sure you can make changes without causing problems to escalate.

This feeling is important because the easier it is to improve, the more likely you are to achieve it. And, the more you suggest and act on improvements, the better your search results will be and the better your team and company will feel.

It's not necessarily an easy path, and the jobs I've described so far involve delaying gratification and rewarding initial effort. But it's worth it, and it's worth imagining some possible improvements.

Better A/B Testing

An effective search service translates user intent into queries, allowing you to A/B test the accuracy of those queries. By continuously running these parallel queries, you can measure user engagement over time and fine-tune your search service to consistently improve intent detection and translation.

Redundancy monitoring

A common pattern of wrong search results is when a user performs a search and the search returns unnecessary results. Redundancy can make users feel like they're using an inaccurate search function, or that the results they're looking for don't exist (even if they do). With proper metrics and monitoring, such anomalies can be searched for and logged so that you can investigate, replicate, and track redundancy back to what might have caused it.

similarity test

There are many ways to detect and measure similarity (some of which include ElasticSearchdocument). The purpose of measuring similarity is usually to ensure that you don't present users with results that appear to be intended based on superficial similarities (such as similar text). With advanced search capabilities, you can apply ranking algorithms to score results based on finer-grained similarities including syntactic and semantic content.

improve performance

Efficiency is one of those values that many consider important but few prioritize. Efficiency is hard to come by, and it's unclear what significant results such efforts will yield.

However, I try to remind people that even small performance improvements can lead to permanent user experience changes. In some cases, speed may be more important than accuracy, as responsive search encourages users to repeat searches and find the right combination more quickly.

If there is a large influx of data or users, performance can quickly become a big problem. However, by using a better schema for searching, you can more easily change server types and change configuration methods. Ultimately, it becomes relatively easy to make changes individually or rebuild the entire cluster as needed.

Intermediate results

Better search architecture also provides flexibility. In the first example, I mentioned the possibility of A/B testing, but that's not always possible - no matter how good the search is. The main problem with A/B testing is that you need a lot of data for any results to be statistically significant.

But with a well-designed architecture, you can interpolate the results, providing a cheaper form of iteration. By inserting results, you can run queries on both the control and experiment sides to combine the results and present them to the user. You'll still get great results, but the process requires far less commitment and action to generate meaningful signals.

break free don't get stuck

In this post and in my talk at Haystack, I've tried to emphasize the optimistic view of re-architecting: it's more possible than you think, and with proper thought and planning, you can get to the point where the pursuit is no longer is characteristic of fear.

But I want to end by pointing out some of the costs of not improving search, many of which have clear but hard-to-quantify consequences:

All systems wear out over time, and the longer it takes to redesign search, the harder it will be and the worse the search results will be.
User expectations can and will change—sometimes quickly and dramatically. A task that once seemed "good enough" can quickly become inadequate.
AI and ML search are evolving and improving every day, and even if you can't implement them today, having a better foundation will allow you to implement and integrate new search technologies more efficiently and effectively.

its authorPhoenix ProjectWhen they write:

“As our work becomes more intertwined, everything becomes more difficult, one step at a time. Smaller actions lead to bigger failures, and we become more afraid and less tolerant of making change .The work required more communication, coordination and approval. Teams had to wait for their dependent work to finish. And our quality kept getting worse. The wheels started rubbing slower and required more effort to turn."

When the search experience is as bad as it used to be, problems become harder to spot and harder to improve — the wheels grind and grind until they stop. Between those two gaps, it's hard to see how good search is.

Here's why I'm finally back on optimism: If you take the time to refine your search from the ground up, the payoff will be greater than you expected.

Why Search Optimization Is Impossible, and How a New Architecture Can Set You Free - Bonsai (2023)