We recently faced the challenge of developing a web application to showcase a large dataset with over 100,000 entries. The application needed an intuitive, easy-to-use search function that allows users to efficiently find information. Additionally, the app features an elegant interface for data visualization, enabling users to easily comprehend the dataset and its purpose.
Building a reliable and performant search proved to be the most challenging aspect of this application. We felt it was important to share some of the issues we encountered and the lessons we learned during this process.
We chose Algolia to power our search due to its robust features and flexibility. The front end of the application is built in Nextjs, and the database is stored on AWS. The implementation presented interesting challenges (just like we love it), which we overcame through various workarounds. In this article, we'll share the most significant issues we faced and the solutions we devised.
Product Requirements
Before exploring the challenges and solutions, it's crucial to outline the key product requirements:
- The data search needed to accommodate two different data entries: Name and Location. This required two distinct search boxes that worked in tandem.
- Users should be able to filter results at any time using a series of filters in the sidebar of the search result page.
- The Name search box, when clicked, displayed a dropdown with filtering options, allowing users to see filtered results upon selection.
- For the Location search box, as the user types, a dropdown list of relevant location options appears for selection.
- Users should be able to sort results using at least three different options.
- The search box remained visible at the top of the page, even outside the search result page.
Why Algolia?
Algolia is a search-as-a-service platform that offers search engine solutions for individual websites. With over a decade in the market, Algolia has become well-known and widely used on the web.
From the outset, we recognized that creating a robust and user-friendly search experience was vital for our web application's success. Given the variety of search services available, developing a search function from scratch seemed impractical.
After evaluating several options and building a small proof of concept, we chose Algolia. Its comprehensive service offering—including features like autocomplete and faceted search—allowed us to leverage out-of-the-box functionalities in developing our application.
Algolia struck the right balance between a powerful search engine and an appropriate level of complexity. It's crucial to select a solution that neither lacks essential features nor is overly engineered for the task at hand. An ill-fitting choice can quickly become overwhelming and expensive.
Challenges
Limited to One InstantSearch Component per page
Given our use of Next.js for the application interface and the project's search requirements, we opted for Algolia Instant Search. This choice was driven by its rapid and precise search results, along with features such as typo tolerance, faceting, and autocomplete.
Our initial prototype, featuring a single search input with Instant Search, successfully incorporated filters, sort options, and autocomplete. We leveraged native InstantSearch solutions, such as the RefinementList component, to maintain compatibility and ease of integration, even though custom divs would have offered more UI control.
Implementing sidebar filters and pagination with InstantSearch proved to be very straightforward—we simply passed a few props to the SearchResults component. The RefinementList component was particularly useful, enabling us to create a filterable list of facets. It automatically queried Algolia for available facets and values, updating search results as users selected filters.
Overall, implementing these core features out of the box took just minutes, with the components handling querying Algolia and updating the UI automatically. Right from the beginning, we were able to have the Names and all related filters up and running.
However, we encountered difficulties when adding a second search bar for the location. Algolia supports only one InstantSearch component per page, making the integration of two searches unexpectedly challenging. Despite attempts to overcome this limitation using refinement lists, query suggestions, sorting, and multiple-form autocompletes, we were unsuccessful.
We transformed the additional Location search box into a RefinementList—a basic widget for faceting or filtering search results based on predefined attributes. This allows users to narrow down results by selecting specific filters. We added states, cities, and zip codes as facets, enabling users to see instant options with similar names filtered by location as they type. This new approach reduced the number of queries and eliminated the need for a second search bar using Instant Search.
Sorting Uses Different Indexes which can triplicate storage need (and costs with Algolia)
An interesting aspect of sorting in Algolia is its use of separate indexes for each sort option. For instance, there might be a posts_timestamp_desc index storing posts sorted by timestamp in descending order, and a posts_title_asc index with posts sorted alphabetically by title.
When users select a different sort option, the search queries the corresponding index. This approach allows for pre-sorting on the server side, rather than sorting client-side after fetching results.
The downside is that maintaining multiple indexes for each sort order requires more storage space, which, in our case, meant additional costs with Algolia. However, the performance benefit of server-side sorting typically outweighs this drawback.
Recognizing that sort orders use separate indexes explains certain behaviors when switching sorting options in Algolia's instant search. It's a straightforward yet effective technique for fast sorting at scale.
Implementing Dropdown Filtering and Preserving Filters During Redirects
The web app features a search-first experience with an ever-present search bar. When users click on the Name search box, a dropdown with filtering options appears. They can select an option to view a filtered results list. We aimed to provide options directly from the search box. If the user starts typing in the Name search box, they will see the instant results below, and clicking on them will redirect them to the corresponding page.
Implementing this dropdown filtering required a workaround to Algolia's standard functionality. We used redirects to build this feature so the user would be taken to the search page with their chosen filter already applied.
However, we encountered an issue when implementing redirects from other pages to the search results page. Searches and filters on these pages would redirect to the main search page, but all filters were lost in the process.
The challenge stemmed from Next.js's push() method, which deleted search filters upon redirection. After some trial and error, we found a simple yet effective solution: using the classic window.location.href instead of Next.js routing. By assigning the new URL directly to location.href, we successfully preserved all search filters during redirection.
Filtering options changed during the search experience
As mentioned earlier, implementing side filters was straightforward. However, we encountered a minor challenge: by default, Algolia filters adapt dynamically as users apply them. Our client preferred a static approach, wanting users to see all available filters throughout their search, even if certain filter combinations yielded no results.
We had to find a workaround for this default Algolia behaviour. We manually customized the order of the filters shown on the custom Refinement list, and we called the custom order for the initial facets on the RefinementList.
By manually setting the filter order and applying it to the initial facets, the team was able to ensure that all available filters remained visible to users throughout their search, regardless of the results returned by specific filter combinations.
Conclusion
Building a search experience comes with numerous challenges. While Algolia InstantSearch offers a powerful starting point, limitations—particularly the constraint of using a single component per page—created roadblocks.
Overall, the project provided an opportunity to deeply understand the intricacies of the Algolia search. Key lessons learned include:
- Start simple—don't attempt to build complex, customized search until the basics work
- Understand the full search architecture, including how indexes and facets interact
- Budget for potential query costs during development and testing
- Leverage InstantSearch strengths like pagination while finding workarounds for limitations
While challenging, developing a custom search experience led to significant learning. With careful planning and a simplified initial scope, the process can be smoother for future projects.