Reforming Search: Both With & Without Claude

AI-powered search is everywhere now. Google has embedded AI-powered search so effortlessly into our lives that we do not even question what's happening behind the query box. Determined to unfold this mystery, I started diving into AI-powered search on a random weekend. I wanted to create something with Claude but I did not want Claude to create it, if you catch my drift. By the time I finished my demo, it became so interesting and live as much as I felt alive!

I started with the basics, integrating semantic matching for logical sense and lexical matching for keyword matching. I wanted both of the matching algorithms part of my search algorithm as I knew one of them independently would not guarantee me satisfactory results. I integrated their individual search results and ranks using the reciprocal rank fusion function. I used the Flask API to fetch me 20 set products and used a free foundational and embedding model did not give me promising results. I knew my own testing would not be enough so i wanted my search to have a run-eval workflow, one specially that test for relevance. I directed Claude to build it after I laid down the parameters for the pipeline.

Once, I had a benchmark score, then it was all about optimizing the search. I noticed there is no gender distinction in my search results. I wanted to learn how is it dealt by typical search models and in what unique way I would establish it in my demo. The Claude implementation humbled me. It crafted some gender specific queries and calculated the cosine similarity of the new queries from these pre-documented queries to put into simple terms. It also kept threshold and margin values. Threshold to ascertain if a query really belonged to a gender class and margin to signify that a query substantially leaned towards one gender class over other. It established all of this by re-using the same embedding and foundational model initially established.

The next stumbling block was too many products being displayed for a query. Even if a product was remotely connected to the query, it got flashed on the screen. I wanted to remedy this. I asked Claude to limit the number of results being displayed. It gave me a lot of ideas on how to establish this. Finally, I choose a solution that could be integrated into my workflow easily. After my demo runs its hybrid search and accounts for gender, all the results being produced at this stage would get re-ranked by a cross-encoder which is running an algorithm to fetch the top 5 results. The problem was not solved however.

I searched for "Gift for Dad" and only one result was relevant out of the 5. The next task was to make the demo dynamic. I guided Claude towards this positioning. The solution was simplistic and genius. If results are within a margin of the top result they made the search results, otherwise they got filtered out. The precision increased from 0.52 to 0.74 in a single pass. That made me think what if the top result is not relevant. Luckily, I searched for a query called "Halloween Costume" that had no corresponding relevant product in the assortment. Ideally, no results show have been shown but Claude had hardcoded a top result approach into the algorithm.

What followed even astonished me. I have never seen Claude struggle like this and I was proud I had worked it so. It brainstormed a different kind of semantic search, setting a threshold to filter out results naturally and pre-seed categorization like it did with gender specific classification but this time with 4 classes. It failed. It was ready to give in when I prompted it one last time. It worked! Claude deduced a standard deviation/variance approach where the the relevant results clustered far away from the average scores in the distribution. So, it set a threshold. The actual mechanism thresholds the spread (standard deviation) of the whole shortlist, not each product's individual score. That distinction is the whole point of why it works when per-product thresholds didn't. To test itself, Claude even came up with a 39 queries calibration list and performed convincingly on the list through this new approach. Also had to adjust recall since recall was being tested on at least 5 results being present but if you only have 1 relevant item being shown. Testing recall on 5 would give you sub-par results. I learned that even testing has to be dynamic. I had studied standard deviation and variance in the statistic class in high school, undergrad and grad school but had never seen it applied it like this. I think this is the beauty of AI and working with AI which people often underestimate or under-utilize. It opens unseen avenues for you.

Luckily the next thing I looked up was " Show me everything" in this new search bar demo I was building with Claude. It showed some specific products. I wondered why not all? That makes sense after all, doesn't it? I prompted Claude and it incorporated my request, using the same logic as established for the gender specific results but this time it did not need a margin to prove the queries alignment with any other class just one " Do we show everything ?". It induced this check before running any hybrid search or re-ranker to optimize for results. Again seed queries such as "bring all products" and " show me the catalog" among many more. If anything pertained remotely and statistically to these seed queries, the whole assortment would be shown without needing any kind of search at all.

Next, I wanted to account for the seasonality effect namely summer and winter clothes. Most of the results were consistent but a summer shirt kept on showing on winter catalogue and "winter scarf" presented results even though it was never a scarf being displayed. For the seasonality effect, Claude tested 5 approaches namely the same standard deviation with margin approach as it implemented for "halloween costume", kneedle knee detection, tail noise-floor z test, cross-signal corroboration and widening the standard floor for these entries which basically means finding a threshold numerically significant to allow search results past it, where each result gets allotted a number. Similarly for the "winter scarf" problem it looked at token coverage, out of vocabulary, per word semantic decomposition and also switched to a larger embedding model on my request. Nothing worked and we had to document these cases as limitations. The problem is general-purpose embeddings are trained for broad topic similarity, not fine-grained product-type discrimination. Having such a small set of products closely spaced to each other in the vector space and not having a fine tuned model to deal with them limits what Claude could establish in this problem space. Although I would love to extend my demo to bigger problems with more resources, keen to see how it scales.

The wonderful thing about working with Claude is efficiency. It recommended when I asked how could we make our demo better to re-use the 39 queries calibration list it wrote to test relevance for the whole demo workflow from scratch to sit alongside the evaluation pipeline. I wanted to close out the demo at this stage but asked Claude for any final integrations and it took care of error handling and wanted to include a regular expression based matching algorithm to filter for pricing. The solution did seem tech-savvy but I wondered if we could just have a filter ad-hoc on the UI and recommended that to Claude. It did a cost vs effort analysis on both approaches and determined that having UI filters is the cheaper and faster solution. Now we have max and min price filtering, part of the user experience. Finally, I got one over Claude!

Here is the final flow diagram:

Building the search logic was half the project. Getting it live for anyone to try taught me just as much.

I planned to host it on Hugging Face Spaces, but their pricing had changed since I last checked, and the free tier no longer supported what I needed. Free is a moving target, worth checking again before assuming it still holds.

I moved to Render instead. The demo broke right away for a reason that had nothing to do with my code. Fake Store API, the data source my demo relied on, started blocking requests from Render entirely. Two services that had simply never been tested together.

The fix ended up better than the original design. Instead of pulling data live every time, I saved a copy once and used that. My search tuning already assumed that data would never change, so a live feed was a weak point before it even broke.

One more wall came up after that. The free tier on Render had a memory limit, and PyTorch was pulling in far more than the demo actually used. I switched to a lighter version of it, and it fit.

None of this was a mistake. It is the ordinary cost of moving from something that works for me to something that works for anyone, and it has little to do with how good the underlying idea is.