LLM Knowledge agent with RAG Fusion on Cohere platform with LangChain and Streamlit
Feb 11, 2024
ThruThink is an online business budgeting app for business projection and forecasting. It is a product of literally decades of experience and careful thought, and thousands of calculations. It got its name from thru-hiking, or through-hiking: the act of hiking an established long-distance trail end-to-end continuously. When budgeting, you need to carefully plan and account for everything, just like when planning a thru-hike.
It is a startup in the incubation phase, and there are no dedicated team members for support chat agent roles. At some point we had a “classic” human chat agent integration; however, that’s on hiatus now, and it would be great to supplement it with an AI chat agent given the rise of large language models.
For this variation of chat agent, I continued from the QnA Boosted RAG concept of the Vectara hackathon.
Cohere is another company that offers Retrieval-Augmented Generation services, but they also have their own state-of-the-art large language model called Command R (and lately Command R+). Just like many companies, they are also moving towards higher-level managed services in this space: their service takes care of a large portion of the RAG pipeline, including chunking, re-ranking, and more.
The most prominent feature of the current set of experiments (this blog post) was RAG Fusion. RAG Fusion takes the original query, generates variations of it, and performs the RAG procedure on all of those variations. Then we employ reciprocal rank fusion on the combined retrievals and their ranks from the variations to determine the final ranking. The variations should stay close to the original query (or rather shouldn’t stray too far); however, they preferably shouldn’t be too similar either, so they wouldn’t just retrieve duplicate results.
The conceptual idea is twofold:
- Sometimes users themselves don’t know what the proper question to ask is about their issue.
- In the case of sparse data, some variations might match indexed chunks better and could potentially uncover more matching results.
These are hypotheses.
As for the technology stack, Cohere’s co.chat has two modes:
- Document mode, which essentially means RAG
- Connector mode, where it can perform a web search to supplement information for an answer
That would perfectly cover our goal for the agent: to be knowledgeable about ThruThink specifics and also be well-versed in general accounting and business projection concepts.
One of the most interesting technical challenges was a prompt engineering one: generating variations is trivial at the beginning of a conversation. However, once we are in the middle, a conversation can contain several contextual references to previously mentioned information. We wouldn’t want to clutter the complete prompt with the entire history; we only want to resolve any context references in the latest user query. I had an extremely hard time achieving that originally; various large language models were performing poorly until I crafted a few-shot prompt. This was an essential early lesson about how crucial few-shot prompting and prompt engineering are alongside the other technical components. This realization would strike me again at my first Kaggle hackathon.