No-Code LLM Agent and RAG Solution at Google Cloud Next 2024

Apr 13, 2024

(In the photo: friends we made at the Startups at the Skyfall pre-conference party, including Anna Nerezova from GDG NYC)

In the previous Google Cloud Next 2024 post I mentioned that there were two hackathons at the conference:

  1. A Kaggle competition
  2. An AI hackathon

For the AI Hackathon, it turned out that the teams would need to leverage the new Agent Builder offerings. I must go back in time to May 2023 at Google I/O Connect in Miami. I was looking around for various LLM solutions for a domain-specific knowledge-based (a.k.a. RAG) chat agent, and when I talked with Christopher Overholt he drew my attention to the DialogFlow ecosystem besides Vertex AI. One didn’t necessarily have to go to the Vertex AI Model Garden and fine-tune a model or build RAG on Search or other vector databases. At the time I exported an HTML-based hierarchical help knowledge base from ThruThink, but I experienced server errors with document chunking/parsing/indexing.

DialogFlow has a way longer history. I have been working on DialogFlow apps for a long time now, helping GDG Fresno ex-secretary Mark Simonian to port and enhance his applications. Mark is a pediatrician and he was the original expert who developed an automated telephone robot protocol to help parents determine Motrin (or Tylenol or Advil) dosages for very young babies. Traditional DialogFlow operated with explicit Intents and Entities, and in the case of applications like the dosage helper, function calling could be very helpful (called fulfillment in DialogFlow lingo). Mark’s original agent had a very deep tree structure handling all possible integer baby weight choices, whereas a fulfillment-converted version can handle any arbitrary weight (fractional weight) and the agent structure is much simpler. However, the fulfillment function requires programming knowledge; it is a special Cloud Function I wrote in JavaScript.

Classic DialogFlow was also capable of employing NLP (Natural Language Processing) AI techniques; we only had to specify a few example conversations and it was able to understand other variations of a question. This was before the GPT era. Since the transformer generative AI boom, DialogFlow has been enhanced in parallel with Vertex AI.

It had a set of offerings such as “Enterprise Search Engine” and “Generative AI App Builder”. These already refactored DialogFlow to utilize large language models and greatly simplified the configuration process to a no-code level. However, there is an even newer generation of LLM and generative AI search-based offerings under the umbrella name of Agent Builder. It provides a no-code way to define:

  1. A very complex agent by simply describing agent behavior using so-called playbooks. The agents can be hierarchical, and the flow can invoke one agent playbook from another.
  2. The agent(s) can also have DataSets:
    • Indexed knowledge bases for RAG
    • Function calls
    • API calls
    • These DataSets can be referenced in the playbooks in a specific markup way, but the whole process is very natural and completely no-code.

The engineer has knobs to tune some parameters, such as chunking size during indexing, RAG retrieval sensitivity, the LLM model used, and so on. After the hackathon was announced we were provided with some training videos (mostly recorded by Pak-Ming Wan), and I studied them at the end of the day at my hotel.

The domain

Since I was participating in the ongoing sustainability session series by GDG Tucson (Dan Stormont), and I had recently learned about SDGs (Sustainable Development Goals), I decided to download the various yearly reports from the United Nations website and index them for Retrieval Augmented Generation. The resulting agent would help answer questions and have a chat about SDGs. There are 17 SDGs, and even though I wish I had time to go over each of the yearly reports, it is so much fun to chat with a domain expert agent.

How I built it

I spent a lot of time curating indexable data for the Sustainable Development Goals. I downloaded yearly reports, extended reports, gender snapshots, and massaged data sheets to help build the data store. I used the Agent Builder interface to develop a playbook where I leverage the established data stores. I experimented with various indexing techniques, thresholds, and parameters for data indexing and retrieval. I also went through many versions of the instruction prompts to cover the intended use cases of the agent.

Challenges I ran into

  1. Data gathering and preparation are still extremely important and a foundational step even for no-code frameworks. Sometimes I needed to convert files to supported data formats.
  2. PDF documents can be very versatile and sometimes tricky to chunk and index, especially if the PDF contains figures and charts: OCR parsing might be better than the default or the layout parsing, but OCR didn’t perform well for me.
  3. I also had to disable grounding. Grounding has five confidence level thresholds, and even when I allowed very low confidence retrievals, the retrieval engine sometimes still could not perform RAG for me, so I resorted to completely disabling that filtering.
  4. It’s a testament to the breakneck pace at which Google is developing under the hood that in the tutorial videos by Pak, the product had a different name from “Agent Builder” and Playbooks were also named differently. By talking with Pak (he was there in person) at the hackathon, I got a clearer picture after a few minutes of showing him my work so far. I’m referring to another YouTube video here.
  5. I learned one of the most important lessons while interacting with Ferdinand Loesch - an engineer who works on this feature - when I showed my examples to him. Originally I simply went on manually and entered questions and answers. However, that would have taught the Agent to avoid using RAG, since my manual entries wouldn’t contain DataStore lookup entries. So the right way to build examples was to initiate a chat in the test sandbox and convert a good conversation into examples. These would contain the DataStore lookups (along with the retrieval phrase and retrieved chunks), and I could still fine-tune the user questions and agent responses.

In the photo, you can see the wrappers of the rice cake snacks the organizers provided for us. They were yummy and a lifesaver because I skipped lunch in favor of the hackathon, so my only other food source was some protein bars from the Certified Professional Lounge.

Accomplishments I’m proud of

The Agent Builder provided a no-code way of developing an agent that was able to interpret vast amounts of information and synthesize it to help with unique and creative conversation pieces.

Agent Builder provides ready-to-use connectors for a wide range of chat platforms such as Slack, Meta Messenger, Discord, Telegram, Viber, Google Chat, etc. I went on to try to integrate the agent into the Fresno State Sustainability Club Slack, but I realized I wasn’t the workspace admin.

In Summary

Google provides a no-code managed framework for developing complex agent scenarios. The details are still extremely important and it’s crucial to discover all possible options of the offered features to reach the best possible outcome.

After the Hackathon

After the hackathon submission, I kept working on the agent post-hackathon, developing a standalone Streamlit front-end (source code) and glue functions called from the front-end to reach Agent Builder. I did my research and had to come to a solution on my own.

Comments loading...