Personalization is what I missed at the Samsung NextGen AI hackathon
Sep 25, 2023
The title of this post is meant to rhyme with “Attention is all we need”. After not pursuing multimodal ideas in favor of business at the Intel Innovation Bridge hackathon, I was keen to do something multimodal. Samsung organized the Next Gen AI hackathon at the breathtaking, historic Fort Mason Center for Arts & Culture.
Besides the historic buildings, we had a view of Alcatraz, and there was even another tech event taking place in an adjacent building.
I started recruiting a team on social media channels, and though I cannot remember from where, Yiru and Coco He expressed interest. The maximum team size was four and they also knew a mobile engineer, so with that, the team was complete. A few weeks before the event, our mobile engineer got stuck in China due to COVID-19, so I managed to recruit Kevin Moore, GDE (Google Developer Expert for Android and Flutter), for the front-end role. The team:
- Yiru and Coco He: product ideation, management, presentation
- Kevin Moore: front-end
- Csaba Toth: back-end and Generative AI expertise
I must emphasize before getting into the details of our project: all of my peers showed such professionalism and dedication that I would hands down work with them anytime in a professional setting.
We threw around ideas before the event. I was up for anything; I just wanted something multimodal. Coco and Yiru like to travel and were thinking about a travel agent app tailored to digital nomads and travelers. They homed in on the problem that we all have trouble organizing our photos and presenting them to friends or a wider audience on social media.
Coco and Yiru put the product concept under scrutiny and interviewed several people, and we landed on an agent that can automatically categorize photos and easily write blog posts and social media entries with minimal instructions. For example, the agent can sense sunny versus cloudy days and reflect that emotional arc in a blog post. But primarily, the agent groups photos by keywords and then summarizes the experience of the matching photos. It can also group by location or date range.
The front-end was an Android mobile application written in Kotlin using Jetpack Compose. This left open the possibility of porting it to iOS, the web, or other platforms via KMP (Kotlin Multiplatform) and CMP (Compose Multiplatform).
As for the back-end, I immediately hit serious roadblocks. I was planning to use Google’s Imagen, which is an image + text multimodal model offered on Vertex AI.
- I would use the multimodal model’s descriptive power to generate very detailed descriptions for each image we processed. The same model would also generate keywords related to each photo. This was a precomputation step for the app: each time an image landed in media storage, a Google Cloud Function was triggered to perform these two actions. We stored the resulting data in Google BigQuery.
- Later, the app user would specify a desired date range, and the back-end would present the unified set of keywords for the images in that scope. The user could then pick a keyword, and a blog post would be generated along with the corresponding image references.
- For the blog post generation step, I used the PaLM 2 LLM (Large Language Model) via a MakerSuite (now Google AI Studio) API key because it was so much easier to set up and use than the Vertex AI Model Garden method.
The foundational problems were:
- Imagen was extremely terse when describing our photos, no matter how much prompt engineering I tried.
- Then I thought I would use an open LLaVA - LLaMA2 model. I looked around quickly on Hugging Face, Replicate, and AnyScale, but was not able to find a model that I could productionize to be called via an API. One of the main problems was how to prompt such a model. At that time, Hugging Face’s transformers library did not support it natively.
- There were AWS engineers on-site and I thought I might utilize Amazon SageMaker; however, it turned out that I didn’t have a high enough GPU instance quota. Today that GPU shortage has eased, but at that time, my quota request was approved too late for the hackathon.
- I tried to reach out to Intel engineers because their LLaVA LLaMA-2 model was running super fast on Intel Cloud—they had optimized everything for Intel Max server GPUs and Habana Gaudi2 accelerators. However, I was not able to connect with them over the weekend.
- Finally, AWS engineers recommended an image+text Generative AI service called Chooch, which was able to describe the images with the great level of detail we needed.
- I should also mention that I communicated Imagen’s terse behavior through multiple channels to Google, and the newer Google Gemini models overcome this terseness.
Coco and Yiru did an excellent job preparing our pitches and managing our progress. We didn’t know it at the time, but the evaluation didn’t put that much weight on project pitches. We didn’t make it to the finalist round, but I gained extremely valuable input at the food-truck social as the event was closing. I wanted to pick Arthur Soroken’s brain (MakerSuite fell under his purview at Google, so he was delighted to hear I was utilizing PaLM 2) on what we could have done better. He pointed out that we were missing a personalized experience. At first, I didn’t understand, since we were already working with very personal photos. But then I realized that we had essentially missed incorporating RAG (Retrieval-Augmented Generation).
Here is an example: let’s imagine Yiru visited Stonehenge ten years ago when the weather was cloudy and rainy. When she revisits Stonehenge ten years later, our software recognizes that she was there before, and the generated blog post can reflect that: “After 10 years, fortunately you had a perfect sunny day at the historic site” (or something similar). To implement that, we could have added extra information embeddings, vector database indexing, and RAG. When preparing the prompt for blog generation, we could then include relevant retrieved experiences from the past. RAG is ubiquitous these days, but I had developed tunnel vision while scrambling to get the other parts working. Even right there speaking with Arthur, I realized what great advice he had given, and I’ll always remember it: personalization!
It was interesting to see other teams’ versatile projects. One that captured my attention was by two Stanford students who combined art, Augmented Reality, and Generative AI. They presented semi-interactive artwork in an augmented reality format. Another project fine-tuned on websites you visited on the fly, allowing you to converse about their content with the power of LLMs. After almost everyone left, only three attendees and the organizer remained chatting in the hacker room. It was an experience I’ll never forget.
Let me close with Coco and Yiru’s gift: Chinese mooncakes. I tasted both the red bean paste and lotus seed paste varieties, and they were delicious!