Chasing the Next Transformer Killer - Part 2

Jul 21, 2024

This is the second part of a post about research toward faster Large Language Models; for the first part, click here.

In the previous part, we looked at efforts that improved the asymptotic complexity of traditional transformers. Where the improvement efforts become exciting is when researchers try to combine the advantages of attention mechanisms and RNNs. Yes, you heard right: there is a way to bring recurrent neural networks back into the picture while still allowing the parallel training of the neural network (to overcome the serial nature of RNNs). The most prominent pioneer of this idea is RWKV (pronounced “RwaKuv” and named after the four major parameters: R, W, K, and V).

It is defined as “Parallelizable RNN with Transformer-level LM Performance” on their main GitHub page. To understand more of the details behind this marvelous model, I recommend two sections of the Hugging Face RWKV article:

  1. Transformer Architecture vs RNNs
  2. RWKV attention formulation: “The model architecture is very similar to classic transformer-based models (i.e., an embedding layer, multiple identical layers, layer normalization, and a Causal Language Modeling head to predict the next token).”

According to the RWKV authors, it is the leading sub-quadratic transformer architecture alternative:

RWKV models can compete with models one category larger; for example, a 3-billion-parameter RWKV can be as fast as 7-billion-parameter transformer models while performing just as well on benchmarks. Not to mention the smaller footprint, which makes edge deployment on mobile devices much more feasible, for example. I was experimenting with MLC-LLM and tried some 7-billion and 3-billion models. A 7-billion-parameter model stretches the capability of today’s phones and can only provide a few tokens per second of inference, while the 3-billion-parameter model can run 10 times as fast. I have not been able to test 7-billion-parameter models with Google MediaPipe mobile edge apps yet, but the 2-billion-parameter Gemma (especially the gemma-2b-it-gpu-int4) worked well.

I was so interested in RWKV that I attended a tech meetup about it in San Francisco, and I discovered that there’s even more to it than just being very efficient. The community around RWKV is very international; it’s much more popular in Asia than in the US. These international roots resulted in a naturally better-performing token vocabulary and tokenizer. If someone tries to use OpenAI’s tokenizer with a non-Latin alphabet, they incur a 3x+ token count blowup. It is somewhat like encoding Chinese or Japanese alphabets with UTF-8. It is well known that generative AI services are not cheap, and if an API bills by token count, this results in a 3x+ cost increase. It is also slower, because inference speed is proportional to prompt length: the longer your prompt, the longer it takes to generate an answer. It doesn’t have to be like that, though.

The vocabulary is an extremely important part of a transformer model; for example, the BloombergGPT developers realized that number representations were not ideal with the traditional OpenAI tokenizer, so they custom-trained their own tokenizer, which handles numbers better. This piece is key to why BloombergGPT performs better on its specialized tasks. Consequently, RWKV’s tokenizer is friendly to non-Latin languages and alphabets. Non-Latin alphabets only incur a 1.5–2.5x increase, while the English alphabet still stays at 1.5x only.

On the other hand, one cannot simply increase the vocabulary to an arbitrary size, because certain algorithms have quadratic asymptotic consequences (for transformer-like models, though not for RWKV), and one would also need much more training data to cover everything well. The availability of good training data is a bottleneck for today’s large language model training. The sweet spot for vocabulary size seems to be between ~32K and ~64K symbols, according to the RWKV researchers.

Another interesting development along the lines of RNN-based models is the introduction of Recurrent Gemma. This underlines that even Google DeepMind (the inventor of the transformer architecture) believes that a mashup of RNNs and transformers can be beneficial and promising. Recurrent Gemma is based on the Griffin architecture and mixes gated linear recurrences with local attention. Fortunately, it is open-source just like the other Gemma variants, and I cannot wait for follow-up versions. These cutting-edge models can play a crucial role on edge devices such as mobile phones. In the future, our computers and phones will run models locally, providing offline capabilities, eliminating server roundtrips, and increasing privacy.

Comments loading...