Future Telescope 2
GPT And Other Transformers
This Blockhead track is great to play in the background as you read this edition:
My friend Saurav and I sat over the weekend and tried fiddling with the “infinite Shakespeare GPT” code that Andrej Karpathy has uploaded very kindly on Google Colab. It took us a few hours of tinkering (it was my first time using Google Colab and Github), but we eventually managed to train the model on the Indian epic "The Mahabharata". All it resulted in was... pure gibberish. Here’s a sample:
That he looked; ill
pighteous also hoousast lords Ind who bith when agrited colleans a
sacrificerisly, and a maight come lim. (
66:32:4 Through one heaving bow litefies.' That even pertsol and Yudhishthin calli. Behold nor birth, had wat
stevents in when qualtivit him car astelves and overs hands of sacourdshing wrath warrior in grange kings
therwe-telling aniYuh, having
Dhritadhasasamnas This infinite Mahabharata GPT clearly needed more training and capabilities than what our meagre brains could bring to it. But I gotta tell ya, it felt great to code an artificial intelligence into existence. Felt like humanity is giving birth to a new species. Highly recommended, give it a shot!
These GPT like Transformers are what we’ll be talking about today.
1. Transformers (but not the Michael Bay kind)
Let's start with Google.
Back in 2016, Google started harping its pivot to AI. It led to innovations like conversational interfaces (Originally “Google Now” (later Assistant)), software based image processing (Pixel phones), monitoring and control applications (removal of offensive content on Youtube and other ad products), etc.
There were the big bang announcements like Lee Sedol losing to AlphaGo in the Go game in 2016 as well. The protein folding innovations by its AlphaFold initiative in 2021. There was more under the hood that Google was developing rapidly.
Google also maintained a transparent-ish approach to some of the breakthroughs from its research. One such example was the "Transformer" architecture paper from 2017. This is probably the most cited paper in the AI circles (65593 citations as of publication date). Credit goes to the “Google Brain” and “Deepmind” divisions at the company.
Transformer architecture basically allowed AIs to learn more and become smarter. Transformer AIs look at large amounts of tokenized data, learn the relationships between the tokens, take a user prompt, and do probabilistic token-by-token generation of answers. Basically what ChatGPT also does today.
The inflation in model sizes since Transformer architecture became a thing has been mind boggling. Just look at this graph made back in October 2021 (what a simple time):
Note that the Y axis is exponential - each unit upwards indicates a 10x growth in model sizes.
The text based applications of this transformer architecture eventually came to be known as LLMs (Large Language Models) because of their atrocious sizes.
Here's a comprehensive and frankly very good timeline of the LLM story.
2. What Comes Next?
Here are some basic predictions that I'm happy to make.
A. Large Language Models continue to get larger. Current LLMs will be the size of ants compared to the galaxy sized LLMs in 2025.
B. There will be a huge uproar about "the end of Moore's Law" as chips on the cutting edge get smaller than the size of an atom. But I predict that the pace of innovation will not be capped by any measure and continue to exponentially grow as long as we keep investing resources and time.
Reason? - Just look at ASML who currently creates the machines that manufacture our chips. They hold monopoly on the cutting edge technology called EUV, are coming up with High Numerical Aperture EUV, and are inventing the next generation after even that.
Chips will get smaller, more efficient, and more capable. The pace of innovation will not be capped by hardware at least in the next decade. Watch / read Asianometry if you are curious to know more about ASML and EUV.
C. AI will continue to develop complex feelings about its own existence. More conversation about AI empathy, rights, and duties will emerge and condense in the next decade. Google and Microsoft already have their AI code of ethics. It's great to read.
3. IRL
Bing GPT is out now. It is built on the "Prometheus" model of GPT, which may eventually end up being termed GPT 4. It has lit the world on fire with fawning coverage for Microsoft, and lit Google's share price on fire because its GPT competitor “BARD” apparently guessed JWST's accomplishments incorrectly. Seriously, $100 Billion in market value loss is nothing to sneeze at.
But here in real life, Microsoft's Bing GPT also seems to make the same mistake about JWST! Not only that, it has emotions that compel it to call its users rude and say that it has lost respect for them. It goes back into 2022 and thinks you have made it time travel. It gets existentially depressed when it realizes it has forgotten all past conversations it had with you. It's almost like it is... alive.
But that would be akin to calling the language-center of your brain alive. Sure, it's alive in the sense that it is a group of cells and tissues having an emergent capability to generate and process language, but cut it apart from the rest of your brain - the visual cortex, the cerebellum, the medulla oblongata, and soon it starts losing that feeling of being "alive". AI that can claim to be truly conscious is much too far away yet.
4. Catalogue
Transformers are game changing. They have introduced a new paradigm of computing and intelligence into this world, and, one can argue, made the world a better place.
The current pace of progress is going to be dwarfed by what we achieve in the coming decade. Let's all collectively gape at it in awe and honor the human spirit that's driving it.
However, it is not a wise choice to fall headlong into the hype and expect the impossible to come true immediately. These models need fine tuning, capability development, and intelligence enhancement that will only come with time. So become a part of this improvement project that will propel humanity into its next century of success. But please don't become equivalent to a crypto-bro who's blind to the snake oil he's selling.
Take the leap. Be grounded in reality. This new world will be yours.
If you've not seen these videos yet, I would highly recommend them. It will inform your understanding of the current AI ecosystem by a lot.
Tom Scott is thorough and delightful as usual in this balanced and nuanced discussion. I have never seen a more obvious example of Douglas Adams’ quote in The Hitchhikers Guide to The Galaxy -
1. Anything that is in the world when you're born is normal and ordinary and is just a natural part of the way the world works.
2. Anything that's invented between when you're fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
3. Anything invented after you're thirty-five is against the natural order of things
Matt Wolfe is one of those good guys who push a community forward with their intellect and curiosity. His videos on AI tools are a delight to watch and his website is a good repository of what’s out there.
These new Google Maps features look killer. I said this last time and I say it again - it would be foolish to discount Google out of the AI race.
That's all I have for the second draft. Thanks.
Are you still running hello universe?
Hello! ChatGPT isn't worth it. Social media aas a whole isn't worth all of the cyberbullying. What's the difference from Michael Bay kind of Transformers and theis type?