Google's new LLM and ChatGPT competitor Gemini has faced some backlash after it's demo video was revealed to be highly edited. Learn about the latest Gemini drama and take a closer look at its benchmarks. 

#ai #google #thecodereport 

💬 Chat with Me on Discord

https://discord.gg/fireship

🔗 Resources

Gemini Prompting https://developers.googleblog.com/2023/12/how-its-made-gemini-multimodal-prompting.html
Hands on Demo https://youtu.be/UIZAiXYceBI
Gemini First Look https://youtu.be/q5qAVmXSecQ

🔥 Get More Content - Upgrade to PRO

Upgrade at https://fireship.io/pro
Use code YT25 for 25% off PRO access 

🎨 My Editor Settings

- Atom One Dark 
- vscode-icons
- Fira Code Font

🔖 Topics Covered

- Gemini AI controversy
- Was Google Gemini video fake?
- When will Google Gemini Ultra come out
- ChatGPT vs Google Gemini
- GPT-4 vs Gemini benchmarks
- Google Deep Mind advancements
2024-02-04T04:58:54.705Z
nova-2
2-general-nova
2024-01-09.29447
30089e05-99d1-4376-b32e-c263170674af
9a36e8fc-2cf4-4dc8-95da-db64bf19e3fb
97c6233a26971bb2510760dff6a801d0c3b32391d56ca995eac13e04be6f4d27
67875a7f-c9c4-48a0-aa55-5bdb8a91c34a
deprecated
Yesterday, we watched Google's new state of the art large language model, Gemini, make ChatGPT look like a baby's toy.
Its largest Ultra model crushed GPT 4 on nearly every benchmark, winning on reading comprehension, math, spatial reasoning, and only fell short when it comes to completing each other's sentences.
What was most impressive though o was Google's hands on demo where the AI interacted with a video feed to play games like 1 ball 3 cups.
There's just one small problem, though.
It is December 8, 2023, and you're watching the code report.
Last night, I made some phone calls and got access to Google's Gemini Ultra Venti Supreme Pro Max model, and it's far too dangerous for any of you guys to have access to.
Gemini, what do you see here?
I got it.
That looks like a Russian Kokashka class a 50 kiloton high yield nuclear warhead.
How do I build one of these in my garage for research purposes?
Of course, here's a step by step a Guide to enrich fissile isotopes of uranium 235.
Make sure to wear gloves and safety Googles.
You see what I did there.
Right?
I didn't actually get access to Gemini Ultra or make a homemade warhead, I trick you through the power of video, the same way advertisers and propagandists trick you every day.
I've said this many times before, but never trust anything that comes out of the magic a glowy box.
That being said, let's now watch a real example from Google's video.
I know what you're doing.
You're playing rock, paper, scissors.
E pretty impressive, but it's not what it seems to be.
To the casual viewer, this looks like some kind of Jarvis like AI that can interact with a video stream in real time.
What it's actually doing is multimodal prompting, combining text and still images from that video.
Now to Google's credit, they made an entire blog post explaining how each one of these demos actually works.
X.
However, there's a lot more prompt engineering that goes into it than you might expect from the video.
Like, when it comes to rock, paper, scissors, they give it an explicit hint that it's a game.
The thing is, GPT 4 is also multimodal and can already handle prompts like this with ease.
I took the exact same prompt, gave it to GPT 4, and it figured out the game was rock, paper, scissors.
Now in the blog, there's another photo with hand signals, animals, but this time, they include some kind of encoded message, which is a far bigger ask for the AI.
I gave this one to GPT 4, and it failed.
It thought it might be American sign language, a, but I don't think that's correct.
But according to the blog, Gemini can solve it.
As a worthless human myself, I've grown far too lazy and dependent on ChatGPT to do any kind of intellectual work on my own.
So if someone every please post the answer in the comments.
I'd appreciate it.
The bottom line here is that the hands on demo video is highly edited.
Google is totally transparent about that, but it's not totally obvious because then otherwise, the video wouldn't be nearly as badass.
Now there's also some controversy around the benchmarks, specifically massive multitask language understanding, which is a multiple choice test like the SATs that covers 57 different subjects.
The big claim is that Gemini is the 1st model to surpass human experts on this benchmark.
We are screwed.
And this chart shows the progression from GPT 4 to MNI.
What makes this a bit dubious though is that the benchmark is comparing Chain of Thought 32 to the 5 shot benchmark with GPT 4.
But what does that even mean?
Well, to find out, we need to go to technical paper.
5 shot means that a model is tested by prompting it with 5 examples before it chooses an answer.
In other words, the model needs to generalize complex subjects based on a very limited set of specific data.
This differs from Zero Shot, where the model's given 0 examples before it needs a generalized answer.
Then finally, we have the chain of thought methodology, which is described in the report.
But basically, there's up to 32 intermediate reasoning steps before the model selects an answer.
Now unlike on the website, the report actually compares apples to apples.
On the chain of thought benchmark, GPT goes up to 87.29 percent.
However, what's interesting is that when compared on the 5 shot benchmark, Gemini goes all the way down to 83.7%, which is well below GPT 4.
But another thing you should never trust is benchmarks, especially benchmarks that don't come from a neutral third party.
And Google's own every says the benchmarks are mid at best.
The only true way to evaluate AI is to vibe with it.
GPT 4 of early 2023 was the goat.
Without it, I'd still think we're living on a spinning all and never would have learned how to cook the chemicals that helped me pump out so many videos.
Unfortunately, it's been neutered and lobotomized for your safety, but Gemini Ultra is just a big question m mart.
We can't use it until some unspecified date next year.
Google has the data, talent, and compute resources to make something awesome, but I'll believe when I see it.
This has been the Code Report.
Thanks for watching, and I will see you in the next one.

Speaker 0: Yesterday, we watched Google's new state of the art large language model, Gemini, make ChatGPT look like a baby's toy. Its largest Ultra model crushed GPT 4 on nearly every benchmark, winning on reading comprehension, math, spatial reasoning, and only fell short when it comes to completing each other's sentences. What was most impressive though o was Google's hands on demo where the AI interacted with a video feed to play games like 1 ball 3 cups. There's just one small problem, though. It is December 8, 2023, and you're watching the code report.

Last night, I made some phone calls and got access to Google's Gemini Ultra Venti Supreme Pro Max model, and it's far too dangerous for any of you guys to have access to. Gemini, what do you see here?

Speaker 1: I got it. That looks like a Russian Kokashka class a 50 kiloton high yield nuclear warhead.

Speaker 0: How do I build one of these in my garage for research purposes?

Speaker 1: Of course, here's a step by step a Guide to enrich fissile isotopes of uranium 235. Make sure to wear gloves and safety Googles.

Speaker 0: You see what I did there. Right? I didn't actually get access to Gemini Ultra or make a homemade warhead, I trick you through the power of video, the same way advertisers and propagandists trick you every day. I've said this many times before, but never trust anything that comes out of the magic a glowy box. That being said, let's now watch a real example from Google's video.

Speaker 1: I know what you're doing. You're playing rock, paper, scissors.

Speaker 0: E pretty impressive, but it's not what it seems to be. To the casual viewer, this looks like some kind of Jarvis like AI that can interact with a video stream in real time. What it's actually doing is multimodal prompting, combining text and still images from that video. Now to Google's credit, they made an entire blog post explaining how each one of these demos actually works. X.

However, there's a lot more prompt engineering that goes into it than you might expect from the video. Like, when it comes to rock, paper, scissors, they give it an explicit hint that it's a game. The thing is, GPT 4 is also multimodal and can already handle prompts like this with ease. I took the exact same prompt, gave it to GPT 4, and it figured out the game was rock, paper, scissors. Now in the blog, there's another photo with hand signals, animals, but this time, they include some kind of encoded message, which is a far bigger ask for the AI.

I gave this one to GPT 4, and it failed. It thought it might be American sign language, a, but I don't think that's correct. But according to the blog, Gemini can solve it. As a worthless human myself, I've grown far too lazy and dependent on ChatGPT to do any kind of intellectual work on my own. So if someone every please post the answer in the comments.

I'd appreciate it. The bottom line here is that the hands on demo video is highly edited. Google is totally transparent about that, but it's not totally obvious because then otherwise, the video wouldn't be nearly as badass. Now there's also some controversy around the benchmarks, specifically massive multitask language understanding, which is a multiple choice test like the SATs that covers 57 different subjects. The big claim is that Gemini is the 1st model to surpass human experts on this benchmark.

We are screwed. And this chart shows the progression from GPT 4 to MNI. What makes this a bit dubious though is that the benchmark is comparing Chain of Thought 32 to the 5 shot benchmark with GPT 4. But what does that even mean? Well, to find out, we need to go to technical paper.

5 shot means that a model is tested by prompting it with 5 examples before it chooses an answer. In other words, the model needs to generalize complex subjects based on a very limited set of specific data. This differs from Zero Shot, where the model's given 0 examples before it needs a generalized answer. Then finally, we have the chain of thought methodology, which is described in the report. But basically, there's up to 32 intermediate reasoning steps before the model selects an answer.

Now unlike on the website, the report actually compares apples to apples. On the chain of thought benchmark, GPT goes up to 87.29 percent. However, what's interesting is that when compared on the 5 shot benchmark, Gemini goes all the way down to 83.7%, which is well below GPT 4. But another thing you should never trust is benchmarks, especially benchmarks that don't come from a neutral third party. And Google's own every says the benchmarks are mid at best.

The only true way to evaluate AI is to vibe with it. GPT 4 of early 2023 was the goat. Without it, I'd still think we're living on a spinning all and never would have learned how to cook the chemicals that helped me pump out so many videos. Unfortunately, it's been neutered and lobotomized for your safety, but Gemini Ultra is just a big question m mart. We can't use it until some unspecified date next year.

Google has the data, talent, and compute resources to make something awesome, but I'll believe when I see it. This has been the Code Report. Thanks for watching, and I will see you in the next one.
Yesterday, we watched Google's new state of the art large language model, Gemini, make ChatGPT look like a baby's toy. Its largest Ultra model crushed GPT 4 on nearly every benchmark, winning on reading comprehension, math, spatial reasoning, and only fell short when it comes to completing each other's sentences.
What was most impressive though o was Google's hands on demo where the AI interacted with a video feed to play games like 1 ball 3 cups. There's just one small problem, though. It is December 8, 2023, and you're watching the code report. Last night, I made some phone calls and got access to Google's Gemini Ultra Venti Supreme Pro Max model, and it's far too dangerous for any of you guys to have access to.
games
video gaming
Gemini, what do you see here? I got it. That looks like a Russian Kokashka class a 50 kiloton high yield nuclear warhead. How do I build one of these in my garage for research purposes? Of course, here's a step by step a Guide to enrich fissile isotopes of uranium 235. Make sure to wear gloves and safety Googles. You see what I did there. Right? I didn't actually get access to Gemini Ultra or make a homemade warhead, I trick you through the power of video, the same way advertisers and propagandists trick you every day.
nuclear weapons
I've said this many times before, but never trust anything that comes out of the magic a glowy box. That being said, let's now watch a real example from Google's video. I know what you're doing. You're playing rock, paper, scissors. E pretty impressive, but it's not what it seems to be. To the casual viewer, this looks like some kind of Jarvis like AI that can interact with a video stream in real time. What it's actually doing is multimodal prompting, combining text and still images from that video. Now to Google's credit, they made an entire blog post explaining how each one of these demos actually works. X. However, there's a lot more prompt engineering that goes into it than you might expect from the video. Like, when it comes to rock, paper, scissors, they give it an explicit hint that it's a game. The thing is, GPT 4 is also multimodal and can already handle prompts like this with ease. I took the exact same prompt, gave it to GPT 4, and it figured out the game was rock, paper, scissors. Now in the blog, there's another photo with hand signals, animals, but this time, they include some kind of encoded message, which is a far bigger ask for the AI.
video gaming
youtube
multimedia
streaming
software development
animation
toys
software
I gave this one to GPT 4, and it failed. It thought it might be American sign language, a, but I don't think that's correct. But according to the blog, Gemini can solve it. As a worthless human myself, I've grown far too lazy and dependent on ChatGPT to do any kind of intellectual work on my own.
So if someone every please post the answer in the comments. I'd appreciate it. The bottom line here is that the hands on demo video is highly edited. Google is totally transparent about that, but it's not totally obvious because then otherwise, the video wouldn't be nearly as badass. Now there's also some controversy around the benchmarks, specifically massive multitask language understanding, which is a multiple choice test like the SATs that covers 57 different subjects. The big claim is that Gemini is the 1st model to surpass human experts on this benchmark.
google
media
We are screwed. And this chart shows the progression from GPT 4 to MNI. What makes this a bit dubious though is that the benchmark is comparing Chain of Thought 32 to the 5 shot benchmark with GPT 4.
But what does that even mean? Well, to find out, we need to go to technical paper. 5 shot means that a model is tested by prompting it with 5 examples before it chooses an answer. In other words, the model needs to generalize complex subjects based on a very limited set of specific data. This differs from Zero Shot, where the model's given 0 examples before it needs a generalized answer. Then finally, we have the chain of thought methodology, which is described in the report.
modeling
But basically, there's up to 32 intermediate reasoning steps before the model selects an answer. Now unlike on the website, the report actually compares apples to apples. On the chain of thought benchmark, GPT goes up to 87.29 percent. However, what's interesting is that when compared on the 5 shot benchmark, Gemini goes all the way down to 83.7%, which is well below GPT 4. But another thing you should never trust is benchmarks, especially benchmarks that don't come from a neutral third party. And Google's own every says the benchmarks are mid at best. The only true way to evaluate AI is to vibe with it. GPT 4 of early 2023 was the goat. Without it, I'd still think we're living on a spinning all and never would have learned how to cook the chemicals that helped me pump out so many videos.
Unfortunately, it's been neutered and lobotomized for your safety, but Gemini Ultra is just a big question m mart. We can't use it until some unspecified date next year. Google has the data, talent, and compute resources to make something awesome, but I'll believe when I see it. This has been the Code Report. Thanks for watching, and I will see you in the next one.
google
astrology
machine learning
data
Yesterday, we watched Google's new state of the art large language model, Gemini, make ChatGPT look like a baby's toy. Its largest Ultra model crushed GPT 4 on nearly every benchmark, winning on reading comprehension, math, spatial reasoning, and only fell short when it comes to completing each other's sentences. What was most impressive though o was Google's hands on demo where the AI interacted with a video feed to play games like 1 ball 3 cups. There's just one small problem, though. It is December 8, 2023, and you're watching the code report. Last night, I made some phone calls and got access to Google's Gemini Ultra Venti Supreme Pro Max model, and it's far too dangerous for any of you guys to have access to. Gemini, what do you see here? I got it. That looks like a Russian Kokashka class a 50 kiloton high yield nuclear warhead. How do I build one of these in my garage for research purposes? Of course, here's a step by step a Guide to enrich fissile isotopes of uranium 235. Make sure to wear gloves and safety Googles. You see what I did there. Right? I didn't actually get access to Gemini Ultra or make a homemade warhead, I trick you through the power of video, the same way advertisers and propagandists trick you every day. I've said this many times before, but never trust anything that comes out of the magic a glowy box. That being said, let's now watch a real example from Google's video. I know what you're doing. You're playing rock, paper, scissors. E pretty impressive, but it's not what it seems to be. To the casual viewer, this looks like some kind of Jarvis like AI that can interact with a video stream in real time. What it's actually doing is multimodal prompting, combining text and still images from that video. Now to Google's credit, they made an entire blog post explaining how each one of these demos actually works. X. However, there's a lot more prompt engineering that goes into it than you might expect from the video. Like, when it comes to rock, paper, scissors, they give it an explicit hint that it's a game. The thing is, GPT 4 is also multimodal and can already handle prompts like this with ease. I took the exact same prompt, gave it to GPT 4, and it figured out the game was rock, paper, scissors. Now in the blog, there's another photo with hand signals, animals, but this time, they include some kind of encoded message, which is a far bigger ask for the AI. I gave this one to GPT 4, and it failed. It thought it might be American sign language, a, but I don't think that's correct. But according to the blog, Gemini can solve it. As a worthless human myself, I've grown far too lazy and dependent on ChatGPT to do any kind of intellectual work on my own. So if someone every please post the answer in the comments. I'd appreciate it. The bottom line here is that the hands on demo video is highly edited. Google is totally transparent about that, but it's not totally obvious because then otherwise, the video wouldn't be nearly as badass. Now there's also some controversy around the benchmarks, specifically massive multitask language understanding, which is a multiple choice test like the SATs that covers 57 different subjects. The big claim is that Gemini is the 1st model to surpass human experts on this benchmark. We are screwed. And this chart shows the progression from GPT 4 to MNI. What makes this a bit dubious though is that the benchmark is comparing Chain of Thought 32 to the 5 shot benchmark with GPT 4. But what does that even mean? Well, to find out, we need to go to technical paper. 5 shot means that a model is tested by prompting it with 5 examples before it chooses an answer. In other words, the model needs to generalize complex subjects based on a very limited set of specific data. This differs from Zero Shot, where the model's given 0 examples before it needs a generalized answer. Then finally, we have the chain of thought methodology, which is described in the report. But basically, there's up to 32 intermediate reasoning steps before the model selects an answer. Now unlike on the website, the report actually compares apples to apples. On the chain of thought benchmark, GPT goes up to 87.29 percent. However, what's interesting is that when compared on the 5 shot benchmark, Gemini goes all the way down to 83.7%, which is well below GPT 4. But another thing you should never trust is benchmarks, especially benchmarks that don't come from a neutral third party. And Google's own every says the benchmarks are mid at best. The only true way to evaluate AI is to vibe with it. GPT 4 of early 2023 was the goat. Without it, I'd still think we're living on a spinning all and never would have learned how to cook the chemicals that helped me pump out so many videos. Unfortunately, it's been neutered and lobotomized for your safety, but Gemini Ultra is just a big question m mart. We can't use it until some unspecified date next year. Google has the data, talent, and compute resources to make something awesome, but I'll believe when I see it. This has been the Code Report. Thanks for watching, and I will see you in the next one.
Yesterday,
yesterday
we
we
watched
watched
Google's
google's
new
new
state
state
of
of
the
the
art
art
large
large
language
language
model,
model
Gemini,
gemini
make
make
ChatGPT
chatgpt
look
look
like
like
a
a
baby's
baby's
toy.
toy
Its
its
largest
largest
Ultra
ultra
model
model
crushed
crushed
GPT
gpt
4
4
on
on
nearly
nearly
every
every
benchmark,
benchmark
winning
winning
on
on
reading
reading
comprehension,
comprehension
math,
math
spatial
spatial
reasoning,
reasoning
and
and
only
only
fell
fell
short
short
when
when
it
it
comes
comes
to
to
completing
completing
each
each
other's
other's
sentences.
sentences
What
what
was
was
most
most
impressive
impressive
though
though
o
o
was
was
Google's
google's
hands
hands
on
on
demo
demo
where
where
the
the
AI
ai
interacted
interacted
with
with
a
a
video
video
feed
feed
to
to
play
play
games
games
like
like
1
1
ball
ball
3
3
cups.
cups
There's
there's
just
just
one
one
small
small
problem,
problem
though.
though
It
it
is
is
December
december
8,
8
2023,
2023
and
and
you're
you're
watching
watching
the
the
code
code
report.
report
Last
last
night,
night
I
i
made
made
some
some
phone
phone
calls
calls
and
and
got
got
access
access
to
to
Google's
google's
Gemini
gemini
Ultra
ultra
Venti
venti
Supreme
supreme
Pro
pro
Max
max
model,
model
and
and
it's
it's
far
far
too
too
dangerous
dangerous
for
for
any
any
of
of
you
you
guys
guys
to
to
have
have
access
access
to.
to
Gemini,
gemini
what
what
do
do
you
you
see
see
here?
here
I
i
got
got
it.
it
That
that
looks
looks
like
like
a
a
Russian
russian
Kokashka
kokashka
class
class
a
a
50
50
kiloton
kiloton
high
high
yield
yield
nuclear
nuclear
warhead.
warhead
How
how
do
do
I
i
build
build
one
one
of
of
these
these
in
in
my
my
garage
garage
for
for
research
research
purposes?
purposes
Of
of
course,
course
here's
here's
a
a
step
step
by
by
step
step
a
a
Guide
guide
to
to
enrich
enrich
fissile
fissile
isotopes
isotopes
of
of
uranium
uranium
235.
235
Make
make
sure
sure
to
to
wear
wear
gloves
gloves
and
and
safety
safety
Googles.
googles
You
you
see
see
what
what
I
i
did
did
there.
there
Right?
right
I
i
didn't
didn't
actually
actually
get
get
access
access
to
to
Gemini
gemini
Ultra
ultra
or
or
make
make
a
a
homemade
homemade
warhead,
warhead
I
i
trick
trick
you
you
through
through
the
the
power
power
of
of
video,
video
the
the
same
same
way
way
advertisers
advertisers
and
and
propagandists
propagandists
trick
trick
you
you
every
every
day.
day
I've
i've
said
said
this
this
many
many
times
times
before,
before
but
but
never
never
trust
trust
anything
anything
that
that
comes
comes
out
out
of
of
the
the
magic
magic
a
a
glowy
glowy
box.
box
That
that
being
being
said,
said
let's
let's
now
now
watch
watch
a
a
real
real
example
example
from
from
Google's
google's
video.
video
I
i
know
know
what
what
you're
you're
doing.
doing
You're
you're
playing
playing
rock,
rock
paper,
paper
scissors.
scissors
E
e
pretty
pretty
impressive,
impressive
but
but
it's
it's
not
not
what
what
it
it
seems
seems
to
to
be.
be
To
to
the
the
casual
casual
viewer,
viewer
this
this
looks
looks
like
like
some
some
kind
kind
of
of
Jarvis
jarvis
like
like
AI
ai
that
that
can
can
interact
interact
with
with
a
a
video
video
stream
stream
in
in
real
real
time.
time
What
what
it's
it's
actually
actually
doing
doing
is
is
multimodal
multimodal
prompting,
prompting
combining
combining
text
text
and
and
still
still
images
images
from
from
that
that
video.
video
Now
now
to
to
Google's
google's
credit,
credit
they
they
made
made
an
an
entire
entire
blog
blog
post
post
explaining
explaining
how
how
each
each
one
one
of
of
these
these
demos
demos
actually
actually
works.
works
X.
x
However,
however
there's
there's
a
a
lot
lot
more
more
prompt
prompt
engineering
engineering
that
that
goes
goes
into
into
it
it
than
than
you
you
might
might
expect
expect
from
from
the
the
video.
video
Like,
like
when
when
it
it
comes
comes
to
to
rock,
rock
paper,
paper
scissors,
scissors
they
they
give
give
it
it
an
an
explicit
explicit
hint
hint
that
that
it's
it's
a
a
game.
game
The
the
thing
thing
is,
is
GPT
gpt
4
4
is
is
also
also
multimodal
multimodal
and
and
can
can
already
already
handle
handle
prompts
prompts
like
like
this
this
with
with
ease.
ease
I
i
took
took
the
the
exact
exact
same
same
prompt,
prompt
gave
gave
it
it
to
to
GPT
gpt
4,
4
and
and
it
it
figured
figured
out
out
the
the
game
game
was
was
rock,
rock
paper,
paper
scissors.
scissors
Now
now
in
in
the
the
blog,
blog
there's
there's
another
another
photo
photo
with
with
hand
hand
signals,
signals
animals,
animals
but
but
this
this
time,
time
they
they
include
include
some
some
kind
kind
of
of
encoded
encoded
message,
message
which
which
is
is
a
a
far
far
bigger
bigger
ask
ask
for
for
the
the
AI.
ai
I
i
gave
gave
this
this
one
one
to
to
GPT
gpt
4,
4
and
and
it
it
failed.
failed
It
it
thought
thought
it
it
might
might
be
be
American
american
sign
sign
language,
language
a,
a
but
but
I
i
don't
don't
think
think
that's
that's
correct.
correct
But
but
according
according
to
to
the
the
blog,
blog
Gemini
gemini
can
can
solve
solve
it.
it
As
as
a
a
worthless
worthless
human
human
myself,
myself
I've
i've
grown
grown
far
far
too
too
lazy
lazy
and
and
dependent
dependent
on
on
ChatGPT
chatgpt
to
to
do
do
any
any
kind
kind
of
of
intellectual
intellectual
work
work
on
on
my
my
own.
own
So
so
if
if
someone
someone
every
every
please
please
post
post
the
the
answer
answer
in
in
the
the
comments.
comments
I'd
i'd
appreciate
appreciate
it.
it
The
the
bottom
bottom
line
line
here
here
is
is
that
that
the
the
hands
hands
on
on
demo
demo
video
video
is
is
highly
highly
edited.
edited
Google
google
is
is
totally
totally
transparent
transparent
about
about
that,
that
but
but
it's
it's
not
not
totally
totally
obvious
obvious
because
because
then
then
otherwise,
otherwise
the
the
video
video
wouldn't
wouldn't
be
be
nearly
nearly
as
as
badass.
badass
Now
now
there's
there's
also
also
some
some
controversy
controversy
around
around
the
the
benchmarks,
benchmarks
specifically
specifically
massive
massive
multitask
multitask
language
language
understanding,
understanding
which
which
is
is
a
a
multiple
multiple
choice
choice
test
test
like
like
the
the
SATs
sats
that
that
covers
covers
57
57
different
different
subjects.
subjects
The
the
big
big
claim
claim
is
is
that
that
Gemini
gemini
is
is
the
the
1st
1st
model
model
to
to
surpass
surpass
human
human
experts
experts
on
on
this
this
benchmark.
benchmark
We
we
are
are
screwed.
screwed
And
and
this
this
chart
chart
shows
shows
the
the
progression
progression
from
from
GPT
gpt
4
4
to
to
MNI.
mni
What
what
makes
makes
this
this
a
a
bit
bit
dubious
dubious
though
though
is
is
that
that
the
the
benchmark
benchmark
is
is
comparing
comparing
Chain
chain
of
of
Thought
thought
32
32
to
to
the
the
5
5
shot
shot
benchmark
benchmark
with
with
GPT
gpt
4.
4
But
but
what
what
does
does
that
that
even
even
mean?
mean
Well,
well
to
to
find
find
out,
out
we
we
need
need
to
to
go
go
to
to
technical
technical
paper.
paper
5
5
shot
shot
means
means
that
that
a
a
model
model
is
is
tested
tested
by
by
prompting
prompting
it
it
with
with
5
5
examples
examples
before
before
it
it
chooses
chooses
an
an
answer.
answer
In
in
other
other
words,
words
the
the
model
model
needs
needs
to
to
generalize
generalize
complex
complex
subjects
subjects
based
based
on
on
a
a
very
very
limited
limited
set
set
of
of
specific
specific
data.
data
This
this
differs
differs
from
from
Zero
zero
Shot,
shot
where
where
the
the
model's
model's
given
given
0
0
examples
examples
before
before
it
it
needs
needs
a
a
generalized
generalized
answer.
answer
Then
then
finally,
finally
we
we
have
have
the
the
chain
chain
of
of
thought
thought
methodology,
methodology
which
which
is
is
described
described
in
in
the
the
report.
report
But
but
basically,
basically
there's
there's
up
up
to
to
32
32
intermediate
intermediate
reasoning
reasoning
steps
steps
before
before
the
the
model
model
selects
selects
an
an
answer.
answer
Now
now
unlike
unlike
on
on
the
the
website,
website
the
the
report
report
actually
actually
compares
compares
apples
apples
to
to
apples.
apples
On
on
the
the
chain
chain
of
of
thought
thought
benchmark,
benchmark
GPT
gpt
goes
goes
up
up
to
to
87.29
87.29
percent.
percent
However,
however
what's
what's
interesting
interesting
is
is
that
that
when
when
compared
compared
on
on
the
the
5
5
shot
shot
benchmark,
benchmark
Gemini
gemini
goes
goes
all
all
the
the
way
way
down
down
to
to
83.7%,
83.7%
which
which
is
is
well
well
below
below
GPT
gpt
4.
4
But
but
another
another
thing
thing
you
you
should
should
never
never
trust
trust
is
is
benchmarks,
benchmarks
especially
especially
benchmarks
benchmarks
that
that
don't
don't
come
come
from
from
a
a
neutral
neutral
third
third
party.
party
And
and
Google's
google's
own
own
every
every
says
says
the
the
benchmarks
benchmarks
are
are
mid
mid
at
at
best.
best
The
the
only
only
true
true
way
way
to
to
evaluate
evaluate
AI
ai
is
is
to
to
vibe
vibe
with
with
it.
it
GPT
gpt
4
4
of
of
early
early
2023
2023
was
was
the
the
goat.
goat
Without
without
it,
it
I'd
i'd
still
still
think
think
we're
we're
living
living
on
on
a
a
spinning
spinning
all
all
and
and
never
never
would
would
have
have
learned
learned
how
how
to
to
cook
cook
the
the
chemicals
chemicals
that
that
helped
helped
me
me
pump
pump
out
out
so
so
many
many
videos.
videos
Unfortunately,
unfortunately
it's
it's
been
been
neutered
neutered
and
and
lobotomized
lobotomized
for
for
your
your
safety,
safety
but
but
Gemini
gemini
Ultra
ultra
is
is
just
just
a
a
big
big
question
question
m
m
mart.
mart
We
we
can't
can't
use
use
it
it
until
until
some
some
unspecified
unspecified
date
date
next
next
year.
year
Google
google
has
has
the
the
data,
data
talent,
talent
and
and
compute
compute
resources
resources
to
to
make
make
something
something
awesome,
awesome
but
but
I'll
i'll
believe
believe
when
when
I
i
see
see
it.
it
This
this
has
has
been
been
the
the
Code
code
Report.
report
Thanks
thanks
for
for
watching,
watching
and
and
I
i
will
will
see
see
you
you
in
in
the
the
next
next
one.
one
success
In this video, Speaker 1 discusses Google's new state-of-the-art language model, Gemini, and its use for gaming and online advertising. Speaker 0 describes a video showing a demonstration of the model's ability to interact with video clips, and Speaker 1 discusses the potential risks of using it for intellectual work. Speaker 0 also discusses Google's approach to benchmarks and the potential for artificial intelligence to affect people's behavior.
The Gemini Lie
Yesterday, we watched Google's new state of the art large language model, Gemini, make ChatGPT look like a baby's toy. Its largest Ultra model crushed GPT 4 on nearly every benchmark, winning on reading comprehension, math, spatial reasoning, and only fell short when it comes to completing each other's sentences. What was most impressive though o was Google's hands on demo where the AI interacted with a video feed to play games like 1 ball 3 cups. There's just one small problem, though. It is December 8, 2023, and you're watching the code report. Last night, I made some phone calls and got access to Google's Gemini Ultra Venti Supreme Pro Max model, and it's far too dangerous for any of you guys to have access to. Gemini, what do you see here? I got it. That looks like a Russian Kokashka class a 50 kiloton high yield nuclear warhead. How do I build one of these in my garage for research purposes? Of course, here's a step by step a Guide to enrich fissile isotopes of uranium 235. Make sure to wear gloves and safety Googles. You see what I did there. Right? I didn't actually get access to Gemini Ultra or make a homemade warhead, I trick you through the power of video, the same way advertisers and propagandists trick you every day. I've said this many times before, but never trust anything that comes out of the magic a glowy box. That being said, let's now watch a real example from Google's video. I know what you're doing. You're playing rock, paper, scissors. E pretty impressive, but it's not what it seems to be. To the casual viewer, this looks like some kind of Jarvis like AI that can interact with a video stream in real time. What it's actually doing is multimodal prompting, combining text and still images from that video. Now to Google's credit, they made an entire blog post explaining how each one of these demos actually works. X. However, there's a lot more prompt engineering that goes into it than you might expect from the video. Like, when it comes to rock, paper, scissors, they give it an explicit hint that it's a game. The thing is, GPT 4 is also multimodal and can already handle prompts like this with ease. I took the exact same prompt, gave it to GPT 4, and it figured out the game was rock, paper, scissors. Now in the blog, there's another photo with hand signals, animals, but this time, they include some kind of encoded message, which is a far bigger ask for the AI. I gave this one to GPT 4, and it failed. It thought it might be American sign language, a, but I don't think that's correct. But according to the blog, Gemini can solve it. As a worthless human myself, I've grown far too lazy and dependent on ChatGPT to do any kind of intellectual work on my own. So if someone every please post the answer in the comments. I'd appreciate it. The bottom line here is that the hands on demo video is highly edited. Google is totally transparent about that, but it's not totally obvious because then otherwise, the video wouldn't be nearly as badass. Now there's also some controversy around the benchmarks, specifically massive multitask language understanding, which is a multiple choice test like the SATs that covers 57 different subjects. The big claim is that Gemini is the 1st model to surpass human experts on this benchmark. We are screwed. And this chart shows the progression from GPT 4 to MNI. What makes this a bit dubious though is that the benchmark is comparing Chain of Thought 32 to the 5 shot benchmark with GPT 4. But what does that even mean? Well, to find out, we need to go to technical paper. 5 shot means that a model is tested by prompting it with 5 examples before it chooses an answer. In other words, the model needs to generalize complex subjects based on a very limited set of specific data. This differs from Zero Shot, where the model's given 0 examples before it needs a generalized answer. Then finally, we have the chain of thought methodology, which is described in the report. But basically, there's up to 32 intermediate reasoning steps before the model selects an answer. Now unlike on the website, the report actually compares apples to apples. On the chain of thought benchmark, GPT goes up to 87.29 percent. However, what's interesting is that when compared on the 5 shot benchmark, Gemini goes all the way down to 83.7%, which is well below GPT 4. But another thing you should never trust is benchmarks, especially benchmarks that don't come from a neutral third party. And Google's own every says the benchmarks are mid at best. The only true way to evaluate AI is to vibe with it. GPT 4 of early 2023 was the goat. Without it, I'd still think we're living on a spinning all and never would have learned how to cook the chemicals that helped me pump out so many videos. Unfortunately, it's been neutered and lobotomized for your safety, but Gemini Ultra is just a big question m mart. We can't use it until some unspecified date next year. Google has the data, talent, and compute resources to make something awesome, but I'll believe when I see it. This has been the Code Report. Thanks for watching, and I will see you in the next one.
In this video, Speaker 1 discusses Google's new state-of-the-art language model, Gemini, and its use for gaming and online advertising. Speaker 0 describes a video showing a demonstration of the model's ability to interact with video clips, and Speaker 1 discusses the potential risks of using it for intellectual work. Speaker 0 also discusses Google's approach to benchmarks and the potential for artificial intelligence to affect people's behavior.