What Is a World Model?
Ask a video AI to 'show me a fox running through a forest,' and you get a clip a few seconds long. But what if the forest path kept going after the clip ended?
What if you could swing the camera around to see the fox's profile, or tell it, 'Stop for a second and look around'?
✦ Video AI vs. World Model
Same starting point, different result
We call this kind of model a world model. A regular video AI finishes one clip. A world model remembers the objects and space that came before, and builds the next scene on top of them. And you can move around inside that space and change it yourself.
Tencent's Hy-Nomas and Google DeepMind's Genie 4, both released recently, are the best-known examples.
Tencent's Hy-Nomas
Tencent's Hy-Nomas was the first to come out. On October 21, the code, model weights and an online demo went public, and the technical paper followed on October 27.
✦ Hy-Nomas at a Glance
Model, generation method and settings
A diffusion model with about 8 billion parameters builds the video in segments and saves earlier segments' information in a KV cache to refer to when it generates the next one, they say.
Users can change the scene or the action with text, and set the camera's path too.
The default inference settings are 720×1280 resolution at 24 frames per second.
The official demo shows scenes like a panda climbing a stone pillar, a snow leopard scaling a palace wall, and a moon base and ancient ruins.
Give a different action in the place it first made, or move the camera, and it builds the next segment from the elements it already created.
If your hardware can keep up, you can keep extending the scene for up to 60 seconds in the official demo and explore the virtual world.
That said, being released doesn't mean you can comfortably run it on a home PC. The single-GPU low-memory setup Tencent describes is built around an RTX 4090 with 24GB of VRAM.
✦ Running Hy-Nomas on a 16GB GPU
What OMNI Labs had to do and how long it took
OMNI Labs, an AI production studio, got it running on a 16GB RTX 5060 Ti too, but it had to set up a Linux environment through WSL and go through four fixes.
It moved the text encoder to the CPU and used 52GB of system memory and 32GB of swap. Even so, one 5-second segment took about 6 minutes, and stringing together 10 seconds took about 13 minutes.
And that was a silent 1280×704 video at 24 FPS. Experiencing the kind of high-quality virtual world we're all hoping for is still out of reach on a personal PC.
Speed isn't the only thing. We need to test how well the scenes hold together, too, right? OMNI Labs made a scene of a fox walking through a forest lit by fireflies, in two 5-second segments.
✦ Fox Forest Test: What Worked and What Didn't
Connection test and camera test results
In the second segment, they told the fox to stop and look around, and its action changed. The fox, the path and the trees carried over naturally too. The camera viewpoint test succeeded as well, with a natural change of angle.
But the command to make glowing blue flowers bloom along the path wasn't reflected, and the night firefly forest turned into an ordinary daytime scene. So it still doesn't follow prompts well enough.
The Hy-Nomas paper also compares it against other models. It split 280 cases into four areas: text instructions, camera movement, using the two together, and spatial memory. Trained evaluators then compared the results.
For example, the models were asked to make a scene where tongs pick up an apple and move it to a plate, or where a spatula cuts a strawberry and puts it in a glass bowl.
In the comparison videos in the paper, Hy-Nomas got the apple-to-plate flow and the strawberry-into-the-bowl process right, while some rival models changed the apple's color or the tongs' shape, or left out the spatula.
That doesn't mean Hy-Nomas won on every item. In the 'spatial memory evaluation,' which checks whether objects and layout stay put when you return to a place you've already passed, Google's Genie 3 did better.
✦ Spatial Memory: Hy-Nomas vs. Genie 3
Hy-Nomas's results over 60 shared scenarios, on Tencent's own benchmark
On 60 shared scenarios, Hy-Nomas had a 13.3% win rate, a 56.7% draw rate and a 30% loss rate.
Keep in mind that this is a result from a benchmark Tencent's own researchers made, and the comparison was against the older Genie 3, not Genie 4. So Hy-Nomas is good, but it's hard to say it clearly beats the model it was compared with.
✦ Hy-Nomas Licenses: Check the Decoder Separately
Worth checking before any commercial use
There's the license issue, too. With this model in particular, you have to look at the decoder license separately.
Hy-Nomas's code and core DiT weights are Apache 2.0, but the decoder that turns the video back into frames is a fine-tuned version of NVIDIA's PixelUMM.
The PixelUMM checkpoint carries a license for non-commercial research and evaluation only. So even if you somehow get Hy-Nomas running on a home PC and produce results, there are limits on using them commercially.
Google's Genie 4
So what makes Google's Genie 4 different? The model, released on October 28 Korean time, is said to be strong at real-time control and audio generation. It also claims to simulate physical phenomena like collisions and aerodynamics.
While you move with the keyboard and swing the camera, you can change the weather or bring in new characters and objects using natural language.
It also generates ambient sound like wind in the forest and waves, background music, and characters' dialogue at the same time.
✦ How Genie 4 Works With Gemini
Each model has its own job
Genie 4 is connected to several Gemini models. First, Gemini Omni keeps the look and movement of the simulation close to reality.
Then Gemini 3.8 Flash keeps watching the scene, splits the user's goal into steps, and checks that each step went well. One of Google's published examples shows a UR5 robot arm using a pipette (a lab tool) to move liquid onto a lab plate.
It works like this: Gemini plans the next move and passes it along as a text instruction, and Genie 4 draws that process in real time.
In another robot simulation, Gemini 4 Argon split up the order of tasks and Genie 4 turned that into scenes.
✦ Who Can Use Genie 4 Right Now
Access is narrow
Sadly, though, Genie 4 isn't a model just anyone can try.
It was a research preview for trusted testers, and all Google has laid out is a schedule to expand it to Project Genie for Google AI Ultra subscribers and to early-access partners on the Gemini API and AI Studio.
You can't download the model weights and run it on a local PC like you can with Hy-Nomas. The announcement's scope was pretty narrow, and going forward it will probably only reach paid Gemini plan users.
Genie 4 has clear limits too. The shape and position of objects in a scene still drift over time, and it has trouble recreating real places or producing accurate text.
Press reports describe a single continuous interaction as lasting about 2 minutes, so generation length is still limited too. Google itself also said that the accuracy of real-time audio is still a research problem that needs improvement.
Wrapping Up
If you want to try a world model right away and your graphics card is strong, download Hy-Nomas and experiment with it. And since world models keep getting better, I'm hoping lighter ones that run far more easily will come out before long.
That's all for this post. See you next time.
Related posts
- We Made AIs Vote, and They Started Copying Each Other
- Magnitude Local LLM Speed: Is It Really 2x Faster?
- Do Paid AI Assistants Beat Free Ones? Test Results
🌐 Korean version: 영상생성을 넘어, 가상세계를 만드는 AI의 시대가 온다. 텐센트 Hy-Nomas와 구글 Genie 4 등장
No comments:
No comments yet. Want to leave the first one?
Post a Comment