OpenAI presents Sora, its new AI capable of generating stunningly realistic videos from a simple textual description. The results are so impressive that they already raise ethical questions.
Nothing seems to be able to stop OpenAI, which is multiplying generative artificial intelligence projects with tools, each more astonishing than the last! While the company behind ChatGPT has announced that it is developing its own search engine to compete with Google – just that! –, the American company has just unveiled a brand new tool. Baptized Sorawhich means “sky” in Japanese, in reference to the infinity of possibilities offered – nothing to do with the hero of Kingdom Hearts –, it allows you to generate videos from a simple text request. Sam Altman had already announced his project in January 2023, without giving many details. Concretely, it works like any image editor powered by AI – like DALL-E –: you just have to write your request for the artificial intelligence to execute, but instead of creating an image, it generates a video, the duration of which can be up to 60 seconds. And the least we can say is that the result is astonishingly realistic! This may worry artists, who generally take several hours to produce this kind of content…
Sora: videos of breathtaking realism
“Sora is capable of generating complex scenes with multiple characters, specific movement types, and precise subject and background details. The model not only understands what the user asked for in the prompt, but also how these things exist in the physical world”, explains OpenAI. These short videos produce diverse and varied scenes, such as a couple walking in the snowy streets of Tokyo, mammoths running in the snow, a monster admiring a candle, or even a visit to an art gallery with numerous works of art in different styles. Here is an example of a prompt: “The beautiful snow-covered city of Tokyo is bustling. The camera moves through the bustling streets, following several characters enjoying the winter weather and shopping. Sakura petals [ndla : fleurs de cerisiers] fly in the air, accompanied by snowflakes.“ We’ll let you judge the result!
The videos are extremely realistic and contain multiple details, with a camera that enjoys twirling in the sky. There’s even a little monster that looks like it came straight out of a Pixar or Dreamworks animated film! Fortunately there is a marker, otherwise it would be difficult to differentiate them from authentic videos. But how is this possible? “Sora builds on previous research on the DALL-E and GPT models. It uses DALL-E 3’s summarization technique, which involves generating highly descriptive captions for visual training data. As a result, the model is able to more closely follow the user’s textual instructions in the generated video”explains Open AI in a press release.
The AI uses a large set of data from videos, which is then analyzed by the system, which will generate millions of descriptive words from the images. When the user sends their request in writing, Sora analyzes it and extracts the keywords that it deems relevant. It then searches for videos that best match the keywords, then extracts them from its database and stitches them together to create the video clip. Better yet, the tool is able to stretch an already existing video by extending it “forward and backward, to produce a seamless infinite loop.” It obviously ensures that the characters move coherently in three-dimensional space.
Sora: an AI that should not be put into everyone’s hands
Still, if we pay attention, certain details leave no doubt as to the origin of the video. For example, the legs of the woman who wanders the streets of Tokyo at night are inverted at the 15th second of the video. As for the couple, we can notice a freeze which is a stain. OpenAI is well aware of this. “The current model has weaknesses. It may encounter difficulties in trying to accurately simulate the physics of a complex scene, and may not understand specific cases of causality. For example, a character may eat a piece of cookie and, the next moment, the cookie remains intact”concedes the company.
Prompt: Animated scene features a close-up of a short fluffy monster kneeling beside a melting red candle. the art style is 3d and realistic, with a focus on lighting and texture. the mood of the painting is one of wonder and curiosity, as the monster gazes at the flame with pic.twitter.com/aLMgJPI0y6
— OpenAI (@OpenAI) February 15, 2024
But we can legitimately wonder about the dangers that can arise from such a powerful tool. How to differentiate fake from real? Even if OpenAI takes care to incorporate markers to identify the videos generated by Sora, most Internet users are likely to miss it and take at face value what they see. And that’s without counting problematic content… Also, for the moment, AI is not intended for the general public. The company announces that it will “several important security measures before making Sora available in OpenAI products“. A team of experts “in the areas of disinformation and hateful content” takes care of testing the tool to push it to its limits in order to put in place the most resistant safeguards possible. Subsequently, Sam Altman’s company plans to deploy access to Sora to a few artists, designers and filmmakers in order to obtain feedback. In short, we are not close to having it in our hands!