


Just now, Pika released a new feature:
Sorry we have been muted before.
Starting from today, everyone can seamlessly generate sound effects for videos - Sound Effects!
There are two ways to generate:
- Either give a prompt and describe the sound you want;
- Or Directly let Pika automatically generate it based on the video content.
And Pika said very confidently: "If you think the sound effect sounds great, that's because it is."
The sound of cars, radios, eagles, swords, cheers... it can be said that the sound is endless, and in terms of effect, it is also highly consistent with the video picture.
Not only has the promotional video been released, but Pika’s official website has also released multiple demos.
For exampleWithout any prompt, the AI just watched the video of roasting bacon and can match the sound effects without any sense of violation.
Another prompt:
Super saturated color, fireworks over a field at sunset.
Super saturated color, fireworks over a field at sunset.
Pika can generate video and add sound at the same time. It is not difficult to see from the effect that the sound stuck at the moment when the fireworks bloom is also quite accurate.
Such a new feature was released during the big weekend. While netizens were shouting Pika "It's so curly and awesome" , some people also thought:
It's collecting all the "infinity stones" for multi-modal AI creation.
So let’s continue to look at how to operate Pika’s Sound Effects.
“make some noise” for videos
Pika’s operation of generating sound effects for videos is also extreme! That! simple! one!
For example, with just one prompt, video and sound effects can be "produced in one pot":
Mdieval trumpet player .
Medieval trumpeter.
#Compared with the previous operation of generating video, now you only need to turn on the "Sound effects" button below.
The second method of operation is to dub it separately after generating the video.
For example, in the video below, click "Edit" below, and then select "Sound Effects":
Then you can describe the sound you want, for example:
Race car revving its engine.
The car is starting its engine.
Then in just a few seconds, Pika can generate sound effects based on the description and video, and there are 6 sounds to choose from!
It is worth mentioning that the Sound Effects function is currently only open for testing to Super Collaborator (Super Collaborator) and Pro users.
However, Pika also said: "We will launch this feature to all users soon!"
Now a group of netizens have started testing this Beta version and said:
The sound effects sound very suitable for the video and add a lot of atmosphere.
What is the principle?
As for the principle behind Sound Effects, although Pika has not made it public this time, after Sora became popular, the voice startup company ElevenLabs has produced a similar dubbing function.
At that time, NVIDIA senior scientist Jim Fan made a more in-depth analysis of this.
He believes that AI learning accurate video to audio mapping also requires modeling some "implicit" physics in the latent space.
He detailed the problems that the end-to-end Transformer needs to solve when simulating sound waves:
- Identify the category, material and Spatial location.
- Recognize higher-order interactions between objects: For example, is it a stick, metal, or drumhead? At what speed does it hit?
- Identify the environment: Is it a restaurant, a space station, or Yellowstone Park?
- Retrieve typical sound patterns of objects and environments from the model's internal memory.
- Use "soft", learned physical rules to combine and adjust the parameters of sound patterns, and even create entirely new sounds on the fly. It's a bit like "procedural audio" in game engines.
- If the scene is complex, the model needs to superimpose multiple sound tracks according to the spatial position of the object.
All of this is not an explicit module, but is achieved by gradient descent learning from a large number of (video, audio) pairs, which are naturally found in most Internet videos. Time alignment. Attention layers will implement these algorithms in their weights to meet the diffusion goal.
In addition, Jim Fan said at the time that Nvidia’s related work did not have such a high-quality AI audio engine, but he recommended a paper from MIT five years ago The Sound of Pixels:
Interested friends can click on the link at the end of the article to learn more.
One More Thing
In terms of multimodal, LeCun’s views in the latest interview are also very popular. He believes:
Language (text) is low-bandwidth : less than 12 bytes/second. Modern LLMs typically use 1x10^13 double-byte tokens (i.e. 2x10^13 bytes) for training. It would take a human being approximately 100,000 years (12 hours a day) to read.
Visual bandwidth is much higher: about 20MB/s. Each of the two optic nerves has 1 million nerve fibers, each carrying about 10 bytes per second. A 4-year-old child spends about 16,000 hours in the awake state, which is about 1x10^15 when converted into bytes.
The data bandwidth of visual perception is approximately 16 million times that of text language data bandwidth.
The data a 4-year-old child sees is 50 times the largest LLM data for all text training publicly available on the Internet.
Thus, LeCun concluded:
If machines are not allowed to learn from high-bandwidth sensory input (such as vision), There is absolutely no way we can achieve human-level artificial intelligence.
So, do you agree with this view?
The above is the detailed content of Pika's amplification trick: starting from today, video and sound effects can be produced 'in one pot'!. For more information, please follow other related articles on the PHP Chinese website!

机器学习是一个不断发展的学科,一直在创造新的想法和技术。本文罗列了2023年机器学习的十大概念和技术。 本文罗列了2023年机器学习的十大概念和技术。2023年机器学习的十大概念和技术是一个教计算机从数据中学习的过程,无需明确的编程。机器学习是一个不断发展的学科,一直在创造新的想法和技术。为了保持领先,数据科学家应该关注其中一些网站,以跟上最新的发展。这将有助于了解机器学习中的技术如何在实践中使用,并为自己的业务或工作领域中的可能应用提供想法。2023年机器学习的十大概念和技术:1. 深度神经网

本文将详细介绍用来提高机器学习效果的最常见的超参数优化方法。 译者 | 朱先忠审校 | 孙淑娟简介通常,在尝试改进机器学习模型时,人们首先想到的解决方案是添加更多的训练数据。额外的数据通常是有帮助(在某些情况下除外)的,但生成高质量的数据可能非常昂贵。通过使用现有数据获得最佳模型性能,超参数优化可以节省我们的时间和资源。顾名思义,超参数优化是为机器学习模型确定最佳超参数组合以满足优化函数(即,给定研究中的数据集,最大化模型的性能)的过程。换句话说,每个模型都会提供多个有关选项的调整“按钮

实现自我完善的过程是“机器学习”。机器学习是人工智能核心,是使计算机具有智能的根本途径;它使计算机能模拟人的学习行为,自动地通过学习来获取知识和技能,不断改善性能,实现自我完善。机器学习主要研究三方面问题:1、学习机理,人类获取知识、技能和抽象概念的天赋能力;2、学习方法,对生物学习机理进行简化的基础上,用计算的方法进行再现;3、学习系统,能够在一定程度上实现机器学习的系统。

截至3月20日的数据显示,自微软2月7日推出其人工智能版本以来,必应搜索引擎的页面访问量增加了15.8%,而Alphabet旗下的谷歌搜索引擎则下降了近1%。 3月23日消息,外媒报道称,分析公司Similarweb的数据显示,在整合了OpenAI的技术后,微软旗下的必应在页面访问量方面实现了更多的增长。截至3月20日的数据显示,自微软2月7日推出其人工智能版本以来,必应搜索引擎的页面访问量增加了15.8%,而Alphabet旗下的谷歌搜索引擎则下降了近1%。这些数据是微软在与谷歌争夺生

荣耀的人工智能助手叫“YOYO”,也即悠悠;YOYO除了能够实现语音操控等基本功能之外,还拥有智慧视觉、智慧识屏、情景智能、智慧搜索等功能,可以在系统设置页面中的智慧助手里进行相关的设置。

人工智能在教育领域的应用主要有个性化学习、虚拟导师、教育机器人和场景式教育。人工智能在教育领域的应用目前还处于早期探索阶段,但是潜力却是巨大的。

阅读论文可以说是我们的日常工作之一,论文的数量太多,我们如何快速阅读归纳呢?自从ChatGPT出现以后,有很多阅读论文的服务可以使用。其实使用ChatGPT API非常简单,我们只用30行python代码就可以在本地搭建一个自己的应用。 阅读论文可以说是我们的日常工作之一,论文的数量太多,我们如何快速阅读归纳呢?自从ChatGPT出现以后,有很多阅读论文的服务可以使用。其实使用ChatGPT API非常简单,我们只用30行python代码就可以在本地搭建一个自己的应用。使用 Python 和 C

人工智能在生活中的应用有:1、虚拟个人助理,使用者可通过声控、文字输入的方式,来完成一些日常生活的小事;2、语音评测,利用云计算技术,将自动口语评测服务放在云端,并开放API接口供客户远程使用;3、无人汽车,主要依靠车内的以计算机系统为主的智能驾驶仪来实现无人驾驶的目标;4、天气预测,通过手机GPRS系统,定位到用户所处的位置,在利用算法,对覆盖全国的雷达图进行数据分析并预测。


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Zend Studio 13.0.1
Powerful PHP integrated development environment

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

SAP NetWeaver Server Adapter for Eclipse
Integrate Eclipse with SAP NetWeaver application server.
