A deep dive into the “AI” hidden in Apple Vision Pro

Text/VR Gyro Yuanqiao

At this year’s Apple Developer Conference (WWDC 2023), Tim Cook continued Jobs’s expression of “one more thing” and released a new generation of spatial computing product-Apple Vision Pro.

This is a historic moment worth remembering. We see another "one more thing" moment coming. More importantly, human-computer interface technology will usher in a new development after Apple releases new products. The virtual/augmented reality level has taken another step forward.

The reason why I say "again" is because before Apple released new products, OpenAI launched ChatGPT and GPT large models, allowing humans to fully feel the extraordinary capabilities of AI. Correspondingly, the application of AI technology in the AR/VR field has also been highlighted. came out and even brought about a new round of changes in the industrial structure.

AI is regarded as a key factor in the development of the virtual/augmented reality industry, and it is also the underlying technology that promotes the deepening of the industry. Apple has always been the "game rule setter" in the industry and has a long-standing layout at the AI ​​level. Although Apple did not mention artificial intelligence at this year's WWDC 2023 conference, we can get a glimpse of it from the products at this conference and the new Vision Pro.

Apple’s AI layout over ten years

On June 2, according to foreign media Mark Gurman’s Twitter, Apple was recruiting talents in artificial intelligence. According to foreign media sources, Apple has posted at least 88 jobs on the recruitment portal since May. Positions related to artificial intelligence, covering areas such as visual generative modeling, active intelligence and applied AI research.

Picture: Mark Gurman Twitter

The position being advertised is part of a team that will "shape how generative artificial intelligence technology changes Apple's mobile computing platform," further indicating that AI applications in products such as the iPhone and Vision Pro will be further expanded.

But Apple’s layout in artificial intelligence is actually earlier. In 2010, Apple acquired the Siri team for US$200 million. In 2011, with the release of iPhone 4s, Siri appeared on the market as an intelligent voice assistant. , is now being expanded to Mac, iPad and even the newly released product Vision Pro.

Picture: Calling Siri in Vision Pro (Source: Internet)

Chart: List of Apple acquisitions (VR gyroscope arrangement)

Compared with this simple operation, it is easy for people to forget that Siri is actually artificial intelligence, but it works differently from ChatGPT. One of the working principles of Siri is that when it receives a command, it will be sent to Apple's data center first. The data center will analyze the content of the command and give a known answer; the second step is to upload it to the cloud and give the answer." I don’t know much about it, but I found this information on the Internet” and so on. ChatGPT has no local processing flow, it directly relies on huge calculations and model reasoning, and can give answers regardless of whether it understands the meaning of the instructions.

Source: Internet

Apple has always been cautious about the development of AI, preferring to emphasize machine learning functions and providing benefits to users. As Cook said in an interview with the media at the WWDC 2023 conference, "We do Integrate it (AI) into our products, but people don't necessarily think of it as artificial intelligence." Because of this, Apple will limit the uncontrolled development of Siri.

On the other hand, Apple’s AI strategy focuses more on two points: performance and security. At the performance level, Apple will integrate AI into its products to improve the use of the product; security is privacy, and privacy is a point Apple has always focused on. For example, in 2015, Apple acquired Perceptio, a company mainly engaged in artificial intelligence photo classification. , but in addition to automatic image classification, Perceptio also protects privacy without storing user data in the cloud.

In 2020, John Giannandrea, Apple's senior vice president of machine learning and artificial intelligence strategy, and Bob Borchers, vice president of product marketing, affirmed Apple's AI strategy in an interview and said that Apple accelerated this process in 2018. Let the ISP in the iPhone work closely with the neural engine (central processing unit), but Apple will not talk too much about its AI capabilities, emphasizing the Transformer language model, machine learning, etc.

Picture: John Giannandrea (Source: Internet)

Giannandrea and Borchers also made it clear: Today, machine learning plays an important role in Apple's product functions, and the application of machine learning in Apple will continue to increase.

AI capabilities revealed in Apple Vision Pro

In this year when generative AI is booming, many people in the industry predict that Apple may also talk about AI at this year’s WWDC 2023 conference. However, during the nearly two-hour press conference, AI was not mentioned even once. It's everywhere in its products. Mr. Gyro has sorted out the AI ​​capabilities displayed by Vision Pro. It is not difficult to see that if AR/VR wants to develop further, it will inevitably be inseparable from the support of AI technology.

1.AI digital clone

The access to AI makes the production of digital people more realistic. Even on some websites, users can create digital people that meet their own needs, and 3D virtual digital people are inevitable for the further development of VR headsets. one period. Meta has released Codec Avatar version 2.0, which is more complete than 1.0 in terms of realistic effects of digital humans.

Source: Internet

Vision Pro scans a person's facial information through the front camera, and based on machine learning technology, the system uses advanced coding neural networks to generate a "digital clone" for the user. And when the user is making a FaceTime call, the digital avatar can dynamically imitate the user's facial and hand movements, retaining the volume and depth of the digital avatar.

Source: Internet

Apple said that their team is actively optimizing this feature and will make digital people look more realistic in future visionOS updates. On the other hand, judging from the results at the press conference, the construction of digital avatars is indeed very realistic, and the operation is also simpler. You only need to use the device to scan facial information. It can be said that it has surpassed the current market. Get some digital avatar software.

2.AI emotion detection

Apple acquired an AI emotion detection company in 2016, and this year, according to foreign media reports, Apple is developing an artificial intelligence health guidance service and new technology for tracking emotions called Quartz.

After the release of Vision Pro, former Apple researcher Sterling Crispin revealed a lot of black technology about Vision Pro on Twitter. The most amazing thing is that Vision Pro can detect the user's mental state through the user's body and brain data during the immersive experience. "Brain-computer interface" or "mind reading" is how Sterling Crispin refers to it.

Picture: Sterling Crispin’s Twitter information

The main principle behind this technology is that everyone's pupils react before they do anything, in part because you expect something to happen after you click. Therefore, Apple can algorithmically monitor your eye behavior. Rewritten sentence: Create biofeedback of the individual brain by redesigning the UI in real time to produce more expected pupil responses.

Source: Internet

3. Smarter input method

In order to further satisfy users' interaction with guidance functions and spatial content, Vision Pro introduces a new input system, which interacts with eyes, gestures and voice, through looking at the application and "tap/flick" gestures. App selection, or use voice commands to browse apps.

Source: Internet

When your eyes look at the browser search box, the search box will enter the dictation input state. At this time, you only need to speak the content you want to retrieve, and you can automatically type the text and then search. Use Siri to quickly open and close apps, play media files, and more.

Source: Internet

Although Apple did not emphasize the input method in the introduction of Vision Pro, it can be seen from the introduction of iOS 17 that Apple’s updated smarter input method can not only correct spelling errors, but also correct the user’s input during the input process. Grammatical errors. More importantly, based on device-side machine learning, the input method will also automatically improve the model based on each user's input, bringing the input method's automatic correction function to unprecedented accuracy.

Source: Internet

4. New operating system "visionOS"

According to Apple, based on the research and development of macOS, iOS and iPadOS, they designed a new operating system VisionOS for Vision Pro from scratch to achieve very high spatial computing capabilities with low latency requirements; respectively embedded It includes iOS framework, spatial computing framework, spatial audio engine, 3D spatial engine, foveated rendering engine, real-time driver subsystem, etc.

Source: Internet

According to Apple, visionOS will have a new AppStore, and there will be more applications developed specifically for Vision Pro in the future, such as 3D anatomy diagrams, etc.

It can be seen from the above that the innovation of Vision Pro is inseparable from the development of AI. In Vision Pro’s spatial audio calculation, eye movement and hand behavior capture, these are the areas where artificial intelligence technology has made great contributions. Apple relies on With the computing power support provided by the M2 and R1 chips, the local deployment of artificial intelligence has been successfully realized.

Source: Internet

In addition to the ubiquitous AI capabilities hidden in Vision Pro, AI technology can be seen in almost all of Apple's products. AI has penetrated into almost every detail of its products, further demonstrating the role of "AI" in The importance of AR/VR level development.

Although Apple made no mention of AI, Cook said in an interview that "He is personally using ChatGPT and is excited about this unique application and said Apple is currently paying close attention to the tool." Cook Also adding, "Large language models show great promise, but there is also the potential for things like bias, misinformation, and in some cases worse. "

In the author’s opinion, AI as a technology should be used to improve the performance of products and better serve human beings. Too much hype or publicity has lost what this technology should be like. At the same time, artificial intelligence or machine learning will become more common in the future. Regarding generative AI, Apple may not develop it in the way that the public hopes, but I believe that after Apple silently integrates it into its products, it will definitely improve the user experience and make People's eyes light up.

“AI”, a key part of the development of AR/VR

Gyro Jun has previously analyzed the impact of "AI" on the industry competition pattern in an article. You can click to read "The disruptor GPT-4 has arrived, and the AR industry competition pattern has entered a new dimension", but aside from the industry Looking at the competitive landscape, "AI" will also bring greater improvements to the performance of AR/VR devices.

This year, Meta released a new model for image segmentation, Segment Anything Model (SAM). According to the Meta blog, SAM has learned a general concept about objects, and it can generate it for any object in any image or video. masks, even including objects and image types not encountered during training. SAM is a versatile technique that can be adapted to a variety of use cases and can be immediately applied to new image domains without the need for additional training.

As shown in the figure below, SAM can automatically segment all the content in the image:

Source: Internet

In the future, SAM is expected to be connected to AR/VR devices. For example, if it is used to identify daily objects through AR glasses, the user only needs to use his eyes to look at the object, and SAM can segment the picture the user sees; with the application of AR glasses, The user is shown reminders and instructions for the object they are looking at.

On the other hand, the progress of large-scale language models can provide a more immersive interactive experience for AR/VR devices with voice interaction and object recognition. Large language models can understand more complex natural languages, complete more complex natural language processing tasks, and then be able to understand and respond to complex instructions given by users in the form of voice, enhancing the voice interaction experience of AR/VR devices. As computing power continues to increase and large models are rapidly iterated, the interactive methods of immersive AR/VR experiences are likely to become richer.

Secondly, the development of generative AI will further reduce the cost and threshold of 3D creation in AR/VR applications and solve the large demand for 3D materials in AR/VR. Regarding the impact of generative AI on 3D creation, Mr. Gyro has previously written a short analysis. You can click to read "Is AI 3D Creation Coming?" "Grabbing for jobs" comes true.

Source: Internet

Finally, the development of "AI" integrated with AR/VR has become a major trend in industry development. Whether it is viewed from Apple's products, Meta's latest layout or the impact of AI on device updates, it all shows Fully integrating AI technology is an important part of the future development of the AR/VR industry.

However, it is worth mentioning that the development of AI has not yet seen the release of clear normative documents. If AI develops without restrictions, the negative impact it will bring to the industry cannot be underestimated, such as infringement of user privacy, fraud, data Leakage, cognitive biases, and more.

Reference article:

1.https://arstechnica.com/gadgets/2020/08/apple-explains-how-it-uses-machine-learning-across-ios-and-soon-macos/

2.https://blog.roboflow.com/apple-vision-pro-visionos-computer-vision/

3. Apple’s artificial intelligence application is different from ChatGPT

4. Apple Vision Pro actually hides a “brain-computer interface”?

5. What AI capabilities does Apple Vision Pro “leak”?

CooperationPlease contact

Contact information for Game Top:

Business CooperationConnection:

Watermelon: (WeChat 1865903032)

Wen Jing: (WeChat mutou_kiki)

The above is the detailed content of A deep dive into the “AI” hidden in Apple Vision Pro. For more information, please follow other related articles on the PHP Chinese website!