Dig hundreds of AR/VR patents and explore all aspects of Apple XR

Recently, everyone has also seen a lot of news about Apple XR. Instead of looking at various revelations, it is better to start with patents and see what different ways Apple XR headset may play. Therefore, we selected from Apple’s hundreds of AR/VR patents in recent years and compiled the ones that may directly determine and affect the XR product experience. Of course, these patents will not be concentrated on one product, and we cannot tell you exactly what Apple XR looks like, but we can still get a glimpse of it through the patents.

When sorting out Apple’s AR/VR patents in recent years, we mainly found the following features:

1) Some AR/VR patents come from previously acquired companies, such as PrimeSense, Metaio, NextVR, SMI, Spaces, etc.;

2) Structured light, LiDAR ranging, VST perspective, digital knobs, dynamic zoom, etc. are the focus of Apple’s innovation;

3) Explore a variety of different optical solutions, diffraction light waveguides, retinal projection, etc.;

4) A number of later patents describe a multi-lens group AR/VR display solution, which is expected to be used in conjunction with the VST camera array of the headset.

Table of contents

• 1, optical display
• 2, eye tracking
• Three, biosensor
• Four, interactive technology
• 5, VST perspective
• 6, spatial audio
• Seven, positioning system
• Eight, other sensors
• 9, appearance design
• 10, UX design
• Eleven, application scenarios

1. Optical display

◎ LBS Micromirror Scanning Solution Diffraction Light Waveguide: No. US20190377181A1, etc.

There are many different light path designs, and it is even possible to insert a reflective polarizer between the phase grating and the waveguide. The characteristic is that the light beam can be switched to change the relative polarization of the reflected light and diffracted light.

◎ Retina scan: No. US10681328B1

Apple describes an AR/VR retinal projection system. The principle is to use micromirrors to scan pixels one by one onto the user's retina. The advantage is that dynamic zoom is supported, and objects and surfaces in AR/VR images can be displayed in the correct visual direction. Shown in depth.

◎ Adjustable Lens Waveguide

In 2018, Apple explored an AR/VR display solution based on LCoS optical waveguides. The optical engines were located on the left and right sides of the glasses frame, injecting images into the optical waveguides in front of the user's eyes. In addition, the solution features an adjustable focus lens.

In another patent (US11221488B1), Apple pointed out that the focus lens can use one or more liquid crystal lenses, or a fluid-filled lens, or an Alvarez lens, which is characterized by supporting vision adjustment and alleviating myopia, hyperopia, etc. question.

◎ Holographic Optical Waveguide Micro LED

No. US20180321432A1. In the patent, Apple describes an optical waveguide solution combined with a holographic optical film, using Micro LED light source. The characteristic is that the light can be redirected according to the viewing angle, thereby forming a three-dimensional visual perception.

In addition to the holographic optical waveguide, Apple mentioned an AR optical solution based on a reflective holographic combiner in another 2019 patent (No. US20190285897A1), which has semi-reflective and semi-transparent properties.

◎ Catadioptric optical system

No. US20190146198A1, which points out a catadioptric AR/VR hook-up difficult-to-learn system. It is characterized by being composed of multi-layer high-transmittance lenses (made of glass or plastic), which may include plano-convex mirrors and plano-concave mirrors. Refractive and reflective properties. The purpose of adopting the folding optical path design is to reduce the thickness of the head-mounted optical module.

◎ Fluid adjustment lens

Patent number US20200096770A1 points out a dynamically adjustable AR/VR lens set, which is characterized by adding a fluid interlayer between the lenses to achieve dynamic focus and diopter adjustment effects. The liquid layer is controlled by an electronic mechanical pump. The more liquid there is between the lenses, the farther the distance is. Since the left and right eye optical modules are independent, the left and right eye focal lengths can be controlled independently.

◎Lenticular Display

In patents such as US20200103669A1, US20210072556A1, US11143865B1, US11187914B2, US20220035160A1, US20220146820A1 and other patents, array optical solutions based on multiple cylindrical lenses/microlenses (arranged in an arc shape) are mentioned. The advantage is that compared with a single aperture lens It is thinner, which helps reduce the size of the head display, and has a large FOV, supports dynamic viewing angle switching, and is adapted to the VST perspective module of a multi-camera array.

In addition, the optimization methods of the optical solution are described in detail, such as using louver film to control the light emission angle of the display screen to reduce crosstalk, or adding reflectors to change the perspective angle and bring the perspective scene "visually closer" to the user. ” to show more accurate scene depth/distance.

◎Multi-layer geometric phase lens solution

In patent number US20210048674A1, Apple proposed an optical solution for displaying a three-dimensional virtual and real fusion picture. It is characterized by using a multi-layer lens structure to display different objects in different focal planes and splicing them into an immersive, three-dimensional scene. . Implementing this solution requires great attention to requirements in optics, eye tracking, software algorithms, etc., in order to successfully simulate the dynamic zoom effect of the human eye.

◎ Color-changing lenses

In a 2019 patent (US11029521B2), Apple proposed an AR lens solution with a photochromic layer superimposed, which is characterized by reducing light transmittance under strong ultraviolet light, making the AR image more visible outdoors. Display clearly. Similar to common color-changing glasses, this mechanism can replace some AR glasses that use hoods.

◎ Dispersion compensation optical solution: No. US20200117003A1

This solution aims to improve the optical efficiency and image quality of near-eye optics such as AR. It is characterized by using a variety of holographic materials as grating media. Each holographic material has different grating vectors, frequencies and directions, so it can be well eliminated Color difference. This solution can be combined with polarizers and projection components to form an optical solution.

◎ Improve Micro LED screen quality

In order to better control the production quality of Micro LED screens, Apple has designed special testing equipment and architecture, claiming that it can capture LED pixels that do not meet the brightness standard before the Micro OLED is bonded to the panel (such as using display optical scanning ), after discovering the problem, the spare LED can be directly replaced, so as to repair it more effectively.

Micro LED is a relatively cutting-edge display technology. In addition to being used in mobile phones and smart watches, it is also expected to be used in AR/VR headsets in the future.

◎ 360° perspective

In patent number US20210092292A1, Apple describes an AR/VR head-mounted display solution equipped with 8 cameras. The feature is that 8 cameras are arranged around the head-mounted display (including the front cover and headband) for capturing and see through the surrounding 360° environment, thereby enhancing the user's environmental perception and being able to see even in blind spots.

◎ Main screen secondary screen design

In a patent published in 2021, Apple designed an AR glasses solution based on the main screen and secondary screen. The feature is that the main and secondary screens can operate independently. The main screen has a higher resolution and is used to display the main AR functions. , and the secondary screen has a lower resolution and mainly displays some simple prompt information, such as countdowns, events, etc., and there is a circle of LED indicators on the outermost edge of the frame to remind that the AirTag is nearby. The advantage of this design is to use the limited screen area more rationally while reducing computing power and power consumption.

In patent number US11450297B1, Apple designed a foveated rendering scheme with one main screen and two secondary screens. These three screens (can use flexible or rigid panels) are arranged in an arc that conforms to the curve of the head. In order to hide the screen borders, Apple added an independent layer of diffusion structure to the lens to optimize the connection between high-resolution and low-resolution screens.

2, eye tracking

Eye tracking technology is a key point repeatedly mentioned in Apple’s AR/VR patents. Its main uses include: gaze rendering, power consumption reduction, dynamic focusing, biological monitoring, vision detection, gaze input, etc. In addition, the patent also describes in detail the implementation and optimization of eye tracking, such as:

• Near infrared solution (can capture pupil dilation and gaze point);
• Scheme based on directional retinal flash;
• Coaxial illumination scheme (specific pulse band light source and image sensor are on the same axis, the advantage is that it can capture brighter pupil images);
• Eye tracking delay compensation and effect optimization.

The following are some of the more distinctive ones:

◎ No. US20200019238A1: Saccadic saccades of the human eye are a common phenomenon, usually several saccades per second (succade). During the short period of saccades, visual sensitivity will also be short-lived. decreases, causing temporary blindness that is not easily noticeable. It is precisely by utilizing this characteristic that the system can adjust the display content without being noticed by the user, thereby reducing power consumption and reducing the risk of screen burn-in.

This solution needs to be implemented in combination with eye tracking technology, and may use CV or IR sensor solutions.

◎No. US20190339770A1: The patent describes a mechanical zoom AR/VR solution based on gaze tracking, which can switch the focus of the picture following changes in the user's gaze. In order to make up for the delay of the zoom system, the patent focuses on a gaze point prediction technology that can predict human eye movement trajectories (eyeball scanning landing points) in advance and pre-render the image. The advantage is that it can make focus switching smoother. slip.

In 2018, Apple applied for a patent for a foveated rendering scheme that can achieve 8K resolution (center visual area). In 2019, Apple patents were also unveiled that mentioned the foveated rendering solution (numbered US20190172399A1, US20190180672A1). It is characterized by the use of two screens (high-definition main screen and low-definition secondary screen). The resolution is expected to reach 8K, and the screen ppi is in Between 1000 and 2000.

◎ No. US20190133440A1: points out a solution to improve and optimize the accuracy and stability of eye tracking by identifying the dominant eye. The dominant eye is also called the dominant eye. Focusing on analyzing the data of the dominant eye in eye tracking can better align the optical axis and visual axis of the eyeball, resulting in better and faster calibration results.

Another benefit of identifying the dominant eye is that it can help users diagnose eye diseases related to vergence, such as strabismus or amblyopia, and optimize the fixation point rendering technology on this basis. In order to further optimize the adaptability of eye tracking to a wide range of people, Apple mentioned in the patent number US11644896B2 that the eye tracking system can extract the eye characteristics of different users as control parameters, and dynamically calibrate the gaze point on this basis.

◎ No. WO2020068447: Apple has proposed a way to open AR/VR applications based on eye tracking. It is characterized by a combination of voice gameplay. The user only needs to stare at the target application logo and say "Open" You can open the app and the control is natural and fast. In another patent published in 2021, Apple also proposed a solution to quickly switch text input boxes using gaze points, allowing you to fill out forms and enter personal information faster in VR.

According to Apple, eye tracking can also dynamically optimize first-person video recording, such as using the gaze point to drive the camera shooting position, placing the user's attention in the center of the video, and quickly moving the camera according to changes in the gaze point (using smoothing algorithm for anti-shake). This will be very suitable for future all-weather AR glasses.

◎ No. US20200278539A1: In order to save the power consumption of eye tracking hardware, Apple designed an event camera-based solution in the patent, which is characterized by only turning on when eye movement is recognized, rather than continuously Start scanning. Another feature of this patent is that it can adjust the light intensity based on the user's biological information. When the user blinks an abnormal number of times or recognizes a child user, the system algorithm will reduce the brightness of the lighting source to improve visual comfort.

Another power-saving solution is to combine low-resolution and high-resolution eye tracking modes to identify the relative movement and accurate position of the eyeballs respectively, and combine it with head sensors to locate the gaze point.

◎ No. US10354136B2: In order to improve eye tracking accuracy, reducing drift is a key. In this patent, Apple introduces a special lens system that it claims can reduce eye tracking drift. The solution is based on ray tracing, which can model the eyeball and more by identifying the data of light reaching the human eye and being reflected.

Some eye tracking patents also come from the SMI (SensoMotoric) company acquired by Apple in 2017, such as US10437327B2, which proposes a solution to capture gaze points through secondary imaging. The characteristic is that the first imaging captures clear eyeballs. It is used to locate the pupil position, while the second imaging has lower resolution and is used to identify bright eyeballs. After the two capture data are combined, the gaze point prediction data can be more accurate.

Three, biosensor

In some patents, Apple proposed adding biosensors to AR/VR headsets to sense the wearer's physical status and change the displayed content accordingly. In terms of details, the headset can identify information such as strength, temperature, humidity, displacement, EEG, capacitance, EMG and heart rate to assess psychological stress and personal health status, and then recommend applications or content to help relax in AR/VR, and even Can proactively prompt medical information.

In a 2019 patent, Apple described a facial tracking solution, which featured a variety of biosensors in AR/VR headsets and glasses, such as PPG (photoplethysmography) and ECG. (Electrocardiogram), GSR (Electrical Skin Signal), Pressure Sensor, eField, Thermal Sensor and more. The device can detect the user's health status by analyzing a variety of complex facial movements such as chewing, blinking, and nose wrinkles.

Four, interactive technology

1, air gesture

Some of Apple’s 3D gesture technology comes from PrimeSense, which was acquired in 2013. It can be found from some related patents that the inventor is a former PrimeSense employee. For example, in a patent, a 3D gesture solution is introduced, which allows users to use grab, push, release and other gestures to operate the 3D user interface to play games or play videos.

Air gestures are a common and important technology in Apple’s AR/VR patents. They can not only be used in AR/VR headsets, but can also be implemented on iPhone, iPad, Mac and other devices.

Apple’s air gesture system can use a variety of solutions, such as structured light, LiDAR, RGB, IR sensors, ultrasonic transducers, VSWR sensors, etc.

For example, in patent number US20210011289A1, Apple describes an AR/VR head-mounted display solution that supports Inside-Out positioning. It is characterized by being equipped with an RGB camera, ToF sensor and IR LED module, and supports gesture tracking and 3D environment ranging. and many other functions.

2, somatosensory/smart gloves

Apple has applied for multiple patents for AR/VR gloves, covering various technical solutions, such as adding magnetic sensors to the fingertips of the gloves to detect magnetic field changes and interference to locate the relative position between the magnetic source and the sensor. position to achieve gesture positioning. Flexible force sensor arrays and light-conducting materials can be used to detect small tactile interactions and dynamic changes in grip force.

In addition, a glove solution based on IMU positioning is also proposed. The feature is that the IMU sensor is built into the knuckle part of the glove, which can accurately detect the movement and position of the palm and fingers without being troubled by factors such as occlusion, so it can identify finger crossing movements. .

In patent number US11397486B2, Apple also proposed a VR glove that uses ultrasonic waves to recognize gestures. It is characterized by the ability to recognize the pressure exerted by the user's palm and the thickness of the fingers.

In addition to tracking gestures, Apple also hopes to give AR/VR gloves somatosensory feedback (No. US10481688B1, US20200257362A1) based on technologies such as friction resistance and flexible patches to simulate "touching" the sea, waves, trees, beaches, and steering wheels. The feeling of texture.

In another 2020 patent, Apple described a flexible body-feeling glove solution based on fabric material. The feature is that the fabric part can expand or contract to adapt to users with different hand shapes.

3, smart bracelet/watch

In a patent, Apple describes a solution for using a smart watch to control VR games. It is based on skin sensing and can infer hand joint movements by comparing and analyzing vein images.

In patents such as US20210303068A1, Apple also proposed a solution to identify the contact movements of both hands on smart watches through camera CV, UWB, EMG, RF, sensing circuits, etc., allowing you to use one finger to touch another Swipe with one palm to simulate cursor movement, dragging, opening, or other commands.

In fact, Apple Watch has launched a gesture recognition function (AssistiveTouch) in 2021, which can recognize movements such as making a fist and pinching fingers, allowing you to control the watch interface with one-handed micro-movements. In the future, this form of interaction may be combined with AR/VR headsets to directly enable natural gesture interaction.

4, smart ring

Currently, the main interaction method for AR/VR on the market is handles, followed by gestures. Companies such as Google, Meta and Apple are studying the use of smart rings to control AR/VR technology. Especially in Apple's AR/VR related patents (US20190004604A1, US10444834B2, etc.), there are often introductions to smart rings, involving various designs.

For example: U-shaped finger rings (elastic/flexible material, adjustable size), rings, finger cots (tracking the movement of five fingers), or fashionable spiral, hinged and other designs. It features an expandable appearance and can be used in multiple units or collaborated with peripherals such as Apple Pencil.

Features include wireless design, optical positioning of fabric, and multiple sensors: touchpad, NFC communication, biosensor, SMI pressure sensor, accelerator needle and other devices, which can recognize skin contact/pinch, tap and other micro-gesture interactions , supports 6DoF positioning, and can even identify environmental temperature, physical surface shape and texture, as well as biological signals such as heart rate, blood pressure, and blood oxygen. Some patents also mention that the smart ring can be equipped with some smart functions close to those of the Apple Watch, such as digital knobs, unlocking Mac computers, etc.

5, stylus interaction

In some patents, Apple has proposed using a stylus (such as Apple Pencil) to create 3D objects in space, which is very different from traditional touch-screen-based stylus solutions.

For example, the number US20180018057A1 points out a stylus design based on non-electronic surface input. The characteristic is that the stylus has a built-in positioning sensor that can identify content written anywhere.

In a 2020 patent, Apple described a stylus that supports 6DoF positioning. It is characterized by supporting shear force feedback. Users can feel the simulated surface, volume, internal texture, etc. when holding the pen. Physical somatosensory properties.

In another patent, Apple also proposed a simple stylus adapter that would bring pressure sensitivity to the Apple Pencil (similar to the trigger keys of a VR controller) and allow users to use the Pencil in space Enter information. In addition, the stylus can also be used as a controller to play games (US20230042447A1), such as tapping, scrolling, rotating, flicking, and sliding in AR/VR.

6, virtual keyboard

No. US10237509B1: In order to help AR/VR users quickly input text, Apple proposed a virtual keyboard design in the patent. The concept is to superimpose a layer of virtual keyboard on the physical keyboard. Compared with the camera directly seeing through the keys, the virtual keyboard Vision looks clearer. In terms of input hardware, the solution is compatible with physical keyboards, touch-screen keyboards for tablets, and gesture tracking gloves.

No. US20200088999A1: This patent envisages a projection-based virtual keyboard solution. It is characterized by a projection device on the headset, which can project the keyboard onto a physical surface such as the desktop, and use a 3D sensor to identify the user. The action of typing on a keyboard. It is worth noting that the headset mentioned in this patent also supports other external screen interactions, such as adding buttons, touch sensors, and touch displays to the casing, allowing users to use it quickly without wearing the headset.

In another patent (US10890952B2), Apple also mentioned a similar interactive surface projection solution, which is characterized by turning any physical surface into an operable menu, such as superimposing a calculator function on a notebook , or overlay interactive UI on the arm.

Five, VST perspective

It can be seen in some patents that part of Apple's VST technology is the perspective technology obtained from the VRVANA company they acquired in 2017. VRVANA technology is characterized by correcting the image distortion of the optical lens, and the VST perspective delay is only a few milliseconds. In a patent published in 2019 (one of the inventors is VRVANA founder Bertrand Nepvue), Apple described an MR solution based on the principle of real-time chroma keying, which can replace the background of people in real scenes with virtual ones. Content, similar to green screen replacement.

In addition, Apple pointed out in another patent in 2019 (US10330935B2) that the headset can be equipped with two high-resolution and low-resolution perspective cameras. This design uses VR’s gaze point rendering technology to reduce the Perspective resolution of the user's peripheral vision to optimize computing tasks and reduce hardware costs.

The patent also points out that the AR/VR headset captures the surrounding three-dimensional environment through two sets of RGB cameras, claiming to be able to simulate the human visual system.

6, spatial audio

Apple has also applied for a number of patents for spatial audio, such as some solutions based on HRTF (head-related transfer function), geometric acoustics and other technologies, or simulating indoor sound fields through real-time sound ray tracing. In addition, we also explored a solution to predict two-channel audio based on head positioning information, which is characterized by dynamically adjusting the simulated distance and direction of the sound based on the user's head movement to improve the experience of VR RPG games, horror movies and other content.

In US2022032965A1, a solution for locating the user's orientation based on the direction of the head to the torso is described, similar to the body orientation mode of AirPods.

In a 2021 patent, Apple pointed out that AR/VR headsets can be equipped with multiple microphone arrays, support directional audio detection, and can calculate the source of a sound in the surrounding environment and visually mark or hint. Such a function can well complement human hearing and help users identify sounds beyond the normal range of perception (a similar concept can also use AR to mark gases, electromagnetic signals, etc. that are invisible to the naked eye).

In a 2023 patent (US2023014111A1), it was also pointed out that spatial filters can be used to add spatial cues to binaural audio according to the user's environment, making the actual experience of spatial audio more immersive. territory.

Seven, positioning system

◎ Ultrasonic positioning

Currently, mainstream VR positioning is based on optical solutions, such as infrared, CV, etc. In patent number US20190094955A1, Apple proposed an ultrasonic positioning solution based on the time-of-flight ranging principle, claiming that it can measure the relative distance between the sensor and walls, targets, and obstacles, generate a 3D map for the surrounding environment, and locate Controllers, headsets, and detecting obstacles and identifying surfaces.

◎ 3D ranging/scanning

Apple has mentioned in patents more than once that AR/VR headsets can be equipped with multiple 3D sensors, including: structured light sensors, binocular 3D image sensor modules (composed of two or more cameras), LiDAR, 3D Radio sensors, infrared or CV cameras, etc. These sensors can be used to locate spaces and identify obstacles.

Eight, other sensors

◎Multi-sensor array

In 2021, the USPTO exposed a key AR/VR patent from Apple (US20210165229A1), which describes a split AR/VR head display equipped with multiple cameras and sensors. Why is it important? First of all, the patent inventors are some key figures at Apple, such as HoloLens co-inventor Avi Bar-Zeev, and Geoff Stahl, who has worked at Apple for more than 20 years. Recently, Bloomberg revealed that Stahl is now responsible for the xrOS operating system, a key application in Apple’s AR/VR project.

In addition, in patent number US20210294104A1, Apple also envisions that AR/VR headsets can be equipped with more sensors, such as cameras, eye tracking modules, lidar, fingerprint recognition modules, and gesture recognition modules. , ultrasonic sensor, optical distance sensor, retinal scanning sensor, temperature sensor, EMG, oxygen sensor, heart rate sensor, blood flow sensor, resistive sensor, strain gauge, accelerometer, magnetic sensor, potentiometer, etc.

◎ Hall sensor (No. US10557724B1)

Usually, AR/VR headsets use distance sensors to sense the wearing status. After detecting that the user is wearing the headset, the system will be turned on, and after detecting that the user has taken off the headset, it will enter the standby state. Compared with distance sensors, Hall sensors are more power-saving. Apple hopes to use multiple Hall sensors and magnetic mechanisms in the headset to determine the wearing status by detecting the rotation angle of the device. In fact, Apple's Bluetooth headphones AirPods also use Hall sensors.

Nine, appearance design

◎ Digital Knob

Apple has mentioned the application of digital knob design on AR/VR headsets in multiple patents (such as US20210089136A1). In fact, the digital knob is a classic interaction method for Apple products. It has previously appeared in Apple Watch and AirPods Max headphones. When used in AR/VR headsets, it can adjust the volume, screen brightness, and even switch between AR and VR. VR mode.

Interestingly, in patent number US20230162929A1, Apple pointed out that digital knobs can support more complex operations, such as combining multiple rotation actions to trigger combination instructions to achieve richer interactions.

◎ Flexible smart textile materials

Currently, AR/VR mainly uses fabric wrapping on the outer casing, and conductive fabric design has not yet been applied. In patent number US20190191557A1, Apple proposed a flexible smart fabric solution, which features built-in grooved conductive cables and can be used in wearable devices such as armbands and AR glasses.

◎ Hidden camera

The essential parts of AR/VR headsets are optical sensors and cameras, which are mainly used for sensing the environment, positioning and gesture recognition. Wearing AR glasses around the clock may cause the cameras and sensors on the glasses to run for too long, causing dissatisfaction from others around you. In order to make the camera look more concealed, Apple proposed an infrared light-transmitting one-way mirror coating in the patent (No. US20190353836A1). The principle is similar to the one-way mirror commonly used in police interrogation rooms. The coating can pass through Infrared light (does not affect SLAM positioning), the internal camera cannot be seen from the outside.

◎ Large area touch control on the casing

In patent number US20210089150A1, Apple proposed a large-area touch-enabled AR/VR head-mounted display housing design, which is characterized by being equipped with sensors, slider buttons, knobs and other operating methods to support flexible external control. .

In other patents, Apple also mentioned some kind of "invisible" button and touch slider scheme that is not easily noticeable to the naked eye to optimize the product aesthetic design of AR/VR headsets.

◎ AR head display box

Apple has proposed a plug-in AR/VR head-mounted display box design in multiple patents. The feature is that the mobile phone can be directly inserted into the lens frame to display content in front of the user's eyes. The temples can also be equipped with touch sensors for operating AR/VR interfaces. In an updated patent (US11258891B2) in 2022, a new feature was also added: support for AR (VST mode)/VR mode switching.

Compared with integrated AR/VR headsets, the advantage of headset boxes is that they are lower in cost, can flexibly use existing iPhones, and can even protect the privacy of viewing mobile phones. However, judging from a series of revelations, it seems that Apple will not launch such a product for the time being.

◎ Modular AR glasses

In a patent published in 2020, Apple conceived a relatively advanced AR glasses product design, which is characterized by lenses, frames and other parts that are highly modular and can be replaced and disassembled, such as according to different needs and scenarios. , replacing different components, such as high-resolution display modules, long-lasting batteries, smaller components, etc. It is worth noting that the AR glasses frame and temples adopt a plug-in mortise and tenon structure, so users can assemble complete AR glasses by themselves without additional tool assistance. This gameplay is similar to the replaceable strap mechanism of Apple Watch, allowing users to personalize the product.

◎ Configurable and expandable AR/VR audio system (No. US20200089008A1)

The earmuffs of the headphones are made of deformable foam material, which can be inflated and deflated to switch between noise reduction and penetration modes to change the audio mode. In the VR scene, you can quickly switch to immersive audio (noise reduction mode), while in the AR scene, it is changed to open audio, which plays sound through the bone conduction principle.

◎ Lens protection mode

Apple engineers envisioned a protective mechanism in a patent published in December 2019 to prevent the lenses of AR glasses from shattering upon impact from falls. This solution can detect the status of AR glasses through sensors. When the glasses are freely falling and are hit, the protection mode of the optical system will be activated, that is, the relevant lenses will be moved and air or special liquid will be filled between the lenses to form a buffer. This reduces accidental losses caused by collisions.

In patent number US20210302745A1, Apple also proposed an optical module self-checking system. When a drop, collision, etc. is recognized, it will automatically detect the misalignment of internal components and automatically perform position compensation to restore the situation. default value.

Interestingly, in another patent (US11340466B2), Apple also proposed an elastic frame design, which is characterized by deformable temples that can act as a buffer when hit, avoiding optical modes such as optical waveguides. Group deformation.

◎ Headset strap optimization

No. US20190072772A1: Usually, users need to adjust the straps according to their head shape before wearing the AR/VR headset. This requires a process, so it cannot be used like a mobile phone. When multiple people share an AR/VR headset, everyone needs to adjust the straps, which can be cumbersome.

In order to simplify the process of wearing the headset, Apple describes a mechanical strapping system in the patent, which is characterized by recognizing the parameters of the user's head and automatically adapting to the size. It is worth noting that the system will also adjust based on gestures and head-mounted content, such as assisting in taking off and wearing actions through the position of the user's hands. When running content that does not require the user to move (such as watching movies), the straps will Will relax.

In addition, Apple also proposed a foldable headband system (US11320659B1) in a patent, which features a combination of flexible straps and magnetic modules. It is characterized by easy storage and supports modular expansion ( For example, attach cameras, batteries, speakers and other components to the headband).

10, UX design

◎ Active cooling system

No. US20190075689A1: In order to improve heat dissipation efficiency, Quest 2 has used fans to actively dissipate heat. In contrast, the solution described in Apple's patent has better ventilation and is characterized by two fans on the upper and lower sides of the headset to achieve up-and-down convection to cool internal parts.

In addition, the fan system can dynamically adjust the intensity according to the rendering task, and can also blow air on the face to cool the skin.

In a 2022 patent, Apple further proposed a fan noise control system, which is characterized by identifying the noise of the cooling system and adjusting the fan speed in real time to reduce noise to avoid disturbing users and making and receiving calls. , Siri voice assistant communication. Interestingly, the fan volume can also be increased to alert the user that there are obstacles around them.

◎ Self-cleaning of optical components

We know that AR/VR headsets are usually based on precise optical structures. If dust or debris enters the gap between the lens and the display screen, it is likely to hinder the spread of the light source, thereby affecting the display effect. However, optical components are usually difficult to disassemble and difficult to clean manually. Therefore, Apple pointed out a self-cleaning optical solution in a patent published in 2021. The concept is simple enough. It mainly detects optical quality. When a problem occurs, it shakes away the dust particles and safely removes them. The phone is out of sight of the user.

◎ Facial Adaptation System

No. US20200081259A1: Apple has proposed a modular head-mounted facial support system, which is characterized by the use of retractable materials, inflatable materials, and elastic rubber materials. The advantage is that it can be adjusted according to different people’s facial structures (forehead, forehead, etc.) Cheeks, temples) adjustment, even if the mask is misaligned due to shaking of the headset, the fit can be adjusted to improve comfort.

◎ Cross-device authentication and unlocking

In a patent called "Authentication Device Assisted User Authentication" (US10893412B2), Apple pointed out a multi-device nearby authentication solution that allows users to wear an AR/VR headset to directly unlock their iPhone, thereby eliminating the need for Repeat the steps of putting on and taking off the headset and unlocking the phone. This cross-device linkage design has been adopted on Apple Watch, allowing you to quickly unlock your iPhone or Mac computer. In addition to using the unlock function, you can also lock devices such as mobile phones by using the head-mounted display, and you can also set up a two-step verification mechanism.

In another patent numbered US20210105614A1, Apple also mentioned a similar multi-device authentication and unlocking system, allowing users to use AR/VR headsets to unlock nearby Apple devices.

◎ Enhanced fusion display

No. US10347050B1, Apple has proposed a solution that uses AR to distort images to enhance the display effect. Its characteristic is to use AR to change the visual effects of the real environment by identifying the user's visual blind spots, making it invisible to the original user. The content can also be visible in some form. The solution is based on eye-tracking technology and requires the use of 3D depth perception to capture the surrounding scene.

◎ Wireless induction charging for headset

In patent number US20200081490A1, Apple proposed an inductive charging stand suitable for AR/VR headsets. The feature is that the headset can be directly hung on it for display, or the battery can be separated and placed on the stand for charging. The advantage of this design is that it can achieve beautiful storage and display of the head display, and the head display can be charged when not in use, making it convenient for users to take and use it at any time.

◎ AirPlay

In multiple patents, Apple has proposed a solution for linking AR/VR headsets with computers, laptops, and TVs. The concept is similar to the existing AirPlay function, which can quickly synchronize the screens of other surrounding devices in the headset. (To play videos, load documents, email, etc.).

◎ Millimeter wave communication system

In a 2020 patent, Apple proposed a multi-person interaction solution based on millimeter waves, which is characterized by realizing large-scale wireless communication with electronic devices through millimeter wave base stations and multiple relay base stations. Communication, such as streaming content to multiple AR/VR headsets, etc.

Eleven, application scenarios

In some patents, Apple has deeply discussed various application scenarios of AR/VR, such as immersive movie viewing, AR navigation, AR video calling systems, etc.

◎ AR Call (Patent No. 20160344972)

Based on camera capture and generation of 3D models, it is used to achieve immersive remote AR calls. This solution can be used on devices such as iPad and Apple TV. Its feature is that it can capture the caller and the objects held by the caller and present them in the video in a virtual form.

It will also be combined with external cameras, equipped with RGB image sensors, depth sensors, IR sensors, etc., which can recognize mid-air gestures and simplify menu control.

◎Virtual Assistant

It is characterized by context-aware capabilities, using computers to generate characters or images for prompts, navigation, and display of additional information to assist users in using AR/VR. In addition to visual assistance, virtual assistants also support voice interaction.

◎ AR Navigation

No. US20170213393A1: Apple’s AR/VR headset may support visual navigation, such as marking additional information on buildings in the real scene, points of interest, or using AR to mark the speakers in the car and other functions. In addition, gesture interaction is supported. The patent comes from Metaio's technology. In addition to AR navigation solutions, Metaio also applied for patents for AR environment recognition (3D environment capture, and data recognition of weather, time, coordinates, etc.).

In another patent in 2018, Apple also proposed a 3D AR map solution that can be viewed from multiple angles.

◎ AR sports broadcast

No. US20200175275A1: Add interactive AR content to sports game broadcasts to enrich the viewing experience. The characteristic of this solution is that it requires the use of external devices to detect the audience's points of interest and display additional information on this basis. For example, when you are interested in a basketball player's shoes, the system will display information about the shoes on the TV. The sneaker model can also be displayed in the form of AR, allowing viewers to view it from multiple angles.

◎ Low latency chroma keying

Zoom and other video software already support the background replacement function, and in VST mixed reality headsets, background replacement can also be well applied, such as replacing the people and objects in front of you with the actual background style and replacing it with the virtual background in real time . Apple's patented solution is different from usual background replacement solutions in that it is more three-dimensional and can present a 3D background rather than a 2D image.

◎ Collaboration with computer

In some patents, Apple mentioned the application of AR/VR in office scenarios, such as using a virtual screen to expand the display area of ​​the computer screen with a Mac computer, or displaying the computer screen content in the headset to protect the User privacy and preventing screen snooping.

◎ Mobile phone, etc. connected to AR/VR head display (No. US20190065027A1)

describes a certain split touch interaction solution, which uses devices such as mobile phones with touch pads, touch buttons and displays to control the content displayed by AR/VR headsets. In other words, using the iPhone as a controller for an AR/VR headset can be seen as another way of interaction besides gestures and controllers, and can be well integrated with Apple's family of devices.

◎ 360° video encoding/decoding technology

Patent number 20190004414 points out an adaptive resolution and projection format, which is characterized by multiple preset encoding configuration schemes, and whether to insert frames according to the imaging effect, thus improving the rendering quality of 360° and MR videos.

While we have these patents, they only reveal a small part of Apple's XR device development, so there's no way to determine what the final look and functionality will be. This article is just a starting point, giving you some clues and inspiration through these patents. Apple's XR device is yet to be announced, and we look forward to giving it a glimpse at WWDC23.

The above is the detailed content of Dig hundreds of AR/VR patents and explore all aspects of Apple XR. For more information, please follow other related articles on the PHP Chinese website!