AR has been a part of the industry for several years, but arguably didn’t become a fixture of gaming until smartphones paved the way for portable and high-quality games. Mobile phones allow developers to bring their games into the real world, and do so in a form that’s more accessible to a wider range of consumers.
However, in many ways the technology is still in its infancy. Despite the success of titles such as Pokémon Go or Ingress, many developers have struggled to create AR experiences which attract wide audiences, with even popular IPs such as Jurassic Park and Harry Potter seeing mixed results in their attempts to leverage the technology.
We spoke to Amber Hu, founder and CEO of Holo Interactive and inventor of the Holokit AR headset, about his work in the field, his hopes of taking AR to the next level, and the barriers developers in the space face.
Tell us about your work at Holo Interactive.
I’m the founder of Holo Interactive, a reality computing lab, and inventor of HoloKit. My recent research interest is to experiment design patterns for face-to-face multiplayer stereoscopic augmented reality, and asymmetric mixed reality. I am a future reality computing inventor, creative, and mixed reality artist who has worked for leading companies including Twitter, DJI, Google, Microsoft, Pinterest, and The Game Company. I have extensive experience across artificial intelligence and machine learning, robotics, drone technology and LiDAR, game development, visual design, and software engineering.
What do you find particularly fascinating about augmented reality?
The world of art, design, and technology has been at the centre of my world for my entire life. Although my degrees are in software engineering, robotics, and AI from Tsinghua University and Stanford University, my passion has always been rooted in art. My long-term research interest is to invent dream-inspired paradigm-shifting medium tools that help others to create artworks for the future. I also create artworks based on my tools for showcasing possibilities and capacities.
Driven by curiosity, I would like to learn almost everything that helps me to build the tools including mixed reality, optics, computational photography, new media art, robotics, sensor fusion, smart contracts, machine learning, computer graphics, graphical design, app development, manufacturing, fundraising and more.
What makes Holo Interactive different from other companies in the same space?
HoloKit Interactive is a reality computing lab, not a corporation or tech startup.We imagine and create use cases and content for our mixed reality future as well as the hardware to make it happen. We are multidimensional; part design firm, part engineering firm, part creative studio, and part community. We combine technology and art to introduce the world to a new type of Augmented reality, called Stereoscopic AR. Powered by the company’s lightweight, yet powerful iPhone accessory Holokit X, Holo channels the best of today’s hardware technology to build apps and experiences that unlock AR’s full potential while introducing and immersing people in new realities.
What are the biggest barriers facing AR technology, and how have you overcome them?
The AR world has been stuck in neutral for many years. We have the limited utility apps that require you to hold your phone in front of your face to view an augmented experience or filter; we have niche use cases designed for expensive hardware, geared towards enterprise users; and finally, we have cheaper devices that simply display content on a larger screen without completely blocking the outside world. As we wait on the anticipated headset from Apple and the accompanying use cases that many hope will bring purpose to AR, or the killer app, we felt that the opportunity was there for the taking-or better yet- showcasing what’s possible today using great design, masterful engineering, and the technology we have in our hands from companies like Apple. Use cases include:
- Phone in Front of Face Apps – Apps for iPhone other smartphones that leverage camera capabilities for AR, including those created using technology like Apple’s ARKit, deliver a largely 2D experience overlaid onto the view of what the camera sees. The use cases, thus, are limited to games, like the popular Pokemon GO, or apps that allow you to add filters or 2D items or place things within a space like furniture. These experiences are not true Stereoscopic AR and they do not allow for a hands-free experience either.
- Limited Utility Products – The Snap’ Spectacles are a good example. These glasses allow somebody to take pictures and apply filters and also view content from friends and family with augmented filters and features. Although affordable, they haven’t really taken off and haven’t evolved much outside of the initial experience.
- Expensive, Enterprise-Focused – The second AR experience today is geared to the enterprise user, with expensive hardware with vertical specific features. Companies like Magic Leap and Microsoft, with Magic Leap 2 and Hololens, are the major players in this space, but the audience for these products are limited, the content industry or vertical specific, and the price point prohibitive for consumers. Our goal is not to train people to act like machines.
- Large, Movie-Style Displays – Yet another category of competitors are devices that mirror the display of the iPhone (and other smartphones), and enlarge it to feel more like a movie screen with a wider field of view. These devices are simply reflecting back to what’s on the display just in a wider field of view.
- Future AR & Mixed Reality Devices – Many consumers are eagerly awaiting new AR and mixed reality headsets from companies like Apple and Meta. These devices will likely come in at higher price points, north of $1,000, but will likely feature a broad range of use cases and applications built in-house and also by their developer communities. These are rumored to come out in 2023 or 2024. In Apple’s case, many of the new technologies they are introducing for iPhone and AirPods absolutely hint at a future AR and Mixed Reality headset, many of which we are taking advantage of with HoloKit X.
What experiences do you think the technology can offer users?
Holokit X is an iPhone accessory designed to bring Stereoscopic AR, and the experiences designed to showcase it, to the world at an affordable price point, with a simple, well-designed, and comfortable form factor. It is the only accessory in the world to harness the full power of the iPhone, activating the whole camera system and tapping into specific technologies and APIS that many developers have yet to tap into.
HoloKit X offers an immersive-hands free experience, unique in today’s largely handheld/2D AR world, giving users more freedom and ways to interact in space and within their new realities. HoloKit X was designed to be multi-modal and interactive with multiple control inputs at launch, and with more to come in the future-to make experiences hands free if needed, but also include physical objects that can be manipulated by a user during a game or within a reality environment.
To date, the stereoscopic games and experiences that Holo Interactive has created – from multiplayer gaming and co-presence with avatars to immersive and interactive art and dream sequences – have been developed in-house. Now, these same tools are available for others to use to create stereoscopic AR experiences that push the boundaries of what is possible while introducing AR to a broader range of consumers..
Why did you choose to release an SDK now?
With HoloKit X in the market, we’ve worked to build content and showcase demos to push the accessory and our software to its limit, to show people what is possible with today’s technology and with the iPhone they already have in their pocket. This includes demos that showcase tracking, human object recognition, LiDAR sensor, and haptics, along with multiplayer games where people can battle against each other or engage with their avatar. With interest from developers in the AR, VR, and XR space, the gaming space, the Web3 community, and media labs at universities, and creatives at marketing agencies, we wanted to start to open up our hardware and tools to third party developers to begin building experiences that will run on HoloKit X.
The HoloKit Unity SDK provides access to HoloKit X hardware, software and tools, with the major areas of focus being Stereoscopic Rendering and Hand Tracking. Stereoscopic Rendering allows the user to see the virtual content in a holographic way, and Hand Tracking enables the user to interact with the virtual content with their hands.
The technology is currently only available on Apple devices – is there a specific reasoning behind this?
Yes, HoloKit X is currently an accessory that works exclusively with the iPhone. HoloKit X, and its software and content, leverages many of the hardware and software features that Apple has available to developers that can be leveraged for AR experiences. These include: Apple Silicon, LiDAR, Camera, ARKit, PHASE (spatial audio), Neural Engine, Vision Framework, the world-class Display, Accelerometer in Apple Watch, Apple Health, and more. The Holo Interactive team has a close relationship with Apple’s developer evangelism team and is an active member of Apple’s iOS developer community. This is also the reason why we have certain requirements in place such as experience with Xcode and ARKit and experience with building on top of the Unity platform and tools for AR.
What can we expect next from Holokit?
We will continue to engage with those audiences that have a curiosity for new types of AR and XR experiences, both purchasing HoloKit X to experiment and play and looking to develop for it. We feel like AR enthusiasts, multiplayer gamers, fandom and in-person role playing fans, Web3 and NFT creators, and early technology adopters will be our main audiences. We will continue to develop our multiplayer content and co-presence experiences, with an eye on showing people the potential that exists for AR and XR and giving them a chance to experience it firsthand and with friends. We also plan to continue to grow our developer community and unlock more tools for developers to create on our hardware and software.