The World’s First Live Streamed Augmented Reality Music Experience: Watch a Gig from the Front Row, Anywhere in the World

How We Used 5G to Deliver a Real Time AR Music Experience in a Different Location

What if you could watch a gig from anywhere in the world and still be right there in the front row? As part of the CORE project, we developed the world’s first live streamed Augmented Reality concert experience. This brings bands to life in real time, wherever you are. With cutting-edge 5G tech, immersive visuals and physical sensations through haptic vests, we’ve reimagined how live music can be experienced.

Meet the Minds Behind the Music Tech: The Client Powering This Live Streamed Augmented Reality Music Experience

Gooii worked alongside a consortium of partners to develop emerging technologies that are designed to test and improve mobile capacity in busy areas of the UK, or ‘High Demand Density’ locations. These included AWTG, Benetel, Ontix, Wolfram, University of Surrey and University of Cambridge. We were also thrilled to be partnered with the historic Cambridge Corn Exchange venue.

Read more about the new 5G Open Ran network. 

Turning the Vision into Augmented Reality: The Brief That Sparked a New Way to Experience a Live Gig

We were asked to design an experience that would celebrate 150 years of the Cambridge Corn Exchange using Augmented Reality. For this, we conceived a live stream experience in AR, broadcasting bands from one location to another.

Key Deliverables that Brought the Live Streamed Augmented Reality Music Experience to Life

Using the 5G network, we were able to:

• Stream 8K 3D video live from the stage at the Corn Exchange to a second location
• Stream live audio direct from the mixing desk 
• Deliver the feed over 5G in under 1.5 seconds from the camera to the receiving location 
• Deliver the experience to multiple users
• Maintain audio and video sync with no lag between headsets.

Live Streamed Augmented Reality Concert: How We Developed The Technology

Nexus Studios Sheffield designed a large bespoke green screen for the project. This was constructed for use on the Cambridge Corn Exchange stage. We then used specially adapted 3D cameras to capture the live bands. 

This followed an extensive R&D stage, where we worked with AI systems to separate performers from the background. We then removed the background on-the-fly, using streaming technologies developed in-house. This was achieved in combination with some excellent third party services.

This allowed us to place a live hologram of the band into a receiving location in front of the audience. 

To facilitate this, we developed live streaming deformation software to unpack the streamed 3D image. Using 8K spatial video, we were able to live stream the footage to Meta Quest and Apple Vision Pro headsets across the 5G Open Ran network. 

Public Trials

We commissioned two bands – folk duo Smith & Brewer and rock band Hungry – to play at the Corn Exchange during a day of public trials. 

Using the 5G network, the bands were live streamed in AR to another location, where participants put on headsets, headphones and haptic vests. The band then played in front of the audience in full 3D AR placed in the real world.

Front Row and On Stage

This created a unique and immersive viewing experience for live music fans. It also gave them a front row view of the live performance. It felt as though you were standing in front of the band and that the artists were performing to you. Compared to watching on TV or even streaming in VR, this replicates a real gig experience. This achieved creating a real emotional connection between the band and viewer. 

With our live AR experience, you can see the musicians up close, much closer than if you were at a real gig. It enables you to see little details happening live, such as finger picking and chord changes. 

One participant said: “The technology was really impressive. It could certainly replace watching live music on TV, which never really seems to engage. It added an extra dimension – literally – and I felt in the room with the performers.”

Client Testimonials

Here’s some of the feedback we got from participants of the trial: 

• “The live stream was clear and uninterrupted and the band was excellent!”
• “Sound was really excellent. Better than the live experience perhaps musically. I felt totally present and engaged.”
• “It felt really close to “being there”.”
• “The immersive clarity and ability to interact with the real world was excellent… Impressive that such an experience was live streamed.”
• “Where this industry and the world is heading.”
• “A profound experience. I found myself smiling and tapping as if the musicians were in the room. Being able to watch their finger picking and chord changes was a huge improvement on the real thing.”
• “Looked like it was real and in the room with you.”
• “A great way for people who have trouble attending shows to get to see bands live.”

Flying Drumsticks – A Real 3D Experience

The beauty of using 3D meant we could capture the dynamic nature of a live music set. Imagine guitars being flung towards you as the lead plays their famous riff, or a drum stick being thrown into the crowd. This really enhances the interactive and immersive experience. It’s enough to make you physically respond, which many participants did. 

Another participant said: “That was really good fun, a unique experience. They’re right in front of you, like you’re being serenaded. It’s interactive – you lean back at first because you think things are coming at you. Really cool! One of the most unique experiences I’ve had watching music.”

Physical Sensations: Feeling the Music

By donning a haptic vest, you physically feel the vibrations of the bass and drums. This replicates the unmistakable feeling of standing in a crowd watching a performance and physically feeling the music. 

“You get the visual, the sound and the body experience. To feel that and take it all in and experience it… It feels like you’re in the front row.”

Opportunities for Using the Technology

The success of this live streamed Augmented Reality music experience opens the door to a wide range of exciting possibilities across the music industry, and beyond. For artists, it means the ability to perform in multiple locations simultaneously, reaching global audiences without the need for travel. For venues, it offers a new revenue stream by selling tickets to sold-out gigs, or exclusive behind-the-scenes performances.

Cambridge band Smith & Brewer told us how they found live streaming a gig in AR. “It’s incredible being in two different places at once! Particularly if you have an audience spread around the world and it’s impractical to go and tour Australia.” 

AR live streaming also presents game-changing opportunities in accessibility. Fans who may not be able to attend live events due to physical, geographical or financial barriers can still enjoy an immersive, high-quality gig experience. Live sign language interpreters could also be added in Augmented Reality whilst a band perform on stage.

Educational institutions and music academies could use the technology for live masterclasses or virtual rehearsals, giving students a front row seat to watch technique in action, right down to finger placement and stage presence. 

Beyond music, this technology can be adapted for:

• Theatre
• Dance
• Sport
• Business conferences
• Lectures
• Documentaries
• Training
• In care homes and hospitals
• Helping deaf and visually impaired people.

Good’s live AR streaming technology offers real-time, spatially-aware, high-quality streams that create presence and emotional connection. As 5G and AR capabilities continue to grow, so too does the potential to redefine how we experience live events. Making them more inclusive, more immersive and more unforgettable than ever before.

The Future of Live Music Experiences

This live streamed Augmented Reality music experience has shown just how far technology can take us in creating more accessible and engaging ways to enjoy live music. With 5G and AR, audiences can now connect with performances in real time, wherever they are. 

Here at Gooii we love live music. Our AR experiences aren’t designed to replace live entertainment but to complement it. For many bands, touring the world is no longer affordable and is tied up with red tape. Artists are already streaming live gigs from venues, basements and garages. With our AR technology we can take this to the next level, so you aren’t just watching a live stream, the band is playing in the room with you. Be it at a ticketed venue, or in your own home. Gigs can also be recorded and sold again, earning repeat revenue.

An Enriching Experience

This has been an incredibly enriching and rewarding experience for Gooii. The trials provided a rare opportunity to gain insight, feedback and ideas that will further improve live AR streaming. It was also immensely gratifying to see the joy and sense of wonder we brought to people. Due to a solid and robust network, there were also no technical issues which allowed us to deliver a fully commercial experience. 

Gooii are proud to have delivered a step forward for live events, whilst opening up new creative opportunities for artists, venues and fans alike.