VRU: SESQUI USES 360° VIDEO TO TELL GROUNDBREAKING STORIES FOR CANADA’s 150th BIRTHDAY
Joel McConvey
Director of Digital at SESQUI
To get you started on your path to VR content creation, we’ve partnered with Udacity to offer Samsung Developer Program members 25% off Term 1 or Term 2 of their VR Nanodegree program. It’s a great opportunity to learn best practices from the experts. As an added bonus, you’ll receive a free Samsung 360 camera after you complete your Nanodegree program.
Welcome to the first Q&A in our VRU (VR to the Power of You) series, where we’re asking some of the world’s best VR content creators how they’re developing groundbreaking work that’s pushing the limits of VR.
In this first entry, we chatted with Joel McConvey, Director of Digital at SESQUI, a Canada 150 signature project that aims to engage millions of Canadians through its immersive cinematic and virtual reality content, online activities and learning programs. SESQUI is using a traveling mobile dome and partnerships with museums, planetariums and science centers to screen its marquee film HORIZON, which features more than 90 breathtaking scenes from across Canada– all in 360°.
Many people associate VR with gaming. Why do you think 360 film has the potential to push VR into the mainstream?
I think what we will ultimately see is the amalgamation of these two forms. You may always have games that hew to game convention, and films that explore the passive experience, but if we think about what is always held up as VR’s Holy Grail – “presence” – that points to a virtual environment that has the verisimilitude of film, combined with the interactivity of gaming. In this very early stage of the medium, we are still exploring, defining parameters and working with the constraints of the hardware. But with continued development, I hope to see VR become less reliant on previously established media forms and categories, and to develop its own grammar – so that something won’t necessarily be a VR game or a VR film, but will just be VR, and that we’ll all understand what that means.
Even with high-end VR headsets, users can experience poor video quality. How do you trouble shoot this?
We employed dozens of tricks. We improved video playback performance via unsynchronized PixelBuffer updates. We tested half a dozen tricks to increase frame-rates. We turned on linear filtering for videos for a smoother look. To be honest, though, the highest video quality we were able to achieve was for our marquee 360 film, HORIZON. It’s pristine – but to get it there, it meant shooting on a custom-built Red Dragon rig housing three red cameras, for a 9K image that required 3000 hours of painstaking stitching and compositing. If you can manage that, I guarantee your videos will look great.
VR is growing, but monetization of the product remains a challenge. What's your advice to VR filmmakers for attracting funding?
My answer is counterintuitive: make stuff. If you experiment with “demo” quality VR setups and amass a library of content that illustrates what you hope to do and what you’re capable of, you will instantly be more fundable than someone who has a great VR idea but no idea how to actually execute it. Like a first-time novelist, there needs to be a willingness to experiment and work without a safety net to reach a point of fundability in an environment in which funding streams are still very rare.
How has VR filmmaking changed the production process both on set and in the studio?
I think it requires a totally new approach, in everything from developing an idea to storyboarding a shoot to actually filming in 360. There are lots of funny stories from our shoots about the ways in which the crew had to hide themselves within the scene, so as not to appear in the picture. There’s no “behind the camera” in VR, so that’s a big consideration. You’d better have a big tree or fence to crouch behind.
In the studio, for our purposes, we were often fundamentally working with code. That meant a sometimes frustrating dissonance in the vocabularies of the various team members. Our 3D artists would create something based on specs, which would then change based on iterations. One thing I learned is that VR is a lot more like an ongoing process than the clean linear production of a film. Sure, you’re finished at some point. But allowing time for testing and iteration is crucial, since you never know what kinds of bugs will pop up, especially in developing for multiple platforms.
So technically, I think the really great VR filmmakers of the near future will likely need to know – or at least be comfortable speaking about – code. But on an even larger scale, I think it’s about thinking differently – letting go of the grammar of traditional film to allow VR to be itself. For instance, storyboarding in 3D or plotting in concentric circles. You need a different toolkit, but also a different mindset.
Where do you think the challenge lies in the next 5 years of hardware development, both from display and capture points of view?
All hardware challenges will revolve around aligning people's high expectation of VR with the reality of the limitations of hardware. If VR is to succeed, it will need to be what people think it already is and that means increasing pixel density, refresh rates, frame rates and allowable poly counts. The expectation of where VR is/should be is a wonderful thing and there is a need from both content creators and hardware manufacturers to meet that expectation to avoid disappointment.
Specifically, for capture, I think there’s an urgent need for a standard in turnkey 360 cameras or rigs. The contenders still vary wildly in quality, price and availability. Stitching feels like a step we’ll eventually just be able to automate. But someone has to combine design, functionality and cost in a way that makes it effortless for people to set up a 360 camera.
For display, the consensus seems to be inside-out tracking. As long as you have external sensors, you’re asking people to modify their physical spaces, which is sort of a cheat for VR, when you think about it. Wireless inside out tracking will lighten VR’s hardware load considerably, and again, ease of use is key.
What should amateur filmmakers know before shooting VR for the first time?
Practically, number your cameras carefully. Establish a clear and efficient pipeline. Watch out for shadows or other things that will make stitching a pain in the rear.
More broadly, you will almost certainly have to collaborate, and you’ll (almost) always be better off listening to those who have more experience than you. Try not to use proprietary software, unless there are already a lot of people using it.
Most importantly, this is just the beginning. You may not be able to achieve exactly what you set out to do, but we have not even seen the tip of the iceberg in terms of VR’s full potential. You are at the crest of the frontier. Keep your eyes on the horizon, but don’t forget to look around.
Thanks to Joel for taking the time to speak with us. To make sure you don’t miss any VRU content, be sure to follow us on Twitter (#BeASamsungDev) and like us on Facebook. Also, if you’re a VR creator and you want your content to be seen, explore Samsung VR. It can help you with gain the reach you need – available on Gear VR, mobile and web platforms, as well as in over 50 countries abroad, it can take your content to a global audience.
Finally, if you have an interest in VR, be sure to join us at the 2017 Samsung Developer Conference (happening on October 18-19 in San Francisco) where you can attend a variety of VR content sessions and meet and mingle with other VR creators.
