On Creating Augmented and Virtual Realities Book, and the Next Edition

Happy Monday and December! Happy Holidays (Christmas/Hanukah/Kwanzaa/Simbang Gabi etc.) all! 🎄🎁❄️☃️🎉🎂

As a December baby, I wish you all a wonderful, restful, and joyful holiday and winter season! It’s December (my birthday month =D yay so shout-out to all you winter babies.

So this was my book, I published in 2019 and worked on in 2018, which you can buy on Amazon and learn more about here. Many of the fundamentals are still relevant, some open source frameworks are no longer maintained, and new headsets have arrived on the market this year, with greater anticipation for Augmented Reality (AR) / Artificial Intelligence (AI) glasses, which is all the rage and very exciting!

Throwback: Me and Creating AR VR Contributors and Reviewers at our book launch at Unity in 2019

While this was a team effort with many contributors and 2 other co-editors and O’Reilly staff (I felt like I was managing about 30-50 people at any given time), the next book I wanted to write was going to be completely solo, and what better time to do it then after the release of a headset I had been waiting many years for (Apple Vision Pro).

Earlier this year, I pitched to O’Reilly Media, an idea for a second edition of the book, or an entirely new one focused on Apple Vision Pro (AVP) and Artificial Intelligence (AI). While I initially thought of writing the book to include Swift development with XCode, and potentially Meta Quest 3, I also had many requests to also include tutorials for cross-platform development (using C#, C++, PolySpatial, so both Unity, Unreal Engine), and this would be a lot to write, by itself (even without those working on cryptocurrency/blockchain/web3 who also traverse into the XR space - the ones who actually build protocols or other helpful things like user control (like MetaGuard) with an actual headset, not the NFT crowd that randomly makes webXR environments and monetizes assets without a headset component. For me, the headset component while seemingly seen as essential for me is just about 3D programming with an actual tangible interface that is different from 2D computing on the web (the other Holy Grail being the creation of the HoloDeck, or 3D holograms in thin air and focus on graphene and physics or AR contacts).

Slide from my talk QCon “Multidimensionality: Spatial Computing x Spatial Intelligence to Create New Worlds - or more accurately “Creating XR with AI”

While AVP sales have fallen short of expectations to many, the technical breakthroughs are impressive. And despite Meta AR Glasses having an open Application Programming Interface (API) and Standard Development Kit (SDK) for third-party developers, it remains popular to consumers. Now with the anticipation for another pair of glasses coming from a partnership between Google, QualComm, and Samsung, we are in it for the next wave of spatial computing / AR VR MR XR working in tandem with Artificial Intelligence (AI).

I also wanted to include crypto/blockchain/web3, which many people outside of that industry find somewhat less palatable, and others simply too much of a challenge to handle multiple topics between cross-platform development and AI (I found that wanting to write tutorials beyond Swift a bit too much to be quite honest, it’s a ton to cover). As I’ve mentioned before in an earlier post about MetaGuard, we have a lot of room for growth in terms of AI Safety that can be applied to XR safety that has not been built quite yet, and this is partially due to the fact that major tech companies dominate this area, which lock in certain levels of control (what some call a walled garden so to speak) and make it more challenging for consumers to have a wider variety of options in terms of product choices (not everyone wants to opt in their eyeballs for eyetracking and arm data to FAANGMULA companies).

To my surprise, I was also approached to consider writing a course on game development for another platform focused on AI, and while I did not pursue that route, much of my internal research and following market trends includes seeing what other new AI creation tools (beyond game engines) can be applied to XR. It’s not quite writing a single prompt and a fully baked application can come right out and be ported into a device, the tooling was never that sophisticated and still required an in-depth knowledge of a human to be able to create assets, meshes, environments, and characters. In my QCon presentation, I talk about generative AI being able to create these elements using a variety of techniques, specific algorithms that are developed (from GANs, Stable Diffusion and onward) to achieve the goal of making it much easier to create. As the cost of compute drops, and developing more inclusive UIs lowers the barrier to entry, software engineers, computational designers, game designers, creators of all kinds are challenged with what makes their creation truly unique and human (as opposed to an AI replicating/mimicking some known style with a few tweaks).

Now that I’m in the last quarter of the year, I would like to know, what other topics would one want to cover for creating XR with AI? You can submit your feedback on this survey and I will pick 3 winners to receive books in Chinese, Korean and English.

SUBMIT FEEDBACK FORM

Next Substacks

Not a huge fan of AR VR and want me to talk about AI more? Are you= productivity and self-development junkie like me and need a a fix? Or are you a crypto/blockchain/web3 daytrader wanting more insight into the market?

In my next Substack, I’ll touch on AI Safety, Differential Privacy (these topics are some of the last lectures the last two weeks of UC Berkeley’s LLM - Large Language Model Agents course I’m completing this term).

And yes, there will be a dedicated post afterward on productivity I’ve been cooking up for quite some time, and web3 and all the other things (how to learn each of these stacks). There’s a ton that’s gone on and all other select and older posts from my social media and blog will be apart of my blog archive in new website to be released soon.

And frequency wise (how often I post), I promise these all won’t come in a single day! I had a giant backlog I’ve been trying to clear for weeks, and a schedule of these Substacks since earlier this year, so want to cover a lot of ground, other than the intro / life story on me.

Erin Jerri Malonzo Pañgilinan is a software engineer and computational designer. She is an internationally acclaimed author, publishing Book Authority’s #2 must-read book on Virtual Reality in 2019, O’Reilly Media book, Creating Augmented and Virtual Realities: Theory and Practice for Next-Generation of Spatial Computing, which has been translated into Chinese, Korean, and distributed in over 2 dozen countries.

She was also previously a fellow in the University of San Francisco (USF) Data Institute’s Deep Learning Program (2017-2018) and Data Ethics Inaugural Class (2020) through fast.ai.

She is currently working on her next books, applications, and films.

Erin earned her BA from the University of California, Berkeley. She is a proud Silicon Valley native.