QCon 2024 and My Talk on Multidimensionality - Spatial Computing x Spatial Intelligence to Create New Worlds

I had a blast speaking at my first ever QCon, check out my speaker profile page and Q&A here. My talk was about creating XR with AI, the original title a bit verbose.

Firstoff, I want to thank Erin Doyle, the host of our track, “Designing Inclusive User Interfaces - UIs” at QCon, for giving me opportunity to speak and attend my very first QCon. Since hearing about it years ago, it’s a conference I have always wanted to go to, but never went, and it was truly an honor and privilege to share my experience and knowledge with this community.

Credit: Victor Dibia “10 Reasons Your Multi-Agent Workflows Fail and What You Can Do About It”

On the first day, I got to see Microsoft Principal Researcher on Human-AI Systems (HAX) at Microsoft, Victor Dibia, speak on multi-agent workflows.

Thanks to Lizzie Matusov, the organizer and founder of Quotient, who also spoke at QCon.

I also was invited to attend a panel on developer productivity with Kathryn Koehler, Director of Productivity Engineering at Netflix Mitali Parthasarathy, Director of Engineering (Engineering Effectiveness) at Yelp Eddie Flaisler, CEO at Sagittarius Labs, prev. VP of Engineering at Lob.

2 Key Takeaways from both talks:

People have overbloated expectations about EVERY use case for AI, when in fact the working use cases is actually much smaller (see slide by Victor above).

AI is doing some unanticipated things - having unintended consequences. The person sitting next to me during my breakout mentioned that at Amazon, an AI agent created an account since a human did not make one, then it called a human to troubleshoot through tech support to create an account. The need for guardrails was high and many people were not completely sold on using AI in production at the company, some stating that genAI (generative AI) was banned from employees using it in the codebase.

To me, AI is all the rage, however it is important to think about AI Safety, in a future post I’ll talk about co-founder of Anthropic’s presentation at UC Berkeley LLM Agents course recently which outlines more of this in detail.

Designing Inclusive UIs Track

From left to right: Erin Doyle (Track Host, Senior Staff Engineer, Lob), Colby Morgan (Technical Director, Mighty Coconut), Dylan Fox (Director of Operations, XR Access, Erin Pañgilinan (me), and Oda Ohana (Senior Software Engineer, Google).

We started off the day with Oda Ohan from Google speaking in detail utilizing Google AR Maps for the visually impaired. I absolutely LOVED the presentation by Dylan Fox, director of operations at XR Access, who gave a comprehensive overview utilizing XR as an extension for those from differently-abled/disabled communities (including ADHD, autistic, and neurodivergent).

My Talk

Since the talk at QCon (video) isn’t quite out yet, I share a couple key slides below outside of my Q&A on the QCon website here.

To me the future of human computer interaction is a combination of AR VR MR XR/spatial computing and AI

The future of UI and front-end engineering is three-dimensional, AR VR MR XR/spatial computing being that thing — or some sort of other public hologram if glasses, contacts, gesture and haptics isn’t your thing.

The future of back-end programming is automation with AI (aiding humans, not completely displacing them).

I give a brief overview and history of hardware (reference slide from Nvidia GTC 2024 with founder and CEO Jensen Huang, the history of headsets, evolution of the tech stack (how we design, code/develop for XR, for AI), and how these two are fields are merging now more than ever before.

I explained the basic concepts of multimodality and multidimensionality and how AI with XR Enables 3 key areas:

Collaboration - increases engagement

Generation - makes humans more expressive

Automation - manual tasks done more efficiently

AI is a thing, XR people

To the XR folks that didn’t understand AI, and to the web3/blockchain/crypto people who looked at me funny also not understanding AI as an early adopter, I can say that being in the space since mid-2014 (data science as we called it back then moreso), yes, it’s a thing.

I want to note that I was taken aback by a comment made about AI being separate from XR during AWE - Augmented World Expo’s keynote this year, and that in fact, it is because of AI and computer vision algorithms, namely SLAM algorithms (Simultaneous Location And Mapping) that enable Pokemon Go or any sort of AR type of experience, and gameAI and simulation (the entire idea of a virtual world) is itself involving AI and AI agents.

While I was obsessed with gameAI and RL (Reinforcement Learning), and the paper “Attention is All You Need” in 2017 caught on, it wasn’t until GPT’s evolution and release of ChatGPT 3.5 that more of the community could see that the 3 different classes of AI algorithms began to merge even more (the three classes being, Natural Language Processing - NLP), Reinforcement Learning (RL), and Computer Vision (CV).

The Goals of XR and AI are about Humanity

See the graphics from my presentation which reference the concept of the creation of memory by Jason Jerald (author of the VR Book), Percy Liang (Stanford Professor) from his lecture recently at UC Berkeley on cybersecurity, and the ending keynote, Shruti Bhat, founding team member of Rockset (acquired by OpenAI) all mentioned.

We must consider the actions and agency by both the technology itself AND humans, let us not despair believing that human agency and power overall is completely lost and displaced just because AI development has advanced faster and now more thane ever. If VR’s aim was to achieve “presence,” and for an AI to be an “embodied intelligent being” as Jim Fan, Senior Research Scientist at Nvidia noted in his recent lectures at UC Berkeley, then the goals of both VR and AI are actually aligned and the same. Both technologies by themselves must be built with and for humans, not replacing them, but augmenting human capability and potential.

More AR / AI Glasses Coming Soon

Given the growth of popularity for the AI / AR glasses, with Meta AR Glasses, Ray-Ban’s being the most popular despite the lack of an Application Programming Interface (API) and Standard Development Kit (SDK) for 3rd party and open source developers, we can anticipate this continuing to be a hot space.

Outside of the latest Snap Spectacles which I got to try during Meta Connect, we have soft announcements/rumors over the years that Google, Samsung, and QualComm are working in partnership for AI glass competitors.

Me and my friend Sasha Menscikova, Oculus Launch Pad alumnus who lent me her Snap Spectacles for a demo.

Confidentially, I know some folks working at Google and can say that we should keep our eyes peeled for what is to come in the future.

Privacy, Responsible AI, AI Safety

I make a couple of other references to friends like Kent Bye about privacy issues we should be aware of, but also note that other advancements in crypto/blockchain/web3 (of all places) is creating new tooling in user control to give humans more agency as AI, XR become more ubiquitous and as human data as input needs to be considered in terms of “ethics” and “responsibility.” I know many people don’t like those words, but ultimately the new rebrand for such a thing is “AI Safety,” and tech safety in general.

Credit: Shruti Bhat

Shruti Bhat, founding member of Rocketset (acquired by OpenAI) ended the keynote with these key words that I believe as software engineers and computational designers have a responsibility as a part of their profession to ensure human/user trust. And yes, teh crypto/blockchain/web3 community talks about being “trustless,” all the time, but in this instance, what it means more deeply is establishing truly human centered design and development that ensures adoption.

Erin Jerri Malonzo Pañgilinan is a software engineer and computational designer. She is an internationally acclaimed author, publishing Book Authority’s #2 must-read book on Virtual Reality in 2019, O’Reilly Media book, Creating Augmented and Virtual Realities: Theory and Practice for Next-Generation of Spatial Computing, which has been translated into Chinese, Korean, and distributed in over 2 dozen countries.

She was also previously a fellow in the University of San Francisco (USF) Data Institute’s Deep Learning Program (2017-2018) and Data Ethics Inaugural Class (2020) through fast.ai.

She is currently working on her next books, applications, and films.

Erin earned her BA from the University of California, Berkeley. She is a proud Silicon Valley native.