Demo Award at IEEE ISMAR 2026

At the International Symposium on Mixed and Augmented Reality (IEEE ISMAR) in Bari, Italy, I received Best Demo Award (Honorable Mention) for: Flow Matching for Augmented Reality Applications.

In this post, I first highlight the demo’s capabilities and limitations. Second, I explain how I made it; I trust that this will be useful for other practitioners!

1. The Effect

A lot of researchers in the AR and VR communities are investigate rendering with Gaussian Splatting and Neural Radiance Fields. I believe that these techniques are fine for VR and pure computer graphics, but are completely wrong for AR. This was one of the main motivations for the demo: show that Diffusion Models are a feasible alternative that deserve careful consideration.

The quickest way to familiarize yourself with the high-level story is to watch this 1-minute explanatory video:

At the conference, I recorded the screen of my laptop while several attendants were playing with the demo (in order of appearance: Ke Li, Doron Friedman, Xinyi Ye, Huyen Nguyen, and Yingfangzhong Sun):

Unfortunately, the screen recording only ran at 7 FPS at 720p 🤦🤦. In order to get closer to the actual rendering performance, I performed post-processing with the FILM frame interpolation model to get to 30 FPS, which led to melting artifacts that were not present in the actual demo.

Effects that can be observed include:

• Virtual light sources illuminate real elements
• Virtual elements are registered in the real world
• Several prompts are stacked successively

There are also defects visible that are inherent to my pipeline:

• Temporal inconsistencies
• The more people are in the frame, the more edits fail

I had great fun playing together with one of my PhD advisors: Steven Feiner. Moronically, I only captured the output, but not the input 🤦. I used Qwen-Image-2.1 to recreate the “before” frame.

2. How it’s Made

Three Key Ingredients

This demo would have been hardly possible without Chinese open-weight models, generously donated to the public by Alibaba Inc. and Z.ai. 谢谢! For reasons that become apparent further down, closed models, such as Anthropic or OpenAI, would not have been feasible to use within a reasonable financial or time budget—they are simply too slow and too expensive for the approach I took!

One of the biggest perks of being a researcher in France is having access to our DALIA GPU beast, which has 64 NVIDIA GB200 GPUs. For lighter tasks, I used the older Jean-Zay with 1456 H100 GPUs. I estimate that I have used the equivalent of 1000 GB200 hours for developing this demo.

The final key ingredient is Andrej Karpathy’s autoresearch, which transposes the classical idea of automatic parameter search into code space. I used this mechanism on several levels of abstraction when developing the demo.

Development Stack

The summary of my development stack from bottom to top is:

• vllm on DALIA for high-performance serving of LLMs. I used autoresearch to optimize startup times (e.g. GLM went from 1 hour to 8 minutes), and inference speed.
• Pi as agent harness
• Herdr to manage agent swarms
• Moshi as mobile phone client so that I could manage my agents from everywhere

Let’s spend a moment to look at the most fundamental part of this stack: the LLMs used and their performance. In terms of speed, I can use leading open weight models at about 10x speed compared to OpenAI models and 5x faster than Anthropic models:

The main autoresearch loop for the demo ran about 1 month continuously. A 10x slower model (such as Anthropic’s Fable) would have simply not been feasible. But hold on! Aren’t the closed source models smarter? Indeed, they are:

A problem that requires an intelligence index over 50 would lead to open weight models running in high-speed circles without ever solving them! To my great delight, I have not yet encountered such a problem.

Demo Development

To attack the challenging performance issues, I ran autoresearch loops for individual components of the inference pipeline (DIT, VAE, Upscalers), as well as the overall system. Key instructions included:

• Increase FPS
• Decrease Latency
• Verify that visual quality is preserved using a test suite with a VLM

Some of the most impactful optimizations happened on the architectural level, such as cross-GPU pipeline parallelism:

The agents discovered an astonishing amount of bugs in NVIDIA’s TensorRT for Blackwell. For example, see above diagram, it was not possible to use TensorRT (TRT) for VAE encoding/decoding at my demo’s resolution (it only worked for smaller ones); so we had to fall back to the much slower torch implementation. Luckily the DIT, RIFE, and ESRGAN components worked with it ;-)

Show Time

3. Conclusions

I am still trying to fully wrap my head around this development experience! It is certainly very different from anything I have ever done in my 31 years of programming!

A mini-postmortem:

• Autoresearch is unreasonably effective.
• 1000 GPU hours were wasteful. I should have planned better.
• Agent swarms on multiple heterogeneous machines are not trivial. Especially for network optimizations, it was useful to have agents on DALIA communicate with agents on my laptop, which regularly ran into the aggressive intrusion detection system on DALIA. In the end, I figured everything out, even how to make persistent connections between moshi on my phone and DALIA. But, this took me too much time.

What comes next? This is a big question that I need to think deeply about. The only thing that is already clear to me is that the next experiment will use about 10x more compute :-)

You can find some more information about the demo in the Demo description as submitted to ISMAR (2-page pdf). Please note that the performance numbers are outdated (I worked a few more weeks on the demo after submission).