Han, Lanyi韩蓝毅한람의

Project · Mixed reality · Meta Quest

AAA Atto, Acco, Anno

A mixed-reality assistant named after the three agents that live in it. You talk to it in plain language; the agents reason about the request and change the room around you: spawning objects, flying to a work spot, or reshaping their own body.

Source on GitHubThe 2025 write-up

Drag to rotate

Ask Atto about Lanyi


        

      

    

    
  


  

    
How one turn runs

    
    
      
  1. You type in the headset panel. The UI client attaches scene context: room anchors from Meta's MR Utility Kit and the objects already spawned.
    2. 
      
  2. Atto reasons in a ReAct loop. Prompt, model call, parse Thought / Action / Observation / FinalAnswer, dispatch the tool, observe, repeat.
    3. 
      
  3. Delegation runs the sub-agent inline. Its real result comes back as Atto's next Observation, so “put a cube there, then turn Anno into stairs” is two passes of the same loop. There is no separate scheduler.
    4. 
      
  4. Tools change the scene. A cube spawns, an agent flies, Anno morphs.
    5. 
    

  


  

    
Design choices

    

      
Local model, no cloud
Everything runs against an OpenAI-compatible endpoint on the same LAN: LM Studio serving Gemma. The endpoint can be retyped inside the headset.

      
Unity is the runtime
Agents, tools, prompts and memory all live in the Unity 6 client. No Python server is required in the loop.

      
Shapes are data
Each preset is 27 integer voxel cells in a JSON file. Adding a shape needs no recompile.

      
Fix the prompt, not the parser
When a small model's output breaks tool parsing, the prompt changes. The parser stays strict.

      
Test both sides
“Give me a cube” passes only if the answer says so and a cube really exists in the scene.

      
SDK behind an interface
Meta SDK calls sit behind a platform layer with a null implementation, so tests run headless.

    

  


  

    
How it got here

    
    
      
  1. Repository starts: AR experiments and a first local LLM.
    2. 
      
  2. AR on OpenXR and MR Utility Kit. Agent routing moves into an MCP server with a Python host.
    3. 
      
  3. AAAHost backend (FastAPI and WebSocket) documented.
    4. 
      
  4. Quest SDK calls abstracted; first version of the Unity AR client; the runtime is simplified.
    5. 
      
  5. Multi-agent coordinator wires Atto to Acco. Agents get a visible body and flight.
    6. 
      
  6. Fly-home and working height; scripted build-and-run for Quest.
    7. 
      
  7. Anno, the shape-shifting agent, joins.
    8. 
    

    
The Python backend from the 2025 design is kept in the repository for reference; it is no longer on the runtime path.