Digital Planet

Personal project during the WPP HEX Creative Tech Apprenticeship. Framed primarily as R&D: validate multimodal orchestration (live Gemini vision + OpenAI Speech-to-Text + RAG / LLM) driving scene and material updates in NVIDIA Omniverse, with Substance-authored assets and Blender API where real-time Substance API access was not available.

Concept

Early creative north star was Solaris-inspired — a sentient, conversational planet — while the build prioritized integration proof (multi-model connectivity and real-time tooling) over a single polished narrative arc.

The pipeline ingests live camera and speech, analyzes vision through Gemini and voice through OpenAI Speech-to-Text, and routes fused intent through OpenAI RAG / LLM to select, arrange, and place content in Omniverse, with Blender API and Substance Designer (Smart Materials) for terrain and material updates.

Visuals

Initial planet surface before multimodal interaction.

Work in progress: terrain generation, materials, and pipeline integration.

Tech

Gemini Pro Vision · GCP · OpenAI Speech-to-Text · OpenAI RAG / LLM · NVIDIA Omniverse · Blender API · Substance Designer · WPP HEX