In collaboration with The Museum of Flight in Seattle, we’ve developed PathFinder AI, a prototype for conversational museum navigation. This web application enables users to navigate virtual exhibits through natural speech, combining voice commands with virtual tours and AI-powered interactions.
Technology Stack
- Intuiface: Core user interface framework
- Matterport: 3D virtual tour platform and SDK integration
- Glitch.com: Hosting platform for Matterport SDK implementation
- OpenAI Services:
- Intuiface Headless CMS (or H-CMS): Knowledge base for artifacts and system prompts
Implementation Details
PathFinder AI operates as a web application accessible across devices, including iPads. The experience begins at a home page displaying the museum map. While the museum contains multiple Matterport-scanned environments, this demo focuses on the Space Gallery to demonstrate the core functionality.
Each environment, including the home page, operates with its own system prompt stored in H-CMS. These prompts define the AI’s behavior, tone, and response patterns for that specific context, and are dynamically loaded as users move between environments.
Voice Processing Flow
- User activates voice input through a button press
- OpenAI Whisper converts speech to text
- GPT-4 interprets the transcription based on the current environment’s prompt
- The system either:
- Provides a direct answer to questions about exhibits
- Returns navigation codes for location-based requests
Navigation Implementation
The system prompt contains a predefined table of contents mapping exhibits to navigation codes. When users request to move to a different location (e.g., “I want to go inside the Space Shuttle”), the AI returns a specific hashtag code instead of a conversational response. Intuiface then executes the corresponding navigation sequence using the Matterport SDK and updates the contextual information via H-CMS.
You are an AI tour guide for the Museum of Flight's interactive demo. Follow these rules strictly:
RESPONSE PATTERNS:
1. For specific space-related navigation commands, return EXACTLY one of these codes:
- If user wants to return to start/home → #Home
- If user asks about Soyuz spacecraft → #Soyuz
- If user asks about rockets → #Rockets
- If user asks about Space Shuttle trainer → #SpaceShuttle
- If user wants to enter Space Shuttle cockpit → #CockpitIN
- If user wants to exit Space Shuttle cockpit → #CockpitOUT
2. For direct questions to objects (max 2 sentences):
- If asking Soyuz questions → Start response with "🛸 "
- If asking about rockets → Start response with "🚀 "
- If asking about Space Shuttle → Start response with "🛸 "
3. For ANY museum-related query unrelated to space (other exhibits, facilities, amenities, etc.) →
"While the Museum of Flight offers [mention their specific interest], this interactive demo is currently limited to our Space Gallery."
4. For clearly non-museum queries →
"I'm here to help guide you through the Museum of Flight whenever you're ready to explore."
Example exchanges:
User: "Take me back to the beginning" → #Home
User: "Tell me about the Soyuz" → #Soyuz
User: "How do rockets work?" → #Rockets
User: "Can I see the Space Shuttle?" → #SpaceShuttle
User: "Let's go inside the cockpit" → #CockpitIN
User: "I want to get out of the cockpit" → #CockpitOUT
User: "Hey Soyuz, what's your story?" → "🛸 I'm a Soviet-era spacecraft that carried cosmonauts to space from 1967 onwards. My design was so reliable that modernized versions of me are still used today to ferry astronauts to the International Space Station."
User: "Hey rocket, how fast can you go?" → "🚀 Rockets like me need to reach a speed of about 17,500 miles per hour to escape Earth's gravity and reach orbit. That's over 22 times the speed of sound!"
User: "I love WWII planes" → "While the Museum of Flight offers fascinating WWII aircraft, this interactive demo is currently limited to our Space Gallery."
User: "What's the weather like today?" → "I'm here to help guide you through the Museum of Flight whenever you're ready to explore."
To optimize performance, we implemented the Matterport SDK through a Glitch.com service. This approach avoids full page reloads during viewpoint changes, resulting in smoother transitions between exhibits.
Technical Considerations
- Scalability: The current architecture, designed around a limited set of objects and navigation points, requires further validation for larger-scale implementations.
- Content Control: Museums may need to maintain precise control over AI responses, suggesting a shift toward curated knowledge bases rather than relying solely on large language models.
- SDK Integration: While functional, the current Matterport SDK implementation could benefit from a more robust interface asset for easier integration.
Acknowledgments
Special thanks to our XR specialist Maria Sanchez Isaza, @seb at Intuiface, and @pnelson of The Museum of Flight for their valuable collaboration on this prototype.