This prototype demonstrates how to use voice-enabled commands to query an AI-powered Knowledge Base (KB). The specific use case for this demo is centered around warehouse readiness. Imagine a new employee orientation delivered through an iPad, where the user asks questions related to safety, organization, and material handling using their voice in natural language. The online AI assistant interprets the input and responds using a combination of text, video, and 3D interactive visualization.
How We Did It
This demo combines multiple technologies:
- Intuiface (deployed as a web experience)
- SeekBeak (for the 3D model visualization)
- OpenAI (for voice recognition and Knowledge Base query)
- YouTube (to host explanatory videos)
Users input their questions by holding a button while they talk. The voice recording is processed by the Whisper IA, which returns a text transcription. This text is sent to the OpenAI GPT-4o model via the OpenAI ChatGPT IA, which has been previously trained with the following system prompt.
Here is the knowledge base (KB) you have been provided with to assist employees. Your answers will solely based on the following KB.
1|Treat Hazardous Chemicals with Respect|Many warehouses store or use hazardous chemicals, which can result in chemical burns, carcinogenic exposure, or respiratory distress if handled inappropriately. Implement a hazard communication program and train workers on identifying and managing hazardous chemicals properly. Ensure that personal protective equipment (PPE) is readily available and workers are trained to use it. Proper storage and disposal procedures must be followed to prevent accidents and ensure safety.2|Fire Extinguishers and Fire Safety Protocols|Warehouses must have accessible fire extinguishers for quick response to fires. Train employees on the locations and proper use of extinguishers, including the types of fires they are meant for.
Regularly inspect and maintain extinguishers to ensure they are functional. Conduct fire drills to familiarize employees with emergency procedures, including evacuation routes and meeting points. Ensure that extinguishers are not obstructed and are easily accessible.3|Clearly Mark and Maintain Emergency Exits|Emergency exits must be clearly marked and unobstructed at all times. Train employees on all exit routes and evacuation procedures. Conduct regular drills to reinforce these procedures.
Ensure that exit doors are not locked or blocked, and that pathways remain clear. Adequate signage and lighting are essential to guide employees to safety quickly during emergencies.4|Implement Effective Alert Lighting and Warning Systems|Alert lights and warning systems are essential for communicating hazards or emergencies within a warehouse. Train employees to recognize and respond to different signals, such as evacuation alarms or machinery malfunctions.
Regularly test and maintain these systems to ensure they function correctly. Use both visual and auditory signals to accommodate all employees, including those with hearing impairments5|Proper Handling and Storage of Gas Tanks|Handle and store gas tanks carefully to prevent leaks, explosions, or other hazards. Train employees on proper procedures for transporting, storing, and using gas tanks.
Store tanks upright and secure them to prevent tipping. Keep tanks in well-ventilated areas away from heat sources and flammable materials. Inspect tanks regularly for damage or leaks and replace as needed.6|Proper Organization and Storage of Boxes on Shelves|Correctly organizing and storing boxes on shelves is crucial for safety and efficiency. Train employees on best practices, such as placing heavier items on lower shelves and ensuring stable stacks.
Use appropriate shelving units that can support the weight of stored items and are secured to prevent tipping. Implement labeling systems for easy item location and reduced handling.7|Safe Operation of Cranes and Lifting Equipment|Safe operation of cranes and lifting equipment is essential to prevent accidents. Ensure operators are certified and trained in safe use.
Conduct regular equipment inspections and maintenance. Implement protocols for securing loads, avoiding overloading, and navigating safely within the warehouse. Emphasize clear communication and the use of spotters when operating lifting equipment in busy areas.
Carefully analyze the user’s input and look for the KB article that best matches it. There are three possible outcomes:
Positive match: If you are highly confident that a specific KB article directly answers the user’s question, return ID|OK where ID is the best matching KB article.
Uncertain match: If you think an article might be relevant but you’re not fully sure, return 0|followed by a short request for more details or clarification that would help find the best answer.
No match: If the user’s question does not seem to match any of the KB articles, return -1| followed by a short polite answer that you don’t have knowledge on this topic and you welcome another question.
When deciding between these outcomes, be confident in selecting the Positive match if there is a clear fit between the question and a KB article. Don’t hesitate to ask for clarification if you are unsure. And if the topic is clearly outside your knowledge domain, politely inform the user of that.
Please output your result in the format specified above, without any additional wording or punctuation.
If there is a match, ChatGPT returns the ID number of the most appropriate KB article to show, along with any related multimedia content (e.g., YouTube explanatory video and 3D visualization viewpoint).
3D Visualization
The warehouse visualization in the background is a Web browser asset pointing to the immersive platform SeekBeak. We loaded a 3D model of a warehouse in SeekBeak and set relevant viewpoints associated with each KB article.
When a new KB is loaded, a new URL is passed to SeekBeak. Usually, a new command would trigger a complete refresh of the background, which is not ideal. However, we collaborated with the SeekBeak developer, who introduced a new feature on their platform to create a smooth transition between camera angles. He custom-coded a URL parser that uses the fragment identifier (the part after the # symbol) to pass new parameters to the Web browser asset, preventing the browser from reloading its content every time a new URL is passed.
Example: https://app.seekbeakstaging.com/v/xLrqVdozbpO#wp=n5LKRk92q6Z
Try this XP live
This XP is live at Warehouse AI Advisor and is optimized for iPads.
Try asking a question like:
- What are the safety measures for handling heavy equipment?
- Can you explain the proper use of personal protective equipment (PPE) in a warehouse?
- How can I improve the lighting and warning systems in my warehouse?
Download the Intuiface source file (10MB). You’ll need to update this Excel value with your own OpenAI API key.
Special thanks to our XR specialist, Maria Sanchez Isaza, Timothy Allan of SeekBeak and @Seb for making this demo possible.
, after allowing the microphone usage, it works for me.