Humanoid – Audio-Visual AI Agent
A full-stack audio-visual AI agent with a real-time animated face, capable of hearing, reasoning, and responding with synchronized voice and facial expressions — not just another chatbot.
Overview
Graduation Project – Ain Shams University & University of East LondonBuilt a complete audio-visual AI agent — not a traditional chatbot. The system features an animated face that listens, thinks, and responds with voice and visuals in real time. The goal was to make users feel they are talking to a human, not a system. The agent runs on a fully integrated pipeline: the user speaks → ASR converts speech to text with high accuracy → the text enters an LLM that understands context and generates an intelligent response → the response is converted to natural speech via TTS → simultaneously, a face generation system takes the audio and produces facial animation with precise lip sync, facial expressions, and head movement — all in real time. The system is also architected to accept connections to external APIs and systems — CRM, databases, or any tool — because an agent should never be isolated, it must be connected.
What was not working
Most AI agents available are either text-only, voice-only, or avatar-only. Combining all three modalities into a single coherent real-time pipeline while maintaining low latency and natural-feeling interactions was the core challenge. Additionally, achieving high-quality Arabic NLP support (not just translation) and real-time facial animation with accurate lip synchronization required pushing beyond standard approaches.
What I built
Used TalkingGaussian v2 for facial animation instead of traditional NeRF, achieving significantly faster real-time performance. GPT-4o serves as the LLM engine, with the option to swap in open-source models like DeepSeek or Jais for privacy-sensitive or self-hosted deployments. The architecture supports full RAG integration — the agent can connect to Vector Databases like Pinecone or Chroma and pull information from company or store-specific knowledge bases, making responses accurate and customized rather than generic. The Arabic character trained in this project achieved the highest performance among all characters on PSNR, LMD, and LPIPS metrics. The system was built, tested, and evaluated with real academic benchmarks.
Stack
08 partsWhat each part of the system does, from the screens the team uses to the services that send messages and reports.