WhatsApp AI agent that understand text, image , audio

Overview 📌 This template is a multimodal WhatsApp assistant that understands text, images, and audio, aggregates media inputs, and returns intelligent replies using Google Gemini. It can fetch knowledge from Google Docs, log conversations into Google Sheets, and respond via WhatsApp — all orchestrated inside n8n. Features ✨ Multimodal input handling: Receives images and audio from WhatsApp, analyzes them, and sends contextual responses. 🖼️🎧 Audio transcription: Converts voice messages to text for analysis. 🔊➡️📝 Image analysis: Extracts objects/labels/text from images to inform responses. 🖼️🔍 Knowledge fetch: Pulls relevant documents from Google Docs for richer answers. 📄📚 Conversation logging: Appends each interaction to Google Sheets for auditing or analytics. 📈 Modular design: Clear subflows for media reception, transcription, image analysis, aggregation, and the AI Agent. ⚙️ Ready for customization: Swap models, change providers, or extend with a vector store. 🔁 Requirements ✅ n8n instance (self-hosted or n8n.cloud) with public webhook access Google Cloud project with these APIs enabled: Vertex AI (Gemini), Cloud Vision API, Cloud Speech-to-Text, Drive & Docs & Sheets 🧠 Google Service Account JSON key (with permissions for above services) 🔐 WhatsApp Business API provider credentials (Twilio, 360dialog, or similar) 📞 (Optional) Pinecone or another vector store if you want embeddings / retrieval augmentation 🗂️ Setup Instructions 🔧 Import the workflow Download the .json from your package and in n8n Editor → Import → From File → select JSON. Save. Create credentials Google Service Account: Create service account in GCP, grant Vertex AI, Vision, Speech, Drive, Docs, Sheets roles, download JSON → create n8n Google Service Account credential and upload JSON. WhatsApp: Add your WhatsApp API credentials (API key/token, phone ID, webhook secret). (Optional) Vector store: Add Pinecone credentials if using embeddings. Configure media receiver nodes WhatsApp Trigger: Ensure webhook URL is set on your WhatsApp provider to forward incoming messages/media to n8n. 🌐 Audio Receiver nodes: point to incoming audio payload path or download URL. 🎧 Image Receiver nodes: point to image URL or attachment field. 🖼️ Transcription & Image Analysis Transcribe Audio: Select Google Speech-to-Text via your Google Service Account credential. Confirm language and sampling settings. 🗣️➡️📝 Analyze Image: Point to Vision API via same Google credential. Choose required outputs (labels, landmarks, OCR). 🔎 Aggregate media Confirm the Aggregate node collects text + transcribed audio + image analysis results into a single payload for the AI Agent. 🧩 AI Agent (Gemini) Open AI Agent node → choose your Google Gemini/Vertex AI credentials. Edit System Prompt to include: how to prioritize image/audio text, where to look up knowledge (Docs), and style/tone. Replace placeholders (business name, policies). ✍️ Knowledge Fetch (Google Docs) Configure Get a document in Google Docs node to point to your knowledge docs folder or specific document IDs. 📂 Conversation Logging Configure Google Sheets node to append rows to your chosen spreadsheet (structure below). 🧾 WhatsApp Reply Map AI Agent output to the WhatsApp Send message node and choose template messaging if needed. 💬 Test end-to-end Activate the workflow, send a text message, an image, and an audio message to your WhatsApp test number. Verify the AI reply and that logs appear in Google Sheets. 🧪 Google Sheet: Conversation Log (suggested columns) 🗒️ Column Type Description Timestamp Date/Time When the message arrived UserID Text WhatsApp user identifier (phone) MessageType Text text / image / audio / mixed MessageText Text Original text or transcribed text ImageAnalysis Text Vision API summary / labels / OCR text AttachmentURL Text Link to image/audio file AIResponse Text Final text reply sent to user Notes Text Any extra flags (e.g., escalation) Customization 🔁 Swap Gemini with another model