Automate document ingestion & RAG system with Google Drive, Sheets & OpenAI

1. Overview The IngestionDocs workflow is a fully automated document ingestion and knowledge management system built with n8n. Its purpose is to continuously ingest organizational documents from Google Drive, transform them into vector embeddings using OpenAI, store them in Pinecone, and make them searchable and retrievable through an AI-powered Q&A interface. This ensures that employees always have access to the most up-to-date knowledge base without requiring manual intervention. ------------------------------------------------------------------------ 2. Key Objectives - Automated Ingestion → Seamlessly process new and updated documents from Google Drive.\ - Change Detection → Track and differentiate between new, updated, and previously processed documents.\ - Knowledge Base Construction → Convert documents into embeddings for semantic search.\ - AI-Powered Assistance → Provide an intelligent Q&A system for employees to query manuals.\ - Scalable & Maintainable → Modular design using n8n, LangChain, and Pinecone. ------------------------------------------------------------------------ 3. Workflow Breakdown A. Document Monitoring and Retrieval - The workflow begins with two Google Drive triggers: - File Created Trigger → Fires when a new document is uploaded.\ - File Updated Trigger → Fires when an existing document is modified.\ - A search operation lists the files in the designated Google Drive folder.\ - Non-downloadable items (e.g., subfolders) are filtered out.\ - For valid files: - The file is downloaded.\ - A SHA256 hash is generated to uniquely identify the file's content. ------------------------------------------------------------------------ B. Record Management (Google Sheets Integration) To keep track of ingestion states, the workflow uses a Google Sheets--based Record Manager:\ - Each file entry contains:\ - Id (Google Drive file ID)\ - Name (file name)\ - hashId (SHA256 checksum)\ - The workflow compares the current file's hash with the stored one:\ - New Document → File not found in records → Inserted into the Record Manager.\ - Already Processed → File exists and hash matches → Skipped.\ - Updated Document → File exists but hash differs → Record is updated. This guarantees that only new or modified content is processed, avoiding duplication. ------------------------------------------------------------------------ C. Document Processing and Vectorization Once a document is marked as new or updated:\ 1. Default Data Loader extracts its content (binary files supported).\ - Pages are split into individual chunks.\ - Metadata such as file ID and name are attached.\ 2. Recursive Character Text Splitter divides the content into manageable segments with overlap.\ 3. OpenAI Embeddings (text-embedding-3-large) transform each text chunk into a semantic vector.\ 4. Pinecone Vector Store stores these vectors in the configured index:\ - For new documents, embeddings are inserted into a namespace based on the file name.\ - For updated documents, the namespace is cleared first, then re-ingested with fresh embeddings. This process builds a scalable and queryable knowledge base. ------------------------------------------------------------------------ D. Knowledge Base Q&A Interface The workflow also provides an interactive form-based user interface:\ - Form Trigger → Collects employee questions.\ - LangChain AI Agent:\ - Receives the question.\ - Retrieves relevant context from Pinecone using vector similarity search.\ - Processes the response using OpenAI Chat Model (gpt-4.1-mini).\ - Answer Formatting:\ - Responses are returned in HTML format for readability.\ - A custom CSS theme ensures a modern, user-friendly design.\ - Answers may include references to page numbers when available. This creates a self-service knowledge base assistant that employees can query in natural language. ------------------------------------------------------------------------ 4. Technologies Used - n8n → Orchestration of the entire workflow.\ - Go