Interactive knowledge base chat with Supabase RAG using AI 📚💬

Google Drive File Ingestion to Supabase for Knowledge Base 📂💾 Overview 🌟 This n8n workflow automates the process of ingesting files from Google Drive into a Supabase database, preparing them for a knowledge base system. It supports text-based files (PDF, DOCX, TXT, etc.) and tabular data (XLSX, CSV, Google Sheets), extracting content, generating embeddings, and storing data in structured tables. This is a foundational workflow for building a company knowledge base that can be queried via a chat interface (e.g., using a RAG workflow). 🚀 Problem Solved 🎯 Manually managing a knowledge base with files from Google Drive is time-consuming and error-prone. This workflow solves that by: - Automatically ingesting files from Google Drive as they are created or updated. - Extracting content from various file types (text and tabular). - Generating embeddings for text-based files to enable vector search. - Storing data in Supabase for efficient retrieval. - Handling duplicates and errors to ensure data consistency. Target Audience: - Knowledge Managers: Build a centralized knowledge base from company files. - Data Teams: Automate the ingestion of spreadsheets and documents. - Developers: Integrate with other workflows (e.g., RAG for querying the knowledge base). Workflow Description 🔍 This workflow listens for new or updated files in Google Drive, processes them based on their type, and stores the extracted data in Supabase tables for later retrieval. Here’s how it works: 1. File Detection: Triggers when a file is created or updated in Google Drive. 2. File Processing: Loops through each file, extracts metadata, and validates the file type. 3. Duplicate Check: Ensures the file hasn’t been processed before. 4. Content Extraction: - Text-based Files: Downloads the file, extracts text, splits it into chunks, generates embeddings, and stores the chunks in Supabase. - Tabular Files: Extracts data from spreadsheets and stores it as rows in Supabase. 5. Metadata Storage: Stores file metadata and basic info in Supabase tables. 6. Error Handling: Logs errors for unsupported formats or duplicates. Nodes Breakdown 🛠️ 1. Detect New File 🔔 - Type: Google Drive Trigger - Purpose: Triggers the workflow when a new file is created in Google Drive. - Configuration: - Credential: Google Drive OAuth2 - Event: File Created - Customization: - Specify a folder to monitor specific directories. 2. Detect Updated File 🔔 - Type: Google Drive Trigger - Purpose: Triggers the workflow when a file is updated in Google Drive. - Configuration: - Credential: Google Drive OAuth2 - Event: File Updated - Customization: - Currently disconnected; reconnect if updates need to be processed. 3. Process Each File 🔄 - Type: Loop Over Items - Purpose: Processes each file individually from the Google Drive trigger. - Configuration: - Input: {{ $json.files }} - Customization: - Adjust the batch size if processing multiple files at once. 4. Extract File Metadata 🆔 - Type: Set - Purpose: Extracts metadata like fileid, filename, mimetype, and webviewlink. - Configuration: - Fields: - fileid: {{ $json.id }} - filename: {{ $json.name }} - mimetype: {{ $json.mimeType }} - webviewlink: {{ $json.webViewLink }} - Customization: - Add more metadata fields if needed (e.g., size, createdTime). 5. Check File Type ✅ - Type: IF - Purpose: Validates the file type by checking the MIME type. - Configuration: - Condition: mimetype contains supported types (e.g., application/pdf, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet). - Customization: - Add more supported MIME types as needed. 6. Find Duplicates 🔍 - Type: Supabase - Purpose: Checks if the file has already been processed by querying knowledgebase. - Configuration: - Operation: Select - Table: knowledgebase - Filter: fileid = {{ $node'Extract File Metadata'.json.fileid }} - Customization: - Add additional duplicate checks (e.g., by file name). 7. Handle Duplicates 🔄 - Type: IF - Purpose: Routes the workflow based on