Migrate large Hugging Face datasets to MongoDB with a looping subworkflow

This n8n template provides a production-ready, memory-safe pipeline for ingesting large Hugging Face datasets into MongoDB using batch pagination. It is designed as a reusable data ingestion layer for RAG systems, recommendation engines, analytics pipelines, and ML workflows. The template includes: - A main workflow that orchestrates pagination and looping - A subworkflow that fetches dataset rows, sanitizes them, and inserts them into MongoDB safely --- 🚀 What This Template Does - Fetches rows from a Hugging Face dataset using the datasets-server API - Processes data in configurable batches (offset + length) - Removes Hugging Face id fields to avoid MongoDB duplicate key errors - Inserts clean documents into MongoDB - Automatically loops until all dataset rows are ingested - Handles large datasets without memory overflow --- 🧩 Architecture Overview Main Workflow (Orchestrator) - Starts the ingestion process - Defines dataset, batch size, and MongoDB collection - Repeatedly calls the subworkflow until no rows remain Subworkflow (Batch Processor) - Fetches a single batch of rows from Hugging Face - Splits rows into individual items - Removes id fields - Inserts documents into MongoDB - Returns batch statistics to the main workflow --- 🔁 Workflow Logic (High-Level) 1. Set initial configuration: - Dataset name - Split (train, test, etc.) - Batch size - Offset 2. Fetch rows from Hugging Face 3. If rows exist: - Split rows into items - Remove id - Insert into MongoDB 4. Increase offset 5. Repeat until no rows are returned --- 📦 Default Configuration | Parameter | Default Value | |---------|--------------| | Dataset | MongoDB/airbnbembeddings | | Config | default | | Split | train | | Batch Size | 100 | | MongoDB Collection | airbnb | All values can be changed easily from the ConfigStart node. --- 🛠 Prerequisites - n8n (self-hosted or cloud) - MongoDB (local or hosted) - MongoDB credentials configured in n8n - Internet access to datasets-server.huggingface.co --- ▶️ How to Use 1. Import the workflow JSON into n8n 2. Configure MongoDB credentials in the MongoDB node 3. Update dataset parameters if needed: - Dataset name - Split - Batch size - Collection name 4. Run the workflow using the Manual Trigger 5. Monitor execution until completion --- 🧠 Why id Is Removed Hugging Face dataset rows often include an id field. MongoDB requires id values to be unique, so reusing these values can cause insertion failures. This template: - Removes the Hugging Face id - Lets MongoDB generate its own ObjectId - Prevents duplicate key errors - Allows safe re-runs and incremental ingestion --- 🔍 Ideal Use Cases ✅ RAG (Retrieval-Augmented Generation) - Store dataset content as source documents - Add embeddings later using OpenAI, Mistral, or local models - Connect MongoDB to a vector database or hybrid search ✅ Recommendation Systems - Build item catalogs from public datasets - Use embeddings or metadata for similarity search - Combine with user behavior data downstream ✅ ML & Analytics Pipelines - Centralize dataset ingestion - Normalize data before training or analysis --- ⚙️ Recommended Enhancements You can easily extend this template with: - Upsert logic using a deterministic hash (idempotent ingestion) - Embedding generation before or after insertion - Schema validation or field filtering - Rate-limit handling & backoff - Parallel ingestion for faster processing --- ⚠️ Notes & Best Practices - Reduce batch size if you encounter memory limits - Verify dataset license before production use - Add indexes in MongoDB for faster downstream querying - Use upserts if you plan to re-run ingestion frequently --- 📄 License & Disclaimer This workflow template is provided as-is. You are responsible for: - Dataset licensing compliance - Infrastructure costs - Downstream data usage Hugging Face datasets are subject to their respective licenses. --- ⭐ Template Summary Category: Data Ingestion Complexity: Intermediate Scalability: High Memory Safe: Yes Prod