Automated FTP File Migration with Smart Cleanup and Email Notifications

Description

Automated FTP File Migration with Smart Cleanup and Email Notifications

Overview
This n8n workflow automates the secure transfer of files between FTP servers on a scheduled basis, providing enterprise-grade reliability with comprehensive error handling and dual notification systems (email + webhook). Perfect for data migrations, automated backups, and multi-server file synchronization.

What it does
This workflow automatically discovers, filters, transfers, and safely removes files between FTP servers while maintaining complete audit trails and sending detailed notifications about every operation.

Key Features:
Scheduled Execution**: Configurable timing (daily, hourly, weekly, or custom cron expressions)
Smart File Filtering**: Regex-based filtering by file type, size, date, or name patterns
Safe Transfer Protocol**: Downloads → Uploads → Validates → Cleans up source
Dual Notifications**: Email alerts + webhook integration for both success and errors
Comprehensive Logging**: Detailed audit trail of all operations with timestamps
Error Recovery**: Automatic retry logic with exponential backoff for network issues
Production Ready**: Built-in safety measures and extensive documentation

Use Cases

🏢 Enterprise & IT Operations
Data Center Migration**: Moving files between different hosting environments
Backup Automation**: Scheduled transfers to secondary storage locations
Multi-Site Synchronization**: Keeping files in sync across geographic locations
Legacy System Integration**: Bridging old and new systems through automated transfers

📊 Business Operations
Document Management**: Automated transfer of contracts, reports, and business documents
Media Asset Distribution**: Moving images, videos, and marketing materials between systems
Data Pipeline**: Part of larger ETL processes for business intelligence
Compliance Archiving**: Moving files to compliance-approved storage systems

🔧 Development & DevOps
Build Artifact Distribution**: Deploying compiled applications across environments
Configuration Management**: Synchronizing config files between servers
Log File Aggregation**: Collecting logs from multiple servers for analysis
Automated Deployment**: Moving release packages to production servers

How it works

📋 Workflow Steps
Schedule Trigger → Initiates workflow at specified intervals
File Discovery → Lists files from source FTP server with optional recursion
Smart Filtering → Applies customizable filters (type, size, date, name patterns)
Secure Download → Retrieves files to temporary n8n storage with retry logic
Safe Upload → Transfers files to destination with directory auto-creation
Transfer Validation → Verifies successful upload before proceeding
Source Cleanup → Removes original files only after confirmed success
Comprehensive Logging → Records all operations with detailed metadata
Dual Notifications → Sends email + webhook notifications for success/failure

🔄 Error Handling Flow
Network Issues** → Automatic retry with exponential backoff (3 attempts)
Authentication Problems** → Immediate email alert with troubleshooting steps
Permission Errors** → Detailed logging with recommended actions
Disk Space Issues** → Safe failure with source file preservation
File Corruption** → Integrity validation with rollback capability

Setup Requirements

🔑 Credentials Needed
Source FTP Server
Host, port, username, password
Read permissions required
SFTP recommended for security

Destination FTP Server
Host, port, username, password
Write permissions required
Directory creation permissions

SMTP Email Server
SMTP host and port (e.g., smtp.gmail.com:587)
Authentication credentials
For success and error notifications

Monitoring API (Optional)
Webhook URL for system integration
Authentication tokens if required

⚙️ Configuration Steps
Import Workflow → Load the JSON template into your n8n instance
Configure Credentials → Set up all required FTP and SMTP connections
Customize Schedule → Adjust cron expression for your timing needs
Set File Filters → Configure regex patterns for your file types
Configure Paths → Set source and destination directory structures
Test Thoroughly → Run with test files before production deployment
Enable Monitoring → Activate email notifications and logging

Customization Options

📅 Scheduling Examples
0 2 * * * # Daily at 2 AM
0 */6 * * * # Every 6 hours
0 8 * * 1-5 # Weekdays at 8 AM
0 0 1 * * # Monthly on 1st
*/15 * * * * # Every 15 minutes

🔍 File Filter Patterns
Documents
\\.(pdf|doc|docx|xls|xlsx)$

Images
\\.(jpg|jpeg|png|gif|svg)$

Data Files
\\.(csv|json|xml|sql)$

Archives
\\.(zip|rar|7z|tar|gz)$

Size-based (add as condition)
{{ $json.size > 1048576 }} # Files > 1MB

Date-based (recent files only)
{{ $json.date > $now.minus({days: 7}) }}

📁 Directory Organization
// Date-based structure
/files/{{ $now.format('YYYY/MM/DD') }}/

// Type-based structure
/files/{{ $json.name.split('.').pop() }}/

// User-based structure
/users/{{ $json.owner || 'system' }}/

// Hybrid approach
/{{ $now.format('YYYY-MM') }}/{{ $json.type }}/

Template Features

🛡️ Safety & Security
Transfer Validation**: Confirms successful upload before source deletion
Error Preservation**: Source files remain intact on any failure
Audit Trail**: Complete logging of all operations with timestamps
Credential Security**: Secure storage using n8n's credential system
SFTP Support**: Encrypted transfers when available
Retry Logic**: Automatic recovery from transient network issues

📧 Notification System
Success Notifications:
Confirmation email with transfer details
File metadata (name, size, transfer time)
Next scheduled execution information
Webhook payload for monitoring systems

Error Notifications:
Immediate email alerts with error details
Troubleshooting steps and recommendations
Failed file information for manual intervention
Webhook integration for incident management

📊 Monitoring & Analytics
Execution Logs**: Detailed history of all workflow runs
Performance Metrics**: Transfer speeds and success rates
Error Tracking**: Categorized failure analysis
Audit Reports**: Compliance-ready activity logs

Production Considerations

🚀 Performance Optimization
File Size Limits**: Configure timeouts based on expected file sizes
Batch Processing**: Handle multiple files efficiently
Network Optimization**: Schedule transfers during off-peak hours
Resource Monitoring**: Track n8n server CPU, memory, and disk usage

🔧 Maintenance
Regular Testing**: Validate credentials and connectivity
Log Review**: Monitor for patterns in errors or performance
Credential Rotation**: Update passwords and keys regularly
Documentation Updates**: Keep configuration notes current

Testing Protocol

🧪 Pre-Production Testing
Phase 1: Test with 1-2 small files (< 1MB)
Phase 2: Test error scenarios (invalid credentials, network issues)
Phase 3: Test with representative file sizes and volumes
Phase 4: Validate email notifications and logging
Phase 5: Full production deployment with monitoring

⚠️ Important Testing Notes
Disable Source Deletion** during initial testing
Use test directories to avoid production data impact
Monitor execution logs** carefully during testing
Validate email delivery** to ensure notifications work
Test rollback procedures** before production use

Support & Documentation

This template includes:
8 Comprehensive Sticky Notes** with visual documentation
Detailed Node Comments** explaining every configuration option
Error Handling Guide** with common troubleshooting steps
Security Best Practices** for production deployment
Performance Tuning** recommendations for different scenarios

Technical Specifications

n8n Version**: 1.0.0+
Node Count**: 17 functional nodes + 8 documentation sticky notes
Execution Time**: 2-10 minutes (depending on file sizes and network speed)
Memory Usage**: 50-200MB (scales with file sizes)
Supported Protocols**: FTP, SFTP (recommended)
File Size Limit**: Up to 150MB per file (configurable)
Concurrent Files**: Processes files sequentially for stability

Who is this for?

🎯 Primary Users
System Administrators** managing file transfers between servers
DevOps Engineers** automating deployment and backup processes
IT Operations Teams** handling data migration projects
Business Process Owners** requiring automated file management

💼 Industries & Use Cases
Healthcare**: Patient data archiving and compliance reporting
Financial Services**: Secure document transfer and regulatory reporting
Manufacturing**: CAD file distribution and inventory data sync
E-commerce**: Product image and catalog management
Media**: Asset distribution and content delivery automation