Back to Work Diary

Scaling AI Systems: Production Batch Processing with Built-In Disaster Recovery

Scaling AI Systems: Production Batch Processing with Built-In Disaster Recovery

systems-architectureai-engineeringdisaster-recoveryautomationproduction-systems

The Challenge: Scale AI Documentation Across 235 Production Systems

When you maintain a plugin marketplace with 235 live integrations, manual documentation doesn’t scale. Each plugin needed 8,000-14,000 byte enhancement files following official Anthropic standards - a multi-week manual effort.

My approach: Build an overnight batch processing system using Vertex AI Gemini 2.0 Flash, staying entirely within free tier limits while maintaining 100% success rate and full disaster recovery capabilities.

This case study demonstrates systems thinking, risk management, and production-grade automation under real constraints.

Systems Design: Starting with Constraints

Most engineers jump straight to implementation. I started by defining hard constraints:

Non-negotiable requirements:

  • Must stay within Vertex AI free tier (1,500 requests/day)
  • 100% success rate (no corrupted production files)
  • Complete audit trail for compliance
  • Disaster recovery plan before processing starts
  • Zero tolerance for quota violations

The math:

  • 235 plugins × 2 API calls each = 470 total calls
  • Free tier: 1,500 calls/day
  • Safety margin required: 3x headroom
  • Maximum rate: 500 calls/day
  • Minimum delay: ~170 seconds per plugin pair

I chose 90-120 seconds initially - ultra-conservative but guaranteed safe.

Phase 1: Build for Reliability First, Speed Second

Architecture Components

1. SQLite Audit Database

Every change tracked with timestamp, status, processing time:

CREATE TABLE enhancements (
    id INTEGER PRIMARY KEY,
    timestamp TEXT NOT NULL,
    plugin_name TEXT NOT NULL,
    plugin_path TEXT NOT NULL,
    enhancement_type TEXT NOT NULL,
    status TEXT NOT NULL,
    processing_time_seconds REAL
)

Why SQLite?

  • Zero external dependencies
  • Queryable for metrics
  • Easy to backup (copy one file)
  • Perfect for audit trails

2. Automatic Backup System

Before any modification:

  • Create timestamped backup directory
  • Copy entire plugin structure
  • Log backup location
  • Verify backup integrity

Recovery time: < 5 minutes to restore any single plugin.

3. Two-Phase AI Processing

Phase 1: Analysis and planning (15-20s) Phase 2: Generation (30-40s)

Why separate? If generation fails, we have the analysis cached. Saves API quota on retries.

4. Smart Rate Limiting

# Base delay with randomness (prevents patterns)
delay = 90.0 + random.uniform(0, 30.0)

# Extra rest every 10 plugins (long-term sustainability)
if idx % 10 == 0:
    extra_delay = random.uniform(30, 60)

The principle: Randomness prevents triggering rate limit algorithms. Regular breaks ensure sustainability over hours.

Phase 2: Observability and Monitoring

The Timeout Problem

First test run: Process appeared stuck.

My debugging process:

  1. Check process still running ✓
  2. Check CPU usage ✓
  3. Check log file… empty?

Root cause: Python output buffering. Script was working fine, but output wasn’t visible in real-time.

Fix: Unbuffered output (python3 -u)

Lesson: Production systems need real-time observability. You can’t debug what you can’t see.

Monitoring Dashboard

Simple but effective:

# Real-time progress
tail -f overnight-enhancement-all-plugins.log

# Success rate
sqlite3 enhancements.db \
  "SELECT status, COUNT(*) FROM enhancements GROUP BY status"

# Performance metrics
sqlite3 enhancements.db \
  "SELECT AVG(processing_time_seconds) FROM enhancements"

Business value: Know exactly when the system will complete, catch failures immediately, prove 100% success rate to stakeholders.

Related: Building Production CI/CD Systems covers similar observability patterns.

Phase 3: Disaster Recovery Planning

Mid-batch, legitimate concern raised: “What if we lose GitHub access?”

With 235 production plugins, GitHub lockout would be catastrophic. Local backups aren’t enough - they’re on the same machine.

I needed off-site backup within 30 minutes.

Turso: Edge SQLite for Disaster Recovery

Why Turso?

  • Edge SQLite database (globally distributed)
  • Free tier: 500 databases, 9GB storage
  • CLI-first (perfect for automation)
  • Git-like branching capabilities

Backup system design:

  1. Compress all plugins (tar.gz)
  2. Calculate SHA256 hashes (integrity verification)
  3. Export enhancement database (SQLite dump)
  4. Upload metadata to Turso (queryable backup records)
  5. Store file references (recovery instructions)
# Run backup
./scripts/turso-plugin-backup.sh

# Creates:
# - plugins-YYYYMMDD-HHMMSS.tar.gz (compressed archive)
# - enhancements-YYYYMMDD-HHMMSS.db (audit trail)
# - plugin-inventory.json (searchable metadata)
# - Turso records (off-site queryability)

Recovery time objective: < 30 minutes to restore complete repository from Turso.

Business impact: Eliminated single point of failure, ensured business continuity, provided compliance-grade audit trail.

Phase 4: Performance Optimization with Data

After 12 hours: 157/235 plugins complete (66%)

Analysis showed:

  • API quota usage: Only 7-14% of daily limit
  • Success rate: 100% (no failures)
  • Safety margin: Excessive (could safely go 2x faster)

Risk assessment:

  • Cutting delays in half: 45-60s per plugin
  • New quota usage: ~28% of daily limit
  • Still 3.5x safety margin
  • Completion time: 2:30 AM instead of 5:30 AM

The decision: Optimize based on real production data.

# Old: Ultra-conservative (testing phase)
RATE_LIMIT_DELAY = 90.0
RATE_LIMIT_RANDOMNESS = 30.0

# New: Conservative but proven safe (production data)
RATE_LIMIT_DELAY = 45.0
RATE_LIMIT_RANDOMNESS = 15.0

Result: 3 hours saved, still 100% success rate, well within safety margins.

Management lesson: Start conservative. Optimize with data. Never optimize blindly.

Phase 5: Smart Processing Logic

Skip What’s Already Done

The system intelligently skips plugins that already meet standards:

if skill_md_exists and len(content) > 8000:
    print("⏭️  Already comprehensive, skipping AI generation")
    # Just backup and validate (saves 45 seconds + API quota)

Business value:

  • Saves API quota (money)
  • Enables safe restarts after failures
  • Allows incremental improvements
  • Idempotent operations (run multiple times safely)

Graceful Degradation

If AI generation fails:

  1. Log detailed error to SQLite
  2. Preserve existing plugin structure (no corruption)
  3. Continue to next plugin (don’t block entire batch)
  4. Report failures in final summary

Zero data loss policy: Never overwrite working plugins with failed generations.

Production Results

Final Metrics (as of 11:30 PM):

  • Plugins processed: 163/235 (69%)
  • Success rate: 100%
  • Average enhancement size: 10,617 bytes
  • Processing time: 60-100s per plugin
  • API quota used: 22% of daily limit
  • Cost: $0 (free tier)

Quality metrics:

  • All files follow official Anthropic standards
  • Comprehensive documentation (8,000-14,000 bytes)
  • Complete backup trail (every change logged)
  • Zero corrupted files

Business impact:

  • 163 plugins × 10KB = 1.63MB of production documentation
  • Generated overnight, unattended
  • Zero manual intervention required
  • Full disaster recovery capabilities

Key Lessons for Engineering Leaders

1. Constraints Drive Better Design

Free tier limits forced me to:

  • Build efficient rate limiting
  • Implement smart skipping
  • Design for restartability
  • Monitor quota usage religiously

Result: Better system than if I had unlimited budget.

2. Disaster Recovery Isn’t Optional

Building Turso backup mid-batch was the right call. In production:

  • Murphy’s Law applies
  • GitHub can go down
  • Servers crash
  • Backups must be off-site

ROI: 30 minutes of engineering = eliminated existential business risk.

3. Observability Enables Optimization

Without real-time monitoring, I couldn’t:

  • Calculate accurate completion times
  • Identify optimization opportunities
  • Prove 100% success rate
  • Debug timeout issues

Investment: 10 minutes to add logging = hours saved in debugging.

4. Start Conservative, Prove Safety, Then Optimize

The 90s → 45s optimization was safe because:

  • I had 12 hours of production data
  • Metrics showed excessive safety margins
  • Success rate was 100%
  • Could monitor effects in real-time

Never optimize without data.

5. Idempotent Operations Enable Fault Tolerance

Smart skipping means:

  • Restarts are cheap
  • Partial failures are recoverable
  • Incremental improvements are possible
  • System is self-healing

Design principle: Every operation should be safely repeatable.

Related: Building Scalable Content Systems demonstrates similar fault-tolerant architecture.

Technical Skills Demonstrated

This project showcases:

Systems Architecture:

  • Rate limiting and quota management
  • Batch processing design
  • Fault-tolerant systems
  • Disaster recovery planning

Production Engineering:

  • Real-time observability
  • Performance optimization with data
  • Risk management under constraints
  • Zero-downtime operations

Data Engineering:

  • SQLite for audit trails
  • Integrity verification (SHA256)
  • Queryable backup metadata
  • Idempotent data operations

AI Engineering:

  • Vertex AI integration
  • Free tier optimization
  • Two-phase AI processing
  • Quality control for AI outputs

DevOps:

  • Automated backup systems
  • Off-site disaster recovery
  • Process monitoring
  • Production debugging

What’s Next

Immediate:

  • Complete batch processing (163/235 done tonight)
  • Run Turso backup after completion
  • Deploy v1.2.0 release

Short-term:

  • Automate weekly Turso backups
  • Build restoration testing procedures
  • Generate quality analytics dashboard
  • Document runbooks for operations

Long-term:

  • Progressive enhancement system (update existing files)
  • A/B testing framework for documentation quality
  • Cost optimization for scale (beyond free tier)
  • Multi-region backup strategy

Open Source Implementation

Full code available: claude-code-plugins

Key files:

  • scripts/overnight-plugin-enhancer.py - Batch processor
  • scripts/turso-plugin-backup.sh - Disaster recovery
  • scripts/TURSO-BACKUP-GUIDE.md - Recovery procedures

The Bottom Line

Processing 235 plugins with AI isn’t about throwing API calls at the problem. It requires:

✅ Systems thinking - Design for constraints, not infinite resources ✅ Risk management - Disaster recovery before you need it ✅ Data-driven optimization - Prove safety before going faster ✅ Production discipline - Observability, audit trails, idempotent operations ✅ Business focus - Zero data loss, complete automation, $0 cost

By 2:30 AM tonight, this system will have generated 2.3MB of high-quality documentation across 235 production plugins - completely unattended, entirely free, with 100% success rate and full disaster recovery.

That’s what production-grade AI engineering looks like.

Interested in AI engineering, systems architecture, or production operations? Connect with me on LinkedIn or check out more case studies on my portfolio.

Historical result: The enhanced plugin marketplace was hosted at the [retired legacy public domain].