@puberty-labs/clits/docs/CLOUDFLARE_WORKAROUND.md

CLiTS (Chrome Logging and Inspection Tool Suite) is a powerful Node.js library for AI-controlled Chrome browser automation, testing, and inspection. Features enhanced CSS selector support (:contains(), XPath), dry-run mode, element discovery tools, and co

# 🛡️ Cloudflare Bot Detection Workaround

## Overview

When using CLiTS with Chrome debugging mode on Cloudflare-protected websites, you may encounter bot detection challenges that prevent automation. This guide provides official workarounds to bypass these restrictions while maintaining full CLiTS functionality.

## Problem Description

### Symptoms
- "Can't verify the user is human. Please try again." error messages
- CAPTCHA challenges appearing during automation
- Authentication failures on Cloudflare-protected sites
- Login forms becoming unresponsive during automated interactions

### Root Cause
Chrome's debugging mode sets automation-related flags that Cloudflare's bot protection detects:
- `navigator.webdriver` property set to `true`
- Chrome DevTools Protocol presence
- Automation-specific browser fingerprints
- Missing or altered user agent strings

## ✅ Official Solution: Anti-Detection Chrome Startup

### Standard Procedure

1. **Stop all Chrome processes**:
   ```bash
   pkill -f "Google Chrome"
   ```

2. **Start Chrome with anti-detection flags**:
   ```bash
   "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
     --remote-debugging-port=9222 \
     --user-data-dir=/tmp/chrome-debug \
     --disable-blink-features=AutomationControlled \
     --exclude-switches="enable-automation" \
     --disable-extensions-except \
     --disable-plugins-discovery \
     --no-first-run \
     --no-default-browser-check &
   ```

3. **Wait for startup and verify**:
   ```bash
   sleep 5
   clits inspect --tabs --chrome-port 9222
   ```

### Automation Script (Recommended)

Create `scripts/start-cloudflare-safe-chrome.sh`:

```bash
#!/bin/bash
# CLiTS Cloudflare-Safe Chrome Launcher
# Official script for bypassing Cloudflare bot detection

set -e

echo "🛡️ CLiTS: Starting Cloudflare-safe Chrome debugging session..."

# Kill any existing Chrome processes
echo "📋 Stopping existing Chrome processes..."
pkill -f "Google Chrome" 2>/dev/null || true
sleep 2

# Clean up any stale debug directories
echo "🧹 Cleaning debug directory..."
rm -rf /tmp/chrome-debug 2>/dev/null || true

# Start Chrome with anti-detection flags
echo "🚀 Launching Chrome with anti-detection configuration..."
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --remote-debugging-port=9222 \
  --user-data-dir=/tmp/chrome-debug \
  --disable-blink-features=AutomationControlled \
  --exclude-switches="enable-automation" \
  --disable-extensions-except \
  --disable-plugins-discovery \
  --no-first-run \
  --no-default-browser-check \
  --disable-dev-shm-usage \
  --no-sandbox > /dev/null 2>&1 &

# Wait for Chrome to fully initialize
echo "⏳ Waiting for Chrome initialization..."
sleep 5

# Verify connection
echo "🔍 Verifying debugging connection..."
if curl -s http://localhost:9222/json/version > /dev/null; then
    echo "✅ Chrome debugging ready on port 9222"
    echo "🛡️ Anti-detection flags active - Cloudflare bypass enabled"
    echo ""
    echo "📋 Connection Details:"
    curl -s http://localhost:9222/json/version | jq -r '"Browser: " + .Browser'
    echo ""
    echo "🎯 Ready for CLiTS automation on Cloudflare-protected sites"
else
    echo "❌ Failed to establish Chrome debugging connection"
    echo "💡 Try running: lsof -i :9222 to check port availability"
    exit 1
fi
```

Make executable:
```bash
chmod +x scripts/start-cloudflare-safe-chrome.sh
```

## 🔧 Anti-Detection Flags Explained

| Flag | Purpose | Cloudflare Impact |
|------|---------|-------------------|
| `--disable-blink-features=AutomationControlled` | Prevents `navigator.webdriver = true` | **Primary bypass method** |
| `--exclude-switches="enable-automation"` | Removes automation indicators | Reduces detection signals |
| `--disable-extensions-except` | Minimizes extension fingerprinting | Cleaner browser profile |
| `--disable-plugins-discovery` | Reduces plugin detection | Lower automation signature |
| `--no-first-run` | Skips Chrome setup dialogs | Smoother automation startup |
| `--no-default-browser-check` | Prevents permission prompts | Uninterrupted automation |

## 🚀 Usage with CLiTS Commands

After starting Chrome with anti-detection flags, all CLiTS commands work normally:

```bash
# Basic interactions
clits extract --chrome --chrome-port 9222
clits interact --click ".login-button" --chrome-port 9222
clits navigate --url "https://cloudflare-site.com" --chrome-port 9222

# Advanced automation
clits automate --script cloudflare-workflow.json --chrome-port 9222
clits vision --screenshot --output "cloudflare-page.png" --chrome-port 9222

# Debugging and inspection
clits inspect --find-clickable --chrome-port 9222
clits extract --console --log-levels error --chrome-port 9222
```

## 🎯 Alternative Solutions

### Option 1: Use Regular Chrome Profile
For sites where you're already logged in:

```bash
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --remote-debugging-port=9222 \
  --disable-blink-features=AutomationControlled \
  --exclude-switches="enable-automation" \
  --no-first-run &
```

**Pros**: Maintains existing cookies and login state  
**Cons**: May affect your regular browsing session

### Option 2: Custom User Agent
For additional stealth:

```bash
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --remote-debugging-port=9222 \
  --user-data-dir=/tmp/chrome-debug \
  --disable-blink-features=AutomationControlled \
  --exclude-switches="enable-automation" \
  --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36" &
```

### Option 3: Different Debug Port
If port 9222 is occupied:

```bash
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --remote-debugging-port=9223 \
  --user-data-dir=/tmp/chrome-debug \
  --disable-blink-features=AutomationControlled \
  --exclude-switches="enable-automation" &

# Use with CLiTS
clits inspect --tabs --chrome-port 9223
```

## 🔍 Testing and Verification

### Basic Connection Test
```bash
# 1. Check if port is listening
lsof -i :9222

# 2. Test Chrome API
curl -s http://localhost:9222/json/version

# 3. Verify CLiTS connection
clits inspect --tabs --chrome-port 9222
```

### Cloudflare Detection Test
```bash
# Navigate to a Cloudflare-protected site
clits navigate --url "https://your-cloudflare-site.com" --chrome-port 9222

# Take screenshot to check for bot detection
clits vision --screenshot --output "cloudflare-test.png" --chrome-port 9222

# Look for authentication elements (should be visible if bypass worked)
clits inspect --find-clickable --chrome-port 9222 | grep -i "login\|sign\|email"
```

## 🛠️ Troubleshooting

### Chrome Won't Start
```bash
# Check for existing processes
ps aux | grep Chrome

# Force kill all Chrome processes
pkill -9 -f "Google Chrome"

# Check port availability
lsof -i :9222

# Clear debug directory
rm -rf /tmp/chrome-debug

# Try alternative port
# Use --remote-debugging-port=9223 instead
```

### Still Getting Bot Detection
```bash
# 1. Verify anti-detection flags are active
# Check Chrome process flags:
ps aux | grep "AutomationControlled"

# 2. Try with regular profile
# Remove --user-data-dir flag

# 3. Add additional stealth flags
--disable-features=VizDisplayCompositor \
--disable-ipc-flooding-protection \
--disable-renderer-backgrounding \
--disable-backgrounding-occluded-windows
```

### CLiTS Connection Issues
```bash
# 1. Verify Chrome debugging API
curl -v http://localhost:9222/json/version

# 2. Check CLiTS can see tabs
clits inspect --tabs --chrome-port 9222

# 3. Test basic interaction
clits extract --url --chrome-port 9222
```

## 📱 Platform-Specific Instructions

### macOS (Primary Support)
```bash
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" [flags]
```

### Linux
```bash
google-chrome [flags]
# or
google-chrome-stable [flags]
```

### Windows
```cmd
"C:\Program Files\Google\Chrome\Application\chrome.exe" [flags]
```

## ⚠️ Security and Performance Notes

### Security Considerations
- `--disable-web-security` creates security vulnerabilities
- Only use anti-detection flags in development/testing environments
- Never expose debugging port externally (`--remote-debugging-address=0.0.0.0`)
- Use temporary data directories for isolation

### Performance Impact
- Startup time increases by ~2-3 seconds
- Memory usage similar to regular Chrome session
- Anti-detection flags have minimal performance impact
- May occasionally need restart after extended use

## 🔄 Integration with CI/CD

### GitHub Actions
```yaml
- name: Start Cloudflare-safe Chrome
  run: |
    ./scripts/start-cloudflare-safe-chrome.sh
    
- name: Run CLiTS automation
  run: |
    clits automate --script tests/cloudflare-workflow.json --chrome-port 9222
```

### Docker
```dockerfile
# Add anti-detection flags to Chrome startup
CMD ["google-chrome", "--remote-debugging-port=9222", "--disable-blink-features=AutomationControlled", "--exclude-switches=enable-automation", "--no-sandbox", "--disable-dev-shm-usage", "--headless=new"]
```

## 📋 Quick Reference

### Start Command (Copy-Paste Ready)
```bash
pkill -f "Google Chrome"; "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --remote-debugging-port=9222 --user-data-dir=/tmp/chrome-debug --disable-blink-features=AutomationControlled --exclude-switches="enable-automation" --disable-extensions-except --disable-plugins-discovery --no-first-run --no-default-browser-check &; sleep 5
```

### Verification Commands
```bash
lsof -i :9222 && curl -s http://localhost:9222/json/version && clits inspect --tabs --chrome-port 9222
```

---

**💡 Key Success Factor**: The `--disable-blink-features=AutomationControlled` flag is the primary method for bypassing Cloudflare detection. This prevents the `navigator.webdriver` property from being set to `true`, which is Cloudflare's main detection vector.

For additional support, see [CLiTS Troubleshooting Guide](./TROUBLESHOOTING.md) or contact the development team.