meta-ads-scraper/README.md

Meta Ad Library scraper with AI-powered competitive analysis using Apify and Gemini 2.0 Flash

# Meta Ads Scraper

A powerful TypeScript library for scraping Meta (Facebook) Ad Library data and performing AI-powered competitive analysis using Google's Gemini 2.0 Flash model.

## Features

- 🚀 **Meta Ad Library Scraping**: Extract ad data from Facebook's Ad Library using Apify
- 🤖 **AI-Powered Analysis**: Analyze competitor ads with Google Gemini 2.0 Flash
- 📊 **Competitive Intelligence**: Generate strategic insights and ad concepts
- 🔄 **Smart Caching**: Reuse existing scraped data to save time and costs
- 📝 **TypeScript Support**: Full type safety and IntelliSense support

## Installation

   ```bash
npm install meta-ads-scraper
```

## Prerequisites

You'll need API keys for:
- **Apify**: For scraping Meta Ad Library data
- **Google Gemini**: For AI analysis

Get your keys:
- [Apify API Token](https://console.apify.com/account/integrations)
- [Google AI Studio API Key](https://makersuite.google.com/app/apikey)

## Quick Start

### Basic Usage

```typescript
import { MetaAdsScraper, GeminiAnalyzer, performCompetitiveAnalysis } from 'meta-ads-scraper';

// Set up your API keys
process.env.APIFY_TOKEN = 'your-apify-token';
process.env.GEMINI_API_KEY = 'your-gemini-api-key';

// Perform complete competitive analysis
const result = await performCompetitiveAnalysis({
  clientDetails: `
    Your company is a leading e-commerce platform specializing in 
    fast delivery and competitive pricing...
  `,
  competitors: [
    {
      name: 'Competitor 1',
      url: 'https://www.facebook.com/ads/library/?active_status=active&ad_type=all&country=ALL&is_targeted_country=false&media_type=all&search_type=page&view_all_page_id=123456789'
    },
    {
      name: 'Competitor 2', 
      url: 'https://www.facebook.com/ads/library/?active_status=active&ad_type=all&country=ALL&is_targeted_country=false&media_type=all&search_type=page&view_all_page_id=987654321'
    }
  ],
  scrapingOptions: {
    maxResults: 20,
    useExistingRun: true
  },
  apiKeys: {
    apifyToken: 'your-apify-token',      // Optional: falls back to APIFY_TOKEN env var
    geminiApiKey: 'your-gemini-api-key'  // Optional: falls back to GEMINI_API_KEY env var
  }
});

console.log('Analysis Results:', result);

// Parse the results into structured JSON format
import { parseCompetitiveAnalysisResult } from 'meta-ads-scraper';

const parsedResult = parseCompetitiveAnalysisResult(result);

console.log('Parsed Restructured Ads:', parsedResult.restructuredAds);
console.log('Parsed Industry Analysis:', parsedResult.industryAnalysis);
```

### Step-by-Step Usage

```typescript
import { MetaAdsScraper, GeminiAnalyzer } from 'meta-ads-scraper';

// 1. Scrape ads
const scraper = new MetaAdsScraper(process.env.APIFY_TOKEN);
    const ads = await scraper.getAds(competitorUrl, {
      maxResults: 10,
      useExistingRun: true
    });
    
// 2. Analyze with AI
const analyzer = new GeminiAnalyzer(process.env.GEMINI_API_KEY);

// Restructure ads by company
const restructuredAds = await analyzer.restructureAds(ads);

// Analyze each company
const companyAnalysis = await analyzer.getTopIdeasPerCompany(
  'Company Name', 
  restructuredAds.text
);

// Generate industry insights
const industryAnalysis = await analyzer.industryWriteup([companyAnalysis]);

// Create ad concepts
const adConcepts = await analyzer.generateAdIdeas(
  [companyAnalysis],
  industryAnalysis,
  'Your company details...'
);
```

## API Reference

### Core Classes

#### `MetaAdsScraper`

Scrapes Meta Ad Library data using Apify.

```typescript
const scraper = new MetaAdsScraper(apiToken?: string);

// Get ads from a Meta Ad Library URL
const ads = await scraper.getAds(url: string, options?: {
  maxResults?: number;
  useExistingRun?: boolean;
});
```

#### `GeminiAnalyzer`

Performs AI-powered analysis of scraped ad data.

```typescript
const analyzer = new GeminiAnalyzer(apiKey?: string);

// Restructure raw ad data into organized format
const restructured = await analyzer.restructureAds(ads: MetaAdData[]);

// Extract company-specific data
const companyText = analyzer.extractCompanyText(text: string, companyName: string);

// Analyze a specific company
const analysis = await analyzer.getTopIdeasPerCompany(companyName: string, text: string);

// Generate industry insights
const industry = await analyzer.industryWriteup(analyses: CompanyAnalysisResult[]);

// Create ad concepts
const concepts = await analyzer.generateAdIdeas(
  companyAnalyses: CompanyAnalysisResult[],
  industryAnalysis: IndustryAnalysisResult,
  clientDetails: string
);
```

### Main Function

#### `performCompetitiveAnalysis(config: CompetitiveAnalysisConfig)`

Performs a complete competitive analysis pipeline.

```typescript
interface CompetitiveAnalysisConfig {
  clientDetails: string;
  competitors: Competitor[];
  scrapingOptions?: ScrapingOptions;
  apiKeys?: {
    apifyToken?: string;
    geminiApiKey?: string;
  };
}

interface Competitor {
  name: string;
  url: string;
}

interface ScrapingOptions {
  maxResults?: number;
  useExistingRun?: boolean;
}
```

### Parsing Functions

Convert the text-based results into structured JSON format:

```typescript
import { 
  parseCompetitiveAnalysisResult, 
  parseRestructuredAds, 
  parseIndustryAnalysis,
  parseCompanyAnalysis
} from 'meta-ads-scraper';

// Parse the complete result
const parsedResult = parseCompetitiveAnalysisResult(result);

// Or parse individual components
const parsedAds = parseRestructuredAds(result.restructuredAds);
const parsedIndustry = parseIndustryAnalysis(result.industryAnalysis.text);
const parsedCompany = parseCompanyAnalysis(result.companyAnalyses[0].text);

// Example parsed structure:
console.log(parsedResult.restructuredAds.companies[0].ads[0]);
// {
//   adId: "768050109455085",
//   assetType: "Unknown (likely motion based on the presence of \"cards\")",
//   format: "Carousel (2 cards)",
//   visualHooks: ["💧 water droplet emoji, implying skincare or hydration."],
//   audiencesAddressed: ["Shoppers interested in Early Bird Deals and the Big Billion Days sale."],
//   ageGroup: "18-45 (typical online shoppers).",
//   graphicIdentity: ["Flipkart branding (colors not specified in data). The focus is on promoting the sale event."],
//   copywritingHooks: ["Early Bird Deals Starts 8th September", "💧 Glow season starts now!"],
//   copywritingAngles: ["Limited-time offer, emphasis on deals, pre-sale hype for Big Billion Days. Focus on skincare/beauty category."],
//   usps: ["Early access to deals before the main Big Billion Days sale."],
//   cta: "Learn more",
//   adSummary: "A carousel ad promoting Flipkart's Early Bird Deals...",
//   engagementMetrics: "Not available",
//   duration: "Null (likely short video or multiple static images)",
//   targetingAssumptions: ["Existing Flipkart users, individuals interested in online shopping..."],
//   publisherPlatform: ["FACEBOOK", "INSTAGRAM"]
// }

// Example parsed company analysis structure:
console.log(parsedResult.companyAnalyses[0]);
// {
//   companyName: "Flipkart",
//   creativeSummary: "The Flipkart ads analyzed consistently promote their Big Billion Days sale...",
//   topCreatives: {
//     description: "All the ads follow a similar structure...",
//     primaryHook: "Early Bird Deals Starts 8th September, combined with Big Billion Days branding...",
//     cta: "Learn More",
//     ctaFraming: "framed as a way to access early deals and wide product selection",
//     visualComposition: "The visuals likely utilize a vibrant and festive color scheme...",
//     copywritingTone: "The tone is promotional and benefit-oriented..."
//   },
//   whatWorks: [
//     {
//       title: "Consistent Branding",
//       description: "Strong, consistent use of Big Billion Days branding..."
//     }
//   ],
//   missedOpportunities: [
//     {
//       title: "Generic Learn More CTA",
//       description: "The consistent use of Learn More is a missed opportunity..."
//     }
//   ],
//   actionableTakeaways: [
//     {
//       title: "Test Category-Specific CTAs",
//       insight: "Generic CTAs limit conversion potential.",
//       suggestedFormat: "Carousel ads with category-specific CTAs on each card...",
//       hypothesis: "Category-specific CTAs will increase click-through rates..."
//     }
//   ]
// }
```

### Types

```typescript
interface MetaAdData {
  metadata: {
    page_name: string;
    ad_archive_id: string;
    scraped_at: string;
  };
  ad_content: {
    title: string;
    body: string;
    cta_text: string;
    cta_type: string;
    link_url: string;
    cards: Array<{
      title: string;
      body: string;
    }>;
  };
  distribution: {
    publisher_platform: string[];
  };
  status: {
    is_active: boolean;
  };
  performance: {
    impressions?: number;
    spend?: number;
    currency: string;
  };
}

interface CompetitiveAnalysisResult {
  clientDetails: string;
  competitors: Competitor[];
  totalAdsScraped: number;
  restructuredAds: string;
  companyAnalyses: CompanyAnalysisResult[];
  industryAnalysis: IndustryAnalysisResult;
  adConcepts: AdConceptResult;
  analysisMetadata: {
    timestamp: string;
    scrapingOptions?: ScrapingOptions;
    totalCompanies: number;
  };
}
```

## Configuration

### API Keys

You can provide API keys in two ways:

#### Option 1: Environment Variables (Recommended for production)

```bash
# Required
APIFY_TOKEN=your-apify-token
GEMINI_API_KEY=your-gemini-api-key

# Optional
NODE_ENV=production
```

#### Option 2: Pass directly in config (Useful for testing or multiple accounts)

```typescript
const result = await performCompetitiveAnalysis({
  clientDetails: 'Your company description...',
  competitors: [...],
  apiKeys: {
    apifyToken: 'your-apify-token',
    geminiApiKey: 'your-gemini-api-key'
  }
});
```

**Note**: If you provide API keys in the config, they will take precedence over environment variables. If neither is provided, the function will throw an error.

### Meta Ad Library URLs

The scraper works with Meta Ad Library URLs in this format:
```
https://www.facebook.com/ads/library/?active_status=active&ad_type=all&country=ALL&is_targeted_country=false&media_type=all&search_type=page&view_all_page_id=PAGE_ID
```

## Advanced Usage

### Custom Analysis Pipeline

```typescript
import { MetaAdsScraper, GeminiAnalyzer } from 'meta-ads-scraper';

async function customAnalysis() {
const scraper = new MetaAdsScraper();
const analyzer = new GeminiAnalyzer();

  // Scrape multiple competitors
  const allAds = [];
  for (const competitor of competitors) {
    const ads = await scraper.getAds(competitor.url);
    allAds.push(...ads);
  }
  
  // Custom analysis steps
  const restructured = await analyzer.restructureAds(allAds);
  
  // Process each company individually
  const analyses = [];
  for (const company of companies) {
    const companyText = analyzer.extractCompanyText(restructured.text, company);
    const analysis = await analyzer.getTopIdeasPerCompany(company, companyText);
    analyses.push(analysis);
  }
  
  return analyses;
}
```

### Error Handling

```typescript
try {
  const result = await performCompetitiveAnalysis(config);
  console.log('Analysis completed successfully');
} catch (error) {
  if (error.message.includes('API key')) {
    console.error('Invalid API key provided');
  } else if (error.message.includes('URL')) {
    console.error('Invalid Meta Ad Library URL');
  } else {
    console.error('Analysis failed:', error.message);
  }
}
```

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Support

- 📧 Email: support@example.com
- 🐛 Issues: [GitHub Issues](https://github.com/yourusername/meta-ads-scraper/issues)
- 📖 Documentation: [GitHub Wiki](https://github.com/yourusername/meta-ads-scraper/wiki)

## Changelog

### v1.0.0
- Initial release
- Meta Ad Library scraping with Apify
- AI-powered competitive analysis with Gemini 2.0 Flash
- Complete TypeScript support
- Smart caching and reuse of existing data