Provider System Overview

Understanding how the provider system works is crucial for building effective scrapers.

The all.ts Registration System

All scrapers must be registered in src/providers/all.ts. This central file exports two main functions that gather all available providers:

// src/providers/all.ts
import { Embed, Sourcerer } from '@/providers/base';
import { embedsuScraper } from './sources/embedsu';
import { turbovidScraper } from './embeds/turbovid';

export function gatherAllSources(): Array<Sourcerer> {
  return [
    cuevana3Scraper,
    catflixScraper,
    embedsuScraper,    // Your source scraper goes here
    // ... more sources
  ];
}

export function gatherAllEmbeds(): Array<Embed> {
  return [
    upcloudScraper,
    turbovidScraper,   // Your embed scraper goes here
    // ... more embeds
  ];
}

Why this matters:

  • Only registered scrapers are available to the library
  • The order in these arrays doesn't matter (ranking determines priority)
  • You must import your scraper and add it to the appropriate function

Provider Types

There are two distinct types of providers in the system:

Sources (Primary Scrapers)

Sources find content on websites and return either:

  • Direct video streams (ready to play immediately)
  • Embed URLs that need further processing by embed scrapers

Characteristics:

  • Handle website navigation and search
  • Process TMDB IDs to find content
  • Can return multiple server options
  • Located in src/providers/sources/

Example source workflow:

  1. Receive movie/show request with TMDB ID
  2. Search the target website for that content
  3. Extract embed player URLs or direct streams
  4. Return results for further processing

Embeds (Secondary Scrapers)

Embeds extract playable video streams from embed players:

  • Take URLs from sources as input
  • Handle player-specific extraction and decryption
  • Always return direct streams (never more embeds)

Characteristics:

  • Focus on one player type (turbovid, mixdrop, etc.)
  • Handle complex decryption/obfuscation
  • Specialized for specific player technologies
  • Located in src/providers/embeds/

Example embed workflow:

  1. Receive embed player URL from a source
  2. Fetch and parse the embed page
  3. Extract/decrypt the video stream URLs
  4. Return playable HLS or MP4 streams

Ranking System

Every scraper has a rank that determines its priority in the execution queue:

How Ranking Works

  • Higher numbers = Higher priority (processed first)
  • Each rank must be unique across all providers
  • Sources and embeds have separate ranking spaces
  • Failed scrapers are skipped, next rank is tried

Rank Ranges

Usually ranks should be on 10s: 110, 120, 130...

// Typical rank ranges (not enforced, but conventional)
Sources:  1-300
Embeds:   1-250

// Example rankings
export const embedsuScraper = makeSourcerer({
  id: 'embedsu',
  rank: 165,    // Medium priority source
  // ...
});

export const turbovidScraper = makeEmbed({
  id: 'turbovid', 
  rank: 122,    // Medium priority embed
  // ...
});

Choosing a Rank

For Sources:

  • 200+: High-quality, reliable sources (fast APIs, good uptime)
  • 100-199: Medium reliability sources (most scrapers fall here)
  • 1-99: Lower priority or experimental sources

For Embeds:

  • 200+: Fast, reliable embeds (direct URLs, minimal processing)
  • 100-199: Standard embeds (typical decryption/extraction)
  • 1-99: Slow or unreliable embeds (complex decryption, poor uptime)

Finding Available Ranks

Before choosing a rank, check what's already taken:

# Search for existing ranks
grep -r "rank:" src/providers/ | sort -t: -k3 -n

Or check the cli to see the ranks.

Duplicate ranks will cause conflicts! Always verify your chosen rank is unique before submitting.

Provider Configuration

Each provider is configured using makeSourcerer() or makeEmbed():

Source Configuration

export const mySourceScraper = makeSourcerer({
  id: 'my-source',           // Unique identifier (kebab-case)
  name: 'My Source',         // Display name (human-readable)
  rank: 150,                 // Priority rank (must be unique)
  disabled: false,           // Whether scraper is disabled
  flags: [],                 // Feature flags (see Advanced Concepts)
  scrapeMovie: comboScraper, // Function for movies
  scrapeShow: comboScraper,  // Function for TV shows
});

Embed Configuration

export const myEmbedScraper = makeEmbed({
  id: 'my-embed',            // Unique identifier (kebab-case)
  name: 'My Embed',          // Display name (human-readable)
  rank: 120,                 // Priority rank (must be unique)
  disabled: false,           // Whether scraper is disabled
  async scrape(ctx) {        // Single scrape function for embeds
    // ... extraction logic
    return { stream: [...] };
  },
});

How Providers Work Together

The provider system creates a powerful pipeline:

1. Source → Embed Chain

User Request → Source Scraper → Embed URLs → Embed Scraper → Video Stream → Player

Pipeline Steps:

  1. User Request - User wants to watch content
  2. Source Scraper - Finds content on websites
  3. Embed URLs - Returns player URLs that need processing
  4. Embed Scraper - Extracts streams from player URLs
  5. Video Stream - Final playable stream
  6. Player - User watches the content

2. Multiple Server Options

Sources can provide multiple backup servers:

// Source returns multiple embed options
return {
  embeds: [
    { embedId: 'turbovid', url: 'https://turbovid.com/abc' },
    { embedId: 'mixdrop', url: 'https://mixdrop.co/def' },
    { embedId: 'dood', url: 'https://dood.watch/ghi' }
  ]
};

3. Fallback System

If one embed fails, the system tries the next:

  1. Try turbovid embed (rank 122)
  2. If fails, try mixdrop embed (rank 198)
  3. If fails, try dood embed (rank 173)
  4. Continue until success or all options exhausted

Best Practices

Naming Conventions

  • IDs: Use kebab-case (my-scraper, not myMyscraper or My_Scraper)
  • Names: Use proper capitalization (VidCloud, not vidcloud or VIDCLOUD)
  • Files: Match the ID (my-scraper.ts for ID my-scraper)

Registration Order

  • The order in all.ts arrays doesn't affect execution (rank does)
  • Group similar scrapers together for maintainability
  • Add imports at the top, organized logically

Testing Integration

Always test that your registration works:

# Verify your scraper appears in the list (interactive mode shows all available)
pnpm cli

# Test your specific scraper
pnpm cli --source-id my-scraper --tmdb-id 11527

Next Steps

Now that you understand the provider system:

  1. Learn the details in Building Scrapers
  2. Study Advanced Concepts for flags and error handling
  3. Look at the Sources vs Embeds guide for more examples
Table of Contents