@awesomate/hosting-mcp/skill/awesomate-knowledge/references/ingestion-sources.md

Awesomate MCP server — lets Claude manage your Awesomate WordPress hosting, plan, limits, n8n automations, and build Node/static apps + databases

# Ingestion — what goes in, caps, cost and time

Source of truth: the platform contract (api-v1.md, frozen 2026-08-27). The
platform enforces the real numbers; these are for setting expectations and
estimating BEFORE asking approval.

## What can be ingested

| Kind | Via | Notes |
|---|---|---|
| Web page / blog post | `add {url}` | One page per URL. A page that can't be reached, or has under 200 characters of readable text, fails the job with the reason (`error_code` `page_unreachable` or `page_unreadable`) rather than succeeding empty |
| Whole site / blog | `add {sitemap, since?}` | A `sitemap.xml`, or a sitemap index that points to other sitemaps. Only pages on the sitemap's own site are read. Up to 500 pages per job: a sitemap listing more adds the first 500 and carries on in a follow-on job (`result_summary.remaining` and `continued_as`), so nothing is cut off silently. The one exception: when the workspace's allowance has run out (`paused: "credit_exhausted"`), no follow-on job is made and nothing resumes by itself; say how many pages were not read, and add the sitemap again after a top-up (pages already in are skipped as unchanged). `since` skips entries whose `<lastmod>` is older; when every entry is older, the job succeeds having added nothing. A sitemap that lists no pages to read fails with a plain reason: the address is a web page rather than a sitemap, it lists no page addresses, its pages are all on another site, or the sitemaps it points to couldn't be opened |
| Link to one Vimeo or Wistia video | `add {url}` | **Copied and transcribed**, with playback, when that provider's library is connected (hub, Knowledge, Video library) and the plan includes audio and video (Pro and above, within the month's media hours): the job's `kind` is `video-link`. Not connected yet, or a channel, showcase or folder rather than one video: read as a web page, which often has too little text and fails (`page_unreadable`); once the library is connected, add the link again. Below Pro, or in a trial, it is refused (403 `upgrade_required`, `capability: "media_ingest"`) |
| A whole Vimeo or Wistia library | hub, Knowledge, Video library | Pro and above. Anyone with full access connects the account with a token that can read the videos, not only the account (Vimeo: a personal access token with Private and Video Files; Wistia: a token with "Read all folder and media data"). The hub lists the library and shows what bringing it in involves before anything is copied or charged: videos with captions come in free, the rest are transcribed at A$15 per 10 hours or brought in without transcription. Only the account owner starts an import. Where the account has the video picker, the owner chooses part of the library instead (one video to try first, a folder, the newest few) and is quoted for just that; a library too big to do alone has "Have our team bring it in" in the hub, which raises a ticket and charges nothing. Read progress with `awesomate_knowledge_sources {action:'video_imports'}` (it carries `progress_words`, the `choice` and the `problems` grouped by reason) and browse with `{action:'video_library', importId, view:'folders'|'videos', folderId?, q?, from?, to?}`; connecting, choosing, the quote, starting, pausing, stopping and asking our team stay in the hub, so never say a video is chosen or coming in from what you read |
| YouTube link | `add {url}` | Refused on every plan (400): "YouTube links are not supported yet". No upgrade changes that, so never offer one for it. Uploading the business's own video files works on Pro and above, or add the page that embeds the video |
| Any other link to a recording (Loom, podcast audio, a file address) | `add {url}` | **Not transcribed.** Read as a web page, which gives at most its title and description; a link straight to an audio or video file has no page to read and fails (`page_unreadable`). Pro and above; below Pro, or in a trial, it is refused (403 `upgrade_required`, `capability: "media_ingest"`). To transcribe the recording, upload the file |
| PDF / book / document / markdown | hub upload page | Text layer used; scanned pages OCR'd automatically (`auto` mode, capped at 300 OCR pages per job — over that the job fails BEFORE any vision spend) |
| Video/audio FILES | hub upload page (≤ 100 MB) | Transcribed (Deepgram, language defaults to English). Pro and above, within the month's media hours. A longer recording: split or compress it and upload each part |
| Images | hub upload page | Vision-described, then indexed |
| Facebook export | hub upload page | The platform's facebook-export lane |
| FAQs | Markdown file via `awesomate_knowledge_upload` | One `##` per question, so each becomes its own citable section. **Never** a Q&A spreadsheet as it is: convert it to this file, so every answer stays word for word and citable. Full recipe: [faqs.md](faqs.md) |
| Spreadsheet (`.csv`, `.tsv`, `.xls`, `.xlsx`, `.xlsm`, `.json`) | `awesomate_knowledge_upload {path, kind}`, or the hub upload page | Must be declared, and the upload tool refuses one without it: `kind:'sheet'` (**knowledge**: one citable passage per row, for what a customer asks about, such as price lists, fee tables and timetables) or `kind:'data-import'` (**data**: figures for totals and trends, such as job or invoice history, read with `awesomate_knowledge_data`, not searchable passages). Ask the owner when it is not obvious; never guess. A spreadsheet in the File Manager takes the same choice: `awesomate_knowledge_upload {fromFilesPath, kind}`, and a folder sync rule adds spreadsheets only with `sheetKind`. FAQs in a sheet: convert to Markdown instead, see [faqs.md](faqs.md) |

Images, and scanned pages that need OCR, are Pro and above too. During a trial,
audio, video, images and scanned documents are paused on every plan and switch
on with the first payment.

Local files go through `awesomate_knowledge_upload`, which streams them from
disk so their contents never enter the conversation (the owner can also use
the hub upload page). 100 MB per file.

## Job lifecycle

`add` returns a job id (202). Poll `awesomate_knowledge_sources
{action:'jobs'}` — statuses `queued → running → succeeded | failed`.
Re-adding identical content is cheap: unchanged pages and already-seen
media are skipped by content hash (`unchanged` / `transcript_cached` in the
job summary). Typical time-to-searchable: minutes for pages; media takes
longer (roughly the media's own duration as a worst case).

To bring a web page up to date, use `refresh` {sourceId}, or `add` the same address again: both update it in place (same source, visibility, tags and collections kept; without `visibility`, an existing source keeps its own). A refresh job's `summary_text` reads "Read again: updated" or "Read again: no change since <date>"; a plain re-add reads as an add ("1 already up to date"). A video or recording link is refused by `refresh`: sent again it would come in as the video. `refresh` on anything but a web page answers `not_a_web_page`: upload the file again instead.

## Cost expectations (for the approval conversation)

- Text (pages, posts, books): cents to a few dollars for a typical corpus.
- Media is the big line: transcription ≈ $0.30/media-hour plus enrichment —
  a 200-hour channel is real money AND real quota (media-hours).
- Scanned PDFs needing OCR cost per page — the 300-page OCR cap fails the
  job before surprise spend.
- Everything counts against the plan allowance first
  ([quotas-and-packs.md](quotas-and-packs.md)); a `pack_required` response
  means the allowance is exhausted, and nothing was purchased or queued.

Always: estimate → state it plainly ("about X hours of video — that's ~$Y
and most of this month's media allowance") → explicit yes → queue.