@studyportals/sp-hs-misc
Version:
Miscellaneous code used in HouseStark's projects
1,262 lines (991 loc) • 53.5 kB
Markdown
# Campaigns Table Revamp & API Migration Plan
This document describes how the `campaigns` table and related ServiceLayer-Private API endpoints will be changed, and what other services that **consume this API** must do to stay compatible.
The core goals are:
- Make all business-critical columns on `campaigns` mandatory and strongly typed.
- Remove deprecated/unused columns from `campaigns`.
- Treat the ServiceLayer `campaigns` endpoints as **read-only** (no inserts/updates/deletes through this API).
- Preserve read behaviour for all dependent services with **no downtime**.
- Delete all legacy data that does not conform to the new schema.
- **Remove all Banner, BannerBox, Lead, LeadCampaign, and LeadCampaignStudy entities entirely** (deprecated functionality).
- **Drop 17 related database tables** after code cleanup is deployed.
> **Audience:** Developers of services that call ServiceLayer-Private campaign endpoints.
---
## AI Assistant Prompt
Use this prompt when analyzing other repositories for required changes:
> Here is a migration plan for ServiceLayer-Private campaigns API. Analyze this repository's code and identify all places that need to be updated according to Phase 1 requirements. Specifically look for:
>
> **ServiceLayer API Changes:**
> - Any PUT/POST/DELETE calls to `/data/campaigns/` endpoints (ALL campaign writes are now blocked)
> - Usage of `/data/campaigns/.../organisations/` or `/data/campaigns/.../countries/` sub-component endpoints for LINKING (REMOVED - use Campaign Management API for org linking)
> - Usage of `countries` decorator data in campaign responses (REMOVED - country data no longer in responses)
> - Usage of dropped fields listed in section 8
> - Usage of deprecated search parameters
> - Code that assumes `crm_id` is a string
> - Enums that include deprecated values (`advanced_nrbp`, `rbp_cpl`, `west`, `bne`, `global`, `act`)
>
> **Important distinction:**
> - `organisations` decorator data **still available** in responses (read-only)
> - `countries` decorator data **removed** from responses (deprecated feature)
> - Organisation/country **linking operations** (PUT/DELETE) are removed from ServiceLayer
>
> **Legacy Endpoint Migration (see Section 10):**
> - `/data/campaigns/all/` → Use `/data/campaigns/any/details/`
> - `/data/campaigns/all/{id}/` → Use `/data/campaigns/any/details/{id}/`
> - `/data/campaigns/details/` → Use `/data/campaigns/any/details/`
> - `/data/campaigns/ids/` → Use `/data/campaigns/any/ids/`
> - `/data/campaigns/search/` → Use `/data/campaigns/any/ids/`
> - `/data/campaigns/sender/` → **REMOVED** (use `/any/details/` with `recipients` decorator)
> - `/data/campaigns/all/{id}/banners/` → **REMOVED** (deprecated feature, no replacement)
> - `/data/campaigns/any/details/{id}/banners/` → **REMOVED** (deprecated feature, no replacement)
> - `/data/campaigns/all/{id}/lead-campaigns/` → **REMOVED** (deprecated feature, no replacement)
> - `/data/campaigns/any/details/{id}/lead-campaigns/` → **REMOVED** (deprecated feature, no replacement)
> - `/data/campaigns/all/{id}/organisations/` → **REMOVED** (use Campaign Management API)
> - `/data/campaigns/all/{id}/countries/` → **REMOVED** (deprecated feature)
>
> **Package Updates Required:**
> - Update `@studyportals/sp-millennium-falcon` to ^3.0.0
> - Update `@studyportals/sp-hs-misc` to ^3.0.5 (or ^3.0.1 minimum)
>
> **hs-misc Breaking Changes (see Section 11):**
> - **For Node.js:** Imports from `bin-browser/` no longer work - use main entry point instead: `import { ... } from '@studyportals/sp-hs-misc'`
> - **For browsers (v3.0.5+):** Browser build restored as ES modules - use `<script type="module">` with imports from `/bin-browser/`
> - References to `externalNotes` on campaign objects — this property has been removed from `ICampaign`
> - Usage of `CampaignType.ADVANCED_NRBP` or `CampaignType.RBP_CPL` — these enum values have been removed
>
> **Search patterns to use:**
> ```
> /data/campaigns/all # legacy endpoint - migrate to /any/details/
> /data/campaigns/details # legacy endpoint - migrate to /any/details/
> /data/campaigns/ids # legacy endpoint - migrate to /any/ids/
> /data/campaigns/search # legacy endpoint - migrate to /any/ids/
> /data/campaigns/sender # legacy endpoint - REMOVED, no equivalent
> sp-hs-misc/bin-browser # broken imports (Node.js) - use main entry point
> externalNotes # removed field
> external_notes # removed field (API response)
> ADVANCED_NRBP # removed enum value
> RBP_CPL # removed enum value
> "/data/campaigns/" # check for POST/PUT/DELETE
> /organisations/.*PUT # organisation linking removed (data still in responses)
> /organisations/.*DELETE # organisation linking removed (data still in responses)
> /countries/ # country linking AND data removed
> campaign.countries # decorator removed - data no longer in responses
> chain # chain decorator removed
> banners # banner decorator and linking REMOVED
> lead-campaigns # lead-campaign linking REMOVED
> banner_ids # search parameter REMOVED
> has_banner # search parameter REMOVED
> lead_campaign_ids # search parameter REMOVED
> automated # dropped field
> tableau_reporting # dropped field
> rating # dropped field
> chain_id # dropped field
> options # dropped field
> adwords_start # dropped field
> adwords_end # dropped field
> capped_budget # search parameter REMOVED (budget always non-null)
> ```
>
> **Write operations status:**
> - BLOCKED: ALL `PUT/POST/DELETE` on `/data/campaigns/` entity endpoints
> - BLOCKED: Organisation linking (use Campaign Management API instead)
> - BLOCKED: Country linking (deprecated, no longer supported)
> - BLOCKED: Banner linking to campaigns (deprecated, no longer supported)
> - BLOCKED: Lead-campaign linking to campaigns (deprecated, no longer supported)
> - ALLOWED: `PUT/DELETE` on `/data/campaigns/any/details/{id}/customproducts/`
> - ALLOWED: `PUT/DELETE` on `/data/campaigns/any/details/{id}/presetproducts/`
>
> **COMPLETELY REMOVED ENTITIES (see Section 12):**
> - Banner entity and all `/data/banners/*` endpoints → **REMOVED** (no replacement)
> - BannerBox entity and all `/data/banner-boxes/*` endpoints → **REMOVED** (no replacement)
> - Lead entity and all `/data/leads/*` endpoints → **REMOVED** (no replacement)
> - LeadCampaign entity and all `/data/lead-campaigns/*` endpoints → **REMOVED** (no replacement)
> - LeadCampaignStudy entity and all `/data/lead-campaign-studies/*` endpoints → **REMOVED** (no replacement)
> - Banner driver (`/banner/*`) → **REMOVED** (no replacement)
> - LeadQualifier driver (`/lead-qualifier/*`) → **REMOVED** (no replacement)
> - Publish lead routes (`/publish/lead-*`, `/publish/email/lead/*`) → **REMOVED** (no replacement)
>
> **Additional search patterns for removed entities:**
> ```
> /data/banners/ # REMOVED - banner entity deleted
> /data/banner-boxes/ # REMOVED - bannerbox entity deleted
> /data/leads/ # REMOVED - lead entity deleted
> /data/lead-campaigns/ # REMOVED - leadcampaign entity deleted
> /data/lead-campaign-studies/ # REMOVED - leadcampaignstudy entity deleted
> /banner/ # REMOVED - banner driver deleted
> /lead-qualifier/ # REMOVED - leadqualifier driver deleted
> /publish/lead- # REMOVED - publish lead routes deleted
> /publish/email/lead/ # REMOVED - publish lead routes deleted
> study.lead_campaigns # REMOVED - decorator deleted
> has_lead_campaign # REMOVED - search parameter deleted
> ```
---
## 1. Schema Changes (MySQL)
The `campaigns` table will be updated as follows:
### 1.1 Columns becoming NOT NULL / typed
The following columns will become non-nullable and (re)typed as shown:
```sql
ALTER TABLE campaigns
MODIFY persist_gl_param_ref_link TINYINT(1) NOT NULL,
MODIFY budget INT NOT NULL,
MODIFY budget_total INT NOT NULL,
MODIFY assistant_id INT NOT NULL,
MODIFY handler_id INT NOT NULL,
MODIFY crm_id INT NOT NULL,
MODIFY custom_products_budget INT NOT NULL,
MODIFY `type` ENUM('rbp','nrbp','hybrid_ebp','full_ebp','free_trial') NOT NULL,
MODIFY region ENUM('other','uk','apac','emea','na','strp') NOT NULL,
MODIFY currency ENUM('EUR','GBP','USD','AUD','CAD') NOT NULL,
MODIFY campaign_type ENUM('custom_products','listing_products') NOT NULL,
MODIFY start_date DATE NOT NULL,
MODIFY end_date DATE NOT NULL;
```
**Important changes to note:**
1. **`crm_id`** changes from `VARCHAR(5)` to `INT NOT NULL` – this is a type change, not just a constraint change.
2. **`type` enum** loses these values:
- `advanced_nrbp` (removed)
- `rbp_cpl` (removed)
3. **`region` enum** loses these values:
- `west` (removed)
- `bne` (removed)
- `global` (removed)
- `act` (removed)
4. **`start_date` and `end_date`** change from nullable to `DATE NOT NULL`. **Important:** While the database stores these as DATE type, the ServiceLayer API **converts them to Unix timestamps** in responses using:
```sql
UNIX_TIMESTAMP(c.start_date) AS start_date,
UNIX_TIMESTAMP(c.end_date) AS end_date
```
This means consuming services will continue to receive integer Unix timestamps, not DATE strings.
5. Prior to this change, **all rows that do not satisfy these constraints will be deleted**. There will be no attempts to coerce or migrate invalid values – if your service relies on such rows, they will disappear.
### 1.2 Dropped columns
The following columns will be removed from the `campaigns` table entirely:
```sql
ALTER TABLE campaigns
DROP COLUMN automated,
DROP COLUMN tableau_reporting,
DROP COLUMN is_scheduled,
DROP COLUMN nullIfOld,
DROP COLUMN automation_notes,
DROP COLUMN options,
DROP COLUMN external_notes,
DROP COLUMN chain_id,
DROP COLUMN rating,
DROP COLUMN adwords_start,
DROP COLUMN adwords_end;
```
After this migration:
- These columns **no longer exist** in the database.
- Any query or ORM mapping in your code that selects/references them must be removed or adapted.
---
## 2. ServiceLayer Campaign API Contract Changes
### 2.1 Campaign endpoints become read-only
Historically, the ServiceLayer allowed creation, update and deletion of campaigns via endpoints such as:
- `/data/campaigns/any/details/`
- Legacy: `/data/campaigns/all/`, `/data/campaigns/details/`, `/data/campaigns/ids/`, `/data/campaigns/sender/`
Going forward:
- These endpoints will be treated as **read-only**. No new campaigns will be created, updated or deleted through ServiceLayer-Private.
- The underlying `Campaign` entity in ServiceLayer will **not** call `INSERT`, `UPDATE` or `DELETE` on the `campaigns` table anymore.
- Any attempt by client services to use `PUT`, `POST` or `DELETE` on campaign endpoints must be removed.
**Sub-component linking changes:**
| Operation | Status | Notes |
|-----------|--------|-------|
| Custom products linking | ✅ Still supported | Use `/data/campaigns/any/details/{id}/customproducts/` |
| Preset products linking | ✅ Still supported | Use `/data/campaigns/any/details/{id}/presetproducts/` |
| Banner linking | ❌ **REMOVED** | Deprecated feature, no longer supported |
| Lead campaign linking | ❌ **REMOVED** | Deprecated feature, no longer supported |
| Organisation linking | ❌ **REMOVED** | Use Campaign Management API instead |
| Country linking | ❌ **REMOVED** | Deprecated feature, no longer supported |
**Legacy endpoints removed:**
All legacy campaign endpoints have been removed entirely (will return 404):
- `/data/campaigns/all/` → Use `/data/campaigns/any/details/`
- `/data/campaigns/details/` → Use `/data/campaigns/any/details/`
- `/data/campaigns/ids/` → Use `/data/campaigns/any/ids/`
- `/data/campaigns/search/` → Use `/data/campaigns/any/ids/` (was already an alias for `/ids/`, marked obsolete)
- `/data/campaigns/sender/` → **Removed** (use `/any/details/` with `recipients` decorator)
- All `/data/campaigns/all/{id}/...` sub-component endpoints → **Removed**
**Action for consumers:**
- Audit your usage of campaign endpoints.
- If you are using `GET`, you are generally fine (subject to field removals below).
- If you are using `PUT`, `POST` or `DELETE` on any `/data/campaigns/...` URL, you **must migrate away from this pattern**. Use your own write path or a new dedicated campaign service.
- If you are linking banners to campaigns, **remove this functionality** (deprecated).
- If you are linking lead-campaigns to campaigns, **remove this functionality** (deprecated).
- If you are linking organisations to campaigns, **migrate to Campaign Management API**.
- If you are linking countries to campaigns, **remove this functionality** (deprecated).
- If you are using legacy `/all/` endpoints, **migrate to `/any/details/`**.
### 2.2 Organisation and Country Data Changes
#### Organisation Data (Decorator Stays, Linking Removed)
The `organisations` decorator **remains available** in campaign API responses. You can still read organisation data linked to campaigns via the decorator.
**What changed:**
- ✅ **READ** - Organisation data is still returned in campaign responses via the `organisations` decorator
- ❌ **WRITE** - Organisation linking operations (PUT/DELETE on `/organisations/` sub-component) are removed
**For organisation linking operations**, use the **Campaign Management API** instead of ServiceLayer.
#### Country Data (Decorator and Linking Both Removed)
The `countries` decorator is **removed** from campaign API responses. Country data will **no longer appear** in campaign responses.
**What changed:**
- ❌ **READ** - Country data is no longer returned in campaign responses
- ❌ **WRITE** - Country linking operations are removed
Country linking is a deprecated feature with no replacement.
#### Chain Decorator (Removed)
The `chain` decorator is removed because the `chain_id` column is dropped from the database.
#### Banner Decorator and Linking (Removed)
The `banners` decorator and all banner linking functionality have been **completely removed**:
**What changed:**
- ❌ **READ** - Banner data is no longer returned in campaign responses
- ❌ **WRITE** - Banner linking operations (PUT/DELETE on `/banners/` sub-component) are removed
- ❌ **SEARCH** - `banner_ids` and `has_banner` search parameters are removed
The `campaigns_banners` linking table is deprecated. Banner functionality for campaigns should use custom products of type `banners` instead.
#### Lead-Campaign Linking (Removed)
Lead-campaign linking to campaigns has been **completely removed**:
**What changed:**
- ❌ **WRITE** - Lead-campaign linking operations (PUT/DELETE on `/lead-campaigns/` sub-component) are removed
- ❌ **SEARCH** - `lead_campaign_ids` search parameter is removed
The `lead_campaigns_campaigns` linking table is deprecated.
**Summary of decorator changes:**
| Decorator | Data in Response | Linking Operations | Notes |
|-----------|------------------|-------------------|-------|
| `organisations` | ✅ Still available | ❌ Removed | Use Campaign Management API for linking |
| `banners` | ❌ Removed | ❌ Removed | Deprecated feature |
| `countries` | ❌ Removed | ❌ Removed | Deprecated feature |
| `chain` | ❌ Removed | N/A | `chain_id` column dropped |
| `attached_organisations` | ✅ Still available | N/A | Different relationship, unaffected |
### 2.3 Fields removed from API responses
Because the underlying columns are dropped, the following fields will **disappear** from campaign JSON responses (or be permanently `null` and then removed):
- `automated`
- `automated_over_performance`
- `tableau_reporting`
- `is_scheduled`
- `nullIfOld`
- `automation_notes`
- `options` / `campaign_options`
- `external_notes`
- `chain_id`
- `rating`
- `adwords_start`
- `adwords_end`
This affects at least:
- `/data/campaigns/any/details/`
- `/data/campaigns/any/list/`
- `/data/campaigns/any/export/`
- `/data/campaigns/any/google-ads/`
**Action for consumers:**
- Search your code for these keys in JSON payloads from ServiceLayer campaigns.
- Remove references, or replace them with alternative signals if your business logic still requires that information.
- Remove any sorting/filtering logic that expects these fields to exist.
### 2.4 Fields with stricter semantics
The following fields remain in responses but will now be **guaranteed non-null** and follow stricter domain rules:
- `type` – now limited to `rbp`, `nrbp`, `hybrid_ebp`, `full_ebp`, `free_trial`.
- `region` – now limited to `other`, `uk`, `apac`, `emea`, `na`, `strp`.
- `currency` – now limited to `EUR`, `GBP`, `USD`, `AUD`, `CAD`.
- `campaign_type` – now limited to `custom_products`, `listing_products`.
- `start_date`, `end_date` – always present (no `null` values) and valid dates.
- `budget`, `budget_total`, `assistant_id`, `handler_id`, `crm_id`, `custom_products_budget`, `persist_gl_param_ref_link` – always non-null numeric values.
**Action for consumers:**
- If your code assumed these fields might be `null` or empty, you can simplify it. However, you should:
- Validate that your enums/typed models match the new value sets.
- Remove any fallbacks that treat `null` as “unknown” for these fields.
### 2.4 Deletion of non-conforming campaigns
As part of the migration, **all campaign rows that do not satisfy the new constraints will be deleted**.
**Potential impact:**
- If your service expects historical campaigns with incomplete/legacy data, those IDs may cease to exist.
- Calls such as `/data/campaigns/any/details/{id}/` for such campaigns will return 404 (or simply no entry) after migration.
**Action for consumers:**
- Check whether you rely on very old campaigns or "draft" campaigns with partial data.
- Adjust your code to:
- Handle missing campaigns gracefully.
- Not assume that *every* historic ID will always stay resolvable.
---
## 3. Impact on Related Entities and Endpoints
Several other ServiceLayer entities and endpoints depend on data from `campaigns`. Examples include:
- Decorators that attach campaign information to other entities (e.g. organisations, studies).
- Linking tables such as `campaigns_banners`, `campaigns_countries`, `campaigns_products2`, `campaigns_entities_configs`, `lead_campaigns_campaigns`, `campaigns_recipients`.
### 3.1 Search/Filter Parameters Being Removed
If you use any of these query parameters when calling campaign endpoints, they will **stop working**:
| Parameter | Reason |
|-----------|--------|
| `automated` | Column dropped |
| `automated_over_performance` | Column dropped |
| `tableau_reporting` | Column dropped |
| `has_adwords` | `adwords_start`/`adwords_end` columns dropped |
| `ratings` | `rating` column dropped |
| `options` / `no_options` | `options` column dropped |
| `banner_ids` | Banner linking deprecated |
| `has_banner` | Banner linking deprecated |
| `lead_campaign_ids` | Lead campaign linking deprecated |
| `capped_budget` | Budget is now always non-null, filter is meaningless |
### 3.2 Search/Filter Parameters Still Available
These parameters will continue to work:
- `types`, `regions`, `crm_id`, `statuses`, `is_running`
- `organisation_ids`, `attached_organisation_ids`, `study_ids`
- `assistant_ids`, `handler_ids`, `user_ids`
- `exclude_from_campaign_lift`
- `keywords`, `rbp`
Key points:
- Relationships (via linking tables) stay intact **only for campaigns that survive the data cleanup**.
- Decorators and search maps that use dropped fields (`tableau_reporting`, `rating`, `adwords_*`, etc.) will be updated or removed in ServiceLayer.
- Your service may see differences in filtered lists or counts if it relied on these filters indirectly.
**Action for consumers:**
- If you use endpoints that indirectly filter on `tableau_reporting`, `rating`, AdWords-related fields or other soon-to-be-dropped properties, expect those filters to stop having any effect.
- If you use `capped_budget` parameter, remove it - budget is now always non-null so this filter is meaningless.
- Review any business logic that interprets such filter results.
---
## 4. Required Actions for Dependent Services
This section is the checklist you should follow if your service uses ServiceLayer campaigns.
### 4.1 Stop using ServiceLayer for campaign writes
- [ ] Search your codebase for `"/data/campaigns/"` (including `any`, `all`, `details`, `sender`, etc.).
- [ ] Ensure there are **no** `PUT`, `POST` or `DELETE` HTTP calls to campaign entity URLs.
- [ ] If you find any write operations:
- Migrate campaign creation/update/deletion to your own service or a new dedicated API that is not this ServiceLayer campaign entity.
- Remove the old write calls and associated error handling.
### 4.2 Migrate organisation linking / Remove deprecated linking operations
- [ ] **Organisation linking:** If you link organisations to campaigns via ServiceLayer:
- Search for `/organisations/` in campaign endpoint paths (PUT/DELETE operations)
- **Migrate linking operations to Campaign Management API**
- Note: Organisation data is **still available** in responses via the `organisations` decorator (read-only)
- [ ] **Country data:** If you use country data from campaign responses:
- Search for `campaign.countries` in code that processes responses
- **Remove this usage** - the `countries` decorator is removed, data no longer in responses
- [ ] **Country linking:** If you link countries to campaigns via ServiceLayer:
- Search for `/countries/` in campaign endpoint paths (PUT/DELETE operations)
- **Remove this functionality** - country linking is deprecated and no longer supported
- [ ] **Banner linking:** If you link banners to campaigns via ServiceLayer:
- Search for `/banners/` in campaign endpoint paths (PUT/DELETE operations)
- **Remove this functionality** - banner linking is deprecated and completely removed
- The sub-component endpoint `/data/campaigns/any/details/{id}/banners/` no longer exists
- [ ] **Lead-campaign linking:** If you link lead-campaigns to campaigns via ServiceLayer:
- Search for `/lead-campaigns/` in campaign endpoint paths (PUT/DELETE operations)
- **Remove this functionality** - lead-campaign linking is deprecated and completely removed
- The sub-component endpoint `/data/campaigns/any/details/{id}/lead-campaigns/` no longer exists
- [ ] **Legacy sub-component endpoints:** All `/data/campaigns/all/{id}/...` sub-component endpoints are **removed**
### 4.3 Remove use of dropped fields and decorators
- [ ] Search for usage of the following JSON keys in your campaign-related code:
- `automated`
- `automated_over_performance`
- `tableau_reporting`
- `is_scheduled`
- `nullIfOld`
- `automation_notes`
- `options`
- `campaign_options`
- `external_notes`
- `chain_id`
- `rating`
- `adwords_start`
- `adwords_end`
- [ ] Search for usage of removed decorators:
- `chain` (removed - chain_id column dropped)
- `countries` (removed - deprecated feature, data no longer in responses)
- `banners` (removed - deprecated feature, decorator and sub-component removed)
- [ ] Note: `organisations` decorator **still available** (read-only data in responses)
- [ ] Remove or replace them with alternative fields.
- [ ] Remove any UI elements or reports that display these fields directly.
- [ ] **Check for deprecated enum values:**
- `type`: remove handling for `advanced_nrbp`, `rbp_cpl`
- `region`: remove handling for `west`, `bne`, `global`, `act`
### 4.4 Align enums and required fields
- [ ] Ensure your internal models for campaigns:
- Treat `type`, `region`, `currency`, `campaign_type` as **closed enums** with the allowed values listed above.
- Treat `start_date`, `end_date` as required fields.
- Treat `budget`, `budget_total`, `assistant_id`, `handler_id`, `crm_id`, `custom_products_budget`, `persist_gl_param_ref_link` as required numeric fields.
- [ ] Update validation and mapping code so that it:
- Does not expect `null` values for these fields.
- Fails fast on unexpected enum values (in case new values are added later).
### 4.4 Handle missing/removed campaigns
- [ ] Review any logic that assumes a campaign ID from the past will always resolve.
- [ ] Update code to handle cases where:
- A previously-known campaign ID now returns 404 or is absent from results.
- Counts or lists of historic campaigns shrink due to deletion of invalid/legacy rows.
### 4.5 Re-run and update tests
- [ ] Update integration/end-to-end tests that rely on:
- Deprecated fields in API responses.
- Creating/updating/deleting campaigns through ServiceLayer.
- [ ] Add tests that assert:
- Your service does not attempt to write to `/data/campaigns/...` endpoints.
- It can handle missing campaigns gracefully.
---
## 5. ServiceLayer Deployment Phases
The ServiceLayer team will deploy changes in this order:
> **Critical:** Data cleanup MUST happen before code deployment. Campaign decorators are present on many other entities (e.g., Product2, Study). If campaigns with invalid enum values (like deprecated `region` or `type` values) exist when the new code deploys, those endpoints will fail with validation errors.
### Phase 1: Data Cleanup (Database - FIRST)
- Invalid campaign rows deleted (rows with deprecated `type` values like `advanced_nrbp`, `rbp_cpl`)
- Invalid campaign rows deleted (rows with deprecated `region` values like `west`, `bne`, `global`, `act`)
- Rows with NULL values in soon-to-be-required fields deleted
- Linking table entries for deleted campaigns cleaned up
- Deprecated linking tables truncated (`campaigns_banners`, `campaigns_countries`, `lead_campaigns_campaigns`)
- **This ensures all remaining data is valid for the new schema and code**
### Phase 2: Code Deployment (ServiceLayer)
- All code changes deployed in single release:
- Write endpoints (`PUT`, `POST`, `DELETE`) disabled on `/data/campaigns/*`
- Legacy endpoints removed (`/data/campaigns/all/`, `/data/campaigns/details/`, etc.)
- Deprecated search parameters removed
- Deprecated decorators removed (`banners`, `countries`, `chain`, `lead_campaigns`)
- Banner, Lead, LeadCampaign entities completely removed
- Deprecated properties removed from responses
- **Your service should be ready for this before Phase 2 deploys**
### Phase 3: Schema Migration (Database - LAST)
- Columns modified to `NOT NULL`
- Deprecated columns dropped (`automated`, `chain_id`, etc.)
- New enum constraints applied (reduced `type` and `region` values)
- Deprecated tables dropped (17 tables listed in Section 12.5)
---
## 6. Migration Timeline for Dependent Services
A suggested sequence for your service:
1. **Audit & refactor (NOW - before Phase 1):**
- Stop using campaign write endpoints (`PUT`, `POST`, `DELETE`).
- Remove dependencies on soon-to-be-dropped fields.
- Remove usage of deprecated search parameters.
2. **Align models:**
- Update enums to match new allowed values.
- Treat all listed fields as required (non-null).
- Make tests match the new expectations.
3. **Deploy your changes:**
- Deploy before ServiceLayer Phase 1 if possible.
- Otherwise, deploy immediately after Phase 1 (read-only mode).
4. **Verify after Phase 2:**
- Confirm your service handles the new response format.
- Check that missing campaigns are handled gracefully.
The exact calendar dates for each phase will be communicated separately.
---
## 7. Questions to Answer Before You Start
When updating your service, it helps to clarify:
- Do you **currently** create, update or delete campaigns via ServiceLayer?
- Do you rely on any of the fields that are being dropped (`automated`, `automated_over_performance`, `rating`, `tableau_reporting`, `chain_id`, `options`, AdWords fields, etc.)?
- Do you use any of the deprecated search parameters (`automated`, `tableau_reporting`, `has_adwords`, `ratings`, `options`, `no_options`, `automated_over_performance`)?
- Do you depend on very old or partially-filled campaigns that may be deleted as part of the cleanup?
- Do you use the `chain` decorator to get related campaigns?
- Do you use the `countries` decorator to get country data from campaigns? (This data is being removed)
- Do you use the `banners` decorator to get banner data from campaigns? (This data is being removed)
- Do you link banners or lead-campaigns to campaigns via PUT/DELETE on sub-component endpoints? (This functionality is removed)
- Do you link organisations or countries to campaigns via PUT/DELETE on sub-component endpoints?
- Do you have custom enums or models for `type`, `region`, `currency`, or `campaign_type` that might include deprecated values (`advanced_nrbp`, `rbp_cpl`, `west`, `bne`, `global`, `act`)?
- Does your code handle `crm_id` as a string? (It's changing to an integer.)
Having these answers ready will make it easier for an AI assistant (or another developer) in your other repository to propose specific code changes.
---
## 8. Quick Reference: Field Status
### Fields Being REMOVED
```
automated
automated_over_performance
tableau_reporting
is_scheduled
nullIfOld
automation_notes
options / campaign_options
external_notes
chain_id
rating
adwords_start
adwords_end
```
### Decorators Being REMOVED (Data No Longer in Responses)
```
banners (banner linking deprecated - data removed from responses)
chain (chain_id column dropped)
countries (deprecated feature - data removed from responses)
```
### Decorators with LINKING REMOVED (Data Still Available)
```
organisations (read-only data stays in response, linking moved to Campaign Management API)
```
### Sub-Component Endpoints Being REMOVED
```
/data/campaigns/any/details/{id}/banners/ (banner linking deprecated)
/data/campaigns/any/details/{id}/lead-campaigns/ (lead-campaign linking deprecated)
```
### Fields Becoming REQUIRED (non-null)
```
persist_gl_param_ref_link (TINYINT)
budget (INT)
budget_total (INT)
assistant_id (INT)
handler_id (INT)
crm_id (INT - was VARCHAR)
custom_products_budget (INT)
type (ENUM: rbp, nrbp, hybrid_ebp, full_ebp, free_trial)
region (ENUM: other, uk, apac, emea, na, strp)
currency (ENUM: EUR, GBP, USD, AUD, CAD)
campaign_type (ENUM: custom_products, listing_products)
start_date (DATE)
end_date (DATE)
```
### Fields Unchanged
```
id
name
title
exclude_from_campaign_lift
internal_notes
geo_targeted
```
### Search Parameters Being REMOVED
```
automated
automated_over_performance
tableau_reporting
has_adwords
ratings
options
no_options
banner_ids (banner linking deprecated)
has_banner (banner linking deprecated)
lead_campaign_ids (lead-campaign linking deprecated)
capped_budget (budget is now always non-null)
```
---
## 9. Example: Before and After API Response
### Before (current response from `/data/campaigns/any/details/{id}/`)
```json
{
"123": {
"id": 123,
"ego": "/data/campaigns/any/details/123/",
"name": "Example Campaign",
"title": "Example Campaign",
"type": "rbp",
"region": "emea",
"currency": "EUR",
"campaign_type": 1,
"start_date": 1609459200,
"end_date": 1640995200,
"budget": 10000,
"budget_total": 50000,
"assistant_id": 42,
"handler_id": 43,
"crm_id": "AB123",
"custom_products_budget": 5000,
"persist_gl_param_ref_link": 1,
"exclude_from_campaign_lift": 0,
"internal_notes": "<p>Some notes</p>",
"chain_id": 5,
"options": "internal,fixed_end_date",
"external_notes": "<p>External</p>",
"automated": 1,
"automated_over_performance": 10,
"geo_targeted": 0,
"tableau_reporting": 1,
"rating": "A",
"automation_notes": "<p>Auto notes</p>",
"adwords_start": 1609459200,
"adwords_end": 1640995200,
"chain": { ... },
"organisations": [ ... ],
"countries": [ ... ],
"components": { ... }
}
}
```
### After (new response format)
```json
{
"123": {
"id": 123,
"ego": "/data/campaigns/any/details/123/",
"name": "Example Campaign",
"title": "Example Campaign",
"type": "rbp",
"region": "emea",
"currency": "EUR",
"campaign_type": "listing_products",
"start_date": 1609459200,
"end_date": 1640995200,
"budget": 10000,
"budget_total": 50000,
"assistant_id": 42,
"handler_id": 43,
"crm_id": 12345,
"custom_products_budget": 5000,
"persist_gl_param_ref_link": 1,
"exclude_from_campaign_lift": 0,
"internal_notes": "<p>Some notes</p>",
"geo_targeted": 0,
"organisations": [ ... ],
"components": { ... }
}
}
```
**Key differences:**
- `crm_id` is now an integer (was string `"AB123"`, now `12345`)
- `campaign_type` is now a string enum (was integer `1`, now `"listing_products"`)
- All dropped fields are gone (no `chain_id`, `options`, `automated`, etc.)
- Removed decorators: `chain`, `banners`, `countries` no longer in response
---
## 10. Full List of Affected Endpoints
### Main Campaign Endpoints (Now Read-Only)
| Endpoint | Current Methods | After Migration |
|----------|-----------------|-----------------|
| `/data/campaigns/any/details/` | GET, PUT, POST, DELETE | **GET only** |
| `/data/campaigns/any/details/{id}/` | GET, POST, DELETE | **GET only** |
| `/data/campaigns/any/list/` | GET | GET (unchanged) |
| `/data/campaigns/any/ids/` | GET | GET (unchanged) |
| `/data/campaigns/any/export/` | GET | GET (unchanged) |
| `/data/campaigns/any/google-ads/` | GET | GET (unchanged) |
### Sub-Component Endpoints (Still Writable)
These endpoints write to **linking tables**, not the `campaigns` table, so they remain fully functional:
| Endpoint | Methods | Linking Table |
|----------|---------|---------------|
| `/data/campaigns/any/details/{id}/customproducts/` | GET, PUT, DELETE | `campaigns_products2` |
| `/data/campaigns/any/details/{id}/presetproducts/` | GET, PUT, DELETE | `campaigns_products2` |
### Sub-Component Endpoints Being REMOVED (Deprecated Functionality)
The following sub-component endpoints are being **completely removed** as the banner and lead-campaign linking functionality is deprecated:
| Endpoint | Previous Methods | Replacement |
|----------|------------------|-------------|
| `/data/campaigns/any/details/{id}/banners/` | GET, PUT, DELETE | **REMOVED** - No replacement, functionality deprecated |
| `/data/campaigns/any/details/{id}/lead-campaigns/` | GET, PUT, DELETE | **REMOVED** - No replacement, functionality deprecated |
> **Note:** The linking tables (`campaigns_banners`, `lead_campaigns_campaigns`) are no longer writable via ServiceLayer campaigns endpoints. This functionality has been deprecated.
### Legacy Endpoints Being REMOVED
The following legacy endpoints are being **completely removed** from ServiceLayer. You must migrate to the equivalent `/any/` endpoint or remove the functionality if no equivalent exists.
#### Endpoints WITH Equivalents (Must Migrate)
| Legacy Endpoint | Equivalent `/any/` Endpoint | Notes |
|-----------------|----------------------------|-------|
| `/data/campaigns/all/` | `/data/campaigns/any/details/` | Same properties available |
| `/data/campaigns/all/{id}/` | `/data/campaigns/any/details/{id}/` | Same properties available |
| `/data/campaigns/details/` | `/data/campaigns/any/details/` | Same functionality |
| `/data/campaigns/ids/` | `/data/campaigns/any/ids/` | Identical functionality |
| `/data/campaigns/search/` | `/data/campaigns/any/ids/` | Was already an alias for `/ids/`, marked obsolete in original code |
#### Endpoints WITHOUT Equivalents (Must Remove)
| Legacy Endpoint | Reason | Action Required |
|-----------------|--------|-----------------|
| `/data/campaigns/sender/` | Specialized endpoint for campaign sending | **Remove usage** - if you need recipient data, query `/data/campaigns/any/details/{id}/` with `recipients` decorator |
#### Sub-Component Endpoints Migration
| Legacy Endpoint | Equivalent | Notes |
|-----------------|------------|-------|
| `/data/campaigns/all/{id}/banners/` | **REMOVED** | Banner functionality deprecated |
| `/data/campaigns/all/{id}/lead-campaigns/` | **REMOVED** | Lead-campaign functionality deprecated |
| `/data/campaigns/all/{id}/organisations/` | **REMOVED** | Use Campaign Management API for linking |
| `/data/campaigns/all/{id}/countries/` | **REMOVED** | Deprecated feature |
### Decorator Changes Summary
| Decorator | Data in Response | Linking Sub-Component | Notes |
|-----------|------------------|----------------------|-------|
| `organisations` | ✅ Still available | ❌ Removed | Linking moved to Campaign Management API |
| `banners` | ❌ Removed | ❌ Removed | Deprecated feature, entity deleted |
| `countries` | ❌ Removed | ❌ Removed | Deprecated feature |
| `chain` | ❌ Removed | N/A | `chain_id` column dropped |
| `lead_campaigns` | ❌ Removed | ❌ Removed | Deprecated feature, entity deleted |
**Note:** The `attached_organisations` decorator remains available (this is a different relationship used for filtering).
---
## 11. Using `@studyportals/sp-hs-misc` with the New Campaign API
If your service uses `@studyportals/sp-hs-misc` to interact with ServiceLayer campaigns, this section explains what you need to do.
### 11.1 Required Package Updates
Update to the latest versions that support the new campaign API:
```bash
npm install @studyportals/sp-millennium-falcon@^3.0.0
npm install @studyportals/sp-hs-misc@^3.0.5
```
**Important:** Update `sp-millennium-falcon` first, then `sp-hs-misc`.
**Version notes:**
- v3.0.1: Added proper entry point (`main` and `types` in package.json)
- v3.0.5: **Restored browser build** for CDN usage (ES2020 modules)
### 11.2 Breaking Changes in v3.0.0
#### Browser Build Status
The browser build (`bin-browser/`) was **removed in v3.0.0** but **restored in v3.0.5** as ES2020 modules.
**Status by version:**
- **v2.x:** Browser build available (global scripts)
- **v3.0.0 - v3.0.4:** Browser build removed (CDN imports broken)
- **v3.0.5+:** Browser build restored (ES2020 modules)
#### Import Path Changes for Node.js
For Node.js services, imports should use the main entry point:
```typescript
// ❌ OLD — bin-browser no longer exists in v3.0.0+
import { ServiceLayerClient } from '@studyportals/sp-hs-misc/bin-browser/src/adapters/service-layer-client.class';
// ❌ OLD — direct bin/ path (still works but not recommended)
import { ServiceLayerClient } from '@studyportals/sp-hs-misc/bin/src/adapters/service-layer-client.class';
// ✅ NEW — use main entry point (RECOMMENDED for Node.js)
import { ServiceLayerClient } from '@studyportals/sp-hs-misc';
```
#### Browser CDN Usage (v3.0.5+)
For browser usage via CDN (jsDelivr, unpkg), use the restored `/bin-browser/` with ES modules:
```html
<!-- v3.0.0 - v3.0.4: BROKEN -->
<script src="https://cdn.jsdelivr.net/npm/@studyportals/sp-hs-misc@3.0.4/bin/index.js"></script>
<!-- Error: exports is not defined -->
<!-- v3.0.5+: WORKING - ES Modules -->
<script type="module">
import { ServiceLayerClient, BaseSuperAgentRequestFactory }
from 'https://cdn.jsdelivr.net/npm/@studyportals/sp-hs-misc@3.0.5/bin-browser/index.js';
const client = new ServiceLayerClient(
new BaseSuperAgentRequestFactory(),
'https://servicelayer.example.com',
3
);
</script>
```
**Browser requirements:**
- Must use `<script type="module">`
- Modern browsers with ES2020 support
- Direct class imports also supported from `/bin-browser/src/.../*.js`
#### Campaign Model Changes
The `ICampaign` interface and `CampaignDto` class no longer include `externalNotes`:
```typescript
// ❌ OLD — no longer valid
const campaign: ICampaign = {
id: "123",
title: "Example",
externalNotes: "Some notes", // REMOVED
// ...
};
// ✅ NEW — externalNotes removed
const campaign: ICampaign = {
id: "123",
title: "Example",
// ...
};
```
#### Campaign Type Enum Changes
The following `CampaignType` enum values have been removed:
- `CampaignType.ADVANCED_NRBP`
- `CampaignType.RBP_CPL`
**Valid campaign types:**
- `CampaignType.RBP`
- `CampaignType.NRBP`
- `CampaignType.HYBRID_EBP`
- `CampaignType.FULL_EBP`
- `CampaignType.FREE_TRIAL`
### 11.3 ServiceLayerDataModelsFactory
The `createCampaign()` method automatically handles the new API response format:
```typescript
import { ServiceLayerDataModelsFactory } from '@studyportals/sp-hs-misc';
const factory = new ServiceLayerDataModelsFactory();
const campaign = factory.createCampaign(serviceLayerResponse);
// Returns ICampaign with:
// - id, title, startDate, endDate, currency, budget, type
// - NO externalNotes property
```
### 11.4 ServiceLayerClient Usage
The `ServiceLayerClient` is a generic HTTP client. After this migration:
```typescript
import { ServiceLayerClient, BaseSuperAgentRequestFactory } from '@studyportals/sp-hs-misc';
const client = new ServiceLayerClient(
new BaseSuperAgentRequestFactory(),
'https://servicelayer.example.com',
3
);
// ✅ READ operations — still work
const campaigns = await client.get('/data/campaigns/any/details/');
const campaign = await client.getCached('/data/campaigns/any/details/123/');
// ❌ WRITE operations on campaign entity — will FAIL after Phase 1
await client.post('/data/campaigns/any/details/', data); // BLOCKED
await client.put('/data/campaigns/any/details/123/', data); // BLOCKED
await client.delete('/data/campaigns/any/details/123/'); // BLOCKED
// ✅ WRITE operations on custom/preset products — still work
await client.put('/data/campaigns/any/details/123/customproducts/', data);
await client.put('/data/campaigns/any/details/123/presetproducts/', data);
// ❌ WRITE operations on banners/lead-campaigns — REMOVED (deprecated)
await client.put('/data/campaigns/any/details/123/banners/', data); // REMOVED
await client.put('/data/campaigns/any/details/123/lead-campaigns/', data); // REMOVED
```
### 11.5 Action Checklist for Your Service
If your service uses `hs-misc` to interact with campaigns:
- [ ] Update `@studyportals/sp-millennium-falcon` to ^3.0.0
- [ ] Update `@studyportals/sp-hs-misc` to ^3.0.5 (or ^3.0.1 minimum)
- [ ] **For Node.js:** Search for `bin-browser` imports and replace with main entry point
- [ ] **For browsers:** Update to v3.0.5+ and use ES module imports from `/bin-browser/`
- [ ] Remove any references to `externalNotes` on campaign objects
- [ ] Remove handling for `CampaignType.ADVANCED_NRBP` and `CampaignType.RBP_CPL`
- [ ] Search for `ServiceLayerClient` write operations (`post`, `put`, `delete`)
- [ ] Verify none target `/data/campaigns/any/details/` or legacy campaign endpoints
- [ ] If any do, migrate to a dedicated campaign service
### 11.6 Deprecated Classes (Unrelated to This Migration)
These classes in `hs-misc` are deprecated but **not affected** by the campaigns revamp:
| Class | Replacement |
|-------|-------------|
| `CognitoAuthenticationServicesProvider` | `@studyportals/client-internal-platform-sso` |
| `UserSessionCookieManager` | `@studyportals/client-internal-platform-sso` |
| `K2SOUserPrivilegesDataHelper` | `@studyportals/client-internal-platform-authorization` |
| `AWS4RequestSigner` | `@studyportals/mb-platform-http-requests` |
| `SignedRequestSigner` | `@studyportals/mb-platform-http-requests` |
---
## 12. Complete Removal of Deprecated Entities
As part of this migration, the following **entire entities** have been removed from ServiceLayer-Private. These were deprecated features that are no longer supported.
### 12.1 Entities Completely Removed
| Entity | Description | Replacement |
|--------|-------------|-------------|
| `Banner` | Banner display management | **None** - deprecated feature |
| `BannerBox` | Banner box container management | **None** - deprecated feature |
| `Lead` | Lead (enquiry) data storage | **None** - deprecated feature |
| `LeadCampaign` | Lead campaign configuration | **None** - deprecated feature |
| `LeadCampaignStudy` | Study-to-lead-campaign linking | **None** - deprecated feature |
### 12.2 API Routes Completely Removed
The following **entire routes** have been removed from ServiceLayer-Private. All paths under these routes will return 404.
| Route | Description |
|-------|-------------|
| `/data/banners/*` | Banner entity endpoints |
| `/data/banner-boxes/*` | Banner box entity endpoints |
| `/data/leads/*` | Lead entity endpoints |
| `/data/lead-campaigns/*` | Lead campaign entity endpoints |
| `/data/lead-campaign-studies/*` | Lead campaign study entity endpoints |
| `/banner/*` | Banner driver (non-data) |
| `/lead-qualifier/*` | Lead qualifier driver |
| `/publish/lead-notify-*` | Publish driver lead notification routes |
| `/publish/email/lead/*` | Publish driver lead email routes |
### 12.3 Search Parameters Completely Removed
The following search parameters have been removed from **all entities** that previously supported them:
| Parameter | Removed From | Reason |
|-----------|--------------|--------|
| `banner_ids` | Campaigns, Countries, Disciplines | Banner entity removed |
| `has_banner` | Campaigns | Banner entity removed |
| `lead_campaign_ids` | Campaigns, Studies, Products | LeadCampaign entity removed |
| `has_lead_campaign` | Studies | LeadCampaign entity removed |
### 12.4 Decorators Completely Removed
| Decorator | Removed From | Reason |
|-----------|--------------|--------|
| `banners` | Campaign | Banner entity removed |
| `countries` | Campaign | Deprecated feature |
| `lead_campaigns` | Study | LeadCampaign entity removed |
### 12.5 Database Tables To Be Dropped
After the code changes are deployed, the following **17 database tables** will be dropped:
#### Banner Tables (7 tables)
```sql
DROP TABLE banners;
DROP TABLE banners_box;
DROP TABLE banners_boxes;
DROP TABLE banners_countries;
DROP TABLE banners_countries_origin;
DROP TABLE banners_disciplines;
DROP TABLE banners_regions;
DROP TABLE banners_scholarship;
```
#### Campaign Linking Tables (2 tables)
```sql
DROP TABLE campaigns_banners;
DROP TABLE campaigns_countries;
```
#### Lead Tables (8 tables)
```sql
DROP TABLE leads;
DROP TABLE lead_campaigns;
DROP TABLE lead_campaigns_campaigns;
DROP TABLE lead_campaigns_countries_nationality;
DROP TABLE lead_campaigns_countries_residence;
DROP TABLE lead_campaigns_pricings;
DROP TABLE lead_campaigns_studies;
```
### 12.6 Impact on Other Entity Responses
The removal of these entities affects **responses from other endpoints**. This section details what data is no longer returned.
#### Study Entity — `lead_campaigns` Decorator REMOVED
**Affected endpoints:**
- `/data/studies/any/details/`
- `/data/studies/any/details/{id}/`
- `/data/studies/public/details/`
- Any endpoint that previously included the `lead_campaigns` decorator
**Before (old response):**
```json
{
"123": {
"id": 123,
"name": "Computer Science MSc",
"lead_campaigns": [
{
"id": 45,
"name": "CS Campaign 2024",
"active": true
}
]
}
}
```
**After (new response):**
```json
{
"123": {
"id": 123,
"name": "Computer Science MSc"
}
}
```
The `lead_campaigns` property **no longer exists** in study responses.
**Search patterns:**
```
study.lead_campaigns
studies[id].lead_campaigns
response.lead_campaigns
?.lead_campaigns
```
**Action:** Remove all code that reads `lead_campaigns` from study responses.
---
#### Study Entity — `lead_campaign_ids` and `has_lead_campaign` Search Parameters REMOVED
**Affected endpoints:**
- `/data/studies/any/details/?q=lead_campaign_ids-45`
- `/data/studies/public/results/?q=has_lead_campaign-1`
These query parameters **no longer work**. Using them will have no filtering effect.
**Search patterns:**
```
lead_campaign_ids
has_lead_campaign
q=lead_campaign_ids
q=has_lead_campaign
```
**Action:** Remove any filtering logic that uses these parameters.
---
#### Campaign Entity — `banners` Decorator REMOVED
**Affected endpoints:**
- `/data/campaigns/any/details/`
- `/data/campaigns/any/details/{id}/`
**Before (old response):**
```json
{
"456": {
"id": 456,
"name": "Summer Campaign",
"banners": [
{
"id": 78,
"name": "Homepage Banner"
}
]
}
}
```
**After (new response):**
```json
{
"456": {
"id": 456,
"name": "Summer Campaign"
}
}
```
The `banners` property **no longer exists** in campaign responses.
**Search patterns:**
```
campaign.banners
campaigns[id].banners
response.banners
?.banners
```
**Action:** Remove all code that reads `banners` from campaign responses.
---
#### Campaign Entity — `banner_ids` and `has_banner` Search Parameters REMOVED
**Affected endpoints:**
- `/data/campaigns/any/details/?q=banner_ids-78`
- `/data/campaigns/any/details/?q=has_banner-1`
These query parameters **no longer work**.
**Search patterns:**
```
banner_ids
has_banner
q=banner_ids
q=has_banner
```
**Action:** Remove any filtering logic that uses these parameters.
---
#### Country Entity — `banner_ids` Search Parameter REMOVED
**Affected endpoints:**
- `/data/countries/any/details/?q=banner_ids-78`
This query parameter **no longer works**.
**Search patterns:**
```
/data/countries.*banner_ids
countries.*q=banner_ids
```
**Action:** Remove any country filtering logic that uses banner-related parameters.
---
#### Discipline Entity — `banner_ids` Search Parameter REMOVED
**Affected endpoints:**
- `/data/disciplines/any/details/?q=banner_ids-78`
This query parameter **no longer works**.
**Search patterns:**
```
/data/disciplines.*banner_ids
disciplines.*q=banner_ids
```
**Action:** Remove any discipline filtering logic that uses banner-related parameters.
---
#### Product Entity — `lead_campaign_ids` Search Parameter REMOVE