@accounter/scraper-app
Version:
Scraper app with Fastify server and React UI
217 lines (189 loc) • 15 kB
Markdown
# @accounter/scraper-app
## 0.0.2
### Patch Changes
- [#3454](https://github.com/Urigo/accounter-fullstack/pull/3454)
[`0b654ef`](https://github.com/Urigo/accounter-fullstack/commit/0b654ef08495e1632ccc75942d6a1a3142924d9f)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency
[`fast-xml-parser@5.7.3` ↗︎](https://www.npmjs.com/package/fast-xml-parser/v/5.7.3) (from
`5.7.2`, in `dependencies`)
- Updated dependency [`msw@2.14.3` ↗︎](https://www.npmjs.com/package/msw/v/2.14.3) (from `2.14.2`,
in `dependencies`)
- Updated dependency [`react@19.2.6` ↗︎](https://www.npmjs.com/package/react/v/19.2.6) (from
`19.2.5`, in `dependencies`)
- Updated dependency [`react-dom@19.2.6` ↗︎](https://www.npmjs.com/package/react-dom/v/19.2.6)
(from `19.2.5`, in `dependencies`)
- [#3465](https://github.com/Urigo/accounter-fullstack/pull/3465)
[`89e36f3`](https://github.com/Urigo/accounter-fullstack/commit/89e36f3f76ac3a1c315bd305d55c0f1704efe8ac)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`msw@2.14.2` ↗︎](https://www.npmjs.com/package/msw/v/2.14.2) (from `2.13.6`,
in `dependencies`)
- [#3481](https://github.com/Urigo/accounter-fullstack/pull/3481)
[`9fcf835`](https://github.com/Urigo/accounter-fullstack/commit/9fcf8357867e437f39c33fc7c056cdb3f749a5fc)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`zod@4.4.3` ↗︎](https://www.npmjs.com/package/zod/v/4.4.3) (from `4.3.6`, in
`dependencies`)
- [#3503](https://github.com/Urigo/accounter-fullstack/pull/3503)
[`dab1752`](https://github.com/Urigo/accounter-fullstack/commit/dab17529ee68305203d59d8a195763191baea3d5)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`msw@2.14.5` ↗︎](https://www.npmjs.com/package/msw/v/2.14.5) (from `2.14.3`,
in `dependencies`)
- [#3506](https://github.com/Urigo/accounter-fullstack/pull/3506)
[`b68dc3a`](https://github.com/Urigo/accounter-fullstack/commit/b68dc3ab5b3653de616077734ffa6ae9401d4307)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`msw@2.14.5` ↗︎](https://www.npmjs.com/package/msw/v/2.14.5) (from `2.14.3`,
in `dependencies`)
- [#3517](https://github.com/Urigo/accounter-fullstack/pull/3517)
[`771084b`](https://github.com/Urigo/accounter-fullstack/commit/771084ba04be2aa0cf42904ab5591534b60af680)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`msw@2.14.6` ↗︎](https://www.npmjs.com/package/msw/v/2.14.6) (from `2.14.5`,
in `dependencies`)
- [#3524](https://github.com/Urigo/accounter-fullstack/pull/3524)
[`dda0668`](https://github.com/Urigo/accounter-fullstack/commit/dda066884da59c5b3a1b962a1d8e1364972e965d)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency
[`fast-xml-parser@5.8.0` ↗︎](https://www.npmjs.com/package/fast-xml-parser/v/5.8.0) (from
`5.7.3`, in `dependencies`)
- [#3534](https://github.com/Urigo/accounter-fullstack/pull/3534)
[`54c0e4b`](https://github.com/Urigo/accounter-fullstack/commit/54c0e4b840e1efea5744647745a1b031cfa0f5de)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`ws@8.20.1` ↗︎](https://www.npmjs.com/package/ws/v/8.20.1) (from `8.20.0`, in
`dependencies`)
- [#3548](https://github.com/Urigo/accounter-fullstack/pull/3548)
[`c6555f5`](https://github.com/Urigo/accounter-fullstack/commit/c6555f5d6a731b05f26ae58a47c485a905e4ec55)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`date-fns@4.2.1` ↗︎](https://www.npmjs.com/package/date-fns/v/4.2.1) (from
`4.1.0`, in `dependencies`)
- [#3551](https://github.com/Urigo/accounter-fullstack/pull/3551)
[`9d9ce57`](https://github.com/Urigo/accounter-fullstack/commit/9d9ce57f4058c0d66b8c1ea383030a57692fe607)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`date-fns@4.2.1` ↗︎](https://www.npmjs.com/package/date-fns/v/4.2.1) (from
`4.1.0`, in `dependencies`)
- [#3559](https://github.com/Urigo/accounter-fullstack/pull/3559)
[`43b3575`](https://github.com/Urigo/accounter-fullstack/commit/43b3575f981c01e3caafc7170cb30358ed227c3b)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`date-fns@4.3.0` ↗︎](https://www.npmjs.com/package/date-fns/v/4.3.0) (from
`4.2.1`, in `dependencies`)
- [#3561](https://github.com/Urigo/accounter-fullstack/pull/3561)
[`4298298`](https://github.com/Urigo/accounter-fullstack/commit/429829867e1f9c70619c6a18d8bd9dee8cab1ea4)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`ws@8.21.0` ↗︎](https://www.npmjs.com/package/ws/v/8.21.0) (from `8.20.1`, in
`dependencies`)
- [#3603](https://github.com/Urigo/accounter-fullstack/pull/3603)
[`cca7787`](https://github.com/Urigo/accounter-fullstack/commit/cca778797ff2252e52bf9dcf0777e2d385ac68d3)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`date-fns@4.4.0` ↗︎](https://www.npmjs.com/package/date-fns/v/4.4.0) (from
`4.3.0`, in `dependencies`)
- [#3613](https://github.com/Urigo/accounter-fullstack/pull/3613)
[`7397aad`](https://github.com/Urigo/accounter-fullstack/commit/7397aad6a9bf7d120218e6a8274a539eb1e76160)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency [`react@19.2.7` ↗︎](https://www.npmjs.com/package/react/v/19.2.7) (from
`19.2.6`, in `dependencies`)
- Updated dependency [`react-dom@19.2.7` ↗︎](https://www.npmjs.com/package/react-dom/v/19.2.7)
(from `19.2.6`, in `dependencies`)
- [#3687](https://github.com/Urigo/accounter-fullstack/pull/3687)
[`2130c0d`](https://github.com/Urigo/accounter-fullstack/commit/2130c0d38bd8f53b16a8d95a1eb87d363d6644ab)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency
[`fast-xml-parser@5.9.0` ↗︎](https://www.npmjs.com/package/fast-xml-parser/v/5.9.0) (from
`5.8.0`, in `dependencies`)
- [#3720](https://github.com/Urigo/accounter-fullstack/pull/3720)
[`452d0ed`](https://github.com/Urigo/accounter-fullstack/commit/452d0ede302ac26f9aad5d76afc53e72d6d99977)
Thanks [@renovate](https://github.com/apps/renovate)! - dependencies updates:
- Updated dependency
[`fast-xml-parser@5.9.3` ↗︎](https://www.npmjs.com/package/fast-xml-parser/v/5.9.3) (from
`5.9.0`, in `dependencies`)
- [#3554](https://github.com/Urigo/accounter-fullstack/pull/3554)
[`5d1b2a3`](https://github.com/Urigo/accounter-fullstack/commit/5d1b2a373dd76f12186fff325d57d80bfd286fec)
Thanks [@gilgardosh](https://github.com/gilgardosh)! - - **New Scraper Implementation**: Added a
comprehensive scraper for Otsar Hahayal, supporting ILS, foreign currency, and credit card
transactions.
- **Infrastructure Integration**: Integrated the new scraper into the existing system, including
GraphQL mutation updates and UI configuration support.
- [#3463](https://github.com/Urigo/accounter-fullstack/pull/3463)
[`2d320ea`](https://github.com/Urigo/accounter-fullstack/commit/2d320ea2340a57b9bac1b8f597a2a16aff61cf5b)
Thanks [@gilgardosh](https://github.com/gilgardosh)! - - **Granular Progress Events**: Replaced
legacy progress messages with 10 precisely-typed WebSocket events, providing real-time visibility
into per-month and per-account scraping progress.
- **Non-Fatal Scrape Loops**: Updated Isracard, Amex, Discount, and Cal scrapers to use a
non-fatal month-by-month loop, allowing tasks to continue even if a specific month fails.
- **UI Progress Tracking**: Added a new TaskSteps sub-component to the UI, enabling users to
monitor granular progress via collapsible sub-rows for months or accounts.
- **Poalim Account Discovery**: Implemented per-account and per-transaction-type progress
reporting for Poalim, including vault status checks.
- [#3515](https://github.com/Urigo/accounter-fullstack/pull/3515)
[`0c2b8d1`](https://github.com/Urigo/accounter-fullstack/commit/0c2b8d1076b5ca1a478d21f98f64d9db723358a7)
Thanks [@gilgardosh](https://github.com/gilgardosh)! - - **New Vault API Endpoints**: Added GET
/api/vault/env-path, GET /api/vault/download, and POST /api/vault/upload to support vault file
management, including atomic writes and binary streaming.
- **Vault Path Configuration**: Introduced a configurable vaultPath in SettingsSchema, allowing
users to specify and update the storage location of their vault file.
- **Atomic Vault Migration**: Implemented moveVaultFile in the server store to atomically relocate
the vault file on disk when the vaultPath setting is updated.
- **UI Integration**: Added file upload functionality to both the setup and unlock screens, and
enabled vault downloading and path editing in the settings tab.
- [#3474](https://github.com/Urigo/accounter-fullstack/pull/3474)
[`1e70928`](https://github.com/Urigo/accounter-fullstack/commit/1e709282f39c9f722bb3ce5289a8edcda826012c)
Thanks [@gilgardosh](https://github.com/gilgardosh)! - - Emit detailed currency rates and Max
scrapers task status
- For Max accounts, the websocket logic was updated to process accounts individually
- [#3459](https://github.com/Urigo/accounter-fullstack/pull/3459)
[`6bd3b40`](https://github.com/Urigo/accounter-fullstack/commit/6bd3b40e1d128429e2fdcf8c06a16af2fbabe3a4)
Thanks [@gilgardosh](https://github.com/gilgardosh)! - - **Scraper App Package**: Added the new
`@accounter-helper/scraper-app` package to the monorepo, including its build configuration and
initial server/UI structure.
- **Vault Configuration**: Updated the codegen configuration to include the new scraper app's
GraphQL mutations and added a comprehensive implementation blueprint for the refactor.
- [#3627](https://github.com/Urigo/accounter-fullstack/pull/3627)
[`4f0c31c`](https://github.com/Urigo/accounter-fullstack/commit/4f0c31c03761d0f915ddbdd5104bcaec0a042dfe)
Thanks [@gilgardosh](https://github.com/gilgardosh)! - - Updated `parseForeignXls` to support
`subAccount` identification, ensuring foreign transactions are correctly associated with their
respective sub-accounts.
- Improved browser launch configuration to explicitly check for `headless` mode.
- Updated types to include `subAccount` in `ForeignAccountMetadata`.
- **Database Migrations:**
- Added a new migration to flag fees in Otsar Hahayal transactions (ILS, foreign, and credit
card) based on description and name patterns.
- Updated the migration runner to include the new fee-flagging migration.
- [#3466](https://github.com/Urigo/accounter-fullstack/pull/3466)
[`249ad87`](https://github.com/Urigo/accounter-fullstack/commit/249ad87809e104c6c0ffc02971f3cee5dac478b1)
Thanks [@gilgardosh](https://github.com/gilgardosh)! - - **Scraper Enhancements**: Updated
currency rate and Poalim scrapers to accept specific date ranges, improving flexibility and data
accuracy.
- **Transaction Ingestion Logic**: Implemented advanced transaction diffing and change detection
for Isracard, Amex, and currency rates to identify updates to existing records rather than just
inserting new ones.
- **Data Type Normalization**: Standardized card identifiers to strings across the ingestion
pipeline and updated GraphQL schemas to reflect these changes.
- [#3497](https://github.com/Urigo/accounter-fullstack/pull/3497)
[`ce62b25`](https://github.com/Urigo/accounter-fullstack/commit/ce62b2576b1bfc86344fe40e272f58cd3b45d956)
Thanks [@gilgardosh](https://github.com/gilgardosh)! - Enhance OTP dialog: enable manual closing,
auto-close on done
- Updated dependencies
[[`b4d8625`](https://github.com/Urigo/accounter-fullstack/commit/b4d8625fcdec44818ffc81f363eab652074a2fa6),
[`82f07c2`](https://github.com/Urigo/accounter-fullstack/commit/82f07c25cd88fbcef8d29ef3d32230cfce168caa),
[`a5e6d19`](https://github.com/Urigo/accounter-fullstack/commit/a5e6d19bf96173dde9097985a8df0065b9c3e499),
[`a92aeb2`](https://github.com/Urigo/accounter-fullstack/commit/a92aeb20551558fc2e32d9ac427e47eab52ddb52),
[`f94b2da`](https://github.com/Urigo/accounter-fullstack/commit/f94b2dadb2431f5a386217d716b72c15cff0ddd0),
[`46f9fc5`](https://github.com/Urigo/accounter-fullstack/commit/46f9fc5592590a765770592efab440a0374e9bde),
[`ba0531a`](https://github.com/Urigo/accounter-fullstack/commit/ba0531acbe4936cca371e553cfe7120329acec25),
[`1ad8a10`](https://github.com/Urigo/accounter-fullstack/commit/1ad8a10f3dd92b38c04ccea15b90cc7f9dc43887),
[`0b654ef`](https://github.com/Urigo/accounter-fullstack/commit/0b654ef08495e1632ccc75942d6a1a3142924d9f),
[`e76cee0`](https://github.com/Urigo/accounter-fullstack/commit/e76cee0916c6a9f6cc445d228642bc3caf767f86),
[`771084b`](https://github.com/Urigo/accounter-fullstack/commit/771084ba04be2aa0cf42904ab5591534b60af680),
[`b7634ad`](https://github.com/Urigo/accounter-fullstack/commit/b7634ad71fa879662cbcbc2ccca796016dc3f8e1),
[`c6555f5`](https://github.com/Urigo/accounter-fullstack/commit/c6555f5d6a731b05f26ae58a47c485a905e4ec55),
[`9d9ce57`](https://github.com/Urigo/accounter-fullstack/commit/9d9ce57f4058c0d66b8c1ea383030a57692fe607),
[`ae5aa31`](https://github.com/Urigo/accounter-fullstack/commit/ae5aa311e4beb3d5b9ded7a0c0fadaf00b796e2e),
[`5d1b2a3`](https://github.com/Urigo/accounter-fullstack/commit/5d1b2a373dd76f12186fff325d57d80bfd286fec),
[`43b3575`](https://github.com/Urigo/accounter-fullstack/commit/43b3575f981c01e3caafc7170cb30358ed227c3b),
[`5d9a79a`](https://github.com/Urigo/accounter-fullstack/commit/5d9a79aab5c5b75cf09c2dcc0df7f76513a05ba9),
[`f7a8d24`](https://github.com/Urigo/accounter-fullstack/commit/f7a8d244660fb59627da16b90ef2263fa8aa7718),
[`cca7787`](https://github.com/Urigo/accounter-fullstack/commit/cca778797ff2252e52bf9dcf0777e2d385ac68d3),
[`7397aad`](https://github.com/Urigo/accounter-fullstack/commit/7397aad6a9bf7d120218e6a8274a539eb1e76160),
[`fb74946`](https://github.com/Urigo/accounter-fullstack/commit/fb7494648b7e0a73d72e802c4dbff791e3583767),
[`452d0ed`](https://github.com/Urigo/accounter-fullstack/commit/452d0ede302ac26f9aad5d76afc53e72d6d99977),
[`64119ef`](https://github.com/Urigo/accounter-fullstack/commit/64119ef336f825399c7cf46bdc33e3f7ae1009dd),
[`5d1b2a3`](https://github.com/Urigo/accounter-fullstack/commit/5d1b2a373dd76f12186fff325d57d80bfd286fec),
[`4f0c31c`](https://github.com/Urigo/accounter-fullstack/commit/4f0c31c03761d0f915ddbdd5104bcaec0a042dfe)]:
- @accounter/modern-poalim-scraper@0.10.6