AI Advisor Testing & Troubleshooting

🧪 AI Advisor Testing & Troubleshooting

🧪 Testing Strategy

Run commands with the Node version pinned in the root package.json engines (>=24.16.0); an older shell default breaks node:sqlite and the Husky hook. Typecheck every touched package as well as running tests: vitest has stayed green while tsc and vue-tsc caught real errors. See Development & Testing.

# Whole repository (Turborepo, concurrency 1)
pnpm typecheck && pnpm test

# Per package
pnpm --filter @kryptofolio/backend typecheck && pnpm --filter @kryptofolio/backend test
pnpm --filter @kryptofolio/frontend typecheck && pnpm --filter @kryptofolio/frontend test
pnpm --filter @kryptofolio/shared-types test
pnpm --filter @kryptofolio/core-domain test
pnpm --filter @kryptofolio/database test

# One spec file
pnpm --filter @kryptofolio/backend exec vitest run src/core/infrastructure/ai/__tests__/mastraChunk.spec.ts

The frontend typecheck script is vue-tsc --build --force; a bare --noEmit on this solution-style tsconfig checks zero files.

Spec (relative to package)	Proves
`shared-types/tests/schemas/advisor-stream.spec.ts`	Frame schemas, strict `failed` union, no `transport-lost`.
`shared-types/tests/schemas/advisor-model-chain.spec.ts`	Local versus metered entry shapes, cloud-suffix rule, `classifyExecutionProfile`.
`shared-types/tests/schemas/advisor-local-budgets.spec.ts`	Local budget formulas, defaults, ceilings.
`core-domain/src/__tests__/{holdingRanking,displayOrdering,downsampleSeries}.spec.ts`	Display-only helpers, `Money`-based comparison.
`database/tests/integration/migration_009_.spec.ts`, `migration_010_.spec.ts`	Table shape, constraints, nullable profile.
`backend/.../domain/__tests__/advisor-domain-isolation.spec.ts`, `models/__tests__/advisor-event.spec.ts`	Domain imports nothing external; event union.
`backend/.../use-cases/__tests__/AskAdvisorUC.spec.ts`	Context resolution, receipt persistence on each terminal event, aborted row.
`backend/.../adapters/__tests__/SqliteAdvisorRunLogAdapter.spec.ts`	Upsert, NULL columns, refusal reason never written.
`backend/.../ai/__tests__/MastraAdvisorAdapter.spec.ts`, `mastraChunk.spec.ts`, `classifyProviderError.spec.ts`	Chunk-to-event mapping, receipts, failure classification, sanitisation.
`backend/.../ai/__tests__/mastraImportZone.spec.ts`, `noTelemetry.spec.ts`	`@mastra/*` import zone; no telemetry dependency.
`backend/.../ai/models/__tests__/*`	Credential filtering, chain resolution, profile derivation, budgets.
`backend/.../ai/tools/__tests__/*`	Each tool projection and budget; `toolSurface.spec.ts` (25 tools, tier counts of 14 core and 11 extended, exact exposure per execution profile, read-only, thin wrappers); `noModelAccountId.spec.ts`; `enforceBudget.spec.ts`.
`backend/.../ai/guardrails/__tests__/`, `agents/__tests__/`	Grounding, disclaimer, topology, two-call delegation, tool-call filtering.
`backend/.../routes/__tests__/advisor.test.ts`, `advisorMounting.spec.ts`, `advisorRollback.spec.ts`, `app-type.spec-d.ts`, `credentials.test.ts`	HTTP contracts and 400s, exactly six routes, rollback with no chain, `AppType` typing, AI providers in the vault registry.
`backend/.../di/__tests__/*`, `dtos/__tests__/advisor.spec.ts`	Lazy composition, disposability, wire mapping.
`frontend/.../adapters/__tests__/RestAdvisorAdapter.*.spec.ts`	SSE parsing (CRLF, split chunks), invalid frames, config calls.
`frontend/.../composables/__tests__/useAdvisorChat.spec.ts`, `useAdvisorQueries.spec.ts`	State machine, retry rules, abort, thread handling.
`frontend/.../components/advisor/__tests__/*`	Panel shell, states, tools, disclaimer, steps, refusal chip, markdown safety, design tokens.
`frontend/.../Settings/components/advisor/__tests__/*`	Chain editor, limits, derived budgets, wiring.

Per project rules, new behaviour needs a failing test first, and a deliberate break confirmed to turn the intended assertion red. The change’s tasks.md records those break checks.

🩺 Troubleshooting and FAQ

The panel says “No model available”. ai_advisor_model_chain is unset or invalid, or every entry was filtered out with the vault unlocked. Add an entry in Settings → AI advisor. With no chain configured both ask routes answer NO_MODEL_AVAILABLE and nothing else in the app changes.

The panel says the vault is locked. Every keyed entry was skipped because the vault is locked; unlock it in Settings → Vault. A backend restart (including tsx watch reloads in development) re-locks it. A chain that contains a local ollama entry still answers while locked.

“No provider answered” with a named provider. Read cause.kind: auth-rejected means the key was rejected (use an API key created in the provider’s account); model-not-found means the model id is wrong for that provider; rate-limited, provider-unavailable, and network are temporary and offer Retry. The server log has one warning per failed run with the HTTP status.

A local model answers badly or truncates. Check the declared contextWindow. Budgets scale with it (15 percent per tool, 60 percent per run). A window larger than the model really supports produces oversized prompts; a much smaller one produces frequent truncated results.

A cloud-suffixed Ollama model is labelled “cloud” although I use the local daemon. Correct: the daemon proxies it to ollama.com, so data leaves the machine. Use a non-suffixed model for local-only operation.

Answers show a “Figures incomplete” badge. A tool reported missing exchange rates or prices (ratesIncomplete or pricesIncomplete). Fix the underlying data rather than the prompt.

An answer was refused. Either the text matched the tax-evasion pattern (always refused), attempted to override the disclaimer, or made an investment claim with no tool activity in the turn. Rephrase using your own data; the chip offers this.

Can the advisor filter by account? Yes, by name, through account_holdings. The model passes an account name, never an id (a model that sends accountId still fails as INVALID_TOOL_INPUT). The server matches the name exactly and case-insensitively first, and tries a prefix only if no name matches exactly. A prefix shared by several accounts returns ambiguous with the candidate names, and an unknown name returns not_found with the top-level account names; the agent is instructed to ask which one you meant. The result is one section per account in the matched account’s tree and it never sums across accounts.

A local run does not offer the extended tools. By design: a local execution profile exposes only the 14 core tools, because small local models choose worse from a long list. Use a metered or mixed chain to get all 25. See Tool Catalogue.

account_holdings answers ambiguous or not_found. Give a longer or exact account name. Nothing was guessed or summed.

A scenario tool answers “not held”, “unvalued”, or “unconvertible cost basis”. These are real outcomes, not errors: not_held, unvalued, empty_portfolio (scenario_position_value, scenario_portfolio_shock), unconvertible_cost_basis and zero_quantity (breakeven_price). The agent must state them as they are and never estimate a replacement figure; fix the underlying price, rate, or cost-basis data instead.

performance_history shows points with a null value. A point with no exchange rate on or before its date comes back with portfolioValue: null, never an EUR value or 0.

A new tool answers truncated. Its per-tool metered budget is too small for that payload. Raise it in Settings, or narrow the request (for example a shorter date range for tx_search). Stored profiles from before the tool existed use the default budget. See Providers & Models.

Does anything leave my machine? Only the model call. See Audit & Privacy.

How do I erase conversations? Stop the backend and delete ai-advisor.db and its -wal, -shm, or -journal files. The audit table is unaffected and holds no content.

A run shows aborted in the audit table. The client disconnected or pressed Stop before a terminal event. Such a row has no provider, tokens, or steps by design.

Stored execution profiles break the advisor with INTERNAL_ERROR. See the warning in API & Streaming.

🚧 Known Divergences

Places where the shipped code differs from, or goes beyond, the OpenSpec design.

Topic	Design says	Code does
Cloud model suffix	D16 classifies an `ollama` entry as cloud when the model id ends `:cloud`.	`isOllamaCloudModelId` also treats `-cloud` (for example `gpt-oss:120b-cloud`) as cloud, and `readModelChain` drops a stale `contextWindow` from such stored entries.
Migration `010`	Loosens a constraint.	Implemented as `DROP TABLE` plus recreate, destroying existing audit rows. Safe only under the documented clean-slate assumption.
Corrupt execution-profile setting	The chain degrades gracefully.	The profile setting has no equivalent guard; a bad value yields HTTP 500 or `INTERNAL_ERROR`.
Task 16.7	Local commit.	Unchecked at the time of the audit; it is checked in the archived change, which is now merged into `openspec/specs/`.
Spot-transaction order	The spot-transaction search orders results newest first.	`tx_search` pages are ordered newest first with an `id_hash` tiebreak. The `all` arm used by `GET /tax/transactions/spot` keeps ledger order (oldest first, from `ORDER BY t.timestamp ASC` in `SQLiteLedgerAdapter`), so that route’s response is unchanged.

Verified consistent with the design: twenty-five tool names in ADVISOR_TOOL_NAMES, the six provider ids, the 009/010 column set, the done frame fields, setting keys, and the absence of the forecast keys from the code.