Operations Runtime

Support-facing runtime notes confirmed in current backend code.

• Main app and realtime server both use `express-session` with `connect-redis`.
• Relevant env vars:
• `SESSION_COOKIE_NAME` default: `sessionId`
• `SESSION_SECRET`
• `COOKIE_TTL`
• `COOKIE_DOMAIN`
• `REDIS_SESSION_HOST`
• `REDIS_SESSION_PORT`
• `REDIS_SESSION_PREFIX`
• Session cookie defaults:
• `sameSite: lax`
• `httpOnly: true`
• `secure: true` in production
• Important TTL nuance:
• cookie max-age defaults to `7 days` unless `COOKIE_TTL` overrides it
• Redis session-store TTL is hard-coded to `24 hours`
• Redis clients now log availability transitions for session and queue clients. Repeated error and reconnect logs are quieted for `60` seconds, while unavailable, ready, and disconnected transitions are logged when state changes.
• `/api/init/whoami` also self-heals org selection:
• if the user has no `currentOrganization`, it selects the first org
• if the user has no orgs at all, it creates `My Organization` with `2000` credits

• Main app health endpoint:
• `GET /health`
• returns `status` and `timestamp`
• in non-production, also returns `env`, `session`, and `uptime`
• Realtime server health endpoint:
• `GET /chat/realtime/health`
• returns `status`, `service: realtime`, and `timestamp`

• User WS tokens use `COOKIE_TTL`, defaulting to `7 days`.
• Webchat WS tokens use `COOKIE_TTL` when set, otherwise default to `24 hours`.
• Webchat JWT session tokens use `WEBCHAT_JWT_TTL`, defaulting to `7 days`.

• Generated widget snippets start from `API_DOMAIN`.
• If org webchat settings set `widgetDomain = proxy` and `API_PROXY_DOMAIN` exists, snippets use the proxy domain instead.
• Current snippets load `/widget/alloy-webchat-widget.js` and initialize with:
• `apiKey`
• `eventId`
• `eventId` is routing metadata for the widget entry point. The runtime execution graph still starts from the published workflow or employee workflow tied to that event.

• In development, the backend can proxy local widget assets when `LOCAL_WIDGET_DEV_PROXY_ENABLED` is enabled or when `NODE_ENV=development`.
• Proxy target comes from `LOCAL_WIDGET_DEV_ORIGIN`.
• Proxied paths include `/widget` and widget asset paths under it.
• `/widget/alloy-webchat-widget.js` and HTML responses get `Cache-Control: no-cache, no-store, must-revalidate`.
• Hashed widget assets under `/widget/assets/` get `Cache-Control: public, max-age=31536000, immutable`.
• If the local widget origin is missing or unavailable, the proxy returns an error instead of silently falling through.

• Storage website previews are optional and require both:
• `STORAGE_WEBSITES_BASE_ORIGIN`: the exact dedicated website origin
• `STORAGE_WEBSITES_FRAME_ANCESTORS`: space-separated exact portal origins allowed to embed previews
• In production these values must be HTTPS origins. The website host must use a registrable domain distinct from authenticated application origins such as the app, API, cookie, and allowed CORS origins. Outside production, loopback HTTP origins are accepted.
• If neither website variable is set, serving is disabled. If configuration was attempted but is invalid, the backend logs `Storage website configuration is invalid; website serving is disabled` with the validation reason, disables only this feature, and continues starting the main application.
• Authenticated capability routes create and revoke preview sites. Creation requires organization access and exactly one of an exact `entry_path` ending in case-sensitive `index.html` or a `.html` `selected_path` that resolves to the nearest parent entry.
• Capability lifetime is `3600` seconds. Limits are `10` active site roots per user, `200` per organization, `16` concurrent requests per capability, `300` requests per minute, and `64 MB` of reserved response bytes per minute.
• Each capability uses an opaque 32-hex-label subdomain. Only `GET` and `HEAD` resources from the pinned, symlink-free site root are served; range requests are refused.
• Allowlisted resource families are HTML, CSS, JavaScript, JSON/manifests, text/XML/CSV/TSV, common web images/icons, and WOFF/WOFF2/TTF/OTF fonts. Resource size is bounded by the resolved storage upload-size limit.
• Responses are `no-store`, carry no referrer, are excluded from indexing, disable range support, and use a restrictive Content Security Policy and Permissions Policy. The frontend iframe uses `sandbox="allow-scripts"` without same-origin access.

• Pending realtime session configs are stored in Redis under:
• `${REDIS_PREFIX || 'alloy_dev'}realtime:{sessionId}`
• TTL is `20 minutes`.
• Configs are one-time use and are deleted on first successful retrieval.
• Ally realtime session creation is:
• `POST /chat/realtime/ally/create-session/:organization_id`
• Ally realtime runs seed runtime context with:
• `targetOrganizationId`
• `userId`

• Main app request-body limits:
• JSON: `300mb`
• URL-encoded: `300mb`
• HTTP server timeouts for both app and realtime:
• `requestTimeout = 600000`
• `headersTimeout = 600000`
• `keepAliveTimeout = 5000`
• Storage uploads still enforce `10 MB` per file.
• Chat attachment limits are deployment-configurable:
• `CHAT_MAX_ATTACHMENTS` (default `10`)
• `CHAT_MAX_ATTACHMENT_SIZE_MB` (default `10`)
• `CHAT_MAX_TOTAL_ATTACHMENT_SIZE_MB` (default `30`)
• An unset or invalid value falls back to the default for that variable.
• These limits apply to inbound Slack, Telegram, and MS Teams attachments, the private employee message API, and are returned to the webchat widget through `GET /api/public/webchat/init` as `attachmentLimits`.
• Internal-chat attachments are limited to:
• max `10` files
• max `10 MB` each
• max `30 MB` total
• allowed MIME families include images, PDF, DOCX, SQLite, plain text, Markdown, XML, CSV, and XLSX
• Code execution (`programming.run_code` and `run_code` workflow steps) builds E2B sandboxes from the standard runtime unless `E2B_RUN_CODE_TEMPLATE_IDS` lists approved template IDs. It is a comma-separated allow-list; a template outside the list is refused, never silently swapped for the default. Empty means only the standard runtime is available.
• Backend attachment storage can optionally enable S3 SSE-C encryption through `HETZNER_S3_SSE_C_KEY`.
• When `HETZNER_S3_SSE_C_KEY` is set, new attachment uploads are written as encrypted S3 objects and each attachment row stores `is_encrypted` so reads can send the matching SSE-C headers.
• Existing plaintext attachments stay readable as plaintext until they are explicitly backfilled.

• `src/scripts/attachments/encryptExisting.ts` re-encrypts existing plaintext attachment objects in place and marks their `attachments.is_encrypted` flag.
• The attachment encryption script refuses to run unless `HETZNER_S3_SSE_C_KEY` is configured.
• It supports `--batch-size` and `--dry-run`.
• The script is resumable: it only targets rows where `is_encrypted = false` and can detect objects that were already encrypted on S3 during an interrupted prior run.
• `src/scripts/workflows/redactAttachmentPayloads.ts` backfills attachment-payload stripping and base64 redaction into stored workflow run and workflow step-log rows.
• The workflow redaction script supports `--organization-id`, `--batch-size`, `--min-bytes`, `--target`, `--from`, `--to`, and `--dry-run`.

• The backend uses a shared Pino application logger.
• `LOG_LEVEL` controls the minimum level and defaults to `info` outside tests.
• Production writes newline-delimited JSON to stdout for log aggregation.
• Development writes a colorized pretty console stream and size-rotated JSON files. `LOG_FILE_SIZE` defaults to `20m`, and rotation retains seven files.
• Tests silence the application logger; other non-production environments use the pretty console stream.
• HTTP request logging accepts an incoming `x-request-id` or generates one, returns it in the response, and makes it available to downstream logs together with request-scoped organization and user correlation fields when known.
• Request logging records method, URL, client IP, response status, and level by outcome: `info` below 400, `warn` for 4xx, and `error` for 5xx or request errors.
• Common authorization, cookie, password, token, API-key, and secret fields are redacted as `[REDACTED]`.
• Redis-backed daemons log their queue name and current depth when they start and every `120` seconds. Failure to read a queue depth is logged as an error; the timer does not keep the process alive or block graceful shutdown.
• Mastra agent generation receives the shared Mastra logger.
• The AI logger redacts sensitive content fields such as prompts, messages, tool payloads, request/response bodies, and headers at the top level and one nesting level deep. Provider-error user IDs are also redacted, while application correlation IDs can remain available outside provider error payloads.
• AI provider errors logged during primary-model and fallback-model retries are reduced to compact fields such as name, message, status, retryability, URL, and provider code/message instead of raw provider payloads.

• Both configured Sequelize connections strip NUL (`U+0000`) characters before model persistence because PostgreSQL rejects them in text and JSONB values.
• Sanitization covers changed strings and nested plain arrays or objects during normal saves, bulk creates, upserts, and model bulk updates. Other control characters are preserved.
• Dates, buffers, model instances, and other non-plain objects are left unchanged. Raw `sequelize.query(...)` calls bypass the model hooks.

• Both servers use `http-graceful-shutdown`.
• Shutdown timeout is `450000 ms`.
• Cleanup closes:
• `redisClient`
• `redisQueueClient`
• Sequelize
• `uncaughtException` and `unhandledRejection` are reported to Sentry and the selected system-alert transport.

• `ALERTS_TRANSPORT` selects `telegram`, `slack`, or `both`. An unset value preserves Telegram delivery. An unsupported value logs a configuration error once and falls back to Telegram.
• Alert topics are `errors`, `events`, `warnings`, `feedbacks`, and `general`.
• `both` sends to Telegram and Slack in parallel. Delivery is best effort, and one transport failing does not reject delivery through the other transport.
• Telegram delivery requires a configured bot and destination chat. Optional topic routing is available for events, errors, feedback, and warnings.
• Slack delivery requires a bot with `chat:write`, one configured channel ID for each alert topic, and bot membership in every private destination channel.
• Slack delivery splits oversized content into safe chunks. Delivery diagnostics omit alert bodies, credentials, authorization headers, request bodies, and raw errors.
• Message feedback changes are debounced for two minutes per feedback item. When several changes occur inside that window, the latest stored state is sent. Redis coordinates this across processes when available, with an in-process fallback.
• Tool-output truncation warnings are aggregated into hourly, environment-scoped summaries grouped by organization. Redis-backed windows can be recovered after process interruption; an in-process fallback is used when Redis is unavailable.
• Workflow runs aborted because an AI teammate reached its daily cost limit do not raise Sentry error captures or system error alerts. The channel still receives the temporary-unavailability reply.

• Every organization has a runtime `status` field: `active` or `inactive`.
• Default for new organizations is `active`. Status changes are performed directly in the database.
• `Organization.safeData()` exposes `status` so the frontend can display the current org state.
• Inactive organizations keep their data and allow authenticated app/admin access, but block all external API usage and runtime automation:
• Public API (`/api/public/*`) returns `403` with `{ error: "Organization is inactive", code: "ORGANIZATION_INACTIVE" }` (except `OPTIONS` for CORS).
• Public trigger API and public webchat/trigger entry points are blocked.
• Private API mutating methods (`POST`, `PUT`, `PATCH`, `DELETE`) are blocked.
• Private `GET /api/private/stargate/?system=...` (long-polling for tasks) is blocked.
• Workflow and trigger runs started from the app UI, AI tools, manual scheduler runs, and realtime voice are blocked.
• Inbound webhooks (Slack, MS Teams, Telegram) return `200` with the `ORGANIZATION_INACTIVE` payload instead of `403`, so external platforms do not retry.
• Hosted MCP endpoints are blocked.
• Read-only private `GET` endpoints (storage listing, metadata, downloads) continue to work.

• Session issues:
• verify Redis availability
• compare cookie TTL vs Redis store TTL
• verify cookie domain and allowed origins
• Wrong widget snippet domain:
• check org webchat widget settings plus `API_DOMAIN` and `API_PROXY_DOMAIN`
• Realtime connect failures:
• check the pending Redis key has not expired or been consumed already
• confirm the workflow/voice models still exist
• Missing system alerts:
• verify the selected transport and its destination configuration
• for Slack, verify the bot is a member of every private destination channel and the configured values are channel IDs rather than channel names
• for Telegram, verify the bot, destination chat, optional topic IDs, and bot access to the target chat
• with `both`, check each transport independently because delivery failures are isolated

• The mailer supports two providers, selected by env var:
• `MAILERSEND_API_TOKEN` — uses MailerSend (commercial email API).
• `SMTP_HOST` — uses SMTP (self-hosted or relay).
• When both are set, MailerSend takes priority.
• SMTP env vars: `SMTP_HOST`, `SMTP_PORT`, `SMTP_SECURE`, `SMTP_USER`, `SMTP_PASS`, `SMTP_FROM`.
• Both providers share EJS templates and the same `sendEmail` contract.
• If neither provider is configured, sending throws at runtime.

• This section applies to customer-hosted deployments: BYOC and On-Premises. It covers deployment-time operator tooling, not product UI or normal customer-organization setup in Shared Cloud or Dedicated VPC.
• Alloy operates this tooling for BYOC. The customer deployment operator is responsible for it on On-Premises installations.
• Operator and developer scripts include:
• the backend root installer, which orchestrates component setup, dependency installation, database startup, database creation, and migrations
• per-component environment installers for the backend, frontend, webchat widget, authentication stack, core Docker stack, and optional Google Workspace MCP stack
• `src/scripts/installer/install.ts` for system organization, Ally, defaults, and optional admin bootstrap
• `src/scripts/installer/syncSystemContent.ts` for publishing bundled system documentation and synchronizing Ally instructions
• `src/scripts/export/organization.ts` for creating an organization SQL dump
• `src/scripts/import/organization.ts` for importing a dump in a transaction
• `scripts/check-search-tools.js` for checking configured storage search tools during install validation
• Environment installers preserve existing values, back up existing environment files, derive or generate values where possible, and prompt for remaining values. They can be rerun to fill missing settings.
• After configuration, the backend root installer starts the core, auth-server, and optional MCP Docker Compose stacks. Individual startup failures are warnings rather than fatal installer errors.
• The core Docker installer renders nginx configuration and local TLS certificates from tracked templates. Rendered configuration, generated certificates, and dynamic environment files are ignored by Git.
• The core Docker Compose stack includes a MinIO S3-compatible object store. The storage client supports the local self-signed TLS setup when the deployment explicitly enables insecure TLS for development; this must not be enabled in production.
• The customer-hosted nginx template proxies `/widget/` to the webchat widget service configured by the installer.
• The auth-server stack includes an optional LDAP source blueprint. LDAP remains disabled when its enable setting is empty, and the installer leaves LDAP connection fields empty unless the operator configures the external directory.
• The backend brand name defaults to `Alloy` when its configured value is missing, empty, or whitespace.
• The application bootstrap installer selects the system admin by `INSTALL_ADMIN_EMAIL` when that value is configured. If it is absent, the installer lists existing users and asks for a user ID interactively.
• `INSTALL_SKIP_ADMIN` skips admin selection, and bootstrap also continues without an admin when no users exist yet. The system organization, global Ally, and defaults are still initialized so system content can be published before the first user signs in.
• If the configured email does not match a registered user, admin initialization is skipped rather than selecting another user.
• Bootstrap resolves or creates the system organization and global Ally, stores the global Ally setting, and seeds missing prompt, model, and fallback settings without overwriting existing values.
• The operator-only system-content sync reads its documentation source from `SYSTEM_DOCS_DIR`. When a system organization and source are available, it replaces the shipped `business docs`, `integrations`, and `tech docs` category folders under `Team space`, queues every published file for indexing, and synchronizes the global Ally instructions from the shared installer defaults.
• System-content sync leaves other `Team space` content untouched. It skips cleanly when the system organization or documentation source is unavailable, and it skips an individual documentation category when that category is absent from the bundle.
• Registration deployments can enable invitation-only account creation and single-organization onboarding through the backend registration policy flags documented in `auth-and-session.md`.
• The organization import script wraps inserts in `BEGIN` / `COMMIT`; insert conflicts abort the import and roll back previous inserts.
• These scripts are operational tooling, not product UI. See `deployment.md` for the customer-hosted deployment model and the BYOC versus On-Premises operating distinction.

• The backend production helper can recursively read a configurable AWS Systems Manager Parameter Store path, request decryption for secure values, and render the result as a dotenv file.
• Parameter path segments are normalized into uppercase underscore-separated keys. Empty keys, normalized-key collisions, no returned parameters, or values that cannot be represented faithfully in dotenv stop the run instead of producing a partial or ambiguous environment.
• File output is created with owner-only permissions. If an output file already exists, the helper preserves a backup before replacing it; stdout mode reads and renders parameters without touching a file.
• The managed production deploy runs environment generation after type checking and linting but before the build. It copies the generated file into the staged artifact with owner-only permissions, then removes generated plaintext and backup files from the source checkout immediately after the copy and again during exit cleanup.
• Failure during environment generation leaves the live release untouched because staging and release switching have not started yet.

• `npm run cron:install` renders `cron/crontab` and replaces only the block between the Alloy managed-cron markers in the current user's crontab.
• The installer requires `BACKEND_PATH`.
• `CRON_LOG_DIR` sets the cron output directory. When omitted, it defaults to the current user's `logs/cron` directory; the installer creates the directory when needed.
• The rendered crontab prepends the Node executable directory to `PATH`, followed by `/usr/local/bin:/usr/bin:/bin`.
• The current managed job runs `npm run cron:refresh-ai-provider-tokens` hourly and appends stdout/stderr to `refresh-ai-provider-tokens.log` in the cron log directory.
• Manual changes inside the managed block are overwritten on the next install; entries outside the block are preserved.

• System organizations can execute AI, channel, voice, and speech-to-text paths even when `available_tokens` is zero.
• Usage and calculated cost still produce token-history records for observability.
• A system organization's token balance is not reduced.
• When `ONE_ORGANIZATION_MODE=true`, system-provider-key usage by the shared non-system organization is also recorded without reducing that organization's `available_tokens` balance.
• Outside single-organization mode, non-system organizations using Alloy system provider keys continue to have their balance reduced.
• The flag parser enables this behavior only for the exact string `true`; an unset value or `false` keeps normal non-system balance deduction enabled.

• `PUT /api/admin/settings/:key` validates that `alloy_employee_id` and `onboarding_employee_id` values reference actual global system AI teammates (employees with `is_system: true` and `owned_by_user_id: null`).
• Setting either key to a personal Ally employee returns `400` with `"Setting value must reference a global system AI teammate"`.
• A migration (`20260710120000-repair-global-system-employee-settings.js`) automatically repairs system settings that were pointing to personal Ally employees, reassigning them to the correct global system employees.