Observability, operational tooling, client relationships, and growth¶

Purpose¶

AndreaWeb runs on a self-managed VPS rather than a platform that supplies application monitoring automatically. This document defines how to make the system understandable and maintainable, and records later ideas for professional outreach, customer relationships, and loyalty.

The objective is not to reproduce every AWS or Azure service. It is to collect enough trustworthy signals to answer practical questions:

Is the public site working?
Which service is slow, failing, or consuming unexpected resources?
Is memory growing over time, is the CPU being saturated, or is the VPS running out of disk?
Are database queries, Redis, object storage, background jobs, or Nginx causing a problem?
Did a deployment, migration, or traffic change introduce the problem?
Can an authorized maintainer generate a report or repair data safely and auditably?
What business outcomes are occurring without exposing personal customer data in telemetry?

Three distinct data planes¶

Keep these concerns separate even when they share dashboards:

Operational telemetry: metrics, logs, traces, profiles, health checks, deployment events, and alerts used to operate the software.
Audience analytics: consent-controlled page and interaction measurement such as GA4, described in the publishing and advertising plan.
Business records: products, inventory, inquiries, orders, customers, payments, professional leads, and consent records stored in their authoritative applications.

Google Analytics is not an error tracker, infrastructure monitor, customer database, or accounting source. Operational logs must not become a shadow customer database.

Observability principles¶

Use OpenTelemetry conventions for new application instrumentation so telemetry can move between backends without rewriting business code.
Start with low-cost, actionable signals and add storage-heavy components only after measuring VPS capacity.
Every dashboard must answer an operational question; every alert must imply a response.
Correlate requests with generated request/trace IDs, never with raw email addresses, telephone numbers, payment identifiers, or message contents.
Keep observability interfaces private. Grafana, metrics endpoints, exporters, database consoles, and profiling endpoints are not public website features.
Pin container versions, configure retention, cap resource usage, and monitor the monitoring stack itself.
Preserve deployment commit/version identifiers in logs and metrics so regressions can be tied to releases.

Staged observability architecture¶

Stage A — launch baseline¶

Use capabilities already available on Ubuntu, Docker, Nginx, Node, PostgreSQL, Redis, Garage, and Medusa before adding a large monitoring stack:

Docker and application health checks;
structured application logs to standard output;
Nginx access and error logs with request ID, host, route, status, latency, upstream latency, and response size;
Docker log rotation and disk-usage controls;
docker stats, docker compose stats, docker inspect, and bounded log commands in runbooks;
host memory, CPU, load, filesystem, inode, network, swap, OOM, and service status commands;
PostgreSQL connection/activity, database size, locks, replication/backup status, and slow-query inspection;
Redis memory, connection, eviction, persistence, and command statistics;
Garage health, disk use, and backup status;
external HTTPS uptime checks for portfolio, storefront, Medusa health, and certificate expiration;
a deployment marker recording time, commit, migration result, and smoke-test result.

This baseline must exist before public launch even if Grafana is deferred.

Stage B — metrics, dashboards, and centralized logs¶

After the VPS resource audit, deploy a private, resource-capped observability Compose project. The candidate stack is:

node_exporter / cAdvisor / service exporters / application metrics
                         |
                         v
               Prometheus-compatible metrics
                         |
                         v
                      Grafana

Docker, Nginx, system, and application logs
                         |
                         v
                  Grafana Alloy -> Loki

Grafana Alloy is the candidate collector for Docker/system logs, Prometheus-style metrics, and later OpenTelemetry data. Prometheus stores time-series metrics, Loki stores indexed logs, and Grafana provides dashboards and alert views.

The final selection depends on available VPS memory, CPU, and disk. For a small site, use short retention and modest scrape intervals; do not consume more resources observing the store than serving it. A hosted OpenTelemetry-compatible backend remains a future option if self-hosted storage becomes operationally expensive.

Stage C — traces and profiling¶

Add distributed traces only when request-level diagnosis across Nginx, storefront, Medusa, PostgreSQL, Redis, and payment providers justifies the overhead.

Instrument Node services with OpenTelemetry and export through an OpenTelemetry Collector or Alloy using OTLP.
Add trace context to structured logs so a failing request can be followed across services.
Sample normal traffic and retain errors/slow requests more aggressively without capturing request bodies or personal data.
Evaluate Tempo or another OpenTelemetry-compatible trace backend.
Add continuous profiling only after metrics identify a CPU or memory problem that profiles can answer.

Heap snapshots and CPU profiles can contain sensitive application data and can pause or heavily load a process. Capture them deliberately, store them securely, and never expose profiling controls publicly.

Required dashboards and signals¶

VPS and containers¶

host CPU utilization, load, steal time, memory, swap, filesystem use, inode use, disk latency, and network traffic;
per-container CPU, resident memory, memory limit, restart count, network I/O, block I/O, health, and uptime;
OOM kills, crash loops, unhealthy services, and unexpectedly growing logs or volumes;
Nginx requests, latency percentiles, active connections, response sizes, and 4xx/5xx/upstream failures;
TLS certificate expiration and DNS/HTTP reachability.

Node, Vinext/Next.js, Astro, and Medusa¶

process resident memory, Node heap used/total, external memory, event-loop delay, garbage-collection duration, CPU, and open handles where supported;
request rate, error rate, and latency by bounded route template rather than unbounded raw URL;
background-job success/failure/duration and queue backlog;
Medusa workflow failures, API errors, Admin authentication failures, and health status;
build/deployment version and process start time.

To investigate a suspected memory leak, compare resident memory, heap use after garbage collection, request volume, and uptime over hours or days. Alert on sustained growth and proximity to the container limit, not a single spike. Reproduce under controlled load before taking a heap snapshot.

PostgreSQL¶

availability, connections and saturation;
transaction rate, long-running transactions, deadlocks, locks, and rollbacks;
cache hit ratio, temporary files, checkpoint/write behavior, table/index size, and growth;
backup age and restore-test status;
slow and frequent normalized statements using pg_stat_statements after reviewing its configuration and privacy implications;
migration version and failures.

Redis and object storage¶

Redis memory, fragmentation, connected clients, command latency, evictions, expired keys, persistence status, and restarts;
Garage API health, request failures, capacity, metadata/data volume growth, and last successful off-VPS backup;
missing or inaccessible product images observed from storefront synthetic checks.

Business health without PII¶

published product and variant counts;
in-stock, low-stock, and out-of-stock counts;
carts and inquiries created by day;
inquiry-to-confirmed-sale count once that state exists;
confirmed orders, refunds, fulfillment age, and abandoned pending payments once automated payment exists;
content publication count and consented aggregate conversions from the analytics system.

Use coarse aggregates. Customer identities, contact text, addresses, payment details, and WhatsApp content do not belong in metrics labels or general logs.

Alerts and reports¶

Initial alerts¶

public endpoint unavailable;
repeated 5xx or high latency;
container unhealthy or repeatedly restarting;
host/container memory near a safe limit or an OOM kill occurs;
sustained CPU saturation;
filesystem, inode, PostgreSQL volume, or Garage volume nearing capacity;
PostgreSQL unavailable, connection pool saturated, or migrations failed;
backup missing or older than its objective;
TLS certificate approaching expiration;
payment webhook failures or stuck background jobs once those features exist.

Alerts should initially go to a controlled operational email. WhatsApp alert delivery is optional later and must not depend on the same failing application. Every alert needs a linked runbook, severity, owner, and silence/escalation rule.

Reporting cadence¶

After every deployment: commit, migrations, service health, smoke tests, and rollback reference.
Weekly operational report: uptime, errors, latency, resource peaks, restarts, backup status, capacity trend, and unresolved alerts.
Monthly business report: products, stock status, inquiries, confirmed sales, fulfillment, content performance, acquisition, and consented conversion trends.
Quarterly maintenance review: dependency/security updates, restore drill, access review, retention, costs, capacity, and whether each dashboard/alert remains useful.

Reports containing business or personal data are generated into protected storage and are not committed to Git.

Database reports and maintenance scripts¶

Repository structure¶

ops/
  reports/       Read-only, parameterized operational and business reports
  maintenance/   Deliberate repair or cleanup commands
  runbooks/      Symptoms, diagnosis, safe actions, rollback, verification
  migrations/    Application-owned schema migrations remain with their module

Safety rules¶

Prefer Medusa Admin, supported APIs, modules, and workflows over direct database writes so invariants and side effects are preserved.
Give routine reports a read-only PostgreSQL role with access only to required schemas/views.
Parameterize queries; never construct SQL from untrusted strings.
Maintenance scripts default to preview/dry-run and require an explicit apply flag.
Before material writes, verify the exact target environment, take/verify an appropriate backup, and print affected-record counts.
Run writes in transactions where safe, make operations idempotent, and define rollback or compensating steps.
Record actor, timestamp, reason/ticket, release, parameters excluding secrets, affected counts, and verification result.
Never echo credentials or dump personal/payment data into terminal logs, telemetry, Git, or chat.
Test reports and maintenance against a disposable restored database before production use.
A direct SQL correction is an exceptional, reviewed operation—not an informal substitute for an Admin feature.

Useful future reports include inventory discrepancies, unpublished products, missing media, stale carts/inquiries, orders awaiting fulfillment, failed webhooks/jobs, orphaned records, database growth, and customer-data retention candidates.

Client and customer data model¶

Andrea may eventually need relationships beyond a simple product customer list. Keep purpose and consent explicit by separating:

architecture prospects and professional clients;
cement-store customers and order recipients;
journal/newsletter subscribers;
supporters/tip contributors;
partners, galleries, markets, suppliers, and collaborators.

Medusa remains the source of truth for store customers, customer groups, carts, orders, promotions, gift cards/store credit, and fulfillment-related information. A future CRM can manage architecture leads, opportunities, meetings, proposals, and professional follow-up. A newsletter platform manages subscriptions and suppression. Do not silently merge all three merely because they share an email address.

The design must support:

source and purpose of collection;
prior, express, and informed authorization where required;
channel-specific marketing consent and timestamp/source;
preferred language and contact channel;
unsubscribe/opt-out and a durable suppression list;
access controls, audit history, correction/export/deletion workflows, and retention limits;
prevention of duplicate or conflicting outreach;
data minimization and separation of public professional notes from private customer/order data.

Colombian personal-data obligations, including Ley 1581 and applicable SIC guidance, require a reviewed privacy and authorization process before marketing databases or uploads are enabled. Legal/accounting advice should confirm the final implementation.

Marketing outreach roadmap¶

Marketing is a later business capability, not a reason to collect contacts during the MVP without purpose.

Potential programs include:

architecture and astronomy publishing with search optimization;
an explicitly opted-in bilingual newsletter;
product launches, restocks, studio/process stories, and seasonal collections;
Instagram or other approved social content linked to durable site pages;
architecture-network outreach, exhibitions, local design markets, galleries, and collaborations;
referral sources and campaign links with consented aggregate attribution;
follow-up reminders for professional leads and prior customers where the relationship and consent permit it.

Do not scrape contacts, buy lists, assume an order grants unrelated marketing permission, or send unsolicited WhatsApp campaigns. Transactional messages, professional one-to-one correspondence, and marketing broadcasts have different purposes and rules.

Before adopting a CRM or marketing-automation product, define the workflow and export requirements. Prefer a system with an API, auditable changes, role-based access, consent/suppression support, and a complete export so client data is not trapped in a vendor.

Loyalty and repeat customers¶

Do not build loyalty points before reliable customer identity, confirmed orders, payments, refunds, and cancellations exist.

Early capabilities¶

Medusa customer groups for repeat customers, VIPs, collaborators, or wholesale buyers;
group-specific promotions or price lists;
manually issued thank-you discounts with clear expiry and terms;
gift cards or store credit after evaluating Medusa's Loyalty Plugin and Colombian accounting/payment implications.

Later loyalty program¶

A points or rewards system should use an append-only ledger of earn, redeem, expire, reverse, and administrative-adjustment events rather than only a mutable balance. Define:

what earns rewards and when they become available;
refund/cancellation reversal behavior;
expiration and notification rules;
maximum value, fraud/abuse handling, and manual adjustments;
whether WhatsApp/manual sales qualify and how they are reconciled;
customer access, deletion, and account-merging behavior;
financial/accounting treatment of credits and gift cards.

Measure whether repeat purchases improve before adding complex tiers, referrals, birthdays, or gamification.

Implementation sequence¶

Establish Stage A health, logs, backup verification, deployment markers, and operational runbooks before public launch.
Complete the VPS resource audit and define memory/disk budgets.
Add external uptime and certificate monitoring.
Add private Grafana/Prometheus metrics with host, container, PostgreSQL, Redis, Garage, Nginx, and application dashboards.
Add centralized logs through Alloy/Loki only if local logs are insufficient and resource budgets allow it.
Instrument application boundaries with OpenTelemetry; add tracing only for concrete diagnostic needs.
Create the read-only reporting role and versioned report scripts.
Add reviewed maintenance tooling as actual failure/data-quality cases arise.
Define privacy, consent, and data-retention policy before building a client/marketing database.
Introduce CRM, newsletter, and loyalty features only after their source-of-truth and business workflows are explicit.

Decisions required later¶

VPS resource budget and retention targets for metrics and logs.
Self-hosted Grafana stack versus an OpenTelemetry-compatible hosted backend.
Private access method for Grafana and operational tools.
Alert destination, severity policy, and on-call expectations.
Acceptable report contents and roles permitted to access them.
CRM and newsletter requirements, data residency, export, and deletion capabilities.
Marketing consent language and approved outreach channels.
Whether customer groups, gift cards/store credit, or points solve a demonstrated loyalty need.