mirror of
https://github.com/sudosylabs/vnidrop.git
synced 2026-08-05 02:29:55 +02:00
feat(diagnostics): harden ingestion and delivery
This commit is contained in:
196
services/diagnostics-api/README.md
Normal file
196
services/diagnostics-api/README.md
Normal file
@@ -0,0 +1,196 @@
|
||||
# VniDrop diagnostics API
|
||||
|
||||
Cloudflare Worker for ingesting batched telemetry, crash reports, and user-submitted
|
||||
bug reports. D1 stores searchable metadata; R2 stores larger stack traces and logs.
|
||||
|
||||
The service is designed for modest traffic and low operating cost:
|
||||
|
||||
- one D1 row is written per telemetry batch, not per event;
|
||||
- crash stacks and bug logs are stored in R2 instead of D1;
|
||||
- request and batch limits reject oversized work before storage writes;
|
||||
- an hourly scheduled cleanup and an R2 lifecycle rule enforce retention;
|
||||
- no Queue, Durable Object, or KV resources are required.
|
||||
|
||||
Cloudflare quotas and prices change over time. Check the current
|
||||
[Workers pricing](https://developers.cloudflare.com/workers/platform/pricing/),
|
||||
[D1 pricing](https://developers.cloudflare.com/d1/platform/pricing/), and
|
||||
[R2 pricing](https://developers.cloudflare.com/r2/pricing/) before relying on a
|
||||
particular free-plan capacity.
|
||||
|
||||
## API
|
||||
|
||||
All ingest routes require:
|
||||
|
||||
```http
|
||||
X-VniDrop-Key: <INGEST_KEY>
|
||||
```
|
||||
|
||||
The client also sends its anonymous installation ID:
|
||||
|
||||
```http
|
||||
X-VniDrop-Install-Id: <anonymous install UUID>
|
||||
```
|
||||
|
||||
| Method | Path | Body |
|
||||
|--------|------|------|
|
||||
| `GET` | `/live` | process liveness; does not touch storage |
|
||||
| `GET` | `/health` | authenticated readiness; checks required configuration and the D1 schema |
|
||||
| `POST` | `/v1/events` | `{ batchId, installId, appVersion?, platform?, events: [...] }` |
|
||||
| `POST` | `/v1/crashes` | app crash payload |
|
||||
| `POST` | `/v1/bugs` | app bug-report payload |
|
||||
|
||||
Batch and report IDs are client-generated UUIDs. A client must reuse the same ID
|
||||
when retrying so D1 can acknowledge the request without storing it twice.
|
||||
|
||||
Accepted reports return `202`. Defaults are a 262,144-byte request limit and at
|
||||
most 50 events per batch. Cloudflare rate-limit bindings allow 30 requests per
|
||||
installation and 120 requests per source, per ingest route, per minute. Source
|
||||
limits run before shared-key verification so rejected traffic is bounded too.
|
||||
These counters are eventually consistent and local to a Cloudflare location, so
|
||||
they are abuse mitigation rather than billing or authorization controls.
|
||||
|
||||
The API is consumed by native Android, iOS, and desktop clients and does not
|
||||
enable cross-origin browser access. If a browser-based client is added later,
|
||||
define a narrow origin allowlist instead of enabling wildcard CORS.
|
||||
|
||||
`/health` requires `X-VniDrop-Key` and uses the source limiter because it performs
|
||||
D1 reads. `/live` is the only unauthenticated probe and never touches storage.
|
||||
|
||||
## Security model
|
||||
|
||||
`INGEST_KEY` fails closed when it is missing, but it is a shared value embedded in
|
||||
released app binaries. It can be extracted and therefore is **not** user
|
||||
authentication, a durable secret, or sufficient abuse protection by itself.
|
||||
|
||||
- Store the Worker value with `wrangler secret put`; never put it in
|
||||
`wrangler.jsonc`, source control, logs, or command arguments.
|
||||
- Rotate the key when it is exposed and ship the matching app configuration.
|
||||
- Keep the two rate-limit namespaces unique within the Cloudflare account. A
|
||||
namespace reused by another Worker shares counters with it.
|
||||
- Use Cloudflare WAF or account-level rate-limiting rules if public abuse exceeds
|
||||
what the Worker bindings can absorb.
|
||||
- Do not log request bodies. Bug reports can contain contact details and attached
|
||||
logs.
|
||||
|
||||
## Provision and deploy
|
||||
|
||||
Run these commands from this directory:
|
||||
|
||||
```bash
|
||||
npm ci
|
||||
npx wrangler login
|
||||
|
||||
npx wrangler d1 create vnidrop-diagnostics
|
||||
npx wrangler r2 bucket create vnidrop-diagnostics
|
||||
```
|
||||
|
||||
Replace the placeholder `database_id` in `wrangler.jsonc` with the UUID returned
|
||||
by `wrangler d1 create`. Set the ingest key interactively, apply the tracked D1
|
||||
migrations, and configure the R2 retention rule once:
|
||||
|
||||
```bash
|
||||
npx wrangler secret put INGEST_KEY
|
||||
npm run db:migrate:remote
|
||||
npx wrangler r2 bucket lifecycle add vnidrop-diagnostics diagnostics-retention --expire-days 90
|
||||
npm run check
|
||||
npm run deploy
|
||||
```
|
||||
|
||||
`npm run deploy` also runs the complete `check` script automatically before
|
||||
Wrangler changes the remote Worker.
|
||||
|
||||
The lifecycle command changes the remote bucket. Before adding or changing a
|
||||
rule, inspect the current state with:
|
||||
|
||||
```bash
|
||||
npx wrangler r2 bucket lifecycle list vnidrop-diagnostics
|
||||
```
|
||||
|
||||
## Local development
|
||||
|
||||
Create an ignored `.dev.vars` file containing a development-only key:
|
||||
|
||||
```dotenv
|
||||
INGEST_KEY=local-development-only
|
||||
```
|
||||
|
||||
Then initialize the local D1 database and run the Worker:
|
||||
|
||||
```bash
|
||||
npm run db:migrate:local
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Wrangler keeps local D1 and R2 state under the ignored `.wrangler/` directory.
|
||||
Use `wrangler dev --test-scheduled` when exercising the hourly cleanup handler.
|
||||
|
||||
## Migrations and generated types
|
||||
|
||||
D1 migrations live in `migrations/` and are recorded in D1's migration ledger.
|
||||
Never edit an applied migration; add the next numbered SQL file instead.
|
||||
|
||||
`worker-configuration.d.ts` is generated from `wrangler.jsonc` and committed so
|
||||
bindings cannot silently drift from the Worker code:
|
||||
|
||||
```bash
|
||||
npm run typegen # regenerate after changing bindings or vars
|
||||
npm run types:check # verify the committed file is current
|
||||
```
|
||||
|
||||
Secrets and optional, commented-out bindings are not generated. The source adds
|
||||
only those narrow extensions to the generated environment type.
|
||||
|
||||
Vitest runs inside the Workers runtime. Its setup applies the same numbered D1
|
||||
migrations to the isolated local database assigned to each test file.
|
||||
|
||||
## Retention
|
||||
|
||||
`RETENTION_DAYS` defaults to 90. The `17 * * * *` cron trigger runs cleanup at
|
||||
17 minutes past every hour. Cleanup works in bounded batches: it deletes each
|
||||
expired report's referenced R2 object before deleting that exact D1 row. The R2
|
||||
lifecycle rule is an independent backstop for stack and log objects, including
|
||||
objects left behind by a partial ingest failure. Each scheduled run can remove
|
||||
8,000 event batches and 7,200 rows from each report table while staying below
|
||||
D1's per-invocation query ceiling. Later hourly runs continue any backlog.
|
||||
Reaching the cap emits a structured warning with the remaining expired-row counts;
|
||||
alert on that warning because
|
||||
retention is necessarily best-effort during sustained distributed abuse.
|
||||
|
||||
The Worker variable and bucket lifecycle are separate configuration surfaces.
|
||||
When changing retention, update both `RETENTION_DAYS` and the R2 lifecycle rule;
|
||||
changing one does not update the other. Cloudflare may delete expired R2 objects
|
||||
after the exact expiration time rather than synchronously at it.
|
||||
|
||||
## App wiring
|
||||
|
||||
Keep the tracked root defaults empty. Configure release builds through the
|
||||
user-level `~/.gradle/gradle.properties` or secured CI Gradle project properties:
|
||||
|
||||
```properties
|
||||
vnidrop.diagnostics.included=true
|
||||
vnidrop.diagnostics.endpoint=https://vnidrop-diagnostics.<your-subdomain>.workers.dev
|
||||
vnidrop.diagnostics.ingestKey=<same value as INGEST_KEY>
|
||||
```
|
||||
|
||||
Both the endpoint and key are required. When both are empty the app uses its
|
||||
offline-safe no-op transport; configuring only one fails the Gradle build.
|
||||
`vnidrop.diagnostics.included=false` disables
|
||||
automatic telemetry and crash upload, but a configured endpoint can still accept
|
||||
an explicit user-submitted bug report. Treat the app-side key as an abuse-control
|
||||
token with the limitations described above.
|
||||
|
||||
## Reading reports
|
||||
|
||||
```bash
|
||||
npx wrangler d1 execute vnidrop-diagnostics --remote \
|
||||
--command "SELECT id, exception_type, platform, occurred_at FROM crashes ORDER BY occurred_at DESC LIMIT 20"
|
||||
|
||||
npx wrangler d1 execute vnidrop-diagnostics --remote \
|
||||
--command "SELECT id, what_happened, status, occurred_at FROM bugs WHERE status = 'open' ORDER BY occurred_at DESC LIMIT 20"
|
||||
```
|
||||
|
||||
R2 object keys use `crashes/<id>/<attempt-id>/stack.txt` and
|
||||
`bugs/<id>/<attempt-id>/logs.txt`. The unique attempt segment prevents a retry
|
||||
from overwriting an already accepted object before D1 detects the duplicate.
|
||||
There is no public administration endpoint; inspect reports through authenticated
|
||||
Cloudflare tools or a future Access-protected dashboard.
|
||||
Reference in New Issue
Block a user