LEO Customer 360 — Documentation

Data-Tracking API - UAT Performance Test Report

6 min read

Data-Tracking API - UAT Performance Test Report

Date: 2026-08-31 Environment: UAT Target: https://beta.leocdp.com/data (real endpoint) Test: stepped-RPS ramp with asynchronous S3 store verification Harness: data-tracking-api/tests/perf_uat_tracking.py Raw data: perf_results_ramp.json (main ramp) - perf_results_10rps.json (initial strict-cap trial) Run id: perf-20260831T100254Z-7a216d8d

Architecture note: This report preserves the measurements from the 2026-08-31 UAT run, but the deployment architecture has since moved to the Redis Streams handoff. Current requests return 202 Accepted after a batch is durably enqueued; a consumer-group worker writes the NDJSON object to S3 and acknowledges the stream entry only after the write succeeds. The results below are historical capacity data from the run and should not be read as a description of a synchronous request-to-S3 path.

1. TL;DR

  • Sent 10000 dummy events to the real UAT ingestion endpoint (50 steps x 200, offered load 10 → 500 RPS) and asynchronously verified every accepted request landed in the S3 object store.
  • Success (accepted AND durably stored): 9998 / 10000 = 99.98%. Only 2 failures, both transient HTTP 502. 100% of accepted events were found in S3.
  • No failure ceiling within 500 RPS - but actual throughput saturates at ~100 req/s. Above ~100 RPS the offered load simply queues (send p50 grows from ~56 ms to ~4 s) instead of erroring; nothing hit the 30 s timeout.
  • Interpretation: the 3-replica stack durably ingests ~100 events/sec sustained, then degrades gracefully (latency up, success flat) rather than dropping requests.
  • Total wall time: 203.2 s.

2. System under test - deployment configuration

Deployed with deployments/server/deploy-tracking.sh uat (this session’s multi-replica + local-LB design).

ItemValue
Env / boxUAT - server key tracking - s-general-1x2
Box IPsfloating 49.213.71.192 (SSH) - private 10.100.1.8 (VPC)
App replicas3 - customer360-tracking-api-1/2/3 (TRACKING_REPLICAS default uat=3)
Imageghcr.io/leo-cdp/leo-customer360/data-tracking-api@sha256:bcebc2518b3e2fc6221cf54af91928b436f2c184779650cb4bea98963f475fdc
Runtimeuvicorn (1 process/replica) on :8010, FastAPI
Private networkdocker bridge c360-tracking - replicas 172.18.0.2/.3/.4, LB 172.18.0.5
Load balancercustomer360-tracking-lb (nginx:alpine) on host :8010, least_conn, max_fails=3 fail_timeout=10s, proxy_next_upstream error timeout http_502/503/504
Durable sink (S3)vStorage https://hcm04.vstorage.vngcloud.vn, region us-east-1, path-style; one bucket per source data-tracking-<data_source_id>, key YYYY-MM-DD-HH/<uuid>.jsonl
Redis Streams + rate/session stateapi box 10.100.1.5:6580 (shared; stream is required, rate/session metadata fail-open)
TracingOTLP → api-box Jaeger http://10.100.1.5:4318, sampler 1.0
Public pathPOST https://beta.leocdp.com/data/api/v1/tracking/logs - health /data/health
Front doorVNG NLB :443 (L4) → Caddy (TLS + handle_path /data/* strip) → DATA_UPSTREAM 10.100.1.8:8010 (nginx LB)

Request + verification path

flowchart LR
  T["Perf test harness<br/>httpx AsyncClient<br/>paced at target RPS"]
  NLB["VNG NLB :443<br/>L4 TCP passthrough"]
  CADDY["Caddy (api box)<br/>TLS terminate<br/>strip /data"]
  LB["nginx LB :8010<br/>customer360-tracking-lb<br/>least_conn"]
  A1["api-1 :8010"]
  A2["api-2 :8010"]
  A3["api-3 :8010"]
  S3[("vStorage S3<br/>bucket data-tracking-&lt;id&gt;<br/>YYYY-MM-DD-HH/uuid.jsonl")]
  R[("Redis 10.100.1.5:6580<br/>Redis Stream + rate/session state")]
  W["Consumer-group workers<br/>one per tracking-api replica"]

  T -->|"1. POST /data/... (HTTPS)"| NLB --> CADDY -->|"DATA_UPSTREAM"| LB
  LB --> A1
  LB --> A2
  LB --> A3
  A1 -->|"2. XADD batch"| R
  A2 --> R
  A3 --> R
  R -->|"3. XREADGROUP / XAUTOCLAIM"| W
  W -->|"4. background PUT .jsonl"| S3
  A1 -.->|rate-limit / session| R
  A2 -.-> R
  A3 -.-> R
  T -.->|"5. async HEAD object (store-check)"| S3

  subgraph box["Tracking box 10.100.1.8 - bridge c360-tracking"]
    LB
    A1
    A2
    A3
    W
  end

3. Rate-limit configuration set for the test (explicit)

The service rate-limits per client IP as seen by the app - which behind Caddy→nginx is the nginx LB IP, i.e. ONE global bucket. It is a fixed window (core/redis_cache.py: Redis INCR + EXPIRE=window). I drove it via a new knob added to deploy-tracking.sh:

Env knob (deploy-tracking.sh)Effect
TRACKING_RATE_LIMIT_RPS=<n>convenience → writes TRACKING_RATE_LIMIT_REQUESTS=<n>, TRACKING_RATE_LIMIT_WINDOW_SECONDS=1
TRACKING_RATE_LIMIT_REQUESTS / TRACKING_RATE_LIMIT_WINDOW_SECONDSset requests/window directly
(unset)app default - 120 req / 60 s (core/config.py)

Values used across this session, each applied by a redeploy (writes /opt/c360/tracking.env, restarts replicas):

PhaseRate limitWhy
Baseline (prod default)120 / 60s (= 2 RPS)shipped default
Trial 1 (strict)TRACKING_RATE_LIMIT_RPS=10 → 10 / 1sfirst 10-RPS trial
Main rampTRACKING_RATE_LIMIT_RPS=100000 → 100000 / 1slimiter out of the way so the ramp measures the service, not the limiter
Restored (after test)(unset) → 120 / 60sUAT returned to the shipped default

Finding from Trial 1: a fixed-window limiter set equal to the offered rate (10/1s vs 10 RPS) rejected ~8% (16/200 → 429) purely from window-boundary jitter - a client paced at exactly N/s does not align to the server’s 1-second windows, so some windows see N+1. To allow a rate you must give headroom; to measure the service you must raise the limiter far above the tested RPS (what the main ramp did).

4. Test methodology

  • Harness: tests/perf_uat_tracking.py - async (httpx.AsyncClient), paced launcher (requests fired at 1/RPS intervals; launch cadence independent of response latency).
  • Ramp: --start-rps 10 --step-rps 10 --max-rps 500 --per-step 200 --success-threshold 0.99. Each step = 200 requests at a fixed offered RPS; step up +10 while success >= 99%.
  • Success definition: a request counts as success only if it is accepted (2xx) AND the returned object_key is found in S3 (HEAD). A 429 counts as a failure (not retried) so the ramp can detect a ceiling.
  • Async store-check: asyncio.to_thread(s3.head_object) with a widened worker pool (256) and botocore max_pool_connections=256, so verification runs concurrently and is not the bottleneck.
  • Perf-data tagging (for later deletion): all events land in one bucket data-tracking-abcdef00-0000-4000-8000-000000000001, and each event carries _perf_test=true, _perf_run_id, _perf_rps, _perf_seq.
  • Bot filter avoided: a normal (non-bot) User-Agent is sent so requests are not dropped as bots.

Reproduce:

# S3 read creds for the store-check (mirror deploy-tracking.sh sources)
export S3_ENDPOINT_URL=https://hcm04.vstorage.vngcloud.vn S3_REGION=us-east-1 S3_FORCE_PATH_STYLE=true
export S3_ACCESS_KEY_ID=... S3_SECRET_ACCESS_KEY=...
python tests/perf_uat_tracking.py --start-rps 10 --step-rps 10 --max-rps 500 --per-step 200 \
  --out tests/reports/perf_results_ramp.json

(Requires the limiter raised on UAT first: TRACKING_RATE_LIMIT_RPS=100000 ./deploy-tracking.sh uat, restored after with a plain redeploy.)

5. Results

Overall: 9998/10000 success (99.98%), 2 failures (both HTTP 502), max achieved throughput 105.2 RPS, wall 203.2 s.

Failures by step: 250 RPS: {‘http:502’: 1}, 410 RPS: {‘http:502’: 1}

Legend: Offered = target RPS - Achieved = 200/step_wall - Send = POST latency ms - Store = async HEAD latency ms.

Offered RPSAchieved RPSSuccessRateFailedSend p50Send p99Store p50
1010.0200/200100.00%056.0416.811.9
2020.0200/200100.00%057.6994.311.2
3029.7200/200100.00%059.9302.211.5
4038.3200/200100.00%064.41951.811.4
5049.2200/200100.00%068.3531.412.2
6037.3200/200100.00%074.14036.312.4
7061.1200/200100.00%0127.2712.218.0
8069.9200/200100.00%0105.2951.715.1
9082.6200/200100.00%0180.6505.619.4
10071.4200/200100.00%0185.11317.817.7
11093.5200/200100.00%0292.9764.621.9
12065.9200/200100.00%0393.51858.229.3
13097.6200/200100.00%0253.5566.021.1
140105.2200/200100.00%0384.6725.227.5
15075.4200/200100.00%0476.51671.630.2
16063.5200/200100.00%0432.62130.436.1
17094.5200/200100.00%0320.7791.021.7
18084.0200/200100.00%0365.81569.431.2
19067.2200/200100.00%0440.61835.243.3
20089.6200/200100.00%0559.11466.441.1
21057.2200/200100.00%01173.32971.743.9
22090.7200/200100.00%0670.71382.441.1
23061.6200/200100.00%01290.43002.543.8
24058.9200/200100.00%01343.42225.337.9
25040.5199/20099.50%12196.44099.262.1
26080.8200/200100.00%01035.52190.742.4
27039.6200/200100.00%02195.44563.263.0
28049.3200/200100.00%01062.83194.546.5
29073.7200/200100.00%0770.62326.535.5
30065.9200/200100.00%0704.32462.454.7
31056.8200/200100.00%01883.83235.048.4
32063.5200/200100.00%01489.22767.136.4
33069.1200/200100.00%01189.62698.347.9
34048.8200/200100.00%01470.73825.056.4
35061.0200/200100.00%01900.22926.581.2
36033.1200/200100.00%02912.85681.274.6
37038.3200/200100.00%03079.94928.591.8
38040.2200/200100.00%02660.24665.458.6
39061.0200/200100.00%01126.82704.652.1
40068.9200/200100.00%01428.22736.238.7
41083.5199/20099.50%1801.72029.246.2
42051.5200/200100.00%01882.33467.948.3
43085.6200/200100.00%01087.71882.5144.1
44032.8200/200100.00%03988.35666.273.9
45039.9200/200100.00%01960.03381.446.7
46076.2200/200100.00%0830.62202.441.2
47049.1200/200100.00%01541.53484.041.6
48077.9200/200100.00%0789.42149.458.7
49042.6200/200100.00%02910.64548.268.8
50025.2200/200100.00%01876.93842.449.5

Reading the numbers

  • Achieved RPS never exceeds ~105 despite offering up to 500 → the pipeline saturates around ~100 req/s. Beyond that, per_step / wall falls (e.g. 500 offered → 25 achieved, 8.0 s for 200) because requests queue.
  • Send p50 climbs from ~56 ms (⇐50 RPS) to ~4 s at high offered load - classic latency-under-saturation. Store-check p50 stays 11-144 ms (verification is not the bottleneck).
  • Store durability = 100% of accepted requests. The only 2 losses were 502s (gateway-level, transient), not storage failures.

6. Conclusions

  1. Correctness: every accepted event was durably persisted to S3 - the ingestion + object-write path is reliable under load (99.98% end-to-end).
  2. Capacity: sustained throughput ~ 100 events/sec for the current 3-replica UAT box; the system degrades gracefully (latency up) rather than shedding load up to at least 500 offered RPS.
  3. To push higher: add replicas (TRACKING_REPLICAS) and/or a bigger box; re-run the ramp to find the new plateau. The current bottleneck should be evaluated across Redis Stream enqueue capacity, consumer-worker throughput, and S3 PUT capacity rather than assuming that every request performs the PUT inline.
  4. Rate limiter caveat: it is global (keyed on the LB IP) and fixed-window - set it well above expected peak, and do not set it equal to a target rate.

7. Perf-data cleanup

All test data is isolated for easy deletion:

  • Bucket: data-tracking-abcdef00-0000-4000-8000-000000000001 (delete all objects to remove every perf event).
  • Per-event markers: _perf_test=true, _perf_run_id (e.g. perf-20260831T100254Z-7a216d8d).

8. Files in this folder

  • PERF_REPORT.md - this report.
  • perf_results_ramp.json - full 50-step ramp (per-step = individual test runs).
  • perf_results_10rps.json - initial strict 10/1s trial (92% - the limiter-headroom finding).

Graph View