Project Redline Building a Trading Platform From Scratch#

What It Is#

Project Redline — branded Redline Engine under WhiteXPawn — is my professional algorithmic trading platform. It is built around one idea: research thoroughly on your machine, promote deliberately through an operator console, and execute live in a controlled cluster environment.

It is not a single app. It is a pipeline of specialized components — backtesting, statistical validation, machine learning, live execution, orchestration, and audit — connected by a single handoff artifact (a strategy bundle) and a central control API.

IMPORTANT
This isn’t a “trading bot” in the YouTube sense. There’s no magic AI predicting the market. It’s a system that tests ideas honestly, executes them precisely, and doesn’t let you skip the steps that keep you from fooling yourself.

The platform separates three worlds on purpose:

World	Who	Where	Purpose
Research	Quant / researcher	Local laptop or CI	Download history, train, backtest, validate, then mock-trade on a live stream
Operations	Operator	Web console	Import strategies, promote to live, emergency stop, settings
Live trading	Unattended systems	Kubernetes cluster	Stream market data, infer signals, size risk, send real orders

Research never spins up cluster backtest jobs. Live never re-runs Monte Carlo validation. The bridge between research and production is one JSON bundle — not ad-hoc scripts or manual copy-paste.

The Big Picture: Research First, Then Live#

Research owns the whole proof. Live owns real money. Mock trading is not a middle world between those two — it is the last step of research, still on the laptop, still with simulated fills.

The research pipeline looks like this:

1
Databento downloaded data → ML (optional) → Backtest → Validate → Live Databento stream (mock execution)

Step	Data	Fills	What it proves
Download	Historical Databento OHLCV on disk	—	You have a dataset
ML (optional)	That same history, walk-forward	—	Models exist without leaking the test set
Backtest	Downloaded historical bars	Simulated	The idea works on history at all
Validate	Same history, MCPT pipeline	Simulated	The result is not noise, luck, or a fragile parameter peak
Mock	Live Databento streamed bars	Simulated	The same logic behaves in real time, still with no capital at risk

Only after that pipeline is done does a strategy leave research. Live is what you already expect: real broker connection (Interactive Brokers; NinjaTrader supported in the live engine), real money, full monitoring on the cluster.

CAUTION
Mock is owned fully by research. A strategy that only looked good on downloaded history but fails mock on a live Databento stream should never be bundled for the live stack.

At the center of the signal path sits AlphaEngine: the ML layer that turns market features into a structured trading signal (confidence, direction, regime). Models are trained offline during research and served live to the execution engine during production.

Phase 1 — Research (MiniRedline)#

Research runs locally via MiniRedline, a CLI-oriented lab workflow. You initialize a workspace, download historical OHLCV (futures via Databento), and work through a linear pipeline that ends with mock execution on a live Databento stream. Everything below is still research. The cluster does not see any of it yet.

Initialize the lab#

Creates a standard folder layout: data directories, strategy compositions, artifact folders for verdicts and promotion bundles, and configuration.

Download data (optional)#

Pulls historical market data into local parquet files for backtesting.

Train models (optional)#

Walk-forward ML training produces model artifacts and metadata when the strategy uses AlphaEngine models (mean-reversion, momentum, regime, and so on).

Backtest#

The HardEdge engine replays historical data through the full strategy stack — features, ML inference, entry gates, risk rules, simulated execution — and produces performance metrics, trade logs, and equity curves.

This is the first serious filter: does the idea work on history at all?

IMPORTANT
I do not use the whole dataset to train a model. I only use a small portion to train, then I use the rest to test — and I will never tweak and test the model again on the same dataset. The more you tweak and retest on the same data, the more you overfit, and the less any of it is a real edge.

Validate (MCPT pipeline)#

This is where Redline gets statistically serious. MCPT (Monte Carlo Permutation Testing) runs a four-phase validation pipeline, ordered from cheap to expensive:

Phase	What it tests	Why it matters
1. Parameter sensitivity	Is performance a fragile peak or a stable plateau?	Catches over-fit parameter tuning
2. Walk-forward out-of-sample	Does edge hold on unseen time periods?	Catches in-sample luck
3. Stress tests	Survival across historical crash periods + slippage stress	Catches strategies that only work in calm markets
4. Monte Carlo permutations	p-value: could random bar order produce this result?	Separates genuine edge from noise

The output is a validation verdict — raw metrics and evidence, not a blind pass/fail gate. Promotion decisions stay with the operator, informed by the report.

NOTE
HardEdge also runs parameter optimisation sweeps — testing thousands of combinations in parallel. The goal isn’t finding the single “best” setting. It’s finding settings that work across a wide range, not just at one lucky point. Because of overfitting I do the validation pipeline first, and only after that will I optimise.

Mock session (still research)#

After backtest and validation, MiniRedline runs the same strategy logic against a live Databento stream with simulated fills. No broker. No cluster. No capital.

This is what is good if you have already done the normal research. Downloaded history can hide timing bugs, session boundaries, and “it only worked because the file was static.” A live stream with mock execution is the last research gate — still fully owned by MiniRedline.

Export bundle#

Packages everything into strategy_bundle_v1: strategy definition, backtest summary, validation verdict, lineage/metadata, optional training artifacts, optional mock session results.

This is the only object that crosses from research into production.

Sync (optional)#

The bundle can be pushed directly to the Redline API from the CLI — or uploaded later via the console.

Phase 2 — Handoff (the Bundle Contract)#

The bundle is the platform’s contract between research and operations. It answers: what is this strategy, what evidence supports it, and where did it come from?

When imported, a strategy enters the engine database in bench state — meaning: research gate passed, waiting for live capacity and operator approval.

There is no automatic cluster chain that runs backtest → MCPT → sim jobs in Kubernetes. That path was removed on purpose. MiniRedline already did that work locally. Duplicating it in the cluster was just a way to lie to yourself twice.

Phase 3 — Operations (Console)#

Operators use a private web console to:

Import strategy bundles (upload or API-driven)
Review lifecycle state, performance, settings
Promote approved strategies toward live trading
Emergency stop and manage risk settings
Configure platform behavior without touching cluster internals

Authentication uses Supabase operator JWTs — browsers never hold service secrets. Backend cluster components use a separate service token model.

Phase 4 — Lifecycle (Orchestrator + States)#

Once imported, strategies move through a finite state machine managed by the Orchestrator — a polling service that watches the engine database and drives Kubernetes deployments.

1
import → bench → promoting_to_live → live_running → retired

State	Meaning
bench	Validated bundle imported; queued for capacity / operator decision
promoting_to_live	Approved for live; deployment in progress
live_running	Sentinel live pipeline active; real or sim broker path depending on namespace
retired	Strategy stopped — rules fired, operator action, or end of life

Promotion is deliberate. Live trading runs real money. The platform assumes a human (or an explicit automated policy) approves the transition.

Retirement is automatic where configured. Each strategy can carry retirement rules in its JSON — drawdown limits, consecutive losses, inactivity, time caps, alpha-decay signals, and more. When a rule fires, the orchestrator retires the strategy and frees capacity for the next candidate.

CAUTION
Sim and live run in separate Kubernetes namespaces with separate orchestrators and separate RBAC. A compromise in sim cannot escalate to live.

Phase 5 — Live Execution (Sentinel)#

When a strategy is live_running, Sentinel (the Rust live engine) runs a six-stage streaming pipeline:

1
Market data → Alpha signal → Risk → Order management → Broker → Trade analysis

Stage	Role
1. Market data	Ingest live (or synthetic) OHLCV bars — typically Databento
2. Alpha	Call AlphaEngine sidecar for ML inference → target position / signal
3. Risk	Apply gates, Kelly sizing, portfolio constraints → approved position
4. EMS	Translate approved intent into limit orders and manage working orders
5. Broker	Execute via Interactive Brokers (or NinjaTrader path in dev)
6. TCA	Transaction cost analysis — slippage, fill quality, feedback logging

Stages communicate through a Redis-backed message bus in production. Each stage is a focused service, not a monolith.

The design goal is typed handoffs: each stage receives a structured object and emits the next — reducing the “stringly typed” trading bugs that plague ad-hoc systems.

TIP
A future closed loop is already in the architecture: TCA feedback → cost model updates → better slippage assumptions in research backtests. The wiring is still evolving. The point is the system is built to learn from fills, not just from history.

Supporting Infrastructure#

Redline API#

The single control plane for the engine: strategy import, lifecycle PATCH, trades, backtest runs, settings, audit logging. All cluster components talk to the API — not directly to Postgres from random scripts.

Data storage (two domains)#

1
Engine Postgres     strategies, trades, backtest runs, hedge state, orchestrator settings
2
Supabase audit      audit log and runtime logs for compliance and debugging

Marketing and contact data lives in a separate Supabase in the WhiteXPawn site. Trading state and marketing state do not share a database.

Kubernetes#

Docker images for the API, orchestrator, risk engine, strategy pods. GitLab CI builds and deploys on push to main.

Sim namespace (trading-sim) and live namespace (trading-live) have different resource quotas — live is intentionally tighter.

CI as operational safety#

The monorepo runs lint, ~2800+ tests, Rust builds, docs, API integration tests, and dependency auditing on every push. Coverage sits around 87%. I treat test discipline as part of not blowing up an account — not as a vanity metric.

End to End#

A researcher develops a futures strategy locally: pull Databento history, optionally train models, backtest, run the four-phase validation pipeline, then mock-execute on a live Databento stream with simulated fills — still on the laptop, still research. Only then is a signed evidence bundle exported. An operator imports that bundle into Redline Engine, reviews the verdict, and promotes it when capacity allows. The orchestrator deploys the strategy into the live Kubernetes stack; Sentinel streams market bars, AlphaEngine serves the ML signal, risk sizes the trade, and orders route to Interactive Brokers — with every stage logged, every retirement rule enforced, and every promotion gated by human judgment.

Research stays on the laptop. Money stays behind the cluster wall.

IMPORTANT
Precision over hype. This platform is built for people who care about p-values, walk-forward out-of-sample tests, and retirement drawdowns. Researchers prove. Operators promote. Machines execute. Bundles carry validation artifacts — not just a JSON config and a hope.