Trading System Development on the CMT Exam
Backtesting and system development are critical for CMT Level 2 and Level 3, where candidates must demonstrate the ability to evaluate trading strategies objectively. This integrates statistical analysis with practical risk management.
For the full guide, see the CMT Exam Guide 2026.
The System Development Process
Step 1: Hypothesis Formation
Every system starts with an observable market behavior:
- "Moving average crossovers generate positive expectancy"
- "Breakout above resistance with volume leads to a trend"
- "Mean reversion from oversold RSI levels is profitable"
Step 2: Rule Definition
Translate the hypothesis into explicit, mechanical rules:
- Entry rules: Exact conditions for trade initiation
- Exit rules: Profit targets, stop losses, trailing stops
- Position sizing: How much capital per trade (position sizing)
- Filters: Market regime, sector, time of day
Step 3: Backtesting
Test the rules against historical data:
- Minimum 1,000 trades for statistical significance
- Cover multiple market cycles (bull, bear, sideways)
- Include realistic transaction costs and slippage
Step 4: Validation
Avoid overfitting through proper validation:
- In-sample (IS): Data used to develop the system
- Out-of-sample (OOS): Data reserved for validation (never seen during development)
- Walk-forward analysis: Rolling IS/OOS windows
Key Performance Metrics
| Metric | Formula / Definition | Good Target |
|---|---|---|
| CAGR | Compound annual growth rate | > Risk-free rate |
| Sharpe Ratio | (Return − Rf) / σ | > 1.0 |
| Sortino Ratio | (Return − Rf) / Downside σ | > 1.5 |
| Max Drawdown | Largest peak-to-trough decline | < 20% |
| Win Rate | Winning trades / total trades | > 40% (trend); > 55% (mean reversion) |
| Profit Factor | Gross Profit / Gross Loss | > 1.5 |
| Expectancy | (Win% × Avg Win) − (Loss% × Avg Loss) | > 0 |
| Recovery Factor | Net Profit / Max Drawdown | > 3 |
Common Pitfalls
Overfitting (Curve Fitting)
The most dangerous mistake in system development:
- Adding too many parameters to fit historical data perfectly
- Results look fantastic in-sample but fail out-of-sample
- Rule of thumb: Fewer parameters = more robust system
- Use statistical significance tests (p-values < 0.05)
Survivorship Bias
Using only currently existing stocks excludes delisted companies, inflating backtested returns.
Look-Ahead Bias
Using information that wasn't available at the time of the simulated trade.
Data-Mining Bias
Running thousands of tests and cherry-picking the best results.
System Evaluation Framework
For CMT Level 3 essays, evaluate systems holistically:
- Robustness: Does it work across different market conditions?
- Simplicity: Fewer rules = more likely to work in the future
- Risk-adjusted returns: Sharpe/Sortino ratios matter more than raw returns
- Drawdown profile: Can you psychologically handle the worst drawdown?
- Capacity: Can it manage the intended capital size?
Link to algorithmic trading for automated implementation.
Practice system evaluation questions in our test bank. Full guide: CMT Exam 2026.
Key Performance Metrics — Good vs. Poor Trading System
Comparison of a robust system vs. an overfitted one in out-of-sample testing
In-Sample vs. Out-of-Sample Equity Curves
Overfitted systems show great IS performance but degrade OOS