Agent skill

markov-regime-features

Debugging constant Markov regime features in RL observations - when HMM probabilities show uniform values instead of dynamic regime estimates

View SKILL.md on GitHub Repository
Stars 163
Forks 31

Install this agent skill to your Project

npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/development/markov-regime-features-smith6jt-cop-skills-registry

SKILL.md

markov-regime-features - Research Notes

Experiment Overview

Item Details
Date 2025-12-13
Goal Debug why Markov features (indices 19-24) appeared constant in observation heatmaps while other features showed high variability
Environment Python 3.10, PyTorch 2.0+, alpaca_trading package
Status Success

Context

When visualizing RL observation features as a heatmap, the Markov regime features (volatility and trend probabilities) appeared as constant horizontal bands with values ~0.33, while all other features showed expected variability over time. This made the Markov features useless for regime detection.

Root Cause

The standalone build_inference_observation() function defaults to uniform priors when markov_vol_probs and markov_trend_probs are not provided:

python
# From inference_obs_builder.py:391-401
if include_markov:
    if markov_vol_probs is not None:
        obs[:, feat_idx:feat_idx+3] = markov_vol_probs
    else:
        obs[:, feat_idx:feat_idx+3] = [0.33, 0.34, 0.33]  # Uniform prior

The 6 Markov features are:

Index Feature Description
19 vol_prob_low P(low volatility regime)
20 vol_prob_medium P(medium volatility regime)
21 vol_prob_high P(high volatility regime)
22 trend_prob_down P(downtrend regime)
23 trend_prob_neutral P(neutral regime)
24 trend_prob_up P(uptrend regime)

Verified Workflow

Solution: Use InferenceObservationBuilder class

The InferenceObservationBuilder class maintains Markov state and updates probabilities based on price history:

python
from alpaca_trading.gpu.inference_obs_builder import InferenceObservationBuilder

# Create stateful builder with GPU Markov system
obs_builder = InferenceObservationBuilder(window=100, use_gpu_markov=True)

# Build observation - Markov states are updated from price history
obs = obs_builder.build(
    prices=prices,
    high=high,
    low=low,
)

# Access current regime estimates
print(f"Volatility: {obs_builder.vol_probs}")  # e.g., [0.15, 0.60, 0.25]
print(f"Trend: {obs_builder.trend_probs}")      # e.g., [0.20, 0.30, 0.50]

Visualizing Regime Evolution

To see how regimes change over time, build observations at multiple points:

python
eval_points = list(range(150, n_bars, 10))
vol_history = {'low': [], 'med': [], 'high': []}

for end_idx in eval_points:
    obs_builder.build(prices=prices[:end_idx], high=high[:end_idx], low=low[:end_idx])
    vol_history['low'].append(obs_builder.vol_probs[0])
    vol_history['med'].append(obs_builder.vol_probs[1])
    vol_history['high'].append(obs_builder.vol_probs[2])

plt.plot(eval_points, vol_history['high'], 'r-', label='P(High Vol)')
plt.plot(eval_points, vol_history['low'], 'g-', label='P(Low Vol)')

Failed Attempts (Critical)

Attempt Why it Failed Lesson Learned
Using build_inference_observation() directly No Markov state passed, defaults to uniform [0.33, 0.34, 0.33] Use InferenceObservationBuilder class for stateful Markov tracking
Setting include_markov=False Removes features entirely, breaks model compatibility Keep Markov features, just ensure proper initialization
Manual Markov probability calculation Complex HMM implementation needed Use GPUMarkovSystem via use_gpu_markov=True

Final Parameters

python
# Correct usage for dynamic Markov features
obs_builder = InferenceObservationBuilder(
    window=100,           # Must match training window
    use_gpu_markov=True   # Enable GPU-accelerated HMM
)

# For live trading - maintain single builder instance across bars
# For backtesting - create new builder per symbol, maintain across timesteps

Key Insights

  • The standalone function is for single-shot inference where Markov state is passed externally
  • The builder class is for stateful inference where Markov state evolves with price data
  • GPU Markov system uses last 20 returns to estimate regime probabilities
  • Uniform priors [0.33, 0.34, 0.33] indicate Markov system not receiving updates
  • In heatmaps, constant horizontal bands at indices 19-24 are a red flag for this issue

References

  • alpaca_trading/gpu/inference_obs_builder.py - Feature building implementation
  • alpaca_trading/gpu/harmonized_gpu.py - GPUMarkovSystem class
  • notebooks/develop_branch_testing.ipynb - Visualization examples

Didn't find tool you were looking for?

Be as detailed as possible for better results