XLK

Historical Returns (6 features) — Percentage price change over 1, 2, 3, 5, 10, and 20 trading days. Formula: (close / close_N_days_ago − 1) × 100. These capture momentum at multiple scales — a stock up 5% in the last 20 days but down 1% today shows divergent short/long momentum.

SMA Ratios (5 features) — Price distance from Simple Moving Averages at 5, 10, 20, 50, and 200-day windows. Formula: (close / SMA − 1) × 100. A stock 3% above its SMA 50 is in a short-term uptrend relative to its medium-term average.

EMA Ratios (5 features) — Same concept but using Exponential Moving Averages, which weight recent prices more heavily. EMAs react faster to price changes than SMAs.

MA Crossover Ratios (2 features) — (SMA20 / SMA50 − 1) × 100 and (SMA50 / SMA200 − 1) × 100. Classic trend signals — when the shorter MA crosses above the longer, it signals trend acceleration (the “golden cross” and “death cross” patterns).

RSI — Relative Strength Index (1 feature) — 14-period momentum oscillator bounded 0–100. Computed as: 100 − 100 / (1 + avg_gain / avg_loss). Above 70 = overbought (may reverse down), below 30 = oversold (may reverse up). The model learns these mean-reversion signals.

MACD (2 features) — Moving Average Convergence Divergence. The MACD line is EMA12 − EMA26, normalized as a percentage of price. The histogram is the difference between the MACD line and its 9-period signal line. Captures momentum shifts and trend exhaustion.

Bollinger Bands (2 features) — Band width: (upper − lower) / SMA20 × 100 measures current volatility relative to average. %B: (close − lower) / (upper − lower) shows position within the bands (0 = at lower band, 1 = at upper band). Bands use 20-day SMA ± 2 standard deviations.

Volume Ratios (2 features) — Today's volume divided by the 5-day and 20-day average volume. A ratio of 2.5 means 2.5x normal volume — high volume on a price move confirms the trend; high volume on a reversal signals a potential turning point.

Volatility (2 features) — Rolling standard deviation of daily returns over 5 and 20 days, expressed as a percentage. Higher volatility means larger expected daily moves. The model uses this to gauge the “normal” noise level for the stock.

Candlestick Ratios (4 features) — Body: (close − open) / close × 100. Upper shadow: (high − max(open,close)) / (high − low). Lower shadow: (min(open,close) − low) / (high − low). Range: (high − low) / close × 100. These encode daily price action patterns that technical analysts have used for centuries.

52-Week Position (2 features) — Distance from the 252-day high and low as percentages. A stock at −2% from its 52-week high is near resistance; at +40% from its 52-week low shows strong recovery. These capture long-range mean-reversion and breakout signals.

Day of Week (1 feature) — Monday=0 through Friday=4. Research shows small but measurable weekday effects in stock returns (e.g., the “Monday effect” and “Friday effect”).

A Random Forest builds 200 independent decision trees, each trained on a random bootstrap sample of the data with a random subset of features at each split. The final prediction is the average of all 200 trees.

How a decision tree works: Starting from all training data, it finds the single feature and threshold that best splits the data into two groups with different average returns. It repeats recursively, creating a tree of if/then rules. For example: “If RSI < 30 AND volume_ratio > 1.5 AND return_5d < −3%, predict +0.8% return.”

Why “random”: Each tree sees only ~63% of the data (bootstrap sampling) and considers only a random subset of features at each split. This forces trees to be diverse — they make different errors, and averaging them cancels out individual mistakes (variance reduction).

Our configuration: n_estimators=200, max_depth=8, min_samples_leaf=5, n_jobs=-1, random_state=42. Max depth 8 limits tree complexity to prevent memorizing noise. Min 5 samples per leaf ensures predictions are based on at least 5 historical examples.

Output: Provides feature importance rankings — features used more often at the top of trees (where splits are most impactful) get higher importance scores.

Similar to RandomForest but with an additional layer of randomization: at each split, instead of finding the optimal threshold, it picks a random threshold for each candidate feature and uses the best among those random splits.

Why this helps: Random thresholds make the model even less likely to overfit to specific data points. The model trades a tiny bit of accuracy on the training data for better generalization to unseen data. In practice, ExtraTrees often outperforms RandomForest on noisy data like stock returns.

Same configuration: 200 trees, max depth 8, min 5 samples per leaf. Also provides feature importance rankings.

A linear model that finds the best weighted combination of all features: prediction = w1×feature1 + w2×feature2 + ... + bias. The weights are learned by minimizing the squared prediction error on training data.

L2 Regularization (alpha=1.0): Adds a penalty term alpha × sum(weights²) to the loss function. This penalizes large weights, preventing the model from relying too heavily on any single feature. Without regularization, linear models can assign huge weights to noisy features and overfit badly.

Limitations: Cannot capture non-linear patterns. If the relationship between RSI and returns is U-shaped (both very high and very low RSI predict reversals), Ridge cannot model this. However, its simplicity is also a strength — it won't hallucinate complex patterns that don't exist.

Interpretation: Each weight directly tells you how much a 1-unit increase in the (standardized) feature changes the predicted return. Positive weight = bullish signal; negative = bearish.

A non-parametric, instance-based model. It doesn't learn explicit rules — instead, for each new prediction, it finds the 20 most similar historical trading days (by Euclidean distance across all selected features) and computes a distance-weighted average of their actual returns.

How it works: Imagine a trading day where RSI is 35, volume ratio is 2.1, and the stock is 5% below SMA50. KNN scans all historical days, finds the 20 closest matches to this feature profile, and predicts the average of what actually happened next on those 20 historical days. Closer matches get higher weight (distance-weighted).

Our configuration: n_neighbors=20, weights=“distance”. 20 neighbors balances having enough data points for a stable average vs being specific enough to capture local patterns.

Strengths: Makes no assumptions about the data distribution, naturally adapts to local patterns, and is intuitive to understand. Weakness: Suffers from the “curse of dimensionality” — in high-dimensional feature spaces, all points become roughly equidistant, which is why feature selection (reducing from 34 to 18 features) significantly helps KNN.

A deep learning recurrent neural network specifically designed for sequential data. Unlike the other models which see each day independently, LSTM processes the last 30 trading days as an ordered sequence, learning temporal patterns (e.g., “when this pattern of features appears across 5 consecutive days, a reversal follows”).

Architecture:

• LSTM Layer 1: 64 hidden units, returns sequences to next layer
• Dropout: 20% of neurons randomly deactivated during training (prevents overfitting)
• LSTM Layer 2: 32 hidden units, returns final hidden state
• Dropout: 20%
• Dense Output: 1 unit (predicted return)

How LSTM cells work: Each LSTM cell has 3 “gates” (forget, input, output) that control information flow. The forget gate decides what to discard from the cell state (e.g., “the price action from 25 days ago is no longer relevant”). The input gate decides what new information to store. The output gate decides what to pass to the next time step. These gates are learned during training.

Training details: Adam optimizer, MSE loss, 20 epochs, batch size 32. The target (return) is scaled separately with its own StandardScaler and inverse-transformed after prediction. The 80/20 train/validation split ensures the last 20% of the time series is used for evaluation.

Why separate target scaling: LSTM is sensitive to input scales. Feature scaling alone isn't enough — the target returns must also be scaled so the network's output neurons operate in a numerically stable range. The inverse transform recovers the actual percentage return.

A gradient boosted tree ensemble that builds 300 trees sequentially, where each new tree is trained to correct the errors (residuals) of all previous trees combined. This is fundamentally different from RandomForest, which builds trees independently.

How boosting works: Tree 1 makes predictions. Tree 2 is trained on the errors of Tree 1. Tree 3 is trained on the remaining errors after Trees 1+2. Each tree adds a small correction, and the cumulative effect builds a powerful predictor from many simple ones.

Regularization: L1 (alpha=0.1) promotes sparsity — some features get zero weight. L2 (lambda=1.0) penalizes large leaf values. 80% row subsampling and 80% column subsampling at each tree introduce randomness similar to RandomForest. Learning rate 0.05 means each tree contributes only 5% correction, requiring many trees but reducing overfitting.

Our configuration: n_estimators=300, max_depth=6, learning_rate=0.05, subsample=0.8, colsample_bytree=0.8, reg_alpha=0.1, reg_lambda=1.0.

Why XGBoost dominates: It's the most successful algorithm in machine learning competitions for tabular data. The sequential error-correction mechanism is particularly effective at finding subtle patterns that random forests miss.

Another gradient boosted tree, but with two key algorithmic innovations that make it 10–20x faster than XGBoost while maintaining comparable or better accuracy.

Leaf-wise growth: Instead of growing trees level-by-level (like XGBoost), LightGBM grows leaf-wise — it always splits the leaf with the highest potential gain. This creates deeper, more asymmetric trees that reach better accuracy with fewer splits.

Histogram binning: Continuous feature values are bucketed into 256 bins. Instead of evaluating every possible split threshold, only 256 boundaries are tested. This dramatically reduces computation time and also acts as a form of regularization (smoothing).

Our configuration: n_estimators=300, max_depth=6, learning_rate=0.05, subsample=0.8, colsample_bytree=0.8, reg_alpha=0.1, reg_lambda=1.0. Same hyperparameters as XGBoost for fair comparison.

A gradient boosted tree with ordered boosting — a technique that addresses a subtle form of overfitting called “prediction shift” that affects all other gradient boosting implementations.

The prediction shift problem: In standard boosting, when computing the residual for training example #500, the model has already been influenced by example #500 during earlier trees. This creates a slight data leakage — the model's errors on training data are systematically smaller than on new data. Over hundreds of trees, this bias accumulates.

Ordered boosting solution: CatBoost uses a random permutation of the data and, for each example, only uses preceding examples (in the permutation) to compute the model's prediction. This is analogous to time-series cross-validation applied within each boosting iteration.

Our configuration: iterations=300, depth=6, learning_rate=0.05, l2_leaf_reg=3.0. CatBoost tends to produce the best out-of-the-box results with less hyperparameter tuning needed.

A feedforward neural network (also called a “fully connected” or “dense” network). Unlike the LSTM which processes data as a time sequence, the MLP sees each day as an independent set of features — similar to the tree and linear models, but with the ability to learn complex non-linear relationships.

Architecture: Two hidden layers with 128 and 64 neurons respectively. Each neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a ReLU activation function (max(0, x)). ReLU introduces non-linearity — allowing the network to learn curved decision boundaries that linear models cannot.

Training: Uses the Adam optimizer (adaptive learning rate per parameter) with an initial learning rate of 0.001 that automatically reduces when progress stalls (adaptive schedule). L2 regularization (alpha=0.001) penalizes large weights. Early stopping monitors validation loss and halts training if no improvement for 10 consecutive iterations, preventing overfitting.

Our configuration: hidden_layer_sizes=(128, 64), activation=relu, solver=adam, max_iter=200, early_stopping=True, validation_fraction=0.15, n_iter_no_change=10, alpha=0.001.

vs LSTM: MLP treats each day independently (no sequential memory), which makes it faster to train but unable to learn temporal patterns like “3 consecutive up days followed by high volume.” It complements LSTM by capturing static feature interactions that don't require sequential context.

A probabilistic sequence model that detects hidden market “regimes” (Bull, Bear, Sideways) from observable price data. Unlike the other models which predict returns directly, HMM first classifies the current market state, then uses learned transition probabilities to forecast the most likely future state and its associated return.

How it works: The model assumes the market is always in one of 3 hidden states, each with its own characteristic return distribution (mean and variance). It observes 4 features — daily log returns, 5-day and 20-day rolling volatility, and volume ratio — and uses the Baum-Welch algorithm (Expectation-Maximization) to learn the state distributions and transition probabilities from historical data.

Prediction: The Viterbi algorithm decodes the most likely regime sequence, identifying the current state. The learned transition matrix then gives the probability of moving to each state tomorrow. The predicted return is the probability-weighted average of each state's mean return. For 5-day and 20-day horizons, the transition matrix is raised to the Nth power to compute multi-step transition probabilities.

State labeling: The 3 states are automatically labeled by their mean returns: highest mean = Bull, lowest = Bear, middle = Sideways. This labeling is data-driven and adapts to each stock's characteristics.

Our configuration: n_components=3, covariance_type=diag, n_iter=200. Diagonal covariance assumes features are conditionally independent given the state — a reasonable simplification that prevents overfitting with limited data.

Unique value: HMM is the only model in the ensemble that explicitly models market regime transitions. While tree and linear models treat each day independently, HMM captures the persistence of market conditions (“bull markets tend to continue”) and the probabilities of regime shifts.

An R² of −0.05 does not mean the model is useless. It means the model explains −5% of the variance in returns — slightly worse than simply predicting “0% change every day.” But predicting exact returns is not the goal — predicting direction (up vs down) can still be profitable even with negative R².

Consider: a model predicts +0.5% when the stock goes up 2%, and −0.3% when it goes down 1.5%. The magnitudes are wrong (negative R²), but the direction is correct both times. This is why we track direction accuracy separately from R².

Academic research consistently shows R² values of 0.01–0.05 (1–5%) for daily stock return prediction, even for state-of-the-art models. Our negative values reflect honest evaluation with proper temporal cross-validation — many commercial “AI trading” products show inflated metrics due to look-ahead bias.