Hands-On AI Trading with Python, QuantConnect, and AWS

JIRI PIK: Founder and CEO of RocketEdge.com. A software architect and cloud computing expert, Jiri Pik specializes in designing high-performance trading systems. He has decades of experience in financial technologies and has worked with some of the world’s leading financial institutions, including Goldman Sachs and JPMorgan Chase. ERNEST P. CHAN: A pioneer in applying machine learning to quantitative trading, Ernest P. Chan founded Predictnow.ai and QTS Capital Management. He is author of books such as Quantitative Trading and Machine Trading. JARED BROAD: Founder and CEO of QuantConnect™, Jared Broad has empowered over 300,000 algorithmic traders worldwide with a platform that simplifies strategy design, backtesting, and live deployment. PHILIP SUN: CEO and Co-founder of Adaptive Investment Solutions, LLC, and a seasoned quantitative fund manager, Philip Sun and his team focus on building state-of-the-art AI-driven risk management platform for wealth advisors and institutional investors. VIVEK SINGH: A product leader at Amazon Web Services (AWS), Vivek Singh spearheads the development of large language models (LLMs) and Generative AI applications, bringing cutting-edge AI technologies to the trading domain.

• Biographies xiiiPreface: QuantConnect xvIntroduction xxiiiPart I Foundations of Capital Markets and Quantitative Trading 1Chapter 1 Foundations of Capital Markets 3Market Mechanics 3Market Participants 4Trading Is the “Play” 4The Stage and Basic Rules of Trading—The Limit Order Book 4Actors—Liquidity Trader, Market Maker, andInformed Trader 5Liquidity Trader 5Market Maker 5Informed Trader 6AI Actors Wanted! 7Data and Data Feeds 7Custom and Alternative Data 9Brokerages and Transaction Costs 10Transaction Costs 11Security Identifiers 13Assets and Derivatives 15US Equities 15US Equity Options 19Index Options 21US Futures 21Cryptocurrency 23Chapter 2 Foundations of Quantitative Trading 25Research Process 25Research 25Backtesting 26Parameter Optimization 26Paper and Live Trading 26Testing and Debugging Tools 26Debuggers 27Logging 27Charting 27Object Store 28Coding Process 28Time and Look-ahead Bias 29Look-ahead Bias 29Market Hours and Scheduling 30Strategy Styles 30Trading Signals 31Allocating Capital 31Regimes and Portfolios of Strategies 32Parameter Sensitivity Testing and Optimization 331. Remove 332. Replace 343. Reduce 34Parameter Sensitivity Testing 34Margin Modeling 35Equities 35Equity Options 36Futures 37Diversification and Asset Selection 37Fundamental Asset Selection 38ETF Constituents Asset Selection 39Dollar-Volume Asset Selection 40Universe Settings 40Indicators and Other Data Transformations 41Automatic Indicators 41Manual Indicators 41Indicator Warm Up 42Storing Objects 42Indicator Events 42Sourcing Ideas 42Hypothesis-driven Testing 43Data Driven Investing 44Quantpedia 44QuantConnect Research and Strategy Explorer 45Part II Foundations of AI and ML in Algorithmic Trading 47Step-by-step Guide for AI-based Algorithmic Trading 48Chapter 3 Step 1: Problem Definition 49Chapter 4 Step 2: Dataset Preparation 53Data Collection 53Exploratory Data Analysis 53Data Preprocessing 54Handling Missing Data 55Handling Outliers 58Feature Engineering 61Normalization and Standardization of Features 62Transforming Time Series Features to Stationary 64Identification of Cointegrated Time Series with Engle-Granger Test 70Feature Selection 76Correlation Analysis 76Feature Importance Analysis 77Auto-identification of Features 78Dimensionality Reduction/Principal Component Analysis 80Splitting of Dataset into Training, Testing, and Possibly Validation Sets 83How to Split Your Data 83Chapter 5 Step 3: Model Choice, Training, and Application 87Regression 88Linear Regression 89Polynomial Regression 91LASSO Regression 93Ridge Regression 96Markov Switching Dynamic Regression 99Decision Tree Regression 103Support Vector Machines Regression withWavelet Forecasting 105Classification 110Multiclass Random Forest Model 110Logistic Regression 114Hidden Markov Models 117Gaussian Naive Bayes 119Convolutional Neural Networks 122Ranking 127LGBRanker Ranking 127Clustering 130OPTICS Clustering 130Language Models 132OpenAI Language Model 132Amazon Chronos Model 135FinBERT Model 137Part III Advanced Applications of AI in Trading and Risk Management 141Getting Started with Source Code 141Chapter 6 Applied Machine Learning 143Example 1—ML Trend Scanning with MLFinlab 143Example 2—Factor Preprocessing Techniques for Regime Detection 148Example 3—Reversion vs. Trending: Strategy Selection by Classification 154Example 4—Alpha by Hidden Markov Models 158Example 5—FX SVM Wavelet Forecasting 170Example 6—Dividend Harvesting Selection ofHigh-Yield Assets 176Example 7—Effect of Positive-Negative Splits 181Example 8—Stop Loss Based on Historical Volatility and Drawdown Recovery 185Example 9—ML Trading Pairs Selection 197Example 10—Stock Selection through ClusteringFundamental Data 207Example 11—Inverse Volatility Rank and Allocate to Future Contracts 214Example 12—Trading Costs Optimization 221Example 13—PCA Statistical Arbitrage Mean Reversion 228Example 14—Temporal CNN Prediction 233Example 15—Gaussian Classifier for Direction Prediction 242Example 16—LLM Summarization of Tiingo News Articles 250Example 17—Head Shoulders Pattern Matching with CNN 256Example 18—Amazon Chronos Model 265Example 19—FinBERT Model 272Chapter 7 Better Hedging with Reinforcement Learning 281Introduction 281A New AI Trading Assistant 281Continuous Hedging Is Not Required 282Machine Learning Comes to the Rescue 283A Simplified but Effective ReinforcementLearning Approach 284Overview of the Reinforcement Learning 285Identification 285Simulation 286Ref inement Training on Actual Market Data 287Testing and Implementation 287Implementation on QuantConnect 288Primary Research Notebook 289The Policy Network 290Model Functions 292Fine-tuning with Market Data 296Results 300Conclusion 303Chapter 8 AI for Risk Management and Optimization 305What Is Corrective AI and ConditionalParameter Optimization? 305Feature Engineering 308Applying Corrective AI to Daily Seasonal Forex Trading 312What Is Conditional Parameter Optimization? 318Applying Conditional Parameter Optimization to an ETF Strategy 319Unconditional vs. Conditional Parameter Optimizations 320Performance Comparisons 322Conditional Portfolio Optimization 322Regime Changes Obliterate Traditional Portfolio Optimization Methods 322Learning to Optimize 324Ranking Is Easier Than Predicting 325The Fama-French Lineage 327Comparison with Conventional Optimization Methods 327Model Tactical Asset Allocation Portfolio 331CPO Software-as-a-Service 333Conclusion 340Definitions of Spread_EMA & Spread_VAR 340Chapter 9 Application of Large Language Models and Generative AI in Trading 341Role of Generative AI in Creating Alpha 341Selecting an LLM for Building a Generative AI Application 342Prompt Engineering 344Prompt Engineering in Practice 345Addressing Model “Hallucination” 346Question Answering Using a Retrieval Augmented Application in SageMaker Canvas 347RAG Application Costs and Optimization Techniques 350Testing Our Infrastructure 351Summarization 356Useful AI Platforms and Services 359ChatGPT 359Gemini 359Bedrock 359SageMaker 359Q Business 360References 361Subject Index 363Code Index 379