51enra · May 18, 2024 04:33 · Feb 11, 2023 · Feb 11, 2023 · Feb 11, 2023 · Feb 11, 2023
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1297,5 +1297,12 @@ Sources: e.g. academic research papers
 * *Factor rank autocorrelation (FRA)*: Spearman autocorrelation of factor coefficients at times t-1 and t; corr(rk(α_t-1),rk(α_t)). If the FRA is high, it means that the ranks of the individual assets don't change much and is thus indicative of a low turnover.
 * "If two alpha factors have similar sharpe ratios, similar quintile performance and similar factor returns, we will prefer the one with a lower turnover ratio / higher FRA". Otherwise, if an alpha factor has high turnover, it must be seen whether the return survives backtesting, paper trading and real trading. Low turnover makes it more feasible to execute trades when stocks are illiquid and to offset transaction costs.
 
+**Quantile Analysis**
+* To get a more detailed picture of the performance of an alpha factor, quantile analysis can be used.
+* Sub-divide the whole considered stock universe in quantiles according to their ranked alpha value.
+* For each quantile, analyse return and variance/volatility over a time period, e.g. 1 or 3 years.
+* Ideally, there should be a monotonic relationship between quantiles and their (risk-adjusted) returns. In specific cases, only the highest and lowest quantile returns may be significantly different from zero or there may be a "middle ground" in the low quantiles and another one in the high quantiles that bear most of the negative/positive returns. It should be considered whether there is a rational reason for this or whether it shows some arbitrariness of the predictive power of the alpha factor. In any case, if the alpha factor is predictive only for a subset of stocks in the universe, this corresponds to a higher risk.
+* Richard G. Sloan (1996), Do Stock Prices Fully Reflect Information in Accruals and Cash Flows About Future Earnings? (Not freely available; only [mindmap](https://coggle.it/diagram/XvdM2xYkPOATYZSo/t/do-stock-prices-fully-reflect-by-richard-g-sloan) or [summary](https://paperroni.com/do-stock-prices-fully-reflect-information-by-sloan/). Message: If quarterly earnings are to a larger degree determined by *accruals* (e.g. accounts receivable) and a lesser degree by *cash flow from operations (CFO)*, this is indicative of a lower "quality" and potentially lower future earnings. This fact is (was) not fully accounted for by stock prices, so that stocks may show out-/underperformance following future earnings reports depending on the relation of CFO to accruals in the current report. "The strategy he (Sloan) described is likely not used due to the high information acquisition costs and operational difficulties."
+* Academic researchers often investigate "their" alpha factors different forom practitioners. They look for "broadly applicable" market phenoma by mostly focusing on the outermost quantiles of non-ranked alpha values. To generate trade decisions for each stock in a portfolio, the alpha factor must be applied to the entire portfolio. Thus, practitioners will have to extend the academic studies before applying those factors.
 
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1294,5 +1294,8 @@ Sources: e.g. academic research papers
 * *Liquidity*: Proxy - *Bid-ask spread*. For highly liquid stocks in mature markets, it is around 3-5 basis points (1 basis point = 1/100 %). For less liquid / illiquid stocks, bid-ask spread may be 20-30 basis points (for a single trade); large compared to an annual return of around 400 basis points of a "good" alpha model.  
 * *Transaction Costs*: For large trading firms, *commissions* are a rather negligible part of total transaction costs. Much more significant: *market impact*. To minimize market impact, trades can be spread over multiple days, but this entails the risk of price movement.
 * *Turnover*: Liquidity and transaction costs are hard to determine for the evaluation of an alpha strategy. As a proxy, turnover can be compared. Turnover is calculated as the sum of absolute weight changes of all portfolio components from one day to the next, turnover(t1, t2) = sum_k(|x_(t1,k) - x_(t2,k)|)
+* *Factor rank autocorrelation (FRA)*: Spearman autocorrelation of factor coefficients at times t-1 and t; corr(rk(α_t-1),rk(α_t)). If the FRA is high, it means that the ranks of the individual assets don't change much and is thus indicative of a low turnover.
+* "If two alpha factors have similar sharpe ratios, similar quintile performance and similar factor returns, we will prefer the one with a lower turnover ratio / higher FRA". Otherwise, if an alpha factor has high turnover, it must be seen whether the return survives backtesting, paper trading and real trading. Low turnover makes it more feasible to execute trades when stocks are illiquid and to offset transaction costs.
+
 
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1275,4 +1275,24 @@ Sources: e.g. academic research papers
 * The **Spearman Correlation** r_S(X,Y) = Cov(rk(X), rk(Y))/(Stddev(rk(X))*Stddev(rk(Y))) where rk(X) is the vector containing the ranked X components (i.e. ordinal numbers between 1 and the dimension of X).
 * The "standard" correlation is called the **Pearson Correlation** ρ(X,Y) = Cov(X,Y)/(Stddev(X)*Stddev(Y)).
 * The Pearson Correlation also equals the square root of the R^2 value of a linear regression between X and Y. The R^2 value measures the propoportion of the variance in Y that can be explained by the variance in X.
-* In alphalens: [alphalens.performance.factor_information_coefficient](https://quantopian.github.io/alphalens/alphalens.html?highlight=factor_information_coefficient#alphalens.performance.factor_information_coefficient) ([code link](https://github.com/quantopian/alphalens/blob/master/alphalens/performance.py))
+* We use the Spearman instead of the Pearson correlation because we are only interested whether the alpha factor predicts the right direction. If the direction is always right but the magnitude of the return varies differently than the alpha factor, this reduces the Pearson correlation but would not mean that the alpha factor is less suitable.
+* In alphalens: [*factor_information_coefficient*](https://quantopian.github.io/alphalens/alphalens.html?highlight=factor_information_coefficient#alphalens.performance.factor_information_coefficient) ([code link](https://github.com/quantopian/alphalens/blob/master/alphalens/performance.py))
+* **Example:** *rank_ic.ipynb*; applying the alphalens information coefficient function to the two factors from the pipline already used above.
+
+**Information Ratio**
+* Recap: *Risk Factors* are considered to drive portfolio variance; *Alpha Factors* are considered to drive returns *after neutralizing the risk factors*. Return driven by risk factors = *systematic return*, return driven by alpha factors = *specific return* (also: *residual return*). r = sum_k(β_k*f_k) + s.
+* **Information Ratio** IR = sqrt(252)* mean(s)/Stdev(s), where s is the specific return. "It's the Sharpe ratio of the performance that the fund manager (responsible for the alpha) contributes to the portfolio." *If* we are building a market and common factor *neutral* portfolio, the portfolio's Sharpe ratio equals the IR!
+
+**Fundamental Law of Active Management** (Richard Grinold, Ronald Kahn)
+* IR = IC * sqrt(B)
+* Information Ratio IR, Information Coefficient IC (see above; ranked IC), Breadth B (the number of *independent* applications of the strategy under invetigation; e.g. in different markets or at multiple times)
+* The formula is a strong simplification and should not be applied directly. Implies that IR can be increased by more active trading. Ignores e.g. transaction costs. Only considers a single isolated strategy, while multiple strategies can benefit from adding a low or negative IR, uncorrelated strategy.
+* [Corporate Finance Institue](https://corporatefinanceinstitute.com/resources/capital-markets/fundamental-law-of-active-management/), [Breaking Down Finance](https://breakingdownfinance.com/trading-strategies/law-of-active-management/)
+* "the IC for even the best quants in the world is relatively not very high. In fact, you might be surprised to learn that the count of profitable trades as a ratio to all trades (a measure similar in spirit to IC) by the best, well-known quants is typically just a bit higher than 50%. And in general, great discretionary investors have the advantage over great quants when it comes to IC. Quants however, have an advantage when it comes to breadth."
+
+**Real World Constraints**
+* *Liquidity*: Proxy - *Bid-ask spread*. For highly liquid stocks in mature markets, it is around 3-5 basis points (1 basis point = 1/100 %). For less liquid / illiquid stocks, bid-ask spread may be 20-30 basis points (for a single trade); large compared to an annual return of around 400 basis points of a "good" alpha model.  
+* *Transaction Costs*: For large trading firms, *commissions* are a rather negligible part of total transaction costs. Much more significant: *market impact*. To minimize market impact, trades can be spread over multiple days, but this entails the risk of price movement.
+* *Turnover*: Liquidity and transaction costs are hard to determine for the evaluation of an alpha strategy. As a proxy, turnover can be compared. Turnover is calculated as the sum of absolute weight changes of all portfolio components from one day to the next, turnover(t1, t2) = sum_k(|x_(t1,k) - x_(t2,k)|)
+
+
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1206,16 +1206,16 @@ Sources: e.g. academic research papers
 * **Raw alpha factor**: a version of an alpha factor before additional processing to improve the signal.
 * **Stock universe**: set of stocks under consideration for the portfolio.
 
-####Methods for coming up with alpha factors
+#### Methods for coming up with alpha factors
 * Reading financial news, studying the markets for curious behavior, studying the methods of famous discretionary or quant investors, talking to industry practitioners, academic papers.
 * Academic papers are publicly available, so any information contained is diffused and doesn't lead to strong signals. However, it can aid in the generation of own ideas and serves as a basis for comparison.
 * Free access only to non-peer reviewed papers from open-source libraries as SSRN. Advantage: papers are more recent and the ideas are still "fresh". Academic papers must be assessed for practical applicability, e.g. do I have access to the data used, is the model still feasible after transaction costs, do they excessively rely on returns of illiquid micro-cap stocks, etc.
 
-####Controlling for risk within an alpha factor
+#### Controlling for risk within an alpha factor
 * **Market risk**: With the simplifying assumption that all individual stock beta factors are equal to one, the market risk is already eliminated by creating mean-zero alpha vectors (long and short positions neutralize each other).
 * **Sector risk**: After market risk elimination, assign each stock in the portfolio to a sector and then subtract the mean over all sector stocks from the alpha value of each sector stock. This (mostly) eliminates sector risk.
 
-####Processing techniques for alpha factors 
+#### Processing techniques for alpha factors 
 
 **Ranking**
 * If the alpha vector changes daily, we would have to rebalance the portfolio daily.
@@ -1241,7 +1241,7 @@ Sources: e.g. academic research papers
 [**zipline Simple Moving Average**](https://zipline.ml4trading.io/api-reference.html?highlight=simple%20moving%20average#zipline.pipeline.factors.SimpleMovingAverage)
 [**zipline Exponential Weighted Moving Average**](https://zipline.ml4trading.io/api-reference.html?highlight=exponential%20weighted%20moving#zipline.pipeline.factors.ExponentialWeightedMovingAverage)
 
-####Evaluation metrics for alpha factors
+#### Evaluation metrics for alpha factors
 
 [Quantopian alphalens python code for performance evaluation](https://github.com/quantopian/alphalens/blob/master/alphalens/performance.py) (Same github repository as zipline pyfolio ("portfolio and risk analysis in Python"), trading_calendars,...)  
 [Alphalens documentation](https://quantopian.github.io/alphalens/alphalens.html)
@@ -1272,6 +1272,7 @@ Sources: e.g. academic research papers
 * Correlation between the alpha vector (prediction) and actual forward return:
 * rank_IC_(t-1) = r_S(α_(t-1), r_(t, t+1))
 * where r_S is the *Spearman Correlation Coefficient*, α_(t-1) the alpha vector for time step t-1, and r_(t, t+1) the forward return in the time interval from t to t+1.
-* The **Spearman Correlation Coefficient** r_S(X,Y) = Cov(rk(X), rk(Y))/(Stddev(rk(X))*Stddev(rk(Y))) where rk(X) is the vector containing the ranked X components (i.e. ordinal numbers between 1 and the dimension of X).
-* The "standard" correlation is called the **Pearson Correlation Coefficient** ρ(X,Y).
-* The Pearson Correlation Coefficient also equals the square root of the R^2 value of a linear regression between X and Y. The R^2 value measures how much of the variance in Y can be explained by the variance in X. 
+* The **Spearman Correlation** r_S(X,Y) = Cov(rk(X), rk(Y))/(Stddev(rk(X))*Stddev(rk(Y))) where rk(X) is the vector containing the ranked X components (i.e. ordinal numbers between 1 and the dimension of X).
+* The "standard" correlation is called the **Pearson Correlation** ρ(X,Y) = Cov(X,Y)/(Stddev(X)*Stddev(Y)).
+* The Pearson Correlation also equals the square root of the R^2 value of a linear regression between X and Y. The R^2 value measures the propoportion of the variance in Y that can be explained by the variance in X.
+* In alphalens: [alphalens.performance.factor_information_coefficient](https://quantopian.github.io/alphalens/alphalens.html?highlight=factor_information_coefficient#alphalens.performance.factor_information_coefficient) ([code link](https://github.com/quantopian/alphalens/blob/master/alphalens/performance.py))
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1268,4 +1268,10 @@ Sources: e.g. academic research papers
 * Key performance indicator to compare alpha factors. "Naked" returns can always be amplified by leveraging, but the sharpe ratio is independent of leverage.
 * S = mean(f)/stdev(f) * sqrt(252)  where f are the daily factor returns (**Note:** risk-free rate is neglected here!)
 
-
+**Ranked Information Coefficient (Rank IC)**:
+* Correlation between the alpha vector (prediction) and actual forward return:
+* rank_IC_(t-1) = r_S(α_(t-1), r_(t, t+1))
+* where r_S is the *Spearman Correlation Coefficient*, α_(t-1) the alpha vector for time step t-1, and r_(t, t+1) the forward return in the time interval from t to t+1.
+* The **Spearman Correlation Coefficient** r_S(X,Y) = Cov(rk(X), rk(Y))/(Stddev(rk(X))*Stddev(rk(Y))) where rk(X) is the vector containing the ranked X components (i.e. ordinal numbers between 1 and the dimension of X).
+* The "standard" correlation is called the **Pearson Correlation Coefficient** ρ(X,Y).
+* The Pearson Correlation Coefficient also equals the square root of the R^2 value of a linear regression between X and Y. The R^2 value measures how much of the variance in Y can be explained by the variance in X. 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -6,6 +6,7 @@
 * [Quantdare](https://quantdare.com/)
 * [Quantrocket (tutorials, zipline, data)](https://www.quantrocket.com/code/)
 * [Zipline documentation (Stephan Jansen)](https://zipline.ml4trading.io/index.html) & link to github
+* [Zipline tutorial (Stephan Jansen)](https://zipline.ml4trading.io/beginner-tutorial)
 * [Alphalens](https://quantopian.github.io/alphalens/index.html) ("Alphalens is a Python Library for performance analysis of predictive (alpha) stock factors.")
 
 * OHLC, ticks, timeframes

diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1,10 +1,12 @@
 # Udacity AI for Trading Notes
 
 ## 1. Concepts
-[Nassim Taleb's home page](https://www.fooledbyrandomness.com/)
+* [Nassim Taleb's home page](https://www.fooledbyrandomness.com/)
 
-[Quantdare](https://quantdare.com/)
-[Quantrocket (tutorials, zipline, data)](https://www.quantrocket.com/code/)
+* [Quantdare](https://quantdare.com/)
+* [Quantrocket (tutorials, zipline, data)](https://www.quantrocket.com/code/)
+* [Zipline documentation (Stephan Jansen)](https://zipline.ml4trading.io/index.html) & link to github
+* [Alphalens](https://quantopian.github.io/alphalens/index.html) ("Alphalens is a Python Library for performance analysis of predictive (alpha) stock factors.")
 
 * OHLC, ticks, timeframes
 * Volume. Higher volume at beginning ('price discovery') and end than in the middle of the day.

diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -4,6 +4,7 @@
 [Nassim Taleb's home page](https://www.fooledbyrandomness.com/)
 
 [Quantdare](https://quantdare.com/)
+[Quantrocket (tutorials, zipline, data)](https://www.quantrocket.com/code/)
 
 * OHLC, ticks, timeframes
 * Volume. Higher volume at beginning ('price discovery') and end than in the middle of the day.
@@ -1189,8 +1190,10 @@ Two-dimensional, correlated data set for demonstrating the PCA class of scikit-l
 
 Alpha factors are parameters expected to be predicitve of future mean returns. They build on *deviations from the efficient market hypothesis* and take advantage of *mispricing*.  
 Sources: e.g. academic research papers  
-Processing techniques to turn alpha vectors into signals: Sector neutralization, ranking, Z-scoring, smoothing, conditiong  
-Evaluation techniques: Sharpe ratio, information coefficient, information ratio, turnover analysis
+
+ **Processing techniques** to turn alpha vectors into signals (explained in more detail below): Sector neutralization, ranking, Z-scoring, smoothing, conditiong  
+
+ **Evaluation Metrics for alpha factors**: Factor returns, Sharpe ratio, information coefficient, information ratio, quantile analysis, turnover analysis
 
 * **Alpha Factors vs. Risk Factors**: Risk factors model the movement of stock prices due to commonly known effects, providing systematic returns and systematic volatility. They may significantly impact portfolio volatility without providing adequate returns. In the world of alpha factors, the portfolio exposure to risk factors is to be *neutralized*, so that the remaining portfolio returns and volatility are due to alpha factors.
 * **Alpha Model**: An algorithm that transforms data into numbers associated with each stock.
@@ -1200,19 +1203,66 @@ Evaluation techniques: Sharpe ratio, information coefficient, information ratio,
 * **Raw alpha factor**: a version of an alpha factor before additional processing to improve the signal.
 * **Stock universe**: set of stocks under consideration for the portfolio.
 
-**Methods for coming up with alpha factors**
+####Methods for coming up with alpha factors
 * Reading financial news, studying the markets for curious behavior, studying the methods of famous discretionary or quant investors, talking to industry practitioners, academic papers.
 * Academic papers are publicly available, so any information contained is diffused and doesn't lead to strong signals. However, it can aid in the generation of own ideas and serves as a basis for comparison.
 * Free access only to non-peer reviewed papers from open-source libraries as SSRN. Advantage: papers are more recent and the ideas are still "fresh". Academic papers must be assessed for practical applicability, e.g. do I have access to the data used, is the model still feasible after transaction costs, do they excessively rely on returns of illiquid micro-cap stocks, etc.
 
-**Controlling for risk within an alpha factor**
+####Controlling for risk within an alpha factor
 * **Market risk**: With the simplifying assumption that all individual stock beta factors are equal to one, the market risk is already eliminated by creating mean-zero alpha vectors (long and short positions neutralize each other).
 * **Sector risk**: After market risk elimination, assign each stock in the portfolio to a sector and then subtract the mean over all sector stocks from the alpha value of each sector stock. This (mostly) eliminates sector risk.
 
+####Processing techniques for alpha factors 
+
 **Ranking**
 * If the alpha vector changes daily, we would have to rebalance the portfolio daily.
 * Outliers may lead to portfolio imbalances. Countermeasure: *winsorizing*, i.e. all alpha values smaller than the fifth percentile are replaced by the value at the fifth percentile; correspondingly above the 95th percentile. Also possible: setting a maximum allowed portfolio weight for any single stock.
-* **Ranking**: (General method in statistics) Transform the alpha signal to be less sensitive to noise and outliers and to avoid excessive trades. It is done by assigning an ordinal number (rank) to each stock accoring to its alpha value and only change the portfolio when the rank changes (too simplified???)
+* **Ranking**: (General method in statistics) *Transform the alpha signal to be less sensitive to noise and outliers*. It is done by assigning an ordinal number (rank) to each stock according to its alpha value.
+* Ranking could be applied to avoid excessive trades. The rank instead of the raw alpha factor could be used to only change the portfolio when the rank changes (too simplified???).
+
+[**zipline rank method**](https://zipline.ml4trading.io/api-reference.html?highlight=rank#zipline.pipeline.Factor.rank) for applying ranking to a factor.
+
+**Z-scoring**
+* Transform the raw alpha factor (numerical value) by subtracting the mean and dividing by the standard deviation. This makes alpha factors more readily comparable.
+* if the distribution resembles a Gaussian, 95% of the values would then be always between approximately -2 and 2, irrespective of the alpha factor.
+* Advantage compared to ranking: Even when comparing alpha factors between different stock universes (with different number of stocks), the value range would be the same, while for ranking, it is between 1 and the number of stocks.
+* Disadvantage: Doesn't protect against outliers and noise.
+* To combine the advantages, one could first rank and then Z-score when alpha vectors are generated from different stock universes.
+* Z-scoring of alpha factors is often applied in academic papers.
+[**zipline zscore method**](https://zipline.ml4trading.io/api-reference.html?highlight=zscore#zipline.pipeline.Factor.zscore)
+
+**Smoothing**
+* Financial data are noisy and can be sparse (sparse due to missing values or, e.g. for fundamental data, because they are only measured in longer intervals like quarterly or monthly)
+* To obtain smoother data, rolling window averaging or weighted moving averaging (more recent = more weight) can be applied
+* (In case of fundamental data, to get a daily time series, one would rather copy the monthly/quarterly data to all days in the subsequent month/quarter)
+[**zipline Simple Moving Average**](https://zipline.ml4trading.io/api-reference.html?highlight=simple%20moving%20average#zipline.pipeline.factors.SimpleMovingAverage)
+[**zipline Exponential Weighted Moving Average**](https://zipline.ml4trading.io/api-reference.html?highlight=exponential%20weighted%20moving#zipline.pipeline.factors.ExponentialWeightedMovingAverage)
 
+####Evaluation metrics for alpha factors
 
+[Quantopian alphalens python code for performance evaluation](https://github.com/quantopian/alphalens/blob/master/alphalens/performance.py) (Same github repository as zipline pyfolio ("portfolio and risk analysis in Python"), trading_calendars,...)  
+[Alphalens documentation](https://quantopian.github.io/alphalens/alphalens.html)
+
+"Alphalens is a Python Library for performance analysis of predictive (alpha) stock factors..."
+
+**Factor returns**
+* Use theoretical returns over *the same time window* and for *the same stock portfolio* to compare different alpha factors
+* "Theoretical portfolio returns" means: We adjust our portfolio daily according to the alpha vector calculated until the previous day. The alpha vector is *standardized* (mean 0, sum of components equal to 1; negative components=short). Use alpha vector values as portfolio weights to calculate the *single-day return* of the portfolio from the individual stock returns. The time series of daily returns is the factor return (based on historical data). 
+
+**Universe construction rule**
+* We must ensure that the stock universe to test our factors at a specific date in the past is only constructed based on information available up to that date to avoid *lookahead bias*. A specific example of lookahead bias is *survivorship bias* which would come into play if we selected the stock universe from currently existing assets. To avoid this, we could use historical constituent data from an index provider like S&P 500.
+
+**Factor returns and leverage**
+* Typical factor returns (i.e. the achieved "alpha") are in the range of 1% to 4%. In order to achieve the returns expected for a professional trading firm / hedge fund, leverage (borrowing money) with a leverage ratio between 2 to 6 is applied.
+* *Leverage ratio* = (value of positions)/((own) capital)
+* For alpha factor analysis, we always apply a leverage ratio of 1!
+* *Return denominator* = sum(|α_i|) where the α_i are the components of the alpha vector. For a *normalized* alpha vector, this is equal to 1 (i.e. the "raw" alpha vector is normalized by dividing each component by its return denominator)
+* Alphalens:
+  * [*factor_returns, factor_weights*](https://quantopian.github.io/alphalens/alphalens.html?highlight=factor_returns#alphalens.performance.factor_returns) with *demean=True*
+  * [*get_clean_factor_and_forward_returns*](https://quantopian.github.io/alphalens/alphalens.html#alphalens.utils.get_clean_factor_and_forward_returns)
+
+**Sharpe ratio**
+* Key performance indicator to compare alpha factors. "Naked" returns can always be amplified by leveraging, but the sharpe ratio is independent of leverage.
+* S = mean(f)/stdev(f) * sqrt(252)  where f are the daily factor returns (**Note:** risk-free rate is neglected here!)
+
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1169,4 +1169,50 @@ The portfolio variance of a two-asset portfolio is calculated using the explicit
 
 **Example**: `PCA_Toy_Problem.ipynb`.  
 Two-dimensional, correlated data set for demonstrating the PCA class of scikit-learn.
+
+**Example**: `pca_basics_solution.ipynb`.
+490 dim. US stock returns data from 2011 to 2016 obtained with zipline. Calculate number of PCA dimensions needed to cover a certain amount of variance. E.g. for 50%, 9 components are required. For 90%, 179 components are required.
+
+** PCA as a factor model:**
+* Calculate PCA components and decide on number of components to be kept = number of factors.
+* Factor model: **r** = **Bf** + **s**; here the factor weights **B** are given by the PCA components, written in the original basis (each stock's return series represents one basis vector), as column vectors. If we kept all components, **r** = **Bf** holds exactly and the "factor returns" matrix **f** is the matrix of eigenvectors. The factor returns are the "return time series" of a portfolio exactly representing a single factor. *Usually this factor cannot be interpreted as a 'real-world' property!*
+* Keeping track of dimensions: For N stocks, T timesteps (return values per stock), and K components, we have: dim(**r**) = NxT, dim(**B**) = NxK, dim(**f**) = KxT, dim(**s**) = NxT  (rows x columns).
+* **f** = **B**^T **r** (because the PCA components are orthonormal, **B**^T**B** = **1** if we keep all components)
+* cov(f) = 1/(T-1) **f**^T**f** is a *diagonal* matrix with the *factor variances* as diagonal components (i.e. the covariance of different factors is zero because the PCA components are orthogonal!)
+* If we use daily time series, we need to multiply by 252 to get the *annual variance* (Standard deviation is square root of variance!)
+* Ideosyncratic returns (not explained by model): **s** = **r** - **Bf**.
+* cov(**s**) = 1/(T-1) **s**^T**s**; The "specific risk" matrix (NxN) **S** is obtained by neglecting the off-diagonal elements of cov(**s**) (which should be small if the selected PCA components cover the most part of the original variance). It describes for each stock how much of its variance is *not* covered by the selected components.
+
+**Example:**
+
+### Alpha Factors
+
+Alpha factors are parameters expected to be predicitve of future mean returns. They build on *deviations from the efficient market hypothesis* and take advantage of *mispricing*.  
+Sources: e.g. academic research papers  
+Processing techniques to turn alpha vectors into signals: Sector neutralization, ranking, Z-scoring, smoothing, conditiong  
+Evaluation techniques: Sharpe ratio, information coefficient, information ratio, turnover analysis
+
+* **Alpha Factors vs. Risk Factors**: Risk factors model the movement of stock prices due to commonly known effects, providing systematic returns and systematic volatility. They may significantly impact portfolio volatility without providing adequate returns. In the world of alpha factors, the portfolio exposure to risk factors is to be *neutralized*, so that the remaining portfolio returns and volatility are due to alpha factors.
+* **Alpha Model**: An algorithm that transforms data into numbers associated with each stock.
+* **Alpha value**: refers to a single value for a single stock, for a single time period. Positive for long, negative for short.
+* **Alpha vector**: has a number (alpha value) for each stock, and the number is proportional to the amount of money we wish to allocate for each stock. The *mean* over all components equals zero and the *sum of absolute values* equals one.
+* **Alpha factor**: a time series of alpha vectors (over multiple time periods).
+* **Raw alpha factor**: a version of an alpha factor before additional processing to improve the signal.
+* **Stock universe**: set of stocks under consideration for the portfolio.
+
+**Methods for coming up with alpha factors**
+* Reading financial news, studying the markets for curious behavior, studying the methods of famous discretionary or quant investors, talking to industry practitioners, academic papers.
+* Academic papers are publicly available, so any information contained is diffused and doesn't lead to strong signals. However, it can aid in the generation of own ideas and serves as a basis for comparison.
+* Free access only to non-peer reviewed papers from open-source libraries as SSRN. Advantage: papers are more recent and the ideas are still "fresh". Academic papers must be assessed for practical applicability, e.g. do I have access to the data used, is the model still feasible after transaction costs, do they excessively rely on returns of illiquid micro-cap stocks, etc.
+
+**Controlling for risk within an alpha factor**
+* **Market risk**: With the simplifying assumption that all individual stock beta factors are equal to one, the market risk is already eliminated by creating mean-zero alpha vectors (long and short positions neutralize each other).
+* **Sector risk**: After market risk elimination, assign each stock in the portfolio to a sector and then subtract the mean over all sector stocks from the alpha value of each sector stock. This (mostly) eliminates sector risk.
+
+**Ranking**
+* If the alpha vector changes daily, we would have to rebalance the portfolio daily.
+* Outliers may lead to portfolio imbalances. Countermeasure: *winsorizing*, i.e. all alpha values smaller than the fifth percentile are replaced by the value at the fifth percentile; correspondingly above the 95th percentile. Also possible: setting a maximum allowed portfolio weight for any single stock.
+* **Ranking**: (General method in statistics) Transform the alpha signal to be less sensitive to noise and outliers and to avoid excessive trades. It is done by assigning an ordinal number (rank) to each stock accoring to its alpha value and only change the portfolio when the rank changes (too simplified???)
+
+
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1164,5 +1164,9 @@ The portfolio variance of a two-asset portfolio is calculated using the explicit
   3. For the next PCA component, we apply the same procedure for the set of points {**x_i - w**}, which is the points **x_i** projected into the (N-1) dimensional space perpendicular to **w**.
   4. etc.
 * In terms of asset returns, the principal components are some linear combination of partial returns of different assets, which may or may not be interpretable.
+* The sum of squares of distances to the origin of all data points is proportional to the variance for the mean-centered data. The squared distance to the origin is the sum of squares of distances ("variances") along each perpendicular coordinate axis for each data point. Thus, the sequence PCA axes represents the sequence of coordinates capturing the largest, second largest... portion of the variance.
+* Usually, we want to find lower-dimensional representation of the data that captures as much as possible of the variance. We can achieve this by keeping only the first PCA components. The variation of data points perpendicular to the selected axes (i.e. along the non-selected PCA components) is the variance that remains unaccounted. 
 
+**Example**: `PCA_Toy_Problem.ipynb`.  
+Two-dimensional, correlated data set for demonstrating the PCA class of scikit-learn.
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1160,4 +1160,9 @@ The portfolio variance of a two-asset portfolio is calculated using the explicit
   * **PCA_Core.ipynb** : Visual example in two dimensions with random data (uses *Bokeh* for interactive visualization).
 * PCA mathematical:
   1. *Mean-centering / mean-normalizing* the data; subtract the means along each dimension so that the data are centered around the origin.
+  2. Determine the vector **w** of the first PCA component (maximising the variance) for a set of points {**x_i**} in an N-dimensional space: We want to maximise the total distance from the origin along **w**, which is sum_i(**x_i w**^2)/w^2 (**x_i w** = x_i w cosΘ is the *dot product*). If **X** is the matrix with **x_i** as row vectors, we can write this as L2-norm ||**X w**/w||_2 = (**X w**)^T(**X w**)/(**w**^T **w**) (*Rayleigh quotient*). To interpret it as a variance, we have to multiply by 1/(N-1). So **w** = argmax(||**X w**/w||_2)
+  3. For the next PCA component, we apply the same procedure for the set of points {**x_i - w**}, which is the points **x_i** projected into the (N-1) dimensional space perpendicular to **w**.
+  4. etc.
+* In terms of asset returns, the principal components are some linear combination of partial returns of different assets, which may or may not be interpretable.
+
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1145,4 +1145,19 @@ The portfolio variance of a two-asset portfolio is calculated using the explicit
 * By repeating for each time step, we get the time series and can again calculate the matrices **B**, **F** and **S**.
 * This approach can also be applied to the fundamantal (Fama-French) factors! Above, we constructed theoretical long-short model portfolios to obtain the time series of factors and determined the factor exposures by linear regression. Here, we define the factor exposures first, e.g. β_value = book/market or β_size = market_cap. Second, we calculate f_size and f_value by doing a multilinear regression of all returns of assets in our portfolio at a fixed time step vs. β_size and β_value. In practice, a smaller *estimation universe* of assets would be used for the linear regression.
 
-
+### Statistical Risk Model (PCA)
+
+* Short recap vector spaces (extensive/ YouTube: [Essence of linear algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab))
+  * *Basis*: minimum set of mutually linear independent vectors that span the vector space
+  * Transformation between bases: A matrix where each column expresses a vector of the target basis in terms of the original basis, i.e. each row is the "share" of a vector of the original basis. For transforming back from the new to the old basis, we get the inverse of that matrix.
+* PCA in words: Finding a basis for a vector space defined by a set of points that fulfills specific criteria:
+  * The first axis is selected to *maximise the variance* of datapoints along that axis, i.e. the projection of the points on that axis is maximally spread out (the sum of squares of the distances along the axis from an origin is maximised).
+  * At the same time, this *minimises the reproduction error*, i.e. the sum of the squared (perpendicular) distances of the points to the axis is minimised. The reason is that the sum of squared distance from origin *along* the axis to the projection of the point (related to variance, see above) and the squared distance from the axis equal the squared distance of the point from the origin, which is a constant.
+  * The second axis must be perpendicular to the first, and again maximises the variance among all perpendicular axes.
+  * The third axis is perpendicular to the first two axes and so on, until the whole vector space is spanned
+  * So the set of axes is sorted by "size of variance" and by selecting the first few axes, the majority of the variance can be covered.
+  * For a set of N assets, in theory we can have N linear independent axes, unless two assets move in exact unison.
+  * **PCA_Core.ipynb** : Visual example in two dimensions with random data (uses *Bokeh* for interactive visualization).
+* PCA mathematical:
+  1. *Mean-centering / mean-normalizing* the data; subtract the means along each dimension so that the data are centered around the origin.
+
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1099,14 +1099,50 @@ Calculate the covariance matrix of two assets by expressing the terms in a two f
 
  * **Portfolio variance of two assets in terms of risk factors**: Var(r_P) = x_i^2 * Var(r_i) + x_j^2 * Var(r_j) + 2 x_i x_j * Cov(r_i,r_j). The formulas above can be used to express the terms by factors and factor exposures. The overall formula can be written in matrix / vector notation.
  * **Portfolio variance of n assets in terms of risk factors (matrix formula)**: Considering the portfolio weight vector **X** (the i-th element is the weight of asset i in the portfolio), we get the matrix formula Var(r_P) = **X^T**(**BFB^T** + **S**)**X**, where (for 2 asssets) **X** = (x_i, x_j)^T, **B** = ((β_(i,1), β_(i,2)), (β_(j,1), β_(j,2)), **F** = ((Var(f_1), Cov(f_1,f_2)), (Cov(f_2,f_1), Var(f_2))), **S** = ((Var(s_i), 0), (0, Var(s_j)))
+   * The term **BFB^T** + **S** is the **covariance matrix of assets**.
 
 **Example**: `portfolio_variance.ipynb`.  
 The portfolio variance of a two-asset portfolio is calculated using the explicit terms described above as well as the matrix formula.
 
 **Types of Risk Models**
- * **Time-Series Risk Models**: E.g. CAPM (single factor = market return), Fama-French 3 factor model.
+ * **Time-Series Risk Models**: E.g. CAPM (single factor = market excess return over risk-free rate), Fama-French 3 factor model.
  * **Cross-Sectional Risk Models**
  * **PCA Risk Models**
 
-
+### Time Series Risk Model
+
+ **Example CAPM as one-factor model:**
+ * For stock i in our stock universe (market), e.g. S&P500, r_i - r_f = β_i * f_m + c_i; r_i is the stock return, r_f the risk free rate; f_m = r_m - r_f is the single factor of this model, namely the market excess return. All these are *time series*. We can estimate the *market exposure* β_i of the CAPM for stock i, which equals the factor exposure in this model, as well as the intercept c_i by doing a linear regression of r_i - r_f vs. f_m
+   * Note 1: It's not a clearly defined methodology; there are other methods which may yield different results for β_i and c_i)
+   * Note 2: The sample (length of the time window) to be used for the regression must be selected. One approach would be to make it dependend on the envisaged trading frequency, e.g. several weeks for daily trading up to years for longer holding periods.
+ * For a factor model, we also need the *residual* s_i, which is the part of the return of stock i *not* explained by the model. the time series for s_i is given by s_i = (r_i - r_f) - (β_i * f_m + c_i). This provides the matrix of specific variances.
+ * By calculating the variances and covariances of the of the different terms over a selected time window (see Note 2 above regarding the choice of time window), we can calculate the portfolio variance with the matrix formula Var(r_P) = **X^T**(**BFB^T** + **S**)**X**
+
+**Multi-factor model**
+* "Classical" Fama-French factors: Market, size (SMB = small minus big) and value (HML = high - low). I.e. besides the average "market" return, research suggests that smaller businesses (by market cap) perform on average better than larger businesses and stocks with a low market value compared to their fundamentals perform on average better than those with a high value compared to fundamentals. This research is taken as *hypothethis* for the formulation of corresponding factors.
+* To find out the effect of a hypothetical factor, it's common to use a *dollar neutral* theoretical portfolio, i.e. go long on assets that have more of a particular trait and short on assets that have less of the trait. Assume an investment of equal dollar amount in each asset.
+* E.g. "size factor" time series f_s = (r_small - r_big)/2; r_small = sum_N_small(r_i)/N_small; r_big = sum_N_big(r_i)/N_big; "small" if the first decile of stocks in our universe sorted by market cap; "big" is the last decile.
+* E.g. "value factor: Sort stocks by book value/market value ratio; those with high book/market are called *Growth(Low)*; those with low book/market are called *Value(High)*. This time the original FF model goes *short* all stocks in the first three deciles and *long* the stocks in the last three deciles. A corresponding formula as for the size factor provides the time series for the value factor f_v.
+* **Complete Fama-French approach**: first sort our market universe (return r_m) by size and create the "small" (s) and "big" (b) portfolios as described above. The middle section of 8 deciles is discarded. Then subdivide each of the two portfolios into three parts according to "value" as described for the value factor, labeled "growth" (g), "neutral" (n) and "value" (v). This provides 6 sub-portfolios with return time series r_sg, r_sn etc. Then the definition of the factors is: 
+ Market f_m = r_m - r_f  
+ SMB f_s = ((r_sg + r_sn + r_sv) - (r_bg + r_bn + r_bv)) / 3
+ HML f_v = ((r_sv + r_bv) - (r_sg + r_bg)) / 2. 
+   * Note: The two "neutral" portfolios are only taken into account for the SMB factor!
+* With these three factor time series, the *covariance matrix* of the factors **F** can be calculated.
+* The factor exposures β_(i,x) (x in m,s,v) of a stock i can be calculated by doing a *multilinear regression* of ri vs. β_(i,m) * f_m + β_(i,s) * f_s + β_(i,v) * f_v. This provides the matrix **B** of factor exposures.
+* The specific return time series can be calculated as s_i = r_i - (β_(i,m) * f_m + β_(i,s) * f_s + β_(i,v) * f_v). The variances of these time series provides the specific variance matrix **S**.
+
+### Cross-Sectional risk model
+
+* "Most practitioners tend to buy commercial risk models that are built and maintained by other companies (e.g. MSCI Barra, Axioma, North Field). It is the task of these models to ensure that the factors and specific returns of stocks are independent of each other. The practitioners then focus on identifying alpha factors" (Video 10 and 16 statements)
+* These are often so called cross-sectional risk models.
+* Cross-sectional risk models may use *categorical variables* like country or sector of a company as factors.
+* The *factor exposure* of an asset can be expressed by one-hot encoding, i.e. β_(i,country_j) = (1 if country_j is home country; 0 otherwise), or a number between 0 and 1 expressing the percentage of revenues in that country.
+* *Important*: The factor exposure is determined by the category, *not* by a regression of the time series as above.
+* In contrast, we need regression now to find the time series of *factors* f_country_j: For a *fixed time step* t_n, we plot r_i(t_n) for all assets i vs. β_(i,country_j). f_country_j(t_n) is the slope of the linear regression line in this plot.
+  * *Note*: Does this work with one-hot encoding, when β_(i,country_j) is either 0 or 1 with no intermediate values?
+* The time series of specific returns can again be calculated as the residuals of the linear regressions, i.e. s_i(t_n) = r_i(t_n) - r_estim(t_n) with r_estim(t_n) = sum_countries(β_(i,country_j) * f_country_j(t_n)) + c_i(t_n)
+* By repeating for each time step, we get the time series and can again calculate the matrices **B**, **F** and **S**.
+* This approach can also be applied to the fundamantal (Fama-French) factors! Above, we constructed theoretical long-short model portfolios to obtain the time series of factors and determined the factor exposures by linear regression. Here, we define the factor exposures first, e.g. β_value = book/market or β_size = market_cap. Second, we calculate f_size and f_value by doing a multilinear regression of all returns of assets in our portfolio at a fixed time step vs. β_size and β_value. In practice, a smaller *estimation universe* of assets would be used for the linear regression.
+
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1101,5 +1101,12 @@ Calculate the covariance matrix of two assets by expressing the terms in a two f
  * **Portfolio variance of n assets in terms of risk factors (matrix formula)**: Considering the portfolio weight vector **X** (the i-th element is the weight of asset i in the portfolio), we get the matrix formula Var(r_P) = **X^T**(**BFB^T** + **S**)**X**, where (for 2 asssets) **X** = (x_i, x_j)^T, **B** = ((β_(i,1), β_(i,2)), (β_(j,1), β_(j,2)), **F** = ((Var(f_1), Cov(f_1,f_2)), (Cov(f_2,f_1), Var(f_2))), **S** = ((Var(s_i), 0), (0, Var(s_j)))
 
 **Example**: `portfolio_variance.ipynb`.  
-
+The portfolio variance of a two-asset portfolio is calculated using the explicit terms described above as well as the matrix formula.
+
+**Types of Risk Models**
+ * **Time-Series Risk Models**: E.g. CAPM (single factor = market return), Fama-French 3 factor model.
+ * **Cross-Sectional Risk Models**
+ * **PCA Risk Models**
+
+
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1098,7 +1098,8 @@ Define two (dummy) "factors" by calculating the daily *mean* and *median* return
 Calculate the covariance matrix of two assets by expressing the terms in a two factor model, weighted with the factor exposures. First by calculating the matrix elements one by one using the formulas above, then by doing numpy matrix calculation using the formula **A** = **BFB^T** + **S**, where **A** is the covariance matrix of asset returns, **B** is the matrix of factor exposures, **F** is the covariance matrix of the factors, **S** is the diagonal matrix of specific variances.
 
  * **Portfolio variance of two assets in terms of risk factors**: Var(r_P) = x_i^2 * Var(r_i) + x_j^2 * Var(r_j) + 2 x_i x_j * Cov(r_i,r_j). The formulas above can be used to express the terms by factors and factor exposures. The overall formula can be written in matrix / vector notation.
- * **Portfolio variance of n assets in terms of risk factors (matrix formula)**: Considering the portfolio weight vector **X** (the i-th element is the weight of asset i in the portfolio), we get the matrix formula Var(r_P) = **X**(**BFB^T** + **S**)**X^T**, where (for 2 asssets) **X** = (x_i, x_j)^T, **B** = ((β_(i,1), β_(i,2)), (β_(j,1), β_(j,2)), **F** = ((Var(f_1), Cov(f_1,f_2)), (Cov(f_2,f_1), Var(f_2))), **S** = ((Var(s_i), 0), (0, Var(s_j)))
-
+ * **Portfolio variance of n assets in terms of risk factors (matrix formula)**: Considering the portfolio weight vector **X** (the i-th element is the weight of asset i in the portfolio), we get the matrix formula Var(r_P) = **X^T**(**BFB^T** + **S**)**X**, where (for 2 asssets) **X** = (x_i, x_j)^T, **B** = ((β_(i,1), β_(i,2)), (β_(j,1), β_(j,2)), **F** = ((Var(f_1), Cov(f_1,f_2)), (Cov(f_2,f_1), Var(f_2))), **S** = ((Var(s_i), 0), (0, Var(s_j)))
+
+**Example**: `portfolio_variance.ipynb`.  
 
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1094,4 +1094,11 @@ Define two (dummy) "factors" by calculating the daily *mean* and *median* return
 * **Covariance of two stocks in a 2 factor model**: Cov(r_i,r_j) = Cov(β_(i,1)f_1 + β_(i,2)f_2 + s_i, β_(j,1)f_1 + β_(j,2)f_2 + s_j). When multiplying out the 2 times 3 terms, we end up with 9 terms. However, by definition, the s_n are uncorrelated with the f_m (and uncorrelated with each other?). Therefore, 5 of the 9 terms containing an s_m factor are equal to 0. The remaining 4 are: Cov(r_i,r_j) = β_(i,1)β_(j,1)Var(f_1) + β_(i,2)β_(j,2)Var(f_2) + β_(i,1)β_(j,2)Cov(f_1,f_2) + β_(i,2)β_(j,1)Cov(f_2,f_1).
 * **Covariance Matrix of assets in terms of factors**: By using the above formulas, the covariance matrix of the two stocks, ((Var(r_i), Cov(r_i,r_j)), ((Cov(r_j,r_i), Var(r_j))), can be expressed entirely in terms of the *risk factor* variances and covariances, multiplied by the constant factor exposures. For a large number of assets, this provides a considerable simplification of the calculation!
 
+**Example**: `covariance_matrix_assets.ipynb`.  
+Calculate the covariance matrix of two assets by expressing the terms in a two factor model, weighted with the factor exposures. First by calculating the matrix elements one by one using the formulas above, then by doing numpy matrix calculation using the formula **A** = **BFB^T** + **S**, where **A** is the covariance matrix of asset returns, **B** is the matrix of factor exposures, **F** is the covariance matrix of the factors, **S** is the diagonal matrix of specific variances.
+
+ * **Portfolio variance of two assets in terms of risk factors**: Var(r_P) = x_i^2 * Var(r_i) + x_j^2 * Var(r_j) + 2 x_i x_j * Cov(r_i,r_j). The formulas above can be used to express the terms by factors and factor exposures. The overall formula can be written in matrix / vector notation.
+ * **Portfolio variance of n assets in terms of risk factors (matrix formula)**: Considering the portfolio weight vector **X** (the i-th element is the weight of asset i in the portfolio), we get the matrix formula Var(r_P) = **X**(**BFB^T** + **S**)**X^T**, where (for 2 asssets) **X** = (x_i, x_j)^T, **B** = ((β_(i,1), β_(i,2)), (β_(j,1), β_(j,2)), **F** = ((Var(f_1), Cov(f_1,f_2)), (Cov(f_2,f_1), Var(f_2))), **S** = ((Var(s_i), 0), (0, Var(s_j)))
+
+
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1090,6 +1090,8 @@ Different investors may treat the same factor as either an alpha or a risk facto
 **Example**: `factor_model_portfolio_return.ipynb`.  
 Define two (dummy) "factors" by calculating the daily *mean* and *median* returns over a large number of assets. Build a portfolio by selecting two of these assets with specific weights. Calculate the *factor exposure* to both factors of the two assets by linear regression of factor return vs. asset return. Calculate portfolio factor exposure and portfolio return. Calculate the *common return* of the portfolio = part of portfolio return explained by the factors and the *specific return* = the remainder which is not explained by the factors.
 
-* **Variance of a single stock in a 2 factor model**: Var(r_i) = β_(i,1)^2 * Var(f_1) + β_(i,2)^2 * Var(f_2) + β_(i,1)*β_(i,2) * Cov(f_1,f_2) + Var(s_i). The first three terms are called *systematic variance* (determined by the factors; the fourth the *specific / ideosncratic variance* (not covered by the factors). f_i are the *factor returns*, β_(i,j) the *factor exposure* of stock i to factor j and s_i the *specific* (factor independent) return of stock i.
+* **Variance of a single stock in a 2 factor model**: Var(r_i) = Var(β_(i,1)f_1) + Var(β_(i,2)f_2) + 2 Cov(β_(i,1)f_1, β_(i,2)f_2) + Var(s_i) = β_(i,1)^2 * Var(f_1) + β_(i,2)^2 * Var(f_2) + 2 β_(i,1)*β_(i,2) * Cov(f_1,f_2) + Var(s_i). The first three terms are called *systematic variance* (determined by the factors; the fourth the *specific / ideosncratic variance* (not covered by the factors). f_i are the *factor returns*, β_(i,j) the *factor exposure* of stock i to factor j and s_i the *specific* (factor independent) return of stock i. *Important to note that the factor exposures can be taken out of the factor variance and covariance calculation on the rhs!*
+* **Covariance of two stocks in a 2 factor model**: Cov(r_i,r_j) = Cov(β_(i,1)f_1 + β_(i,2)f_2 + s_i, β_(j,1)f_1 + β_(j,2)f_2 + s_j). When multiplying out the 2 times 3 terms, we end up with 9 terms. However, by definition, the s_n are uncorrelated with the f_m (and uncorrelated with each other?). Therefore, 5 of the 9 terms containing an s_m factor are equal to 0. The remaining 4 are: Cov(r_i,r_j) = β_(i,1)β_(j,1)Var(f_1) + β_(i,2)β_(j,2)Var(f_2) + β_(i,1)β_(j,2)Cov(f_1,f_2) + β_(i,2)β_(j,1)Cov(f_2,f_1).
+* **Covariance Matrix of assets in terms of factors**: By using the above formulas, the covariance matrix of the two stocks, ((Var(r_i), Cov(r_i,r_j)), ((Cov(r_j,r_i), Var(r_j))), can be expressed entirely in terms of the *risk factor* variances and covariances, multiplied by the constant factor exposures. For a large number of assets, this provides a considerable simplification of the calculation!
 
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1086,4 +1086,10 @@ Different investors may treat the same factor as either an alpha or a risk facto
 
  * **Portfolio return in a factor model**: r_P = sum(β_(P,i) * f_i) + s_P, with β_(P,i) = sum(x_j * β_j) and s_P = sum(x_j * s_j), where β_(P,i) is the portfolio factor exposure to factor f_i and β_j the exposure of portfolio component j. s_P is the portfolio specific return, s_j the specific return of component j and x_j the weight of component j in the portfolio.
  * **Factor Model of portfolio variance**: var(r_P) = **X^T**(**BFB^T** + **S**)**X**; **X**: vector of portfolio weights; **B**: Matrix of portfolio constituent factor exposures; **F**: covariance matrix of factors; **S**: Matrix of specific (ideosyncratic) (co)variances - ideally a diagonal matrix.
+
+**Example**: `factor_model_portfolio_return.ipynb`.  
+Define two (dummy) "factors" by calculating the daily *mean* and *median* returns over a large number of assets. Build a portfolio by selecting two of these assets with specific weights. Calculate the *factor exposure* to both factors of the two assets by linear regression of factor return vs. asset return. Calculate portfolio factor exposure and portfolio return. Calculate the *common return* of the portfolio = part of portfolio return explained by the factors and the *specific return* = the remainder which is not explained by the factors.
+
+* **Variance of a single stock in a 2 factor model**: Var(r_i) = β_(i,1)^2 * Var(f_1) + β_(i,2)^2 * Var(f_2) + β_(i,1)*β_(i,2) * Cov(f_1,f_2) + Var(s_i). The first three terms are called *systematic variance* (determined by the factors; the fourth the *specific / ideosncratic variance* (not covered by the factors). f_i are the *factor returns*, β_(i,j) the *factor exposure* of stock i to factor j and s_i the *specific* (factor independent) return of stock i.
+
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1082,4 +1082,8 @@ Different investors may treat the same factor as either an alpha or a risk facto
 
  ### Risk Factor Models
 
+ Purpose: Model, control, if possible neutralize the portfolio's risk.
+
+ * **Portfolio return in a factor model**: r_P = sum(β_(P,i) * f_i) + s_P, with β_(P,i) = sum(x_j * β_j) and s_P = sum(x_j * s_j), where β_(P,i) is the portfolio factor exposure to factor f_i and β_j the exposure of portfolio component j. s_P is the portfolio specific return, s_j the specific return of component j and x_j the weight of component j in the portfolio.
+ * **Factor Model of portfolio variance**: var(r_P) = **X^T**(**BFB^T** + **S**)**X**; **X**: vector of portfolio weights; **B**: Matrix of portfolio constituent factor exposures; **F**: covariance matrix of factors; **S**: Matrix of specific (ideosyncratic) (co)variances - ideally a diagonal matrix.
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1040,3 +1040,46 @@ Solution: `project_3_smart_beta_and_portfolio_optimization.ipynb`
   * **B** is the NxK matrix of factor exposures for each asset (K = number of factors),
   * **F** = E(**ff^T**) the KxK matrix of factor covariances,
   * **S** = E(**ss^T**) the NxN diagonal matrix of the variances of the residuals (as the off-diagonal covariances are zero).
+* For a factor model, we have **Return** *r* = **Bf** + **s** and **Risk** E(**rr^T**) = **BFB^T** + **S**
+* Portfolio factor exposure for a portfolio with weights **x**: **B^Tx**
+* **Risk and Alpha Factors**:
+   * Assumption: we can subdivide factors into "Risk factors" that are predictive of the variance of the portfolio and "Alpha factors" that are descriptive of the returns. In portfolio optimization, we want to put constraints on the variance but not on the returns. It is therefore possible (in terms of portfolio optimization) to disregard any alpha factors and consider **F** as the covariance matrix of all factors that have a considerable impact of variance, while **S** is the (co)variance that cannot be explained by these factors, including the variance related to the (dropped) alpha factors.
+ * In this model, when using E(**rr^T**) = **BFB^T** + **S** to constrain portfolio risk, the terms **B** and **F** don't include any information about alpha and **S** doesn't include explicit information. "They are usually bought by practitioners from commercial providers."
+ * The alpha factors are identified separately and enter into the objective function for the portfolio optimization.
+
+ #### Risk factors vs. alpha factors
+
+ "Real" factors will be not purely risk or alpha factors. When "ranking" factors, there will be a gradual variation of these characteristics.
+
+In general, risk factors are significant contributors to the variance of asset returns, and less predictive of the mean of returns. Risk factors are identified to control risk. One way to do control an asset's exposure to a risk factor is to hold an equal amount long as short. For instance, a dollar neutral portfolio with equal amounts long and short is controlling for risks that the overall market may move up or down.
+
+In general, factors that are significant in describing the mean of asset returns can be candidates for alpha factors. Alpha factors are used to give some indication of whether each stock in the portfolio may have positive expected returns or negative expected returns. For example, a former alpha factor was the market capitalization of a stock. Small cap stocks tend to have higher future returns compared to large cap stocks.
+
+Usually, we'd choose 20 to 60 risk factors that describe overall stock variance as much as possible. So risk factors as a whole account for more of the overall movement of stocks. On the other hand, alpha factors contribute to smaller movements of stocks, which is okay, because we seek to identify these alpha factors because they give some indication of the direction of expected returns, even if they're small compared to risk factors. An important reason why it's important to identify risk factors and then neutralize a portfolio's exposure to risk factors is that if we didn't, the asset movements due to risk factors would overwhelm the movements that are due to the alpha factors.
+
+Risk factors are well-known by the investment community, so investors will track those factors when optimizing their portfolios. This also means that it's unlikely that any one investor can gain a competitive advantage (higher than normal returns) using risk factors (Investors "trade away" the competitive advantage; this has for example happened to the small cap factor after its publication in 1981).
+
+Alpha factors are less well-known by the investment community, because they're generated by in-house research (i.e. they are proprietary) to help the fund generate higher than normal returns. So alpha factors are said to be drivers of the mean of returns because they're used to help push a portfolio's overall returns higher than what would be expected from a passive buy and hold strategy.
+
+When a former alpha factor becomes more and more known, it turns into a risk factor because more and more people are following its signal, thus amplifying price movements in either direction.
+
+Different investors may treat the same factor as either an alpha or a risk factor. Part 2: Use AI to select factors.
+
+#### Factor models and types of factors 
+
+* Alpha factors can be broadly categorized into *momentum* and *reversal* factors. Example of a momentum factor: numerical value of 12 month return. Example of a reversal factor: negative of the weekly return (assuming that higher weekly returns will lead to profit taking).
+* *Price-Volume factors*: Refers to all factors that are constructed from asset price and trading volume information. It can be measured at different frequencies; adjusted or unadjusted prices; bid-ask; OHCL; any combination thereof; statistical properties like (popular) the first four momenta: mean, variance (squared volatility), skewness (asymmetry of the distribution), kurtosis (fat-tailedness); Min and max over different time windows. This information is broadly available for all stocks (as opposed to e.g. analyst estimated as factors) and frequently updated (as opposed to e.g. most fundamental data as factors). It may lead to higher trading / rebalancing frequency, requiring a careful cost - benefit analysis.
+* *Volume factors*: "Net buy" or "net sell", volume as indicator for price significance or significance of other factors like momentum (momentum "conditioned" on volume), short interest (high short interest may propel momentum if prices rise - short squeeze)
+* *Fundamental factors*: Usually updated quarterly. Lower trading frequency may allow for larger position sizes. Example: Market cap ("size effect").
+* *Fundamental ratios*: earnings p. share/price (can become negative!), book/price; less influenced by accounting decisions: cash flow or EBITDA. Cash flow is more volatile than earnings because it immediately reflects the effect of large investments.
+* *Event driven factors*: e.g. natural disasters, major accidents, government changes, M&A / spin-offs, new inventions/discoveries (one-off events). Some regularity: interest rate changes, index changes or major weight changes; new earnings releases or earnings guidance; product announcements.
+* *Pre and post event*: E.g. for earnings announcements vs analyst expectations. "Post earnings announcement drift": prices tend to drift in the same direction for about two months after an earnings announcement. Well known so less exploitable.
+* *Analyst ratings*: Sentiment factor. Herd behavior of analysts. "Sell side" recommendations (banks) for "buy side" (fund managers). Factor could be e.g. a weighted average of analyst ratings (weighted by some reputation measure) or upgrades - downgrades.
+* *Alternative data*: Social media, consumer behavior, satellite images
+* *Sentiment analysis on news and social media*: Following financial news is important for quant analysts. Sentiment analysis using NLP can transform news and social media messages into "buy"/"hold"/"sell" categories. High public attention may indicate the end of a substantial movement (overextension vs. fundamentals).
+* *NLP can be used to enhance fundamental research*: Automated pre-processing / labeling of quarterly reports, financial statements, regulatory filings can filter data for subsequent fundamental analysis. Sentiment analysis of [10-K](https://www.investor.gov/introduction-investing/general-resources/news-alerts/alerts-bulletins/investor-bulletins/how-read) filings (SEC; 1 per year. Plus three 10-Q = quarterly). E.g. amount of legal risks, perceived uncertainties from competitors, customers, suppliers... can be transformed into "business outlook" factors. [Sector classification using 10-K](https://www.winton.com/research/systematic-methods-for-classifying-equities): The article describes (among others) the use of NLP to assign S&P500 stocks to industry sectors. Sector assignment is indicative of correlation and can also be used in factor models. It reviews critically the standard GICS sector classification of the S&P500, which is in parts not very indicative for correlation among stocks.
+* *Other alternative data*: Tracking usage of supermarket car parks or fill level of floating-roof oil tanks from satellite images; self-representation of companies in social media / employment offers; marketing activity in social media (counter indicator);... Examples for sources for alternative data: https://www.buildfax.com/, https://www.thinknum.com/, https://orbitalinsight.com/.
+
+ ### Risk Factor Models
+
+
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1017,7 +1017,26 @@ Solution: `project_3_smart_beta_and_portfolio_optimization.ipynb`
 * *Notional*: The dollar value associated with a portfolio.
 * *Leverage*: The leverage ratio is the sum of the (absolute) magnitudes of all positions in dollars, divided by the notional. By rescaling the portfolio weights (i.e. dividing by the sum of absolute weights), we ensure a portfolio leverage of one.
 
-## ZIPLINE
+#### ZIPLINE
 [Zipline](https://zipline.ml4trading.io/index.html) is a Pythonic event-driven system for backtesting, developed and used as the backtesting and live-trading engine by crowd-sourced investment fund Quantopian. Since it closed late 2020, the domain that had hosted these docs expired. The library is used extensively in the book [Machine Larning for Algorithmic Trading](https://ml4trading.io/) by Stefan Jansen who is trying to keep the library up to date and available to his readers and the wider Python algotrading community.
 
-*Code examples*: `Zipline_Pipeline _Primer.ipynb`, `Zipline_Pipeline _Exercise.ipynb`
+*Code examples*: `Zipline_Pipeline _Primer.ipynb`, `Zipline_Pipeline _Exercise.ipynb`
+
+#### Factor Models
+* A *Factor Model* tries to identify a number of *latent variables that can model / explain the *common variability* of asset returns of certain groups of assets.
+* Linear factor model: Return of asset i r_i = b_i1 * f_1 + b_i2 *f_2 + ... b_iK * f_K + s_i where f_n are numerical values for the factor at a given time and s_i is an "error term" that describe the portion of the returns that can not be associated to the factors.
+* terminology:
+  * Factor returns (the f_k) may be: macro-economic variables, returns on pre-specified portfolios, returns on zero-investment strategies (long and short positions of equal value) giving maximum exposure to fundamental or macro-economic factors, returns on benchmark portfolios representing asset classes, or something else.
+  * The b_{ij} coefficients may be called: factor exposures, factor sensitivities, factor loadings, factor betas, asset exposures, style or something else.
+  * The s_i term may be called: idiosyncratic return, security-specific return, non-factor return, residual return, selection return or something else.
+
+* **Factor model assumptions**:
+  * Factor returns are uncorrelated to the residuals, corr(f_n, s_i) = 0 for every n and i. (This can always be achieved by selecting suitable b_in, e.g. through multi-linear regression)
+  * Residuals of different stocks are uncorrelated, corr(s_i, s_j) = 0 for every i != j. This means that all correlation between two assets is assumed to be captured in the factors. s_i captures the *ideosyncratic risk* of a specific asset. It requires that the factor model includes a sufficient number of important factors to capture the correlation. However, too many factors capture too much "noise", i.e. overfit.
+* **Covariance matrix using the factor model**: When using the two uncorrelation assumptions described and when transforming the returns to have zero means (by subtracting any non-zero mean), it can be shown that the asset return covariance E(**rr^T**) can be expressed as
+  * E(**rr^T**) = **BFB^T** + **S** where
+  * E() denotes the expectation value,
+  * **r** is the (N dimensional) vector of asset returns,
+  * **B** is the NxK matrix of factor exposures for each asset (K = number of factors),
+  * **F** = E(**ff^T**) the KxK matrix of factor covariances,
+  * **S** = E(**ss^T**) the NxN diagonal matrix of the variances of the residuals (as the off-diagonal covariances are zero).
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1017,4 +1017,7 @@ Solution: `project_3_smart_beta_and_portfolio_optimization.ipynb`
 * *Notional*: The dollar value associated with a portfolio.
 * *Leverage*: The leverage ratio is the sum of the (absolute) magnitudes of all positions in dollars, divided by the notional. By rescaling the portfolio weights (i.e. dividing by the sum of absolute weights), we ensure a portfolio leverage of one.
 
-
+## ZIPLINE
+[Zipline](https://zipline.ml4trading.io/index.html) is a Pythonic event-driven system for backtesting, developed and used as the backtesting and live-trading engine by crowd-sourced investment fund Quantopian. Since it closed late 2020, the domain that had hosted these docs expired. The library is used extensively in the book [Machine Larning for Algorithmic Trading](https://ml4trading.io/) by Stefan Jansen who is trying to keep the library up to date and available to his readers and the wider Python algotrading community.
+
+*Code examples*: `Zipline_Pipeline _Primer.ipynb`, `Zipline_Pipeline _Exercise.ipynb`
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1012,6 +1012,7 @@ Solution: `project_3_smart_beta_and_portfolio_optimization.ipynb`
 * **Note**: "Weight" in the following means the portfolio share of each asset. "Factor value" is a numerical representation for the strength of the considered factor for an asset. E.g. for *momentum*, the factor value could be the 12 month return.
 * *De-mean*: Sum of (standardized) weights equals zero; achieved by *subtracting the mean* of raw factor values from each raw factor value.
 * *Rescale*: Sum of absolute (standardized) weights equals one; achieved by *dividing through the sum* of absolute raw factor values for each raw factor value.
+* *The sum of the demeaned weights is always zero, the sum of the rescaled weights is always zero, the sum of the absolute value of the rescaled weights is always one, the sum of the rescaled short positions is always -0.5*
 * Reason for de-meaning: Making the portfolio *Dollar neutral* so that its development is (mostly) independent of the overall market trend ("isolate the effect of the factors"). Long and short positions are balanced (even though the "raw" weights could describe a long-only portfolio.
 * *Notional*: The dollar value associated with a portfolio.
 * *Leverage*: The leverage ratio is the sum of the (absolute) magnitudes of all positions in dollars, divided by the notional. By rescaling the portfolio weights (i.e. dividing by the sum of absolute weights), we ensure a portfolio leverage of one.

diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -1009,8 +1009,11 @@ Solution: `project_3_smart_beta_and_portfolio_optimization.ipynb`
 * *Risk Factors*: Drivers of volatility
 
 * To make different factors better comparable, it is standard practice to *de-mean* and *rescale* factor weights.
+* **Note**: "Weight" in the following means the portfolio share of each asset. "Factor value" is a numerical representation for the strength of the considered factor for an asset. E.g. for *momentum*, the factor value could be the 12 month return.
 * *De-mean*: Sum of (standardized) weights equals zero; achieved by *subtracting the mean* of raw factor values from each raw factor value.
 * *Rescale*: Sum of absolute (standardized) weights equals one; achieved by *dividing through the sum* of absolute raw factor values for each raw factor value.
-* Reason for de-meaning: Making the portfolio *Dollar neutral* so that its development is independent of the overall market trend ("isolate the effect of the factors"). 
+* Reason for de-meaning: Making the portfolio *Dollar neutral* so that its development is (mostly) independent of the overall market trend ("isolate the effect of the factors"). Long and short positions are balanced (even though the "raw" weights could describe a long-only portfolio.
+* *Notional*: The dollar value associated with a portfolio.
+* *Leverage*: The leverage ratio is the sum of the (absolute) magnitudes of all positions in dollars, divided by the notional. By rescaling the portfolio weights (i.e. dividing by the sum of absolute weights), we ensure a portfolio leverage of one.
 
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -3,6 +3,8 @@
 ## 1. Concepts
 [Nassim Taleb's home page](https://www.fooledbyrandomness.com/)
 
+[Quantdare](https://quantdare.com/)
+
 * OHLC, ticks, timeframes
 * Volume. Higher volume at beginning ('price discovery') and end than in the middle of the day.
 * Pre-market: typ. 4:00 - 9:00am; post-market: typ. 16:00 - 20:00pm. Normally low volume, but can be used for additional trading information.
@@ -291,7 +293,7 @@ Steps for creation of a cross-sectional strategy:
 
 **Trading signal**: Any numerical signal that informs a trade.
 
-**Alpha vector**: Vector of numbers representing the portfolio weight of each stock in our universe (can be negative if short). (The alphas in the vector are a specifci type of trading signal)
+**Alpha vector**: Vector of numbers representing the portfolio weight of each stock in our universe (can be negative if short). (The alphas in the vector are a specific type of trading signal)
 
 **Risk analysis**: The strategy should include a mathematical model of the risks.
 * Systematic risks (e.g. inflation, recession, interest rates, GDP...)
@@ -879,7 +881,7 @@ The SML is also used to compare similar securities with approximately similar re
 **Simple example: 2 asset portfolio - minimum variance portfolio**
 * Objective function: minimize sigma_P^2 = x_A^2 sigma_A^2 + x_B^2 sigma_B^2 + 2 x_A x_B sigma_A sigma_B rho_(r_A,r_B) where rho is the correlation between the returns r_A and r_B.
 * constraint: x_A + x_B = 1
-* Can be analytically solved (quadratic equation); solution:
+* Can be analytically solved (*quadratic equation*); solution:
 * x_(A,min) = (sigma_B^2 - sigma_A sigma_B rho)/(sigma_A^2 + sigma_B^2 - 2 sigma_A sigma_B rho), x_B = 1 - x_A
 * mu_P = mu_A x_A + mu_B x_B; mu_P is the portfolio mean return; mu_B and mu_B the individual asset mean returns.
 
@@ -939,7 +941,10 @@ What if you want to minimize portfolio variance, but have the portfolio track an
 
 *objective:* minimize: **x^TPx** + λ ∥**x** − **q**∥_2, q = a set of index weights, λ = a tradeoff parameter
 
-#### Python package for optimization problems: cvxpy
+#### Backtracking algorithm for asset allocation with constraints 
+Besides the solution approach described below (with *cvxpy*), another approach is the [*backtracking algorithm*](https://quantdare.com/asset-allocation-with-constraints-using-backtracking/), which is basically an "exhaustive search" in the space of all possible weights for each asset. To be finite, a step size for the weights search has to be defined (e.g. 1%). Additionally, the search tree should be "pruned" by applying boundary conditions to the weights, like sum of weights equal to one or maximum/minimum allowed weights per asset.
+
+ #### Python package for optimization problems: cvxpy
 
 [cvxpy](https://www.cvxpy.org/) is a Python package for solving convex optimization problems. It allows you to express the problem in a human-readable way, calls a solver, and unpacks the results.
 
@@ -996,5 +1001,16 @@ In this project, you will build a smart beta portfolio and compare it to a bench
 
 Solution: `project_3_smart_beta_and_portfolio_optimization.ipynb`
 
+## 4. Factors
 
+* *Factors* contain potentially predictive information for the future movement of stocks, e.g. momentum, fundamental information, signals from social media.
+* Factors must be transformed into weights / signals for individual stocks
+* *Alpha Factors*: Drivers of mean returns
+* *Risk Factors*: Drivers of volatility
+
+* To make different factors better comparable, it is standard practice to *de-mean* and *rescale* factor weights.
+* *De-mean*: Sum of (standardized) weights equals zero; achieved by *subtracting the mean* of raw factor values from each raw factor value.
+* *Rescale*: Sum of absolute (standardized) weights equals one; achieved by *dividing through the sum* of absolute raw factor values for each raw factor value.
+* Reason for de-meaning: Making the portfolio *Dollar neutral* so that its development is independent of the overall market trend ("isolate the effect of the factors"). 
+
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -941,7 +941,7 @@ What if you want to minimize portfolio variance, but have the portfolio track an
 
 #### Python package for optimization problems: cvxpy
 
-cvxpy is a Python package for solving convex optimization problems. It allows you to express the problem in a human-readable way, calls a solver, and unpacks the results.
+[cvxpy](https://www.cvxpy.org/) is a Python package for solving convex optimization problems. It allows you to express the problem in a human-readable way, calls a solver, and unpacks the results.
 
 **How to use cvxpy**
 
@@ -990,6 +990,11 @@ cvxpy is a Python package for solving convex optimization problems. It allows yo
 * Single-period optimization: The optimization described above has one fixed timeframe in mind. If different time horizons were considered, different optimal actions (e.g. long/short a stock) could result in different time periods. This is considered by [multi-period optimization](https://stanford.edu/~boyd/papers/pdf/cvx_portfolio.pdf).
 * Transaction costs could be considered as constraints in the optimization problem.
 
+#### Smart Beta and Portfolio Optimization
+
+In this project, you will build a smart beta portfolio and compare it to a benchmark index. To find out how well the smart beta portfolio did, you’ll calculate the tracking error against the index. You’ll then build a portfolio by using quadratic programming to optimize the weights. Your code will rebalance this portfolio and calculate turn over to evaluate the performance. You’ll use this metric to find the optimal rebalancing Frequency. For the dataset, we'll be using the end of day from Quotemedia.
+
+Solution: `project_3_smart_beta_and_portfolio_optimization.ipynb`
 
 
 
diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -882,7 +882,9 @@ The SML is also used to compare similar securities with approximately similar re
 * Can be analytically solved (quadratic equation); solution:
 * x_(A,min) = (sigma_B^2 - sigma_A sigma_B rho)/(sigma_A^2 + sigma_B^2 - 2 sigma_A sigma_B rho), x_B = 1 - x_A
 * mu_P = mu_A x_A + mu_B x_B; mu_P is the portfolio mean return; mu_B and mu_B the individual asset mean returns.
-
+
+*Example*: Covariance matrix with numpy, `m3l4_covariance.ipynb`
+
 #### Formulating portfolio optimization problems
 
 So far, we've discussed one way to formulate a portfolio optimization problem. We learned to set the portfolio variance as the objective function, while imposing the constraint that the portfolio weights should sum to 1. However, in practice you may frame the problem a little differently. Let's talk about some of the different ways to set up a portfolio optimization problem.
@@ -969,7 +971,24 @@ cvxpy is a Python package for solving convex optimization problems. It allows yo
 
 *Examples:* `m3l4_cvxpy_basic.ipynb`, `m3l4_cvxpy_advanced.ipynb`
 
+#### Rebalancing strategies
+
+* Rebalancing serves to bring back asset weights to the "optimal" values after assets have appreciated at different rates over some time. The optimization problem must be re-run with updated parameters.
+
+* Rebalancing costs: Transaction costs, capital gains tax, time & labor. To estimate rebalancing costs, one could assume that they are proportional to the portfolio *turnover*, i.e. the sum of the absolute differences of all asset weights before and after the rebalancing.
+
+* Rebalancing triggers: Cash flows (dividends, capital gains, new contributions. Contributions / withdrawals can be used to rebalance), changes in model parameters (e.g. due to mergers and acquisitions. To be decided from which size of deviation rebalancing should occur)
+
+* Rebalancing strategies: At fixed temporal intervals; when deviations from the desired weights reach a certain threshold
+
+#### Problems of portfolio optimization
 
+* Past returns may not be a good predictor of future returns
+* Variance of stocks may not be a good measure of risk (e.g. if the distribution of returns is asymmetric / non-gaussian)
+* For estimating the covariance of a large number of stocks reliably, many datapoints are required (e.g. for 50 stocks, about 5 years of daily data should be a availabl)
+* Estimates (of returns and variances) are noisy. "robust optimization" takes ito account confidence interval. Instead of actual return values, only the relative ranks could be considered.
+* Single-period optimization: The optimization described above has one fixed timeframe in mind. If different time horizons were considered, different optimal actions (e.g. long/short a stock) could result in different time periods. This is considered by [multi-period optimization](https://stanford.edu/~boyd/papers/pdf/cvx_portfolio.pdf).
+* Transaction costs could be considered as constraints in the optimization problem.
 
                                   
 

diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -883,7 +883,7 @@ The SML is also used to compare similar securities with approximately similar re
 * x_(A,min) = (sigma_B^2 - sigma_A sigma_B rho)/(sigma_A^2 + sigma_B^2 - 2 sigma_A sigma_B rho), x_B = 1 - x_A
 * mu_P = mu_A x_A + mu_B x_B; mu_P is the portfolio mean return; mu_B and mu_B the individual asset mean returns.
 
-####Formulating portfolio optimization problems
+#### Formulating portfolio optimization problems
 
 So far, we've discussed one way to formulate a portfolio optimization problem. We learned to set the portfolio variance as the objective function, while imposing the constraint that the portfolio weights should sum to 1. However, in practice you may frame the problem a little differently. Let's talk about some of the different ways to set up a portfolio optimization problem.
 
@@ -937,7 +937,7 @@ What if you want to minimize portfolio variance, but have the portfolio track an
 
 *objective:* minimize: **x^TPx** + λ ∥**x** − **q**∥_2, q = a set of index weights, λ = a tradeoff parameter
 
-####Python package for optimization problems: cvxpy
+#### Python package for optimization problems: cvxpy
 
 cvxpy is a Python package for solving convex optimization problems. It allows you to express the problem in a human-readable way, calls a solver, and unpacks the results.
 

diff --git a/AI_for_Trading.md b/AI_for_Trading.md
@@ -937,6 +937,40 @@ What if you want to minimize portfolio variance, but have the portfolio track an
 
 *objective:* minimize: **x^TPx** + λ ∥**x** − **q**∥_2, q = a set of index weights, λ = a tradeoff parameter
 
+####Python package for optimization problems: cvxpy
+
+cvxpy is a Python package for solving convex optimization problems. It allows you to express the problem in a human-readable way, calls a solver, and unpacks the results.
+
+**How to use cvxpy**
+
+*Import:* First, you need to import the package: import cvxpy as cvx
+
+*Steps:* Optimization problems involve finding the values of a variable that minimize an objective function under a set of constraints on the range of possible values the variable can take. So we need to use cvxpy to declare the variable, objective function and constraints, and then solve the problem.
+
+*Optimization variable:* Use cvx.Variable() to declare an optimization variable. For portfolio optimization, this will be **x**, the vector of weights on the assets. Use the argument to declare the size of the variable; e.g. x = cvx.Variable(2) declares that **x** is a vector of length 2. In general, variables can be scalars, vectors, or matrices.
+
+*Objective function:* Use cvx.Minimize() to declare the objective function. For example, if the objective function is (**x** − **y**)^2, you would declare it to be: objective = cvx.Minimize((x - y)**2).
+
+*Constraints:* You must specify the problem constraints with a list of expressions. For example, if the constraints are **x** + **y** = 1 and **x** − **y** ≥ 1 you would create the list: constraints = [x + y == 1, x - y >= 1]. Equality and inequality constraints are elementwise, whether they involve scalars, vectors, or matrices. For example, together the constraints 0 <= x and x <= 1 mean that every entry of **x** is between 0 and 1. You cannot construct inequalities with < and >. Strict inequalities don’t make sense in a real world setting. Also, you cannot chain constraints together, e.g., 0 <= x <= 1 or x == y == 2.
+
+*Quadratic form:* Use cvx.quad_form() to create a quadratic form. For example, if you want to minimize portfolio variance, and you have a covariance matrix **P**, the quantity cvx.quad_form(x, P) represents the quadratic form **x^TPx**, the portfolio variance.
+
+*Norm:* Use cvx.norm() to create a norm term. For example, to minimize the distance between **x** and another vector, **b**, i.e. ∥**x** − **b**∥^2, create a term in the objective function cvx.norm(x-b, 2). The second argument specifies the type of norm; for an L2-norm, use the argument 2.
+
+*Constants:* Constants are the quantities in objective or constraint expressions that are not Variables. You can use your numeric library of choice to construct matrix and vector constants. For instance, if x is a cvxpy Variable in the expression A*x + b, A and b could be Numpy ndarrays, Numpy matrices, or SciPy sparse matrices. A and b could even be different types.
+
+*Optimization problem:* The core step in using cvxpy to solve an optimization problem is to specify the problem. Remember that an optimization problem involves minimizing an objective function, under some constraints, so to specify the problem, you need both of these. Use cvx.Problem() to declare the optimization problem. For example, problem = cvx.Problem(objective, constraints), where objective and constraints are quantities you've defined earlier. Problems are immutable. This means that you cannot modify a problem’s objective or constraints after you have created it. If you find yourself wanting to add a constraint to an existing problem, you should instead create a new problem.
+
+*Solve:* Use problem.solve() to run the optimization solver.
+
+*Status:* Use problem.status to access the status of the problem and check whether it has been determined to be unfeasible or unbounded.
+
+*Results:* Use problem.value to access the optimal value of the objective function. Use e.g. x.value to access the optimal value of the optimization variable.
+
+*Examples:* `m3l4_cvxpy_basic.ipynb`, `m3l4_cvxpy_advanced.ipynb`
+
+
+
No results found