Una Differenza fondamentale tra Machine Learning e Statistica

I strongly recommend watching these two videos by Josh Angrist

• What’s the Difference Between Econometrics and Data Science?
• What’s the Difference Between Econometrics and Statistics?

The error of a supervised learning algorithm has three components:

• Bias: error caused by erroneous assumptions
• Variance: error caused by overfitting
• Noise: irreducible error, due to the random nature of the variable

A central problem in classical statistics (Econometrics) is to find the MVUE, the estimator with minimum variance among all unbiased estimators.

Suppose all assumptions of Gauss-Markov theorem holds, OLS is the BLUE but it doesn’t mean it yield the minimum MSE, indeed there are non linear biased estimator (james-stein estimator) with lower variance and linear biased estimator with lower MSE (Ridge, Lasso).

A central problem in Machine Learning (ML) is to minimize overall errors: capture the regularities in the data (low bias), while generalizing well to unseen data (low variance).

Econometric models are willing to sacrifice performance in exchange for zero bias while ML models find a balance between bias and variance, in order to maximize performance.

ML methods can achieve lower MSE than Econometric models through Cross-validation, Complexity reduction, Regularization, Feature selection, dimensionality reduction and Ensemble methods (Bootstrap aggregation (Bagging),Boosting and Stacking).