Machine Learning in Data Analysis

1. 1.1 Start from the decision7 min
2. 1.2 Prediction is not explanation7 min
3. 1.3 The cost of being wrong7 min
4. 1.4 The running example and the plan6 min

Objectives. Define supervised and unsupervised learning and give a health example of each. Distinguish prediction from explanation and say which one a given question needs. State the decision a model serves and the cost of a false positive and a false negative. Name three situations where a simple rule or a regression is the better choice. Describe the running example and the outreach decision it supports.

1. 2.1 Features and the label7 min
2. 2.2 Split first7 min
3. 2.3 Fit, predict and the pipeline7 min
4. 2.4 Three leakage checks6 min

Objectives. Prepare a feature table and a label from the analysis table with the decision time respected. Split the data into training and test sets before any preprocessing, and explain why. Fit a logistic regression and a decision tree in scikit-learn and produce scores for new cases. Build a scikit-learn pipeline so that preprocessing is learned only from the training data. Run the three leakage checks before believing any performance figure.

1. 3.1 The confusion matrix7 min
2. 3.2 The curve and its area7 min
3. 3.3 Calibration7 min
4. 3.4 The performance paragraph6 min

Objectives. Build a confusion matrix at a chosen threshold and compute the four rates from it. Explain sensitivity, specificity, positive and negative predictive value in words a ward understands. Read a receiver operating curve and its area, and say what the area does not tell you. Check calibration and explain why a well ranked model can still lie about risk. Choose a threshold from costs and capacity and evaluate at the top of the list.

1. 4.1 Trees and forests7 min
2. 4.2 A fair comparison7 min
3. 4.3 Importance and explanation7 min
4. 4.4 Finding groups without a label6 min

Objectives. Explain how a decision tree makes a prediction and why a single tree overfits. Fit a random forest and compare it fairly with the logistic baseline by cross validation. Read feature importance and permutation importance and say what they do not mean. Use clustering to describe groups of facilities and interpret the groups. Decide when a more complex model is worth its loss of explainability.

1. 5.1 Fair on average, unfair to some7 min
2. 5.2 Validated elsewhere, monitored always7 min
3. 5.3 Principles and governance7 min
4. 5.4 Capstone, your model plan7 min

Objectives. Check a model performance separately for the groups it could treat unfairly and interpret the gaps. Explain why a model must be validated on data from a different place or time before it is trusted there. Design monitoring for a deployed model and state the rule that would stop it. Apply the WHO principles for artificial intelligence in health and the national data governance rules to a model project. Write the capstone, a full model plan from decision to stop rule.