Readings

Required readings

• 🗎 Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.F. and Dennison, D., 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in neural information processing systems (pp. 2503-2511).

Debugging?

What went wrong? Where? How to fix?

Debugging Questions beyond Interpretability

• Can we reproduce the problem?
• What were the inputs to the model?
• Which exact model version was used?
• What data was the model trained with?
• What pipeline code was the model trained with?
• Where does the data come from? How was it processed/extracted?
• Were other models involved? Which version? Based on which data?
• What parts of the input are responsible for the (wrong) answer? How can we fix the model?

Model Chaining: Automatic meme generator

Version all models involved!

Complex Model Composition: ML Models for Feature Extraction

Breakout Discussion: Movie Predictions

How to Version Large Datasets?

InquiryID,CustomerID,InquiryDate,LoanType,LoanAmount,AccountStatus,PaymentStatus
1001,001,2020-01-15,Mortgage,250000,Open,Current
1002,002,2020-02-20,Auto Loan,20000,Closed,Paid Off
1003,003,2020-03-05,Credit Card,5000,Open,Late (30 days)
1004,004,2020-04-10,Personal Loan,10000,Open,Current
1005,005,2020-05-15,Student Loan,30000,Closed,Paid Off
1006,001,2020-06-20,Mortgage,200000,Open,Current
1007,002,2020-07-25,Credit Card,7000,Open,Late (60 days)
1008,003,2020-08-30,Auto Loan,15000,Closed,Paid Off
1009,004,2020-09-10,Personal Loan,8000,Open,Current
1010,005,2020-10-15,Credit Card,10000,Open,Late (90 days)

(example customer data from the credit scenario)

Recall: Event Sourcing

• Append only databases
• Record edit events, never mutate data
• Compute current state from all past events, can reconstruct old state
• For efficiency, take state snapshots
• Similar to traditional database logs

createUser(id=5, name="Christian", dpt="SCS")
updateUser(id=5, dpt="ISR")
deleteUser(id=5)

Versioning Strategies for Datasets

1. Store copies of entire datasets (like Git), identify by checksum
2. Store deltas between datasets (like Mercurial)
3. Offsets in append-only database (like Kafka), identify by offset
4. History of individual database records (e.g. S3 bucket versions)
   • some databases specifically track provenance (who has changed what entry when and how)
   • specialized data science tools eg Hangar for tensor data
5. Version pipeline to recreate derived datasets ("views", different formats)
   • e.g. version data before or after cleaning?

Aside: Git Internals

Versioning Models

Versioning Models

Usually no meaningful delta/compression, version as binary objects

Any system to track versions of blobs

Versioning Pipelines

Associate model version with pipeline code version, data version, and hyperparameters!

Versioning Dependencies

Pipelines depend on many frameworks and libraries

Ensure reproducible builds

• Declare versioned dependencies from stable repository (e.g. requirements.txt + pip)
• Avoid floating versions
• Optionally: commit all dependencies to repository ("vendoring")

Optionally: Version entire environment (e.g. Docker container)

Test build/pipeline on independent machine (container, CI server, ...)

ML Versioning Tools (MLOps)

Tracking data, pipeline, and model versions

Modeling pipelines: inputs and outputs and their versions

• automatically tracks how data is used and transformed

Often tracking also metadata about versions

• Accuracy
• Training time
• ...

Example: DVC

dvc add images
dvc run -d images -o model.p cnn.py
dvc remote add myrepo s3://mybucket
dvc push

• Tracks models and datasets, built on Git
• Splits learning into steps, incrementalization
• Orchestrates learning in cloud resources

https://dvc.org/

DVC Example

stages:
  features:
    cmd: jupyter nbconvert --execute featurize.ipynb
    deps:
      - data/clean
    params:
      - levels.no
    outs:
      - features
    metrics:
      - performance.json
  training:
    desc: Train model with Python
    cmd:
      - pip install -r requirements.txt
      - python train.py --out ${model_file}
    deps:
      - requirements.txt
      - train.py
      - features
    outs:
      - ${model_file}:
          desc: My model description
    plots:
      - logs.csv:
          x: epoch
          x_label: Epoch
    meta: 'For deployment'
    # User metadata and comments are supported

Experiment Tracking

Log information within pipelines: hyperparameters used, evaluation results, and model files

Many tools: MLflow, ModelDB, Neptune, TensorBoard, Weights & Biases, Comet.ml, ...

ModelDB Example

from verta import Client
client = Client("http://localhost:3000")

proj = client.set_project("My first ModelDB project")
expt = client.set_experiment("Default Experiment")

# log the first run
run = client.set_experiment_run("First Run")
run.log_hyperparameters({"regularization" : 0.5})
run.log_dataset_version("training_and_testing_data", dataset_version)
model1 = # ... model training code goes here
run.log_metric('accuracy', accuracy(model1, validationData))
run.log_model(model1)

# log the second run
run = client.set_experiment_run("Second Run")
run.log_hyperparameters({"regularization" : 0.8})
run.log_dataset_version("training_and_testing_data", dataset_version)
model2 = # ... model training code goes here
run.log_metric('accuracy', accuracy(model2, validationData))
run.log_model(model2)

Google's Goods

Automatically derive data dependencies from system log files

Track metadata for each table

No manual tracking/dependency declarations needed

Requires homogeneous infrastructure

Similar systems for tracking inside databases, MapReduce, Sparks, etc.

From Model Versioning to Deployment

Decide which model version to run where

• automated deployment and rollback (cf. canary releases)
• Kubernetis, Cortex, BentoML, ...

Track which prediction has been performed with which model version (logging)

Logging and Audit Traces

Key goal: If a customer complains about an interaction, can we reproduce the prediction with the right model? Can we debug the model's pipeline and data? Can we reproduce the model?

• Version everything
• Record every model evaluation with model version
• Append only, backed up

<date>,<model>,<model version>,<feature inputs>,<output>
<date>,<model>,<model version>,<feature inputs>,<output>
<date>,<model>,<model version>,<feature inputs>,<output>

Logging for Composed Models

Ensure all predictions are logged

Data Provenance

Excursion: Provenance Tracking in Databases

Whenever value is changed, record:

• who changed it
• time of change
• history of previous values
• possibly also justifcation of why

Embedded as feature in some databases or implemented in business logic

Possibly signing with cryptographic methods

Tracking Data Lineage

Document all data sources

Identify all model dependencies and flows

Ideally model all data and processing code

Avoid "visibility debt"

(Advanced: Use infrastructure to automatically capture/infer dependencies and flows as in Goods)

Feature Provenance

How are features extracted from raw data?

• during training
• during inference

Has feature extraction changed since the model was trained?

Recommendation: Modularize and version feature extraction code

Example?

Advanced Practice: Feature Store

Stores feature extraction code as functions, versioned

Catalog features to encourage reuse

Compute and cache features centrally

Use same feature used in training and inference code

Advanced: Immutable features -- never change existing features, just add new ones (e.g., creditscore, creditscore2, creditscore3)

Model Provenance

How was the model trained?

What data? What library? What hyperparameter? What code?

Ensemble of multiple models?

In Real Systems: Tracking Provenance Across Multiple Models

Complex Model Composition: ML Models for Feature Extraction

Summary: Provenance

Data provenance

Feature provenance

Model provenance

Breakout Discussion: Movie Predictions (Revisited)

Assume you are receiving complains that a child gets mostly recommendations about R-rated movies

Discuss again, updating the previous post in #lecture:

• How would you identify the model that caused the prediction?
• How would you identify the code and dependencies that trained the model?
• How would you identify the training data used for that model?

On Terminology

Replicability: ability to reproduce results exactly

• Ensures everything is clear and documented
• All data, infrastructure shared; requires determinism

Reproducibility: the ability of an experiment to be repeated with minor differences, achieving a consistent expected result

• In science, reproducing important to gain confidence
• many different forms distinguished: conceptual, close, direct, exact, independent, literal, nonexperiemental, partial, retest, ...

"Reproducibility" of Notebooks

Practical Reproducibility

Ability to generate the same research results or predictions

Recreate model from data

Requires versioning of data and pipeline (incl. hyperparameters and dependencies)

Nondeterminism

• Model inference almost always deterministic for a given model
• Many machine learning algorithms are nondeterministic
  • Nondeterminism in neural networks initialized from random initial weights
  • Nondeterminism from distributed computing, random forests
  • Determinism in linear regression and decision trees
• Many notebooks and pipelines contain nondeterminism
  • Depend on time or snapshot of online data (e.g., stream)
  • Initialize random seed
  • Different memory addresses for figures
• Different library versions installed on the machine

Recommendations for Reproducibility

• Version pipeline and data (see above)
• Document each step
  • document intention and assumptions of the process (not just results)
  • e.g., document why data is cleaned a certain way
  • e.g., document why certain parameters chosen
• Ensure determinism of pipeline steps (-> test)
• Modularize and test the pipeline
• Containerize infrastructure -- see MLOps

Further Readings

• Sugimura, Peter, and Florian Hartl. “Building a Reproducible Machine Learning Pipeline.” arXiv preprint arXiv:1810.04570 (2018).
• Chattopadhyay, Souti, Ishita Prasad, Austin Z. Henley, Anita Sarma, and Titus Barik. “What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities.” In Proceedings of the CHI Conference on Human Factors in Computing Systems, 2020.
• Sculley, D, et al. “Hidden technical debt in machine learning systems.” In Advances in neural information processing systems, pp. 2503–2511. 2015.

Recall: Composing Models: Ensemble and metamodels

Recall: Composing Models: Decomposing the problem, sequential

Recall: Composing Models: Cascade/two-phase prediction

Decomposing the Image Captioning Problem?

State of the Art Decomposition (in 2015)

Blame assignment?

Nonmonotonic errors

Chasing Bugs

• Update, clean, add, remove data
• Change modeling parameters
• Add regression tests
• Fixing one problem may lead to others, recognizable only later

Partitioning Contexts

Overrides

Ideas?