You are here

Lessons Learned from Testing Machine Learning Software

The models will learn what you teach them to learn. A phrase that describes the main problem which you face when you testing machine learning software, since not only must test the software but also the learning model.Throughout these three years in my team, we have learned that the software built around machine learning algorithms presents a number of challenges and peculiarities for testers, ranging from the application of statistical elements to the knowledge and understanding of neural learning models, therefore I see starting to emerge the figure of a new type of tester the "Experimental Data Scientist" that as in any other scientific discipline is responsible to validate, check and ensure the accuracy and legitimacy of what theoretical models predict.

But not only this, the ecosystem and the necessary infrastructure to conduct experiments is also peculiar, it is here where the testers have to use a variety of technologies and tools capable of supporting anything that involves exercising this kind of software, ranging from containers to the most specialized math libraries.

With all this as a preamble, in this talk, we will see all the things that we fail when faced with this kind of software as well as how to build the necessary ecosystem, and show you some examples of challenges and practical solutions.We'll see how by many failures, errors, and mistakes we have learned some lessons on how to deal with such problems. Some of the things you will see are:

  • Experimental Data Scientist
  • Knowing and learning from the model (Everything is around of training data, objective functions, and metrics)*When the model is wrong (Example with spurious correlations, and adversarial examples)
  • Property-based testing applied to models
  • Supervised versus unsupervised learning, or together
  • Big Data, how many really we used?
  • Dependencies, the output of a model is always the input of the next model (Example showing a real model)
  • Infrastructure, when your laptop is no longer enough
  • Managing expectations, this is not magic!


Key takeaways:

  • Basic knowledge about machine learning
  • Stories about real problems that we faced in testing machine learning software
  • Some techniques of software testing applied in machine learning