the tools we've discussed in this book are great for many machine learning applicationsand allow very quick analysis and prototyping python and scikit-learn are also used in production systems in many organizations--even very large ones like international banks and global social media companies howevermany companies have complex infrastructureand it is not always easy to include python in these systems that is not necessarily problem in many companiesthe data analytics teams work with languages like python and that allow the quick testing of ideaswhile production teams work with languages like goscalac++and java to build robustscalable systems data analysis has different requirements from building live servicesand so using different languages for these tasks makes sense relatively common solution is to reimplement the solution that was found by the analytics team inside the larger frameworkusing high-performance language this can be easier than embedding whole library or programming language and converting from and to the different data formats regardless of whether you can use scikit-learn in production system or notit is important to keep in mind that production systems have different requirements from one-off analysis scripts if an algorithm is deployed into larger systemsoftware engineering aspects like reliabilitypredictabilityruntimeand memory requirements gain relevance simplicity is key in providing machine learning systems that perform well in these areas critically inspect each part of your data processing and prediction pipeline and ask yourself how much complexity each step createshow robust each component is to changes in the data or compute infrastructureand if the benefit of each component warrants the complexity if you are building involved machine learning systemswe highly recommend reading the paper "machine learningthe high interest credit card of technical debt"published by researchers in google' machine learning team the paper highlights the trade-off in creating and maintaining machine learning software in production at large scale while the issue of technical debt is particularly pressing in large-scale and long-term projectsthe lessons learned can help us build better software even for short-lived and smaller systems testing production systems in this bookwe covered how to evaluate algorithmic predictions based on test set that we collected beforehand this is known as offline evaluation if your machine learning system is user-facingthis is only the first step in evaluating an algorithmthough the next step is usually online testing or live testingwhere the consequences of employing the algorithm in the overall system are evaluated changing the recommendations or search results users are shown by website can drastically change their behavior and lead to unexpected consequences to protect against these surprisesmost user-facing services employ / testinga form of blind user study in from prototype to production