Quantifying the Efficacy of Machine Learning Algorithms

There are many articles and books and talks and so on all making claims about machine learning algorithms. Out of those, how many actually QUANTIFY THE EFFICACY OF THE ALGORITHM?.

Below are some references which will hopefully save people some leg work and help others quantify the performance of their algorithms.

1. Articles
   
   1. Advanced non-parametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining by S García
      
   2. Building and deploying large-scale machine learning pipelines by Ben Lorica
      
   3. Characterization and evaluation of similarity measures for pairs of clusterings by Pfitzner, Leibbrandt, Powers
      
   4. Cost Curves: An Improved Method for Visualizing Classifier Performance by Chris Drummond, Robert C. Holte
      
   5. Evaluating Machine Learning Methods: Scored Receiver Operating Characteristics (sROC) Curves by William Klement
      
   6. Evaluation of Classifiers by Jin Tian
      
   7. Multiple Criteria for Evaluating Machine Learning Algorithms for Land Cover Classification from Satellite Data by DeFries, Chan
      
   8. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation by Cawley, Talbot
      
   9. Performance Evaluation of Machine Learning Algorithms by Mohak Shah, Nathalie Japkowicz
      
   10. Projection - Based Framework for Classifier Performance Evaluation by Nathalie Japkowicz, Pritika Sanghi, Peter Tischer
       
   11. Publications by Chris Drummond
       
   12. Visualization of Tradeoff in Evaluation: from Precision - Recall & PN to LIFT, ROC & BIRD by Powers
       
   13. Visualizing Classifier Performance on Different Domains by Alaiz-Rodrıguez, Japkowicz, Tischer
       
   14. What are commonly used performance measures for machine learning algorithms? at Research Gate
       
   15. Wikipedia: confusion matrix
       
2. Books
   
   1. Assessing and Improving Prediction and Classification by Timothy Masters
      
      1. Amazon
         
   2. Evaluating and Comparing the Performance of Machine Learning Algorithms by Melanie Mitchell
      
      1. Part 1
         
      2. Part 2
         
   3. Evaluating Learning Algorithms: A Classification Perspective by Japkowicz, Shah
      
      1. Amazon
         
         1. Customer Reviews
            
            1. Howard B. Bandy on July 7, 2014
               
               1. ... examples presented are all of stationary data ...
                  
      2. Cambridge.Org
         
         1. Looking for an examination copy?
            
            1. If you are interested in the title for your course we can consider offering an examination copy. To register your interest please contact collegesales@cambridge.org providing details of the course you are teaching.
               
      3. Google
         
         1. Has table of contents and you can look at some randomly selected pages
            
      4. MohakShah.Com
         
         1. Email Address: eval@mohakshah.com
            
         2. For electronic editions, ...
            
         3. Computing Resources
            
            1. WEKA Machine Learning Toolkit
               
            2. Visualization Software for Clasifier Evaluation
               
   4. Evaluation and Analysis of Supervised Learning Algorithms and Classifiers by Niklas Lavesson
      
   5. Machine Learning and Data Mining: 14 Evaluation and Credibility by Pier Luca Lanzi
      
      1. slideshare.net
         
      2. Data Mining Course
         
         1. Classification: Evaluation