Skip to main content

AutoML-Papers

Awesome-AutoML-Papers

A curated list of automated machine learning papers, articles, tutorials, slides and projects.

Introduction to AutoML

Machine learning (ML) has achieved considerable successes in recent years and an ever-growing number of disciplines rely on it. However, this success crucially relies on human machine learning experts to perform the following tasks:
  • Preprocess the data
  • Select appropriate features
  • Select an appropriate model family
  • Optimize model hyperparameters
  • Postprocess machine learning models
  • Critically analyze the results obtained.
As the complexity of these tasks is often beyond non-ML-experts, the rapid growth of machine learning applications has created a demand for off-the-shelf machine learning methods that can be used easily and without expert knowledge. We call the resulting research area that targets progressive automation of machine learning AutoML.
AutoML draws on many disciplines of machine learning, prominently including
  • Bayesian optimization
  • Regression models for structured data and big data
  • Meta learning
  • Transfer learning, and
  • Combinatorial optimization.

Table of Contents

Papers

Automated Feature Engineering

  • Expand Reduce

    • 2017 | AutoLearn — Automated Feature Generation and Selection | Ambika Kaul, et al. | ICDM | PDF
    • 2017 | One button machine for automating feature engineering in relational databases | Hoang Thanh Lam, et al. | arXiv | PDF
    • 2016 | Automating Feature Engineering | Udayan Khurana, et al. | NIPS | PDF
    • 2016 | ExploreKit: Automatic Feature Generation and Selection | Gilad Katz, et al. | ICDM | PDF
    • 2015 | Deep Feature Synthesis: Towards Automating Data Science Endeavors | James Max Kanter, Kalyan Veeramachaneni | DSAA | PDF

  • Hierarchical Organization of Transformations

    • 2016 | Cognito: Automated Feature Engineering for Supervised Learning | Udayan Khurana, et al. | ICDMW | PDF

  • Meta Learning

    • 2017 | Learning Feature Engineering for Classification | Fatemeh Nargesian, et al. | IJCAI | PDF

  • Reinforcement Learning

    • 2017 | Feature Engineering for Predictive Modeling using Reinforcement Learning | Udayan Khurana, et al. | arXiv | PDF
    • 2010 | Feature Selection as a One-Player Game | Romaric Gaudel, Michele Sebag | ICML | PDF

Architecture Search

  • Evolutionary Algorithms

    • 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR | PDF
    • 2002 | Evolving Neural Networks through Augmenting Topologies | Kenneth O.Stanley, Risto Miikkulainen | Evolutionary Computation | PDF

  • Local Search

    • 2017 | Simple and Efficient Architecture Search for Convolutional Neural Networks | Thomoas Elsken, et al. | ICLR | PDF

  • Meta Learning

    • 2016 | Learning to Optimize | Ke Li, Jitendra Malik | arXiv | PDF

  • Reinforcement Learning

    • 2018 | AMC: AutoML for Model Compression and Acceleration on Mobile Devices | Yihui He, et al. | ECCV | PDF
    • 2018 | Efficient Neural Architecture Search via Parameter Sharing | Hieu Pham, et al. | arXiv | PDF
    • 2017 | Neural Architecture Search with Reinforcement Learning | Barret Zoph, Quoc V. Le | ICLR | PDF

  • Transfer Learning

    • 2017 | Learning Transferable Architectures for Scalable Image Recognition | Barret Zoph, et al. | arXiv | PDF

  • Network Morphism

    • 2018 | Efficient Neural Architecture Search with Network Morphism | Haifeng Jin, et al. | arXiv | PDF

  • Continuous Optimization

    • 2018 | Neural Architecture Optimization | Renqian Luo, et al. | arXiv | PDF

Frameworks

  • 2017 | Google Vizier: A Service for Black-Box Optimization | Daniel Golovin, et al. | KDD |PDF
  • 2017 | ATM: A Distributed, Collaborative, Scalable System for Automated Machine Learning | T. Swearingen, et al. | IEEE | PDF
  • 2015 | AutoCompete: A Framework for Machine Learning Competitions | Abhishek Thakur, et al. | ICML | PDF

Hyperparameter Optimization

  • Bayesian Optimization

    • 2018 | Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features | Mojmír Mutný, et al. | NeurIPS | PDF
    • 2018 | High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groups. | PMLR | PDF
    • 2018 | A Tutorial on Bayesian Optimization. | PDF
    • 2016 | Bayesian Optimization with Robust Bayesian Neural Networks | Jost Tobias Springenberg， et al. | NIPS | PDF
    • 2016 | Scalable Hyperparameter Optimization with Products of Gaussian Process Experts | Nicolas Schilling, et al. | PKDD | PDF
    • 2016 | Taking the Human Out of the Loop: A Review of Bayesian Optimization | Bobak Shahriari, et al. | IEEE | PDF
    • 2016 | Towards Automatically-Tuned Neural Networks | Hector Mendoza, et al. | JMLR | PDF
    • 2016 | Two-Stage Transfer Surrogate Model for Automatic Hyperparameter Optimization | Martin Wistuba, et al. | PKDD | PDF
    • 2015 | Efficient and Robust Automated Machine Learning | PDF
    • 2015 | Hyperparameter Optimization with Factorized Multilayer Perceptrons | Nicolas Schilling, et al. | PKDD | PDF
    • 2015 | Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization | Martin Wistua, et al. | PDF
    • 2015 | Joint Model Choice and Hyperparameter Optimization with Factorized Multilayer Perceptrons | Nicolas Schilling, et al. | ICTAI | PDF
    • 2015 | Learning Hyperparameter Optimization Initializations | Martin Wistuba, et al. | DSAA | PDF
    • 2015 | Scalable Bayesian optimization using deep neural networks | Jasper Snoek, et al. | ACM | PDF
    • 2015 | Sequential Model-free Hyperparameter Tuning | Martin Wistuba, et al. | ICDM | PDF
    • 2013 | Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms | PDF
    • 2013 | Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures | J. Bergstra | JMLR | PDF
    • 2012 | Practical Bayesian Optimization of Machine Learning Algorithms | PDF
    • 2011 | Sequential Model-Based Optimization for General Algorithm Configuration(extended version) | PDF

  • Evolutionary Algorithms

    • 2018 | Autostacker: A Compositional Evolutionary Learning System | Boyuan Chen, et al. | arXiv | PDF
    • 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR | PDF
    • 2016 | Evaluation of a tree-based pipeline optimization tool for automating data science | Randal S. Olson, et al. | GECCO | PDF
    • 2016 | Automating biomedical data science through tree-based pipeline optimization | Randal S. Olson, et al. | ECAL | PDF

  • Lipschitz Functions

    • 2017 | Global Optimization of Lipschitz functions | C´edric Malherbe, Nicolas Vayatis | arXiv | PDF

  • Local Search

    • 2009 | ParamILS: An Automatic Algorithm Configuration Framework | Frank Hutter, et al. | JAIR | PDF

  • Meta Learning

    • 2008 | Cross-Disciplinary Perspectives on Meta-Learning for Algorithm Selection | PDF

  • Particle Swarm Optimization

    • 2017 | Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks | Pablo Ribalta Lorenzo, et al. | GECCO | PDF
    • 2008 | Particle Swarm Optimization for Parameter Determination and Feature Selection of Support Vector Machines | Shih-Wei Lin, et al. | Expert Systems with Applications | PDF

  • Random Search

    • 2016 | Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization | Lisha Li, et al. | arXiv | PDF
    • 2012 | Random Search for Hyper-Parameter Optimization | James Bergstra, Yoshua Bengio | JMLR | PDF
    • 2011 | Algorithms for Hyper-parameter Optimization | James Bergstra, et al. | NIPS | PDF

  • Transfer Learning

    • 2016 | Efficient Transfer Learning Method for Automatic Hyperparameter Tuning | Dani Yogatama, Gideon Mann | JMLR | PDF
    • 2016 | Flexible Transfer Learning Framework for Bayesian Optimisation | Tinu Theckel Joy, et al. | PAKDD | PDF
    • 2016 | Hyperparameter Optimization Machines | Martin Wistuba, et al. | DSAA | PDF
    • 2013 | Collaborative Hyperparameter Tuning | R´emi Bardenet, et al. | ICML | PDF

Miscellaneous

  • 2018 | Accelerating Neural Architecture Search using Performance Prediction | Bowen Baker, et al. | ICLR | PDF
  • 2017 | Automatic Frankensteining: Creating Complex Ensembles Autonomously | Martin Wistuba, et al. | SIAM | PDF

Tutorials

Bayesian Optimization

  • 2010 | A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning | PDF

Meta Learning

  • 2008 | Metalearning - A Tutorial | PDF

Articles

Bayesian Optimization

  • 2016 | Bayesian Optimization for Hyperparameter Tuning | Link

Meta Learning

  • 2017 | Why Meta-learning is Crucial for Further Advances of Artificial Intelligence? | Link
  • 2017 | Learning to learn | Link

Slides

Automated Feature Engineering

  • Automated Feature Engineering for Predictive Modeling | Udyan Khurana, etc al. | PDF

Hyperparameter Optimization

Bayesian Optimization

  • Bayesian Optimisation | PDF
  • A Tutorial on Bayesian Optimization for Machine Learning | PDF

Books

Meta Learning

  • 2009 | Metalearning - Applications to Data Mining | Springer | PDF

Projects

  • Advisor | Python | Open Source | Code
  • Auger | Python | Commercial | Link
  • auto-sklearn | Python | Open Source | Code
  • Auto-Keras | Python | Open Source | Code
  • Auto-WEKA | Java | Open Source | Code
  • Hyperopt | Python | Open Source | Code
  • Hyperopt-sklearn | Python | Open Source | Code
  • SigOpt | Python | Commercial | Link
  • SMAC3 | Python | Open Source | Code
  • RoBO | Python | Open Source | Code
  • BayesianOptimization | Python | Open Source | Code
  • Scikit-Optimize | Python | Open Source | Code
  • HyperBand | Python | Open Source | Code
  • BayesOpt | C++ | Open Source | Code
  • Optunity | Python | Open Source | Code
  • TPOT | Python | Open Source | Code
  • ATM | Python | Open Source | Code
  • Cloud AutoML | Python | Commercial| Link
  • H2O | Python | Commercial | Link
  • DataRobot | Python | Commercial | Link
  • MLJAR | Python | Commercial | Link
  • MateLabs | Python | Commercial | Link
  • FAR-HO | Python | Open Source | Code
  • TransmogrifAI | Scala | Open Source | Code
  • DEvol | Python | Open Source | Code
  • HyperparameterHunter | Python | Open Source | Code
  • NNI | Python & Typescript | Open Source | Code
  • Tune | Python | Open Source | Code | Docs
  • Milano | Python | Open Source | Code
  • Katib | Python | Open Source | Code
  • nasbot | Python | Open Source | Code
  • rbfopt | Python | Open Source | Code
  • amla | Python | Open Source | Code
  • HpBandSter | Python | Open Source | Code

Prominent Researchers

Acknowledgement

Special thanks to everyone who contributed to this project.

Licenses

Awesome-AutoML-Papers is available under Apache Licenses 2.0.

Contact & Feedback

If you have any suggestions (missing papers, new papers, key researchers or typos), feel free to pull a request. Also you can mail to:
Labels:

Comments

Post a Comment

Popular posts from this blog

Introduction to Machine Learning in Python

Python tutorials for introduction to machine learning Introduction to Machine Learning in Python This repository provides instructional material for machine learning in python. The material is used for two classes taught at NYU Tandon by  Sundeep Rangan : EE-UY / CS-UY 4563: Introduction to Machine Learning (Undergraduate) EL-GY 6123: Introduction to Machine Learning (Graduate) Anyone is free to use and copy this material (at their own risk!). But, please cite the material if you use the material in your own class. Pre-requisites All the software can be run on any laptop (Windows, MAC or UNIX).  Instructions  are also provided to run the code in Google Cloud Platform on a virtual machine (VM). Both classes assume no python or ML experience. However, experience with some programming language (preferably object-oriented) is required. To follow all the mathematical details and to complete the homework exercises, the class assumes undergraduate probability, linear algebra
2 comments
Read more

R tutorials for Data Science, NLP and Machine Learning

R Data Science Tutorials This repo contains a curated list of R tutorials and packages for Data Science, NLP and Machine Learning. This also serves as a reference guide for several common data analysis tasks. Curated list of Python tutorials for Data Science, NLP and Machine Learning . Comprehensive topic-wise list of Machine Learning and Deep Learning tutorials, codes, articles and other resources . Learning R Online Courses tryR on Codeschool Introduction to R for Data Science - Microsoft | edX Introduction to R on DataCamp Data Analysis with R Free resources for learning R R for Data Science - Hadley Wickham Advanced R - Hadley Wickham swirl: Learn R, in R Data Analysis and Visualization Using R MANY R PROGRAMMING TUTORIALS A Handbook of Statistical Analyses Using R , Find Other Chapters Cookbook for R Learning R in 7 simple steps More Resources Awesome-R Repository on GitHub R Reference Card: Cheatsheet R bloggers: blog aggregator R Resources
Post a Comment
Read more

Machine Learning API tutorial

Fast Style Transfer in  TensorFlow Add styles from famous paintings to any photo in a fraction of a second!  You can even style videos!     It takes 100ms on a 2015 Titan X to style the MIT Stata Center (1024×680) like Udnie, by Francis Picabia. Our implementation is based off of a combination of Gatys'  A Neural Algorithm of Artistic Style , Johnson's  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , and Ulyanov's  Instance Normalization . Running on FloydHub It is very easy to train, evaluate and serve fast style transfer on [Floydhub][ https://www.floydhub.com/ ]. Follow the instructions below: Visit  Floydhub  site to create an account if you do not have one. Clone this repository to your machine. Run  floyd init <project name>  inside the directory. Now you can: Train a new model Evaluate an existing model Serve a trained model at a public API. Video Stylization Here we transformed every frame in a video, then co
Post a Comment
Read more