1. 程式人生 > >Why Applied Machine Learning Is Hard

Why Applied Machine Learning Is Hard

How to Handle the Intractability of Applied Machine Learning.

Applied machine learning is challenging.

You must make many decisions where there is no known “right answer” for your specific problem, such as:

  • What framing of the problem to use?
  • What input and output data to use?
  • What learning algorithm to use?
  • What algorithm configuration to use?

This is challenging for beginners that expect that you can calculate or be told what data to use or how to best configure an algorithm.

In this post, you will discover the intractable nature of designing learning systems and how to deal with it.

After reading this post, you will know:

  • How to develop a clear definition of your learning problem for yourself and others.
  • The 4 decision points you must consider when designing a learning system for your problem.
  • The 3 strategies that you can use to specifically address the intractable problem of designing learning systems in practice.

Let’s get started.

Overview

This post is divided into 6 sections inspired by chapter 1 of Tom Mitchell’s excellent 1997 book Machine Learning; they are:

  1. Well-Posed Learning Problems
  2. Choose the Training Data
  3. Choose the Target Function
  4. Choose a Representation of the Target Function
  5. Choose a Learning Algorithm
  6. How to Design Learning Systems

Well-Posed Learning Problems

We can define a general learning task in the field of applied machine learning as a program that learns from experience on some task against a specific performance measure.

Tom Mitchell in his 1997 book Machine Learning states this clearly as:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

We take this as a general definition for the types of learning tasks that we may be interested in for applied machine learning such as predictive modeling. Tom lists a few examples to make this clear, such as:

  • Learning to recognize spoken words.
  • Learning to drive an autonomous vehicle.
  • Learning to classify new astronomical structures.
  • Learning to play world-class backgammon.

We can use the above definition to define our own predictive modeling problem. Once defined, the task becomes that of designing a learning system to address it.

Designing a learning system, i.e. an application of machine learning, involves four design choices:

  1. Choosing the training data.
  2. Choosing the target function.
  3. Choosing the representation.
  4. Choosing the learning algorithm.

There might be a best set of choices that you can make for a given problem given infinite resources, but we don’t have infinite time, compute resources, and knowledge about the domain or learning systems.

Therefore, although we can prepare a well-posed description of a learning problem, designing the best possible learning system is intractable.

The best we can do is use knowledge, skill, and available resources to work through the design choices.

Let’s look at each of these design choices in more detail.

Choose the Training Data

You must choose the data your learning system will use as experience from which to learn.

This is the data from past observations.

The type of training experience available can have a significant impact on success or failure of the learner.

It is rarely well formatted and ready to use; often you must collect the data you need (or think you might need) for the learning problem.

This may mean:

  • Scraping documents.
  • Querying databases.
  • Processing files.
  • Collating disparate sources
  • Consolidating entities.

You need to get all of the data together and into a normalized form such that one observation represents one entity for which an outcome is available.

Choose the Target Function

Next, you must choose the framing of the learning problem.

Machine learning is really a problem of learning a mapping function (f) from inputs (X) to outputs (y).

1 y = f(X)

This function can then be used on new data in the future in order to predict the most likely output.

The goal of the learning system is to prepare a function that best maps inputs to outputs given the resources available. The underlying function that actually exists is unknown. If we knew the form of this function, we could use it directly and we would not need machine learning to learn it.

More generally, this is a problem called function approximation. The result will be an approximation, meaning that it will have error. We will do our best to minimize this error, but some error will always exist given noise in the data.

… we have reduced the learning task in this case to the problem of discovering an operational description of the ideal target function V. It may be very difficult in general to learn such an operational form of V perfectly. In fact, we often expect learning algorithms to acquire only some approximation to the target function, and for this reason the process of learning the target function is often called function approximation.

This step is about selecting exactly what data to use as input to the function, e.g. the input features or input variables and what exactly will be predicted, e.g. the output variable.

Often, I refer to this as the framing of your learning problem. Choosing the inputs and outputs essentially chooses the nature of the target function we will seek to approximate.

Choose a Representation of the Target Function

Next, you must choose the representation you wish to use for the mapping function.

Think of this as the type of final model you wish to have that you can then use to make predictions. You must choose the form of this model, the data structure if you’d like.

Now that we have specified the ideal target function V, we must choose a representation that the learning program will use to describe the function Vprime that it will learn.

For example:

  • Perhaps your project requires a decision tree that is easy to understand and explain to stakeholders.
  • Perhaps your stakeholders prefer a linear model that the stats guys can easily interpret.
  • Perhaps your stakeholders don’t care about anything other than model performance so all model representations are up for grabs.

The choice of representation will impose constraints on the types of learning algorithms that you can use to learn the mapping function.

Choose a Learning Algorithm

Finally, you must choose the learning algorithm that will take the input and output data and learn a model of your preferred representation.

If there were few constraints on the choice of representation, as is often the case, then you may be able to evaluate a suite of different algorithms and representations.

If there were strong constraints on the choice of function representation, e.g. a weighted sum linear model or a decision tree, then the choice of algorithms will be limited to those that that can operate on the specific representations.

The choice of algorithm may impose its own constraints, such as specific data preparation transforms like data normalization.

How to Design Learning Systems

Developing a learning system is challenging.

No one can tell you the best answer to each decision along the way; the best answer is unknown for your specific learning problem.

Mitchell helps to clarify this with a depiction of the choices made in designing a learning system for playing checkers.

Depiction of Choices in Designing a Checker-Playing Learning System

The depiction of Choices in Designing a Checker-Playing Learning System.
Taken from “Machine Learning”, 1997.

The choices act as points of constraint on the design process. Mitchell goes on to say:

These design choices have constrained the learning task in a number of ways. We have restricted the type of knowledge that can be acquired to a single linear evaluation function. Furthermore, we have constrained this evaluation function to depend on only the six specific board features provided. If the true target function V can indeed be represented by a linear combination of these particular features, then our program has a good chance to learn it. If not, then the best we can hope for is that it will learn a good approximation, since a program can certainly never learn anything that it cannot at least represent.

I like this passage as it really drives home both the importance of these constraints to simplify the problem, and the risk of making choices that limit or prevent the system from learning the problem sufficiently.

Generally, you cannot analytically calculate the answer to these choices, e.g. what data to use, what algorithm to use, or what algorithm configuration to use.

Nevertheless, all is not lost; here are 3 tactics that you can use in practice:

  1. Copy. Look to the literature or experts for learning systems on problems the same or similar to your problem and copy the design of the learning system. It is very likely you are not the first to work on a problem of a given type. At the very worst, the copied design provides a starting point for your own design.
  2. Search. List available options at each decision point and empirically evaluate each to see what works best on your specific data. This may be the most robust and most practiced approach in applied machine learning.
  3. Design. After completing many projects via the Copy and Search methods above, you will develop an intuition for how to design machine learning systems.

Developing learning systems is not a science; it is engineering.

Developing new machine learning algorithms and describing how and why they work is a science, and this is often not required when developing a learning system.

Developing a learning system is a lot like developing software. You must combine (1) copies of past designed that work, (2) prototypes that show promising results, and (3) design experience when developing a new system in order to get the best results.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Summary

In this post, you discovered the intractable nature of designing learning systems in applied machine learning and how to deal with it.

Specifically, you learned:

  • How to develop a clear definition of your learning problem for yourself and others.
  • The 4 decision points you must consider when designing a learning system for your problem.
  • The 3 strategies that you can use to specifically address the intractable problem of designing learning systems in practice.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

相關推薦

Why Applied Machine Learning Is Hard

Tweet Share Share Google Plus How to Handle the Intractability of Applied Machine Learning. Appl

Why AI, machine learning is driving data lakes to data hubs

The data lake was a critical concept for companies looking to put information in one place and then tap it for business intelligence, analytics and big dat

Applied Machine Learning is a Meritocracy

Tweet Share Share Google Plus When making a start in a new field it is common to feel overwhelme

Dr. Data Show Video: Why Machine Learning Is the Coolest Science

Watch the premiere episode of The Dr. Data Show, which answers the question, "What makes machine learning the coolest science?" This new web series breaks

Python is the Growing Platform for Applied Machine Learning

Tweet Share Share Google Plus You should pick the right tool for the job. The specific predictiv

A Gentle Introduction to Applied Machine Learning as a Search Problem (譯文)

​ A Gentle Introduction to Applied Machine Learning as a Search Problem 原文作者:Jason Brownlee 原文地址:https://machinelearningmastery.com/applied-m

Machine Learning is Fun! Part 3: Deep Learning and Convolutional Neural Networks

We can train this kind of neural network in a few minutes on a modern laptop. When it’s done, we’ll have a neural network that can recognize pictures of “8

Machine Learning is Fun!

What is machine learning?Machine learning is the idea that there are generic algorithms that can tell you something interesting about a set of data without

Machine Learning is Fun Part 5: Language Translation with Deep Learning and the Magic of Sequences

Making Computers TranslateSo how do we program a computer to translate human language?The simplest approach is to replace every word in a sentence with the

Machine Learning is Fun Part 7: Abusing Generative Adversarial Networks to Make 8

The goal of Generative ModelsSo why exactly are AI researchers building complex systems to generate slightly wonky-looking pictures of bedrooms?The idea is

Why use Machine Learning in DevOps?

Machine Learning (ML): is an application of Artificial Intelligence which allows systems to learn and improve from experience, rather than being manually p

Machine learning -- Is the emperor wearing clothes?

Machine learning uses patterns in data to label things. The core concepts are actually embarrassingly simple. I say "embarrassingly" because if someone mad

Machine learningIs the emperor wearing clothes?

Algorithms for hipstersThese days, no data science hipster is into the humble straight line. Flexible, squiggly shapes are all the rage among today’s fashi

Source Machine Learning Is Free, As In Beer | AITopics

Machine learning (ML) continues to amaze us with its abilities and is set to transform the economic structure of many industries -- from producers of widge

Man Machine Cybersecurity: Machine Learning is Essential to Fighting Attacks

Something is materially broken with web application security and, as a result, critical attacks are being missed. According to a new report from Kaspersky

IoT + Machine Learning is Going to Change the World

IoT + Machine Learning is Going to Change the WorldAnd it’s going to be awesomeThe Internet of Things is slowly permeating every aspect of our lives. If yo

IoT Machine Learning is Going to Change the World

The Internet of Things is slowly permeating every aspect of our lives. If you don't already own an IoT device, you've surely heard of them. From smart ther

How Machine Learning is Revolutionizing Casino and Gambling Industry

Though gambling remains a controversial and highly-regulated industry across the United States, it is also an immensely lucrative one that continues to gro

How machine learning is helping marketers get the edge

Some say marketing is an art. Sure, being creative with your slogans, copywriting and other marketing collateral is important but the'art' component should

Educating the next generation of medical professionals with machine learning is essential

"The general public has become quite aware of AI and the impact it can have on health care outcomes such as providing clinicians with improved diagnostics