Automated Learning

Ada:

Hello! It’s me again, Ada Lovelace, and today I’m pleased to be speaking with Florian. He’s an expert in automated learning at the ADA Lovelace Center for Analytics, Data and Applications – which, by the way, was named after me, the inventor of the first computer program.

I would like to know more about the artificial intelligence applications and methods this project is researching, so I’ll ask him.

Florian, how is automated learning defined? That is, how do you explain this method?

 

Florian:

Many people are now familiar with the concept of machine learning, and what we ultimately want is for computers to learn from existing data as independently as possible. For this to happen, however, we first have to pick the right model, and when we’ve found the right model, we still need to determine the best configuration. Every machine learning model has a certain number of variables, called hyperparameters, that can be adjusted and that then ultimately affect how the model behaves and also how well it performs.

This entails a lot of manual effort for researchers and data scientists, since they have to test several models and several configurations before they can really say, “Now I’ve found a good model.” And this is precisely where automated machine learning (AutoML) comes in and tries to automate these processes.

Basically – that is, in theory, AutoML covers the entire machine learning process, from task setting, data acquisition, and sensor preprocessing to model selection and hyperparameter optimization. I would consider the latter three to be the core competence of AutoML, and that’s also what most of the publications and the current research focus on. But there are certainly also efforts to automate something like data analysis.

In principle, one can try to automate anything that uses machine learning or anything that involves the use of machine learning. We can also look at AutoML as a sort of meta method that can basically be applied to all kinds of problems and data situations.

 

Ada:

I see. So, in principle, it’s about automating machine learning, or about sub-aspects of the overall process. That’s interesting. And what kind of data can be used for this?

 

Florian:

In theory, AutoML can handle all types of data. In practice, most methods deal with tables and image data. If we look at machine learning for image data in recent years, neural networks clearly dominate the field. Accordingly, a subfield has also developed in AutoML, namely neural architecture search. This focuses on automatically finding the optimum network architectures just so data scientists don’t have to design and test different architectures manually. Beyond tables and image data, AutoML systems can be designed to support other data, as well. However, different types of data require different operations, which doesn’t always make it very easy. For example, the [preprocessing] of data is very, very specific, but the type of models can also be very specific. Very particular data types like time series data or multimodal data, when we have images and text, for instance, pose particular problems for AutoML solutions, since this is precisely where we need the very specific methods. And the AutoML system ultimately has to be able to access these methods in order to be able to generate functioning machine learning pipelines. This is generally a major challenge with AutoML: it first has to be decided what the AutoML system will have at its disposal, or in other words, what it will be able to try out. If I limit the AutoML system too much, the search for a good model will still work and will also quickly deliver a result, but I can never be certain that I haven’t perhaps forgotten a couple of good options.

On the other hand, if I give the AutoML system really very many possibilities, the search will take a very, very long time and, in the end, perhaps not be quite as robust.

 

Ada:

Ok. That means the type of data also determines, in a way, the type of AutoML system needed. And can you tell us how large the volume of data should be or what level of quality it should have?

 

Florian:

Yes, as is so often the case with machine learning, here, too, the more data we have, the better. Personally, though, what I also find extremely interesting are the scenarios where we maybe have only small datasets or a difficult data situation, for example with imbalanced data – that is, when there are very few examples of one class. In my view, researchers pay too little attention to these kinds of scenarios. They usually start with very clean benchmark datasets and a generally very neat data situation, which, however, rarely corresponds to reality.

As for the volume of data, of course this should also be somewhat proportional to the complexity of the search space – in other words, to the number of models I want to test.

If I have too little data to reasonably train a certain model – that is, regardless of AutoML – then of course there is also no point in including that model in the AutoML process.

The flip side, of course, to some extent, is that, with a lot of data, training machine learning models quickly becomes expensive. And then if, for AutoML, I want to test a great many models or a great many configurations, of course it becomes twice as expensive. Fortunately, however, there are a couple of approaches that deal with this, and a lot of what is being done in AutoML focuses on developing algorithms that yield very good results as quickly as possible. And AutoML is also commonly used in combination with meta learning.

In this setup, meta learning acts as a kind of recommender system that can recommend good configurations and models early on. In essence, this involves using information from preceding tasks that were very similar or had a very similar data situation, and ideally I then know from the previous tasks what worked well, and hope that these models might now also work well for my current task.

My favorite catchphrase is “learning to learn.” And it’s really quite fitting, because we want to use previously seen machine learning tasks to derive information that will now help us in our search for a good model for the current task.

 

Ada:

That is interesting, when machines teach themselves things. And now I also understand how the method works. But what exactly can you do with it? What result does it produce?

 

Florian:

The major goal of AutoML is to automatically find optimal machine learning models or optimal machine learning pipelines, that then often yield better results than manually configured models. For me, the major result is that it makes life easier for those who use machine learning and who can now focus on other important tasks.

The second result is a kind of democratization of machine learning, since a functioning AutoML system then gives all users direct access to optimal machine learning pipelines without having to know exactly which models underlie them and which configurations might be good.

Of course AutoML can’t replace the innovation or the creativity of experts, or even invent new methods. While there are frequently thought experiments along these lines, and a 2019 publication entitled “AutoML-Zero: Evolving Machine Learning Algorithms From Scratch” received a great deal of attention, we still have a long way to go to automate innovation.

AutoML-Zero was interesting in that no specific models are specified for it, only simple mathematical operations. And in the course of the AutoML process, certain machine-learning methods really were “reinvented” here. AutoML-Zero, for example, succeeded in developing neural networks and also rediscovered optimization – something like gradient descent, for instance. But of course, as of today, those are simply thought experiments. And the fantastic research that is being done day after day at universities or at Fraunhofer Institutes is simply not yet replaceable.

 

Ada:

It’s good to hear that experts like ourselves are still needed to refine or even reinvent these methods, and that machines aren’t just about to take over the world. Nevertheless, there are undoubtedly many possible applications for AutoML. Where is this method used? In which fields or industries is it useful?

 

Florian:

AutoML is used anywhere it can reasonably save time and resources. There are a wide variety of domains and industries where this is the case. There are AutoML tools for medical prognosis, for industrial production, or even for financial data. So it actually really covers the full spectrum. One conceivable scenario is, for example, a major industrial company whose production runs on many similar production lines and now wants to run its quality control with the aid of machine learning. The production lines differ slightly. There are different products and different conditions, and because of that, it actually makes sense to develop a separate machine learning model each time, or at least to test different machine learning models and different configurations. But it doesn’t make sense to bother an expert to do this for each individual production line, and that’s exactly where AutoML can help.

Especially in a scenario like this, where the tasks are always extremely similar, where, for instance, it’s always about quality control for production lines, AutoML can be quite interesting because we can use it to preselect models. There are models that are particularly suited for performing quality control because we have, for example, very imbalanced data here. It won’t normally be 50 percent of the products we see here that are bad, but only very, very few. Still, we want to detect them very reliably. This means we can give AutoML – or our AutoML system – exactly the methods it would need to perform such a task as quality control. This, in turn, means we can limit somewhat in advance what our AutoML should try.

 

Ada:

In addition to the various potential applications, research is also being done in other domains at the Ada Lovelace Center. So of course I’m interested to know which of those methods AutoML can be combined with and where, perhaps, that’s already being done.

 

Florian:

Since, as I said before, AutoML is a meta method, it can, generally speaking, quite easily be combined with other methods. And it’s also, in a sense, designed to build on other methods, or to automate them.

One particularly interesting interaction in my view is with Explainable AI, that is, with interpretable machine learning. AutoML really only cares whether it can achieve maximum performance. The algorithm is relatively indifferent as to whether the model it produces is interpretable or not. So AutoML produces many machine learning pipelines that aren’t really interpretable, and where users can no longer really understand how the machine learning system really makes a decision. This becomes particularly extreme when an ensemble of methods ultimately makes the decision – that is, when there is a sort of consultation between various machine learning methods. Of course there are some interesting approaches that in turn then make these kinds of complex or unclear models explainable, or there are AutoML methods that are designed to produce only explainable models. One current topic, for example, is multi-criteria AutoML – that is, no longer focusing on just one target variable, namely the error I want to minimize or the performance I want to achieve, but also on secondary targets, such as explainability. Other interesting secondary targets might be the model’s energy efficiency, its robustness, or perhaps even its fairness.

Seldom can real-world problems be reduced to a single metric, namely performance. So I’ll often also be interested in this kind of secondary metric. For instance, when I want to deploy a machine learning model on a smartphone. Then it’s not only important that it ultimately performs well, but maybe also that it requires little energy or takes up little disk space.

 

Ada:

Thank you! That was very interesting. Many people are already familiar with machine learning in connection with AI, but the automation of these processes is undoubtedly not yet as widely known, but still practical in many respects and especially also highly versatile.

And what you say about combining it with other methods used at the Ada Lovelace Center makes me curious. I’m excited to learn more about it in my next interview. I hope you are, too. So until next time!