Putting the scientific in machine learning
One of the two current Strategic Research Initiatives (SRIs) of the 4TU.Applied Mathematics Institute is called 'Bridging Numerical Analysis and Machine Learning'. Co-founder Dr. Alexander Heinlein is busy organising the SRI’s next workshop, taking place in Delft on 25 and 26 April. His aim: create a solid foundation for Dutch initiatives in the field of scientific machine learning.
Scientific machine learning as opposed to non-scientific machine learning?
No, no, not at all. Scientific machine learning is a combination of scientific computing and machine learning. Over the last decades, these fields have developed more or less independently. In scientific computing, methods are mostly based on first-principle models. For example, Newton's law or other things we know from physics, biology or chemistry provide us with a mathematical description for what happens. Such numerical models can be used to simulate processes like blood flow in an artery, ice sheet motion in Antarctica, or the microstructure in steel. Machine-learning or data-driven methods are generally not based on prior knowledge. Instead, you often have a lot of data and try to find a model that fits the data well.
Why combine these two fields?
On the one hand, there is an increasing demand for the use of data in modeling because we have a lot of phenomena which we do not fully understand. Take, for example, the weather. There are so many things that come into play that it's impossible to find a physical model to describe everything. We do have a lot of data, but in classical numerical models, it's challenging to bring in those data. On the other hand, in data-driven models, it can be difficult to bring in the actual knowledge that we have about what happens physically. Over the past five, six years, these two fields have been finding each other in the new field of scientific machine learning.
Is it a happy marriage?
A first success are physics-informed neural networks, a quite flexible and powerful framework for combining data with physical knowledge. What makes these networks attractive compared to purely data-driven techniques is that the addition of a physical model allows us to deal much better with less data or low-quality data, because at every point where we don't know the data, we can still use the physics. Compared to first-principle modelling, the technique is quicker and requires less computational effort, especially when problems become more complex. The framework inherits the advantages as well as the disadvantages from the two techniques.
That sounds very promising.
Internationally, it's a very big thing now, and a lot of funding is going into it. I don’t have the precise numbers, but I feel like in Europe, and also in the Netherlands, unfortunately, we are a few years behind, for example, the US. It took a bit longer for people here to pick it up, and unfortunately, it still is not really a community here. This is also because scientific machine learning is a very, very interdisciplinary field. Part mathematics, part computer science, natural sciences and engineering, part machine learning.
That is why the SRI was founded?
Two years ago, when Dr. Mengwu Guo at the University of Twente and I started working on this, we found that there were actually quite a few people from the different universities separately working in this field. Our SRI now has participants from Twente, Delft, Eindhoven and Wageningen. Our idea was to start with a series of workshops to connect people. There was a lot of interest, especially from young researchers, whom we hope will embrace this topic and continue working on it. Our SRI is now at the heart of developments in the Netherlands and even beyond, in Europe.
Did you encounter any language differences during those workshops?
We are already seeing the development of a common language. Of course, there are differences, not just in terms of language. For example, what is the definition of success in the field? In standard machine learning, it is common to benchmark against the error level of the state-of-the-art technique. In the computing field, we also like to consider the overall characteristics of the model, such as its adherence to the laws of physics or the alignment of computations with mathematically proven properties, not just its accuracy. So, in scientific machine learning, what is the best way of validating results? This is why we need to bring people together to talk about these things.
Any other challenge the SRI community is working on?
A big issue and another motivation for our SRI, is that the mathematical foundations of the methods are not sufficiently understood. For example, there's a lot of randomness in training machine-learning models. It’s not robust. It's very sensitive to which parameters you choose. Before you start training the model, you don’t know if you will get a good model, so you may need to repeat it many times, tuning the parameters. We want to understand why these models converge or don’t converge.
What drew you personally to this new field?
Actually, I taught a project-oriented course, where groups of students worked on real-world problems submitted by companies. Their specific questions could not easily be solved with classical algorithms. They were data problems. So, I had to learn about machine learning to give the lectures on this topic. That’s when I noticed that the two worlds share a lot of ideas and methods but use different terminologies. If we can bridge that gap, the two fields could learn from each other.
Of the two fields involved, one is experiencing an unprecedented hype...
There has been a general hype in AI and machine learning over the past years. Among students, there's overwhelming interest in machine learning. It has become more difficult to find students to implement, for example, a classical solver for a finite-element problem. Implemening such a model typically takes weeks, while a neural network takes maybe a few hours. The usability of current machine learning codes is simply better, the documentation is better, and the community is more connected.
What are your goals, as an SRI?
I hope that in five years from now, we will have an understanding of how to train, in a robust way, a neural network to approximate some physical system. But I'm a bit skeptical if this will be possible. I also hope that within five years, there will be new textbooks on these things, and more of a consensus about the opportunities and requirements for combining machine learning and numerical computing. I hope that we've gathered a community in the Netherlands such that we can be more visible, and maybe also have more impact.
Where will this impact be felt?
It could be anywhere. Models for aircraft, cars, any kind of engineering structure. Civil engineering, offshore engineering, renewable energy, health. I think the technique is relevant to all those disciplines where we’re using physical or first-principle models, but where we could also incorporate data. We’re already seeing evidence that if you combine even a simplified model with measurement data, you can get better predictions than if you just use one of the two. And this is just the beginning.
