Introduction to Scientific Machine Learning
Table of Contents
Welcome! This textbook is designed to provide foundational knowledge necessary to understand the core concepts and applications of Scientific Machine Learning (SciML). SciML presents a new paradigm for solving complex scientific and engineering problems by combining domain knowledge from various scientific fields, such as physics, engineering, and chemistry, with state-of-the-art machine learning techniques. In many engineering disciplines, it holds the potential to offer innovative solutions to a wide range of problems, including modeling, simulation, control, and design optimization.
Scientific Machine Learning (SciML) is an interdisciplinary research field that addresses scientific and engineering problems by integrating data-driven machine learning methodologies with domain knowledge (such as physical laws and governing equations). It aims to overcome issues faced by traditional data-centric machine learning, such as data scarcity, lack of extrapolation capabilities, and absence of physical consistency, to build more robust and interpretable models.
Key Characteristics:
Mechanical engineering deals with complex physical systems and phenomena, and has traditionally relied on numerical analysis (e.g., the Finite Element Method and the Finite Volume Method) and on physical experiment to interpret and predict them. Both approaches are mature, rigorous, and remarkably successful. Both are also reaching their practical limits in exactly the regimes where modern engineering questions are being asked. Understanding why is the best motivation for Scientific Machine Learning. SciML is not proposed because machine learning is fashionable, but because the classical toolchain and the purely data-driven toolchain each fail in ways that the other can repair.
Classical simulation and experiment face three structural limitations.
Machine learning offers a route around all three limitations, because it rests on a fundamentally different premise: given a sufficient quantity of data, an expressive model can learn an input-output relationship without being told anything about the mechanism that produces it. A neural network is never informed that a fluid conserves mass or that a material obeys a constitutive law. It observes pairs of inputs and outputs and adjusts its parameters until the discrepancy is small. If enough data is available, a legitimate level of predictive performance can be achieved without domain knowledge.
Set against the three limitations above, the appeal is immediate.
In short, the data-driven approach converts observations directly into predictions, bypassing the difficult and often unfinished task of writing down the underlying model.
The property that makes this approach powerful, namely its indifference to the underlying mechanism, is also the source of its principal weaknesses. Naively applying deep learning to scientific tasks typically produces three recurring failures.
(1) Poor generalization to out-of-distribution data
A statistical model is a statement about the distribution it was trained on. Within that distribution it interpolates well; outside it, there is no guarantee whatsoever. This matters more in engineering than in most application domains of machine learning, because engineering questions are posed precisely about the untested regime: a new geometry, an off-design operating condition, an extreme load, a time horizon longer than any experiment has run.
The failure mode is especially dangerous because it is silent. A purely data-driven model carries no internal representation of what lies outside its training distribution, and nothing within it insists that mass be conserved or that entropy not decrease. It will extrapolate confidently and return a physically impossible answer with the same numerical assurance as a correct one.
(2) Lack of interpretability
In engineering practice a prediction is rarely the final product. It feeds a decision, whether to certify a component, to actuate a controller, to modify a design, or to publish a mechanism, and such decisions require an account of why the prediction takes the value it does. A black-box mapping offers no such account.
The scientific requirement is stronger still. The goal of an investigation is often not a number but a scientifically interpretable output: a governing equation, a dimensionless group, an identified parameter carrying physical units, or a dominant mechanism. A model that cannot express its findings in the vocabulary of the discipline cannot contribute to that discipline's knowledge, however accurate its predictions may be.
(3) Large data requirements
Generic deep architectures are sample-inefficient by design. Having assumed almost nothing, they must learn almost everything from examples. Scientific data, however, is expensive in a way that internet-scale data is not. Wind-tunnel time is booked months in advance, a single molecular dynamics trajectory may consume days of GPU time, fatigue tests destroy the specimen, and field or clinical measurements are limited by cost and ethics. Data scarcity is the normal condition of scientific machine learning, not an edge case. This means that the one assumption the purely data-driven approach depends on, that enough data is available, is usually the assumption that fails first.
(4) The question underneath all three
These symptoms share a single root cause, best posed as a question:
$\qquad$Do neural networks really "understand" the scientific tasks they are being applied to?
A purely data-driven model learns whatever correlations suffice to reduce the training loss on the observed data manifold, which is not the same thing as capturing the mechanism that generated it. A network trained on projectile trajectories recorded on Earth can predict those trajectories to arbitrary precision without ever forming any representation of gravitational acceleration; sent to Mars, it fails completely. Fitting the data and capturing the law are distinct achievements, and standard supervised learning aims only at the first.
The two families of methods fail in complementary ways. Classical simulation encodes the physics exactly but is expensive, brittle where the physics is unknown, and awkward to reconcile with imperfect data. Machine learning is cheap to evaluate, tolerant of noisy data, and indifferent to missing equations, but it generalizes poorly, explains nothing, and demands data that scientific practice cannot supply. Neither deficit is repairable from within, and each is repairable by the other.
This is the premise of SciML. Physical laws, conservation principles, symmetries, boundary conditions, and known asymptotic behavior all constitute prior knowledge, and incorporating them restricts the hypothesis space to physically admissible functions before a single data point is seen. Leveraging scientific knowledge in the analysis of scientific data yields three direct benefits.
A fourth benefit follows from the same mechanism. Because the prior does part of the work, far fewer observations are needed to identify the remainder. Constraining the hypothesis space simultaneously reduces the data requirement and improves the quality of what is learned, which is why data efficiency and physical consistency appear together among the defining characteristics of SciML.
The payoff is practical. SciML aims to overcome the limitations of both traditions, providing faster, more accurate, and more powerful tools for mechanical engineering research and industry. Representative applications include accelerating simulations, designing new materials, and diagnosing and predicting failures in complex systems. The remainder of this book develops the methods that make this possible.
The main methodologies constituting SciML are as follows. They are presented here only in outline; each is developed in detail in a later notebook.
PINN is a method that approximates the solutions of Partial Differential Equations (PDEs) using neural networks. By incorporating differential equations representing physical laws into the neural network's loss function, the network learns not only from data but also from physical laws. This allows for physically consistent solutions even in situations with scarce or noisy data.
For systems where traditional physics-based model building is challenging, data is directly used to model system behavior. Operator learning, in particular, aims to learn the operator mapping between the input and output of a physical system to rapidly predict outputs for new inputs. Examples include DeepONet and Fourier Neural Operator (FNO).
While the methods above assume that the governing equations are known, equation discovery treats the equations themselves as the unknown and identifies them directly from data. The result is not a trained network but a symbolic expression that can be read, analyzed, and inserted into a conventional solver. Sparse regression methods such as SINDy select a small number of terms from a library of candidate functions, based on the observation that physical laws are typically composed of only a few terms, while symbolic regression searches the space of mathematical expressions directly. This approach is particularly valuable for systems whose governing equations are unknown or incomplete.
These methods are complementary rather than competing, and they are often combined. An equation recovered by equation discovery can be supplied to a physics-informed neural network as its residual, and a physics-informed formulation can be applied to operator learning so that the trained operator respects the governing equations. Each of these possibilities is taken up in the corresponding chapter.
For students wishing to delve deeper into this topic, the following resources are recommended.
Overview
Physics-Informed Neural Networks
Neural Operator Learning
Equation Discovery
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')