A Hands on Introduction to Applied Scientific Machine Learning / Physics-Informed Learning
May 11 2025 in Julia, Scientific ML, Uncategorized | Tags: ml, neural networks, sciml | Author: Christopher Rackauckas
Presented at JuliaEO25
This is a hands-on introduction to Scientific Machine Learning that does not assume a background in machine learning. We start scratch, showing the mathematical basis of “what is a neural network?” all the way up through adding physical intuition to the neural network and using it solve problem in epidemic outbreaks to improving sensor tracking of Formula 1 cars.
Open Source Component-Based Modeling with ModelingToolkit
May 5 2025 in Differential Equations, Julia, Mathematics, Programming | Tags: | Author: Christopher Rackauckas
Component-based modeling systems such as Simulink and Dymola allow for building scientific models in a way that can be composed. For example, Bob can build a model of an engine, and Alice can build a model of a drive shaft, and you can then connect the two models and have a model of a car. These kinds of tools are used all throughout industrial modeling and simulation in order to allow for “separation of concerns”, allowing experts to engineer their domain and compose the final digital twins with reusable scientific modules. But what about open source? In this talk we will introduce ModelingToolkit, an open source component-based modeling framework that allows for composing pre-built models and scales to large high-fidelity digital twins.
PyData is an … READ MORE
The Numerical Analysis of Differentiable Simulation: Automatic Differentiation Can Be Incorrect
April 20 2025 in Differential Equations, Julia, Mathematics, Scientific ML | Tags: | Author: Christopher Rackauckas
The Numerical Analysis of Differentiable Simulation: How Automatic Differentiation of Physics Can Give Incorrect Derivatives
Scientific machine learning (SciML) relies heavily on automatic differentiation (AD), the process of constructing gradients which include machine learning integrated into mechanistic models for the purpose of gradient-based optimization. While these differentiable programming approaches pitch an idea of “simply put the simulator into a loss function and use AD”, it turns out there are a lot more subtle details to consider in practice. In this talk we will dive into the numerical analysis of differentiable simulation and ask the question: how numerically stable and robust is AD? We will use examples from the Python-based Jax (diffrax) and PyTorch (torchdiffeq) libraries in order to demonstrate how canonical … READ MORE
JuliaSim: Building a Product which improves Open Source Sustainability
January 26 2025 in Differential Equations, HPC, Julia, Scientific ML | Tags: | Author: Christopher Rackauckas
How do you build products that support open source communities? In this non-technical talk with OpenTeams I discuss how the MIT Julia Lab, PumasAI, and JuliaHub have all been essential pillars of the julialang opensource community in its goal to achieve sustainable open science. If you’ve ever been curious about what the difference is between the Julia Lab and JuliaHub is, the evolution of these groups, and what kinds of different contributions they make to the open source community, in this talk I go through as many details as I could!
Differences Between Methods for Solving Stiff ODEs
April 6 2024 in Differential Equations, Mathematics | Tags: | Author: Christopher Rackauckas
I found these notes from August 2018 and thought they might be useful so I am posting them verbatim.
A stiff ordinary differential equation is a difficult problem to integrator. However, many of the ODE solver suites offer quite a few different choices for this kind of problem. DifferentialEquations.jl offers almost 200 different choices for example. In this article we will dig into what the differences between these integrators really is so that way you can more easily find which one will be most efficient for your problem.
Quick Overview (tl;dr)
- A BDF, Rosenbrock, ESDIRK method are standard
- For small equations, Rosenbrock methods have performance advantages
- For very stiff systems, Rosenbrock and Rosenbrock-W methods do not require convergence of Newton’s method and thus can take larger steps, being more efficient
- BDF integrators are only L-stable (and A-stable) to order 2, so if the problem is … READ MORE
Symbolic-Numerics: how compiler smarts can help improve the performance of numerical methods (nonlinear solvers in Julia)
February 13 2024 in Julia, Mathematics, Programming | Tags: julia, nonlinear solve, pydata, symbolic-numerics | Author: Christopher Rackauckas
Many problems can be reduced down to solving f(x)=0, maybe even more than you think! Solving a stiff differential equation? Finding out where the ball hits the ground? Solving an inverse problem to find the parameters to fit a model? In this talk we’ll showcase how SciML’s NonlinearSolve.jl is a general system for solving nonlinear equations and demonstrate its ability to efficiently handle these kinds of problems with high stability and performance. We will focus on how compilers are being integrated into the numerical stack so that many of the things that were manual before, such as defining sparsity patterns, Jacobians, and adjoints, are all automated out-of-the-box making it greatly outperform purely numerical codes like SciPy or NLsolve.jl.
PyData Global 2023
Semantic Versioning (Semver) is flawed, and Downgrade CI is required to fix it
February 11 2024 in Julia, Programming | Tags: continuous integration, downgrade, package managers, semver, unit testing, versioning | Author: Christopher Rackauckas
Semantic versioning is great. If you don’t know what it is, it’s just a versioning scheme for software that goes MAJOR.MINOR.PATCH, where
- MAJOR version when you make incompatible API changes
- MINOR version when you add functionality in a backward compatible manner
- PATCH version when you make backward compatible bug fixes
That’s all it is, but it’s a pretty good system. If you see someone has updated their package from v3.2.0 to v3.2.1, then you know that you can just take that update because it’s just a patch, it won’t break your code. You can easily accept patch updates. Meanwhile, if you see they released v3.3.0, then you know that some new features were added, but it’s safe for you to update. This allows you to be compatible with v3.3.0 so that if a different package requires it, great you can both use it! … READ MORE
ChatGPT performs better on Julia than Python (and R) for Large Language Model (LLM) Code Generation. Why?
November 19 2023 in Julia, Programming | Tags: ai, artificial intelligence, chatgpt, julia, large language models, llm, machine learning, MATLAB, python, r | Author: Christopher Rackauckas
Machine learning is all about examples. The more data you have, the better it should perform, right? With the rise of ChatGPT and Large Language Models (LLMs) as a code helping tool, it was thus just an assumption that the most popular languages like Python would likely be the best for LLMs. But because of the increased productivity, I tend to use a lot of Julia, a language with an estimated user-base of around a million programmers. For this reason, people have often asked me how it fairs with ChatGPT, Github Copilot, etc., and so I checked out those pieces and… was stunned. It’s really good. It seemed better than Python actually?
The data is in: Julia does well with ChatGPT
This question was recently put to the test by a researcher named Alessio Buscemi in A Comparative Study … READ MORE
DDPS Seminar Talk: Generalizing Scientific Machine Learning and Differentiable Simulation Beyond Continuous models
November 12 2023 in Uncategorized | Tags: data-driven physics, ddps, physics-informed machine learning, piml, sciml | Author: Christopher Rackauckas
I’m pleased to share a talk I gave in the DDPS seminar series!
Data-driven Physical Simulations (DDPS) Seminar Series
Abstract: The combination of scientific models into deep learning structures, commonly referred to as scientific machine learning (SciML), has made great strides in the last few years in incorporating models such as ODEs and PDEs into deep learning through differentiable simulation. However, the vast space of scientific simulation also includes models like jump diffusions, agent-based models, and more. Is SciML constrained to the simple continuous cases or is there a way to generalize to more advanced model forms? This talk will dive into the mathematical aspects of generalizing differentiable simulation to discuss cases like chaotic simulations, differentiating stochastic simulations like particle filters and agent-based models, and solving … READ MORE
Summary of Julia Plotting Packages
June 17 2023 in Julia | Tags: data science, ggplot2, julia, plots, programming language, startup time, ttfx, visualization | Author: Christopher Rackauckas
This is a repost of my response on the Julia Discourse on this topic. I was asked to make a blog post so here you go!
The “Main” Plotting Packages
Here’s a quick summary of the most widely used plotting packages. I may have missed one, but I haven’t missed one that is very widely used.
- Plots.jl is the most used. It’s probably the most documented, used in the most tutorials, and is used in many videos.
- Pros: Its main draw is that it has a lot of plugins to other packages through its recipes system, which means that a lot of odd things like `plot(sol::ODESolution)` or showing the sparsity of a `BandedMatrix` just works. With all of these integrations, it’s normally what I would recommend first to newcomers since they will generally get the most done with the least work. It … READ MORE
- Pros: Its main draw is that it has a lot of plugins to other packages through its recipes system, which means that a lot of odd things like `plot(sol::ODESolution)` or showing the sparsity of a `BandedMatrix` just works. With all of these integrations, it’s normally what I would recommend first to newcomers since they will generally get the most done with the least work. It … READ MORE