Differences Between Methods for Solving Stiff ODEs
April 6 2024 in Differential Equations, Mathematics | Tags: | Author: Christopher Rackauckas
I found these notes from August 2018 and thought they might be useful so I am posting them verbatim.
A stiff ordinary differential equation is a difficult problem to integrator. However, many of the ODE solver suites offer quite a few different choices for this kind of problem. DifferentialEquations.jl offers almost 200 different choices for example. In this article we will dig into what the differences between these integrators really is so that way you can more easily find which one will be most efficient for your problem.
Quick Overview (tl;dr)
- A BDF, Rosenbrock, ESDIRK method are standard
- For small equations, Rosenbrock methods have performance advantages
- For very stiff systems, Rosenbrock and Rosenbrock-W methods do not require convergence of Newton’s method and thus can take larger steps, being more efficient
- BDF integrators are only L-stable (and A-stable) to order 2, so if the problem is … READ MORE
Symbolic-Numerics: how compiler smarts can help improve the performance of numerical methods (nonlinear solvers in Julia)
February 13 2024 in Julia, Mathematics, Programming | Tags: julia, nonlinear solve, pydata, symbolic-numerics | Author: Christopher Rackauckas
Many problems can be reduced down to solving f(x)=0, maybe even more than you think! Solving a stiff differential equation? Finding out where the ball hits the ground? Solving an inverse problem to find the parameters to fit a model? In this talk we’ll showcase how SciML’s NonlinearSolve.jl is a general system for solving nonlinear equations and demonstrate its ability to efficiently handle these kinds of problems with high stability and performance. We will focus on how compilers are being integrated into the numerical stack so that many of the things that were manual before, such as defining sparsity patterns, Jacobians, and adjoints, are all automated out-of-the-box making it greatly outperform purely numerical codes like SciPy or NLsolve.jl.
PyData Global 2023
Semantic Versioning (Semver) is flawed, and Downgrade CI is required to fix it
February 11 2024 in Julia, Programming | Tags: continuous integration, downgrade, package managers, semver, unit testing, versioning | Author: Christopher Rackauckas
Semantic versioning is great. If you don’t know what it is, it’s just a versioning scheme for software that goes MAJOR.MINOR.PATCH, where
- MAJOR version when you make incompatible API changes
- MINOR version when you add functionality in a backward compatible manner
- PATCH version when you make backward compatible bug fixes
That’s all it is, but it’s a pretty good system. If you see someone has updated their package from v3.2.0 to v3.2.1, then you know that you can just take that update because it’s just a patch, it won’t break your code. You can easily accept patch updates. Meanwhile, if you see they released v3.3.0, then you know that some new features were added, but it’s safe for you to update. This allows you to be compatible with v3.3.0 so that if a different package requires it, great you can both use it! … READ MORE
ChatGPT performs better on Julia than Python (and R) for Large Language Model (LLM) Code Generation. Why?
November 19 2023 in Julia, Programming | Tags: ai, artificial intelligence, chatgpt, julia, large language models, llm, machine learning, MATLAB, python, r | Author: Christopher Rackauckas
Machine learning is all about examples. The more data you have, the better it should perform, right? With the rise of ChatGPT and Large Language Models (LLMs) as a code helping tool, it was thus just an assumption that the most popular languages like Python would likely be the best for LLMs. But because of the increased productivity, I tend to use a lot of Julia, a language with an estimated user-base of around a million programmers. For this reason, people have often asked me how it fairs with ChatGPT, Github Copilot, etc., and so I checked out those pieces and… was stunned. It’s really good. It seemed better than Python actually?
The data is in: Julia does well with ChatGPT
This question was recently put to the test by a researcher named Alessio Buscemi in A Comparative Study … READ MORE
DDPS Seminar Talk: Generalizing Scientific Machine Learning and Differentiable Simulation Beyond Continuous models
November 12 2023 in Uncategorized | Tags: data-driven physics, ddps, physics-informed machine learning, piml, sciml | Author: Christopher Rackauckas
I’m pleased to share a talk I gave in the DDPS seminar series!
Data-driven Physical Simulations (DDPS) Seminar Series
Abstract: The combination of scientific models into deep learning structures, commonly referred to as scientific machine learning (SciML), has made great strides in the last few years in incorporating models such as ODEs and PDEs into deep learning through differentiable simulation. However, the vast space of scientific simulation also includes models like jump diffusions, agent-based models, and more. Is SciML constrained to the simple continuous cases or is there a way to generalize to more advanced model forms? This talk will dive into the mathematical aspects of generalizing differentiable simulation to discuss cases like chaotic simulations, differentiating stochastic simulations like particle filters and agent-based models, and solving … READ MORE
Summary of Julia Plotting Packages
June 17 2023 in Julia | Tags: data science, ggplot2, julia, plots, programming language, startup time, ttfx, visualization | Author: Christopher Rackauckas
This is a repost of my response on the Julia Discourse on this topic. I was asked to make a blog post so here you go!
The “Main” Plotting Packages
Here’s a quick summary of the most widely used plotting packages. I may have missed one, but I haven’t missed one that is very widely used.
- Plots.jl is the most used. It’s probably the most documented, used in the most tutorials, and is used in many videos.
- Pros: Its main draw is that it has a lot of plugins to other packages through its recipes system, which means that a lot of odd things like `plot(sol::ODESolution)` or showing the sparsity of a `BandedMatrix` just works. With all of these integrations, it’s normally what I would recommend first to newcomers since they will generally get the most done with the least work. It … READ MORE
- Pros: Its main draw is that it has a lot of plugins to other packages through its recipes system, which means that a lot of odd things like `plot(sol::ODESolution)` or showing the sparsity of a `BandedMatrix` just works. With all of these integrations, it’s normally what I would recommend first to newcomers since they will generally get the most done with the least work. It … READ MORE
Integrating equation solvers with probabilistic programming through differentiable programming
November 24 2022 in Julia, Programming, Science, Scientific ML | Tags: differential equations, julia, probabilistic programming, scientific machine learning, sciml | Author: Christopher Rackauckas
Part of the COMPUTATIONAL ABSTRACTIONS FOR PROBABILISTIC AND DIFFERENTIABLE PROGRAMMING WORKSHOP
Abstract: Many probabilistic programming languages (PPLs) attempt to integrate with equation solvers (differential equations, nonlinear equations, partial differential equations, etc.) from the inside, i.e. the developers of the PPLs like Stan provide differential equation solver choices as part of the suite. However, as equation solvers are an entire discipline to themselves with many active development communities and subfields, this places an immense burden on PPL developers to keep up with the changing landscape of tens of thousands of independent researchers. In this talk we will explore how Julia PPLs such as Turing.jl support of equation solvers from the outside, i.e. how the tools of differentiable programming allows equation solver libraries to be compatible with … READ MORE
Direct Automatic Differentiation of (Differential Equation) Solvers vs Analytical Adjoints: Which is Better?
October 11 2022 in Differential Equations, Julia, Mathematics, Science, Scientific ML | Tags: automatic differentiation, differentiable programming, sciml | Author: Christopher Rackauckas
Automatic differentiation of a “solver” is a subject with many details for doing it in the most effective form. For this reason, there are a lot of talks and courses that go into lots of depth on the topic. I recently gave a talk on some of the latest stuff in differentiable simulation with the American Statistical Association, and have some detailed notes on such adjoint derivations as part of the 18.337 Parallel Computing and Scientific Machine Learning graduate course at MIT. And there are entire organizations like my SciML Open Source Software Organization which work day-in and day-out on the development of new differentiable solvers.
I’ll give a brief summary of all my materials here below.
Continuous vs Discrete Differentiation of Solvers
AD of a solver can be done in essentially two different ways: either directly performing automatic … READ MORE
Is Differentiable Programming Actually Necessary? Can’t you just train the neural networks separately?
October 4 2022 in Scientific ML | Tags: | Author: Christopher Rackauckas
Is differentiable programming actually necessary, or can you just train the neural network in isolation against data and then stick the trained neural network into the simulation? We looked at this problem in detail in our new manuscript titled Capturing missing physics in climate model parameterizations using neural differential equations.
The goal of this project is to understand temperature mixing in large eddy simulations, essentially columns of water in the ocean. I.e., can we take a “true” 3D Navier-Stokes and use that to build very quick and accurate models for how heat flows up and down in the water?
This isn’t a new problem: climate … READ MORE
Accurate and Efficient Physics-Informed Learning Through Differentiable Simulation (ASA Seminar Talk)
July 14 2022 in Uncategorized | Tags: | Author: Christopher Rackauckas
Abstract: Scientific machine learning (SciML) methods allow for the automatic discovery of mechanistic models by infusing neural network training into the simulation process. In this talk we will start by showcasing some of the ways that SciML is being used, from discovery of extrapolatory epidemic models to nonlinear mixed effects models in pharmacology. From there, we will discuss some of the increasingly advanced computational techniques behind the training process, focusing on the numerical issues involved in handling differentiation of highly stiff and chaotic systems. The viewers will leave with an understanding of how compiler techniques are being infused into the simulation stack to increasingly automate the process of developing mechanistic models
Bio: Dr. Chris Rackauckas is the Director of Scientific Research at Pumas-AI, the Director of … READ MORE