Keynote: New Horizons in Modeling and Simulation with Julia (Modelica Conference 2021)
June 25 2022 in Uncategorized | Tags: | Author: Christopher Rackauckas
Keynote Address: New Horizons in Modeling and Simulation in Julia
Presenters: Viral Shah (Julia Computing, CEO and Co-Founder), Chris Rackauckas (Julia Computing, Director of Modeling and Simulation and Christopher Laughman (Mitsubishi Electric Research Laboratories, Principal Member Research Staff)
Abstract: As modeling has become more ubiquitous, our models keep growing. The time to build models, verify their behavior, and simulate them is increasing exponentially as we seek more precise predictions. How will our tools change to accommodate the future? Julia’s language design has led to new opportunities. The combination of multiple dispatch, staged compilation, and Julia’s composable libraries have made it possible to build a next generation symbolic-numeric framework. Julia’s abstract interpretation framework enables capabilities such as automatic differentiation, automatic surrogate generation, symbolic tracing, uncertainty propagation, and automatic parallelism. … READ MORE
Composing Modeling and Simulation with Julia (2021 Modelica Conference)
April 8 2022 in Differential Equations, Julia, Science, Scientific ML | Tags: | Author: Christopher Rackauckas
In this paper we introduce JuliaSim, a high-performance programming environment designed to blend traditional modeling and simulation with machine learning. JuliaSim can build accelerated surrogates from component-based models, such as those conforming to the FMI standard, using continuous-time echo state networks (CTESN). The foundation of this environment, ModelingToolkit.jl, is an acausal modeling language which can compose the trained surrogates as components within its staged compilation process. As a complementary factor we present the JuliaSim model library, a standard library with differential-algebraic equations and pre-trained surrogates, which can be composed using the modeling system for design, optimization, and control. We demonstrate the effectiveness of the surrogate-accelerated modeling and simulation approach on HVAC dynamics by showing that the CTESN surrogates accurately capture the dynamics of a HVAC cycle … READ MORE
Engineering Trade-Offs in Automatic Differentiation: from TensorFlow and PyTorch to Jax and Julia
December 25 2021 in Julia, Programming, Science, Scientific ML | Tags: automatic differentiation, compilers, differentiable programming, jax, julia, machine learning, pytorch, tensorflow, XLA | Author: Christopher Rackauckas
To understand the differences between automatic differentiation libraries, let’s talk about the engineering trade-offs that were made. I would personally say that none of these libraries are “better” than another, they simply all make engineering trade-offs based on the domains and use cases they were aiming to satisfy. The easiest way to describe these trade-offs is to follow the evolution and see how each new library tweaked the trade-offs made of the previous.
Early TensorFlow used a graph building system, i.e. it required users to essentially define variables in a specific graph language separate from the host language. You had to define “TensorFlow variables” and “TensorFlow ops”, and the AD would then be performed on this static graph. Control flow constructs were limited to the constructs that could be represented statically. For example, an `ifelse` function statement is very different from … READ MORE
Improved ForwardDiff.jl Stacktraces With Package Tags
December 19 2021 in Uncategorized | Tags: | Author: Christopher Rackauckas
You may have seen some hilariously long stacktraces when using ForwardDiff. In the latest releases of OrdinaryDiffEq.jl we have fixed this, and the fix is rather safe. I want to take a second to describe some of the technical details so that others can copy this technique.
The reason for this is the tag parameter. The Dual number type is given by Dual{T,V,N} where V is an element type (usually Float64), N is a chunksize (some integer), and T is the tag. What the tag does is prevent perturbation confusion by erroring if two incompatible dual numbers try to interact. The key requirement for it to prevent perturbation confusion is for the type to be unique in the context of the user. For example, if the user is differentiating f, and then differentiating x->derivative(f,x), you want the tag to be … READ MORE
The Use and Practice of Scientific Machine Learning
November 18 2021 in Uncategorized | Tags: | Author: Christopher Rackauckas
The Use and Practice of Scientific Machine Learning
Scientific machine learning (SciML) methods allow for the automatic discovery of mechanistic models by infusing neural network training into the simulation process. In this talk we will start by showcasing some of the ways that SciML is being used, from discovery of extrapolatory epidemic models to nonlinear mixed effects models in pharmacology. From there, we will discuss some of the increasingly advanced computational techniques behind the training process, focusing on the numerical issues involved in handling differentiation of highly stiff and chaotic systems. The viewers will leave with an understanding of how compiler techniques are being infused into the simulation stack to increasingly automate the process of developing mechanistic models.
Benchmarks behind this talk can be found at READ MORE
When does the mean and variance define an SDE?
November 1 2021 in Uncategorized | Tags: | Author: Christopher Rackauckas
I recently saw a paper that made the following statement:
“Innes et al. [22] trained neural SDEs by backpropagating through the operations of the solver, however their training objective simply matched the first two moments of the training data, implying that it could not consistently estimate diffusion functions.”
However, given that the statement “could not consistently estimate diffusion functions” had no reference to it and no proof in the appendix, I was interested to figure out the mathematical foundation behind the claim. Furthermore, I know from the DiffEqFlux documentation example that there was at least one case where second order method of moments seems to estimate the diffusion function. So a question arose, when does the mean and variance define an SDE?
Of course, in 2021 a Twitter thread captures the full discussion. But I want to take a step back … READ MORE
Learning Epidemic Models That Extrapolate, AI4Pandemics
July 25 2021 in Differential Equations, Julia, Mathematics, Science, Scientific ML | Tags: epidemics, scientific machine learning, sciml | Author: Christopher Rackauckas
I think this talk was pretty good so I wanted to link it here!
Title: Learning Epidemic Models That Extrapolate
Speaker Chris Rackauckas, https://chrisrackauckas.com/
Abstract:
Modern techniques of machine learning are uncanny in their ability to automatically learn predictive models directly from data. However, they do not tend to work beyond their original training dataset. Mechanistic models utilize characteristics of the problem to ensure accurate qualitative extrapolation but can lack in predictive power. How can we build techniques which integrate the best of both approaches? In this talk we will discuss the body of work around universal differential equations, a technique which mixes traditional differential equation modeling with machine learning for accurate extrapolation from small data. We will showcase how incorporating different variations of the technique, such … READ MORE
Useful Algorithms That Are Not Optimized By Jax, PyTorch, or Tensorflow
July 21 2021 in Julia, Programming, Science, Scientific ML | Tags: automatic differentiation, differentiable programming, julia, machine learning | Author: Christopher Rackauckas
In some previous blog posts we described in details how one can generalize automatic differentiation to give automatically stability enhancements and all sorts of other niceties by incorporating graph transformations into code generation. However, one of the things which we didn’t go into too much is the limitation of these types of algorithms. This limitation is what we have termed “quasi-static” which is the property that an algorithm can be reinterpreted as some static algorithm. It turns out that for very fundamental reasons, this is the same limitation that some major machine learning frameworks impose on the code that they can fully optimize, such as Jax or Tensorflow. This led us to the question: are there algorithms which are not optimizable within this mindset, and why? The answer is now published at ICML 2021, so lets dig into … READ MORE
ModelingToolkit, Modelica, and Modia: The Composable Modeling Future in Julia
April 19 2021 in Differential Equations, Julia, Mathematics, Programming, Science, Scientific ML | Tags: acausal, dae, language, modelica, modeling, modelingtoolkit, modia | Author: Christopher Rackauckas
Let me take a bit of time here to write out a complete canonical answer to ModelingToolkit and how it relates to Modia and Modelica. This question comes up a lot: why does ModelingToolkit exist instead of building on tooling for Modelica compilers? I’ll start out by saying I am a huge fan of Martin and Hilding’s work and we work very closely with them on the direction of Julia-based tooling for modeling and simulation. ModelingToolkit, being a new system, has some flexibility in the design space it explores, and while we are following a different foundational philosophy, we have many of the same goals.
Composable Abstractions for Model Transformations
Everything in the SciML organization is built around a principle of confederated modular development: let other packages influence the capabilities of your own. This is highlighted in a … READ MORE
Generalizing Automatic Differentiation to Automatic Sparsity, Uncertainty, Stability, and Parallelism
March 10 2021 in Differential Equations, Julia, Mathematics, Programming, Science, Scientific ML | Tags: abstract interpretation, automatic differentiation, non-standard interpretation, Pantelides algorithm | Author: Christopher Rackauckas
Automatic differentiation is a “compiler trick” whereby a code that calculates f(x) is transformed into a code that calculates f'(x). This trick and its two forms, forward and reverse mode automatic differentiation, have become the pervasive backbone behind all of the machine learning libraries. If you ask what PyTorch or Flux.jl is doing that’s special, the answer is really that it’s doing automatic differentiation over some functions.
What I want to dig into in this blog post is a simple question: what is the trick behind automatic differentiation, why is it always differentiation, and are there other mathematical problems we can be focusing this trick towards? While very technical discussions on this can be found in our recent paper titled “ModelingToolkit: A Composable Graph Transformation System For Equation-Based Modeling” and descriptions of methods like intrusive uncertainty quantification, I want … READ MORE