Symbolic-Numerics: how compiler smarts can help improve the performance of numerical methods (nonlinear solvers in Julia)
February 13 2024 in Julia, Mathematics, Programming | Tags: julia, nonlinear solve, pydata, symbolic-numerics | Author: Christopher Rackauckas
Many problems can be reduced down to solving f(x)=0, maybe even more than you think! Solving a stiff differential equation? Finding out where the ball hits the ground? Solving an inverse problem to find the parameters to fit a model? In this talk we’ll showcase how SciML’s NonlinearSolve.jl is a general system for solving nonlinear equations and demonstrate its ability to efficiently handle these kinds of problems with high stability and performance. We will focus on how compilers are being integrated into the numerical stack so that many of the things that were manual before, such as defining sparsity patterns, Jacobians, and adjoints, are all automated out-of-the-box making it greatly outperform purely numerical codes like SciPy or NLsolve.jl.
PyData Global 2023
ChatGPT performs better on Julia than Python (and R) for Large Language Model (LLM) Code Generation. Why?
November 19 2023 in Julia, Programming | Tags: ai, artificial intelligence, chatgpt, julia, large language models, llm, machine learning, MATLAB, python, r | Author: Christopher Rackauckas
Machine learning is all about examples. The more data you have, the better it should perform, right? With the rise of ChatGPT and Large Language Models (LLMs) as a code helping tool, it was thus just an assumption that the most popular languages like Python would likely be the best for LLMs. But because of the increased productivity, I tend to use a lot of Julia, a language with an estimated user-base of around a million programmers. For this reason, people have often asked me how it fairs with ChatGPT, Github Copilot, etc., and so I checked out those pieces and… was stunned. It’s really good. It seemed better than Python actually?
The data is in: Julia does well with ChatGPT
This question was recently put to the test by a researcher named Alessio Buscemi in A Comparative Study … READ MORE
Summary of Julia Plotting Packages
June 17 2023 in Julia | Tags: data science, ggplot2, julia, plots, programming language, startup time, ttfx, visualization | Author: Christopher Rackauckas
This is a repost of my response on the Julia Discourse on this topic. I was asked to make a blog post so here you go!
The “Main” Plotting Packages
Here’s a quick summary of the most widely used plotting packages. I may have missed one, but I haven’t missed one that is very widely used.
- Plots.jl is the most used. It’s probably the most documented, used in the most tutorials, and is used in many videos.
- Pros: Its main draw is that it has a lot of plugins to other packages through its recipes system, which means that a lot of odd things like `plot(sol::ODESolution)` or showing the sparsity of a `BandedMatrix` just works. With all of these integrations, it’s normally what I would recommend first to newcomers since they will generally get the most done with the least work. It … READ MORE
- Pros: Its main draw is that it has a lot of plugins to other packages through its recipes system, which means that a lot of odd things like `plot(sol::ODESolution)` or showing the sparsity of a `BandedMatrix` just works. With all of these integrations, it’s normally what I would recommend first to newcomers since they will generally get the most done with the least work. It … READ MORE
Integrating equation solvers with probabilistic programming through differentiable programming
November 24 2022 in Julia, Programming, Science, Scientific ML | Tags: differential equations, julia, probabilistic programming, scientific machine learning, sciml | Author: Christopher Rackauckas
Part of the COMPUTATIONAL ABSTRACTIONS FOR PROBABILISTIC AND DIFFERENTIABLE PROGRAMMING WORKSHOP
Abstract: Many probabilistic programming languages (PPLs) attempt to integrate with equation solvers (differential equations, nonlinear equations, partial differential equations, etc.) from the inside, i.e. the developers of the PPLs like Stan provide differential equation solver choices as part of the suite. However, as equation solvers are an entire discipline to themselves with many active development communities and subfields, this places an immense burden on PPL developers to keep up with the changing landscape of tens of thousands of independent researchers. In this talk we will explore how Julia PPLs such as Turing.jl support of equation solvers from the outside, i.e. how the tools of differentiable programming allows equation solver libraries to be compatible with … READ MORE
Engineering Trade-Offs in Automatic Differentiation: from TensorFlow and PyTorch to Jax and Julia
December 25 2021 in Julia, Programming, Science, Scientific ML | Tags: automatic differentiation, compilers, differentiable programming, jax, julia, machine learning, pytorch, tensorflow, XLA | Author: Christopher Rackauckas
To understand the differences between automatic differentiation libraries, let’s talk about the engineering trade-offs that were made. I would personally say that none of these libraries are “better” than another, they simply all make engineering trade-offs based on the domains and use cases they were aiming to satisfy. The easiest way to describe these trade-offs is to follow the evolution and see how each new library tweaked the trade-offs made of the previous.
Early TensorFlow used a graph building system, i.e. it required users to essentially define variables in a specific graph language separate from the host language. You had to define “TensorFlow variables” and “TensorFlow ops”, and the AD would then be performed on this static graph. Control flow constructs were limited to the constructs that could be represented statically. For example, an `ifelse` function statement is very different from … READ MORE
Useful Algorithms That Are Not Optimized By Jax, PyTorch, or Tensorflow
July 21 2021 in Julia, Programming, Science, Scientific ML | Tags: automatic differentiation, differentiable programming, julia, machine learning | Author: Christopher Rackauckas
In some previous blog posts we described in details how one can generalize automatic differentiation to give automatically stability enhancements and all sorts of other niceties by incorporating graph transformations into code generation. However, one of the things which we didn’t go into too much is the limitation of these types of algorithms. This limitation is what we have termed “quasi-static” which is the property that an algorithm can be reinterpreted as some static algorithm. It turns out that for very fundamental reasons, this is the same limitation that some major machine learning frameworks impose on the code that they can fully optimize, such as Jax or Tensorflow. This led us to the question: are there algorithms which are not optimizable within this mindset, and why? The answer is now published at ICML 2021, so lets dig into … READ MORE
JuliaCall Update: Automated Julia Installation for R Packages
January 18 2021 in Differential Equations, Julia, Mathematics, Programming, R | Tags: devops, differentialequations, installation, julia, juliacall, modelingtoolkit, r | Author: Christopher Rackauckas
Some sneakily cool features made it into the JuliaCall v0.17.2 CRAN release. With the latest version there is now an install_julia function for automatically installing Julia. This makes Julia a great high performance back end for R packages. For example, the following is an example from the diffeqr package that will work, even without Julia installed:
install.packages("diffeqr")
library(diffeqr)
de <- diffeqr::diffeq_setup()
lorenz <- function (u,p,t){
du1 = p[1]*(u[2]-u[1])
du2 = u[1]*(p[2]-u[3]) - u[2]
du3 = u[1]*u[2] - p[3]*u[3]
c(du1,du2,du3)
}
u0 <- c(1.0,1.0,1.0)
tspan <- c(0.0,100.0)
p <- c(10.0,28.0,8/3)
prob <- de$ODEProblem(lorenz,u0,tspan,p)
fastprob <- diffeqr::jitoptimize_ode(de,prob)
sol <- de$solve(fastprob,de$Tsit5(),saveat=0.01)
Under the hood it’s using the DifferentialEquations.jl package and the SciML stack, but it’s abstracted from users so much that Julia is essentially an alternative to Rcpp with easier interactive development. The following example really brings the seamless … READ MORE
How Inexact Models and Scientific Machine Learning Can Guide Decision Making in Quantitative Systems Pharmacology
December 15 2019 in Biology, Julia, Pharmacology, Science | Tags: differential equations, julia, pharmacology, pumas, qsp | Author: Christopher Rackauckas
Pre-clinical Quantitiative Systems Pharmacology (QSP) is about trying to understand how a drug target effects an outcome. If I effect this part of the biological pathways, how will it induce toxicity? Will it be effective?
Recently I have been pulling in a lot of technical collegues to help with the development of next generation QSP tooling. Without a background in biological modeling, I found it difficult to explain the "how" and "why" of pharmacological modeling. Why is it differential equations, and where do these "massively expensive global optimization" runs come from? What kinds of problems can you solve with such models when you know that they are only approximate?
To solve these questions, I took a step back and tried to explain a decision making scenario with a simple model, to showcase how playing with a model can allow one to distinguish … READ MORE
Neural Jump SDEs (Jump Diffusions) and Neural PDEs
June 5 2019 in Differential Equations, Julia, Mathematics, Stochastics | Tags: CUDA, differentiable programming, DifferentialEquations.jl, gpu, julia, stochastic differential equations | Author: Christopher Rackauckas
This is just an exploration of some new neural models I decided to jot down for safe keeping. DiffEqFlux.jl gives you the differentiable programming tools to allow you to use any DifferentialEquations.jl problem type (DEProblem) mixed with neural networks. We demonstrated this before, not just with neural ordinary differential equations, but also with things like neural stochastic differential equations and neural delay differential equations.
At the time we made DiffEqFlux, we were the “first to the gate” for many of these differential equations types and left it as an open question for people to find a use for these tools. And judging by the Arxiv papers that went out days after NeurIPS submissions were due, it looks like people now have justified some machine learning use cases for them. There were two separate papers on neural … READ MORE
Video Blog: Developing and Editing Julia Packages
May 16 2017 in Julia, Programming | Tags: git, Gitkraken, julia, packages | Author: Christopher Rackauckas
Google Summer of Code is starting up, so I thought it would be a good time to share my workflow for developing my own Julia packages, as well as my workflow for contributing to other Julia packages. This does not assume familiarity with commandline Git, and instead shows you how to use a GUI (GitKraken) to make branches and PRs, as well as reviewing and merging code. You can think of it as an update to my old blog post on package development in Julia. However, this is not only updated but also improved since I am now able to walk through the “non-code” parts of package developing (such as setting up AppVeyor and code coverage).
Enjoy! (I quite like this video blog format: it was a lot less work)