Kyle Murphy

Kyle Murphy

Data Analyst/Researcher


Vectorizing and Parallelizing JB2008

1 minute read

Published:

JB2008 is an emperical atmospheric neutral density model (long name Jacchia-Bowman 2008). It is the most recent itteration of the Jacchia-Bowman family at atmospheric models and incorporates both Solar and Geomagnetic activity into the modelling of atmospheric neutral denisty, temperature, and exospheric temperature.

The model is written in Fortran and has been translated to other languages including Python (e.g., Pyatmos) and Matlab(e.g., DESTO). Models predictions can also be requested through NASA’s CCMC.

The Pyatmos is particularly useful as it provides a wrapper to run JB2008 in Python, arguabally the most popular language for research, data science and machine learning. However, Pyatmos can be slow when performing a large number of predictions. This can inhibit larger studies which require a larger number of predictions (e.g., statistical or machine learning studies).

To that end I set out to speed up the Python version of JB2008 by vectorizing portions of the code as well as leveraging parallelization. The vectorization is applied to a number of input and data prep functions. The parallelization takes advantage of the swifter module which first aims to vectorize a function call and then determines if Dask should can be used to speed up code via parallel processing. The combination of vectorization and swifter signifcantly speeds up JB2008 predictions. When ran on colab the vector-parallelized codes sees an ~11x speed boost compared to Pyatmos, 40s compared to nearly 8 minutes when predicting density for ~70,000 points (see below). On faster machines, the same calculation can take as little as 20 seconds!

The vector-parallelized code can be found here.

JB2008 Vector-Parallelized

Fast JB2008

JB2008 Pyatmos

Slow JB2008

You May Also Enjoy

skops - Machine Learning Model Persistence

2 minute read

Published:

Persisting ML/AI models (or model serialization) allows users to save models that they’ve trained so that they can be reloaded and reused. Model persistence is important as it can save time and allows models to be shared with other users without the need to retrain the model (which can be computationally expensive and time consuming) or share the training data set (which can be large).

Loading Data from the Web

1 minute read

Published:

Many of Pandas read functions have the ability to read files from the internet. This can be helpful you’re writing tutorials or examples and want them to work even if users don’t have the data. Or if you want to quickly look at data without downloading it.

Anaconda and VS Code

less than 1 minute read

Published:

Having problems accessing Anaconda from VS Code (Python: Select Interpreter)?

Python Paths

1 minute read

Published:

Often when working in science/research we are rapidly developing and modifying code (and maybe not following the very best practices for coding). In these instances we may not be writing, developing, or installing Python modules which can be accessed by the Python interpreter. Rather we may have a directory with set of scripts, functions, or classes. In Python in order to access these without installing a module we need to add the file path of our directory to the Python path. This can be easily acomplished using the sys module.