Parallel Processing

Two methods of parallel processing are available: carrying out operations on multi-file datasets in parallel, and defining a processing chain in NCToolkit and using Python's multiprocessing package to run it over many files.

Multi-file datasets in parallel

If you have a multi-file dataset, processing the files within it in parallel just needs a core count:

python

nc.options(cores=6)

NCToolkit will process the files in multi-file datasets in parallel using that many cores, capped at the number of cores available on your machine.

Using multiprocessing

A common task is taking a folder of files, doing something to each, and saving a modified version of each into a new folder. Before using Python's multiprocessing package for this, change the global settings:

python

import nctoolkit as nc
nc.options(parallel=True)

This tells NCToolkit it's about to run in parallel. Behind the scenes, NCToolkit constantly creates and deletes temporary files, tracked via a safe-list of files currently in use. Adding to that list from multiple processes at once needs a process-safe list type — this setting switches to one.

A worked example: convert a folder of files with temperature in Celsius to Kelvin, saving each output to a new directory. First, a function that processes one input file and writes the result to a new location:

python

def process_chain(infile):
    # converts temperature to Kelvin, saves the output in a new directory
    outfile = infile.replace("ensemble", "new")
    if not os.path.exists(os.path.dirname(outfile)):
        os.mkdir(os.path.dirname(outfile))
    ds = nc.open_data(infile)
    ds.assign(tos=lambda x: x.sst + 273.15)
    ds.to_nc(outfile)

Then loop through the files in the ensemble folder and apply it, using a pool of 3 worker processes:

python

ensemble = nc.create_ensemble("ensemble")
import multiprocessing as mp
import os

pool = mp.Pool(3)
for ff in ensemble:
    pool.apply_async(process_chain, [ff])
pool.close()
pool.join()

Once parallel processing is done — especially in an interactive session or notebook — reset the setting:

python

nc.options(parallel=False)

This matters because of how manually terminating commands interacts with the process-safe lists multiprocessing mode uses internally — an edge case that's hard to avoid entirely, so resetting afterwards is the safest habit.

Multi-file datasets in parallel

Using multiprocessing