NCToolkit is built to handle multi-file datasets easily and efficiently — parallel processing, ensemble averaging and merging are all straightforward.
Multi-file datasets are created the same way as single-file ones — a list of paths or a wildcard:
ds = nc.open_data("foo/*.nc") ds.tmean() # applied independently to each file in the ensemble
This operation only applies to individual members of the multi-file dataset — to calculate statistics across the ensemble, see Ensemble statistics below.
Two merge strategies are available via merge:
ds.merge("time") # files share variables/grid, distinct times ds.merge("variable") # files share time steps; this is the default
Merge by time when files have the same variables and grids but distinct times. Merge by variable when files share time steps, or one file has at most one time step.
To calculate a statistic across ensemble members (e.g. across climate models) rather than within each one, use the ensemble methods: ensemble_mean, ensemble_percentile, ensemble_stdev, ensemble_var, ensemble_max, ensemble_min, ensemble_range and ensemble_sum.
ds.ensemble_mean() # e.g. monthly mean across 20 climate models
Set the number of cores used for processing files in a multi-file dataset in parallel. It's almost always faster to process files with a high core count before merging, rather than merging first:
nc.options(cores=6)
For processing an ensemble with your own Python function via multiprocessing, see Parallel processing.