Datasets

Opening datasets

NCToolkit requires netCDF data that follows the GDT, COARDS or CF conventions. Its computational backend is CDO, which can carry out most operations regardless of strict compliance — but if you're unsure whether your files are compliant, see Supported data for how to check.

There are three ways to create a dataset:

• open_data(path)

  Opens local files. Accepts a single path, a list of paths, or a wildcard such as "data/*.nc".

• open_url(url)

  Downloads a file at a URL or ftp address to a temporary folder, then opens it.

• open_thredds(url)

  Opens data served over a thredds server or OPeNDAP. The URL must end in .nc.

import nctoolkit as nc

ds = nc.open_data(infile)             # a single file
ds = nc.open_data(file_list)          # a list of files
ds = nc.open_data("data/*.nc")         # a wildcard, i.e. a multi-file dataset

When you open a multi-file dataset, standard methods apply to each member file individually; to combine members, see Ensembles & multi-file datasets. For a quick overview of what's inside a dataset, check contents — a dataframe of variables, units and long names:

ds = nc.open_thredds("https://psl.noaa.gov/thredds/dodsC/Datasets/COBE2/sst.mon.ltm.1981-2010.nc")
ds.contents

Checking validity of source data

NCToolkit should work out of the box with most netCDF data. However, the format could be incompatible with the system libraries NCToolkit relies on, or the files could simply be corrupt. Run a general check on the data with check:

ds.check()

This runs basic checks on data format compatibility. Install cfchecker if you also want check to verify CF-compliance.

To check whether the files in a dataset are corrupt, use is_corrupt — it reads and writes the source data to a temporary file, which is enough to catch corruption:

ds.is_corrupt()

Modifying datasets

To modify a dataset, just use NCToolkit's built-in methods — they operate directly on the dataset itself. The example below selects the first time step of a sea-surface temperature dataset and plots it:

ds = nc.open_thredds("https://psl.noaa.gov/thredds/dodsC/Datasets/COBE2/sst.mon.ltm.1981-2010.nc")
ds.subset(time=0)
ds.plot()

Underlying datasets are temporary files representing the current state of the dataset, accessible via the current attribute:

ds.current

In this case, that's a single temporary file. Temporary files are generated and deleted automatically as needed — there's no need to manage them yourself.

Lazy evaluation by default

By default, NCToolkit evaluates lazily: operations are recorded, not run, until a result is actually needed. Look at the processing chain below — it carries out four operations, so we don't want a temporary file written at every step:

ds = nc.open_thredds("https://psl.noaa.gov/thredds/dodsC/Datasets/COBE2/sst.mon.ltm.1981-2010.nc")
ds.assign(sst=lambda x: x.sst + 273.15)
ds.subset(months=1)
ds.subset(lon=[-80, 20], lat=[30, 70])
ds.spatial_mean()

NCToolkit has been told what to do to the dataset, but hasn't been told to actually do any of it. Checking ds.current at this point still shows the starting file. To evaluate the chain, use run() — or any method that needs a result, such as plot():

ds.run()
ds.current

This chaining ability comes from Climate Data Operators (CDO), NCToolkit's backend computational engine. You don't need to understand CDO to use NCToolkit, but if you want to see the underlying commands, use the history attribute — before running, it shows the four lines of Python queued up; after running, a single CDO command:

ds.history    # before run() -- queued Python operations
ds.run()
ds.history    # after run() -- the single CDO command actually used

Method chaining

NCToolkit does not allow method chaining the way pandas and xarray do — this will not work:

(
    ds
    .tmean()
    .spatial_mean()
    .add(1)
)

This kind of chaining needs each method to return an object, but NCToolkit's methods generally don't return objects — they modify the dataset in place. Call them on separate lines instead:

ds.tmean()
ds.spatial_mean()
ds.add(1)

Dataset attributes

Useful attributes for inspecting a dataset:

Temporary files are created and deleted automatically as needed — there's no need to manage them yourself. See Q&A for cleaning them up manually if you ever need to.