API Reference

nc.options(**kwargs)

Define session options.

Parameters

kwargs
Define options using key/value pairs. Set thread_safe=True if HDF5 was built to be thread-safe. Set lazy=False if you want methods to evaluate non-lazily. Set cores=n to process the individual files in a multi-file dataset in parallel — only affects multi-file datasets. Set temp_dir="/foo" to change the directory NCToolkit uses to save temporary files. Set progress to "on" or "off" to always or never show a progress bar for multi-file processing; defaults to "auto".

nc.open_data(x=[], checks=True, **kwargs)

Read netCDF data as a Dataset object

Parameters

x
A string or list of netCDF files or a single url. The function will check the files exist. If x is not a list, but an iterable it will be converted to a list. If a *.nc style wildcard is supplied, open_data will use all files available. By default an empty dataset is created, ie. using open_data() will create an empty dataset that can then be expanded using append.
checks
Do you want basic checks to ensure cdo can read files? Default to True. Setting to False can result in a minor speed up.
kwargs
Optional arguments for internal use by open_thredds and open_url.

nc.open_url(x=None, ftp_details=None, wait=None, file_stop=None)

Read netCDF data from a url as a DataSet object

Parameters

Required:

xrequired
A string with a url. Prior to processing data will be downloaded to a temp folder.

Optional:

ftp_details
A dictionary giving the user name and password combination for ftp downloads: {"user":user, "password":pass}
wait
Time to wait, in seconds, for data to download. A minimum of 3 attempts will be made to download the data.
file_stop
Time limit, in minutes, for individual attempts at downloading data. This is useful to get around download freezes.

nc.open_thredds(x=None, wait=None, checks=False)

Read thredds data as a Dataset object

Parameters

Required:

xrequired
A string or list of thredds urls, which must end with .nc.

Optional:

checks
Do you want to check if data is available over thredds?
wait
Time to wait for thredds server to be checked. Limitless if not supplied.

nc.open_geotiff(x=[])

Open a geotiff and convert to a Dataset. This requires rioxarray to be installed.

Parameters

x
A string or list of geotiff files or a single url. This requires rioxarray to be installed.

nc.from_xarray(ds)

Convert an xarray dataset to an nctoolkit dataset. This will first save the xarray dataset as a temporary netCDF file.

Parameters

dsrequired
xarray dataset you want to convert to nctoolkit DataSet.

ds.copy()

Make a deep copy of an DataSet object. Note: This will not make disk copies of the temporary files underlying datasets, so it will be disk-space efficient.

nc.create_ensemble(path='', recursive=True)

Generate an ensemble of files from a directory.

Parameters

Required:

pathrequired
The directory to search for netCDF files

Optional:

recursive
True/False depending on whether you want to search the path recursively. Defaults to True.

ds.variables

List variables contained in a dataset

ds.contents

Detailed list of variables contained in a dataset. This will only display the variables in the first file of an ensemble.

ds.times

List times contained in a dataset

ds.years

List years contained in a dataset

ds.months

List months contained in a dataset

ds.levels

List levels contained in a dataset

ds.size

The size of an object. This will print the number of files, total size, and smallest and largest files in an DataSet object.

ds.current

The current file or files in the DataSet object

ds.history

The history of operations on the DataSet

ds.start

The starting file or files of the DataSet object

ds.calendar

List calendars of dataset files

ds.ncformat

List formats of files contained in a dataset

ds.subset(**kwargs)

A method for subsetting datasets to specific variables, years, longitudes etc. Operations are applied in the order supplied.

Parameters

All arguments are optional keywords, applied in the order supplied. NCToolkit matches on partial names, so year, month, var etc. also work.

variables
A variable or list of variables to select. Accepts wildcards, e.g. "var*" selects every variable starting with var.
years
Year(s) to select, as a list, range or int.
months
Month(s) to select, as a list, range or int.
days
Day(s) to select, as a list, range or int.
hours
Hour(s) to select, as a list, range or int.
seasons
Season(s) to select: "DJF", "MAM", "JJA" or "SON".
timesteps
Time step(s) to select by index, e.g. 0 for the first time step.
range
A [date_min, date_max] list selecting times on or after date_min and before date_max. Dates can be datetime objects or "DD/MM/YYYY"-style strings.
lon
Longitude range to select, as [lon_min, lon_max] between -180 and 180.
lat
Latitude range to select, as [lat_min, lat_max] between -90 and 90.
levels
Vertical level/depth range to select, as [min_level, max_level].

ds.crop(lon=[-180, 180], lat=[-90, 90], nco=False, nco_vars=None)

Crop to a rectangular longitude and latitude box

Parameters

lon
The longitude range to select. This must be two variables, between -180 and 180 when nco = False.
lat
The latitude range to select. This must be two variables, between -90 and 90 when nco = False.
nco
Do you want this to use NCO for cropping? Defaults to False, and uses CDO. Set to True if you want to call NCO. NCO is typically better at handling very large horizontal grids.
nco_varsif nco=True
If using NCO, the variables you want to select

ds.drop(**kwargs)

Remove variables, days, months, years or time steps from a dataset. This will remove stated variables from files in the dataset.

Parameters

All arguments are optional keywords, applied in the order supplied. NCToolkit matches on partial names, so year, month, variable etc. also work.

var
A variable or list of variables to drop. Accepts wildcards, e.g. "var*".
year
Year(s) to drop.
month
Month(s) to drop.
day
Day(s) to drop.
time
Time step(s) to drop by index. Negative indices are accepted.

ds.set_date(year=None, month=None, day=None, base_year=1900)

Set the date in a dataset. You should only do this if you have to fix/change a dataset with a single, not multiple dates.

Parameters

Required:

yearrequired
The year
monthrequired
The month
dayrequired
The day

Optional:

base_year
The base year for time creation in the netCDF. Defaults to 1900.

ds.set_day(x)

Set the day for each time step in a dataset

Parameters

xrequired
Day to set dataset to

ds.shift(**kwargs)

Shift times in dataset by a number of hours, days, months, or years. Operations are applied in the order supplied.

Parameters

kwargs
hours maps to shift_hours days maps to shift_days months maps to shift_months years maps to shift_years. Note: this uses partial matches. So hour, day, month, year will also work.

nc.cor_time(x=None, y=None)

Calculate the temporal correlation coefficient between two datasets. This will calculate the temporal correlation coefficient, for each time step, between two datasets. The datasets must either have the same variables or only have one variable.

Parameters

xrequired
First dataset to use
yrequired
Second dataset to use

nc.cor_space(x=None, y=None)

Calculate the spatial correlation coefficient between two datasets. This will calculate the spatial correlation coefficient, for each time step, between two datasets. The datasets must either have the same variables or only have one variable.

Parameters

xrequired
First dataset to use
yrequired
Second dataset to use

ds.rolling_mean(window=None, align='right')

Calculate a rolling mean based on a window

Parameters

Required:

windowrequired
The size of the window for the calculation of the rolling mean

Optional:

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.rolling_min(window=None, align='right')

Calculate a rolling minimum based on a window

Parameters

Required:

windowrequired
The size of the window for the calculation of the rolling minimum

Optional:

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.rolling_max(window=None, align='right')

Calculate a rolling maximum based on a window

Parameters

Required:

windowrequired
The size of the window for the calculation of the rolling maximum

Optional:

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.rolling_sum(window=None, align='right')

Calculate a rolling sum based on a window

Parameters

Required:

windowrequired
The size of the window for the calculation of the rolling sum

Optional:

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.rolling_range(window=None, align='right')

Calculate a rolling range based on a window

Parameters

Required:

windowrequired
The size of the window for the calculation of the rolling range

Optional:

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.rolling_stdev(window=None, align='right')

Calculate a rolling standard deviation based on a window

Parameters

Required:

windowrequired
The size of the window for the calculation of the rolling standard deviation

Optional:

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.rolling_var(window=None, align='right')

Calculate a rolling variance based on a window

Parameters

Required:

windowrequired
The size of the window for the calculation of the rolling variance

Optional:

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.annual_anomaly(baseline=None, metric='absolute', window=1, align='right')

Calculate annual anomalies for each variable based on a baseline period. The anomaly is derived by first calculating the climatological annual mean for the given baseline period. Annual means are then calculated for each year and the anomaly is calculated compared with the baseline mean. This will be calculated on a per-file basis in a multi-file dataset.

Parameters

Required:

baselinerequired
Baseline years. This needs to be the first and last year of the climatological period. Example: a baseline of [1980,1999] will result in anomalies against the 20 year climatology from 1980 to 1999.

Optional:

metric
Set to "absolute" or "relative", depending on whether you want the absolute or relative anomaly to be calculated.
window
A window for the anomaly. By default window = 1, i.e. the annual anomaly is calculated. If, for example, window = 20, the 20 year rolling means will be used to calculate the anomalies.

ds.monthly_anomaly(baseline=None)

monthly:anomaly: Calculate monthly anomalies based on a baseline period. The anomaly is derived by first calculating the climatological monthly mean for the given baseline period. Monthly means are then calculated for each year and the anomaly is calculated compared with the baseline mean. This is calculated separately for each file in a multi-file dataset.

Parameters

baselinerequired
Baseline years. This needs to be the first and last year of the climatological period. Example: a baseline of [1985,2005] will result in anomolies against 20 year climatology from 1986 to 2005.

ds.tmean(over='time', align='right', window=None)

Calculate the temporal mean of all variables. Useful for: monthly mean, annual/yearly mean, seasonal mean, daily mean, daily climatology, monthly climatology, seasonal climatology

Parameters

over
Time periods to average over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
window
This determines the number of time steps to calculate the mean over to calculate over, on a non-rolling basis. This is useful if you need to calculate the mean every 5 days, for example.

ds.tmin(over='time', align='right', window=None)

Calculate the temporal minimum of all variables. Useful for: monthly minimum, annual/yearly minimum, seasonal minimum, daily minimum, daily climatology, monthly climatology, seasonal climatology

Parameters

over
Time periods to average over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R, with over acting as the grouping.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
window
This determines the number of time steps to calculate the minimum over to calculate over, on a non-rolling basis. This is useful if you need to calculate the minimum every 5 days, for example.

ds.tmedian(over='time', align='right')

Calculate the temporal median of all variables. Useful for: monthly median, annual/yearly median, seasonal median, daily median, daily climatology, monthly climatology, seasonal climatology

Parameters

over
Time periods to average over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R, with over acting as the grouping.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.tpercentile(p=None, over='time', align='right')

Calculate the temporal percentile of all variables. Useful for monthly percentile, annual/yearly percentile, seasonal percentile, daily percentile, daily climatology, monthly climatology, seasonal climatology

Parameters

Required:

prequired
Percentile to calculate

Optional:

over
Time periods to average over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R, with over acting as the grouping.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.tmax(over='time', align='right', window=None)

Calculate the temporal maximum of all variables. Useful for: monthly maximum, annual/yearly maximum, seasonal maximum, daily maximum, daily climatology, monthly climatology, seasonal climatology

Parameters

over
Time periods to average over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R, with over acting as the grouping.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
window
This determines the number of time steps to calculate the maximum over to calculate over, on a non-rolling basis. This is useful if you need to calculate the maximum every 5 days, for example.

ds.tsum(over='time', align='right', window=None)

Calculate the temporal sum of all variables.

Parameters

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
over
Time periods to count the sum over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R, so you can supply combinations of these to get the sum over each year, month or day.
window
This determines the number of time steps to sum over, on a non-rolling basis. This is useful if you need to calculate the sum every 5 days, for example.

ds.trange(over='time', align='right', window=None)

Calculate the temporal range of all variables. Useful for: monthly range, annual/yearly range, seasonal range, daily range, daily climatology, monthly climatology, seasonal climatology

Parameters

over
Time periods to average over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R, with over acting as the grouping.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
window
This determines the number of time steps to calculate the range over to calculate over, on a non-rolling basis. This is useful if you need to calculate the range every 5 days, for example.

ds.tstdev(over='time', align='right', window=None)

Calculate the temporal standard deviation of all variables. Useful for: monthly standard deviation, annual/yearly standard deviation, seasonal standard deviation, daily standard deviation, daily climatology, monthly climatology, seasonal climatology

Parameters

over
Time periods to average over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R, with over acting as the grouping.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
window
This determines the number of time steps to calculate the standard deviation over to calculate over, on a non-rolling basis. This is useful if you need to calculate the standard deviation every 5 days, for example.

ds.tcumsum(align='right')

Calculate the temporal cumulative sum of all variables

Parameters

align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"

ds.tvar(over='time', align='right', window=None)

Calculate the temporal variance of all variables. Useful for: monthly variance, annual/yearly variance, seasonal variance, daily variance, daily climatology, monthly climatology, seasonal climatology

Parameters

over
Time periods to average over. Options are 'year', 'month', 'day'. This operates in a similar way to the groupby method in pandas or the tidyverse in R, with over acting as the grouping.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
window
This determines the number of time steps to calculate the variance over to calculate over, on a non-rolling basis. This is useful if you need to calculate the variance every 5 days, for example.

ds.cor_space(var1=None, var2=None)

Calculate the correlation correct between two variables in space. This is calculated for each time step. The correlation coefficient coefficient is calculated using values in all grid cells, ignoring missing values. This calculates the Pearson correlation coefficient.

Parameters

var1required
The first variable
var2required
The second variable

ds.cor_time(var1=None, var2=None)

Calculate the correlation correct in time between two variables. The correlation is calculated for each grid cell, ignoring missing values. This calculates the Pearson correlation coefficient.

Parameters

var1required
The first variable
var2required
The second variable

ds.spatial_mean()

Calculate the area weighted spatial mean for all variables. This is performed for each time step.

ds.spatial_min()

Calculate the spatial minimum for all variables. This is performed for each time step.

ds.spatial_max()

Calculate the spatial maximum for all variables. This is performed for each time step.

ds.spatial_percentile(p=None)

Calculate the spatial percentile for all variables. This is performed for each time step.

Parameters

prequired
Percentile to calculate. 0<=p<=100.

ds.spatial_range()

Calculate the spatial range for all variables. This is performed for each time step.

ds.spatial_sum(by_area=False)

Calculate the spatial sum for all variables. This is performed for each time step.

Parameters

by_area
Set to True if you want to multiply the values by the grid cell area before summing over space. Default is False.

ds.spatial_stdev()

Calculate the spatial standard deviation for all variables. This is performed for each time step.

ds.spatial_var()

Calculate the spatial variance for all variables. This is performed for each time step.

ds.centre(by='latitude', by_area=False)

Calculate the latitudinal or longitudinal centre for each year/month combination in files. This applies to each file in an ensemble.

Parameters

by
Set to 'latitude' if you want the latitudinal centre calculated. 'longitude' for longitudinal.
by_area
If the variable is a value/m2 type variable, set to True, otherwise set to False.

ds.zonal_mean()

Calculate the zonal mean for each time step

ds.zonal_min()

Calculate the zonal minimum for each time step

ds.zonal_max()

Calculate the zonal maximum for each time step

ds.zonal_range()

Calculate the zonal range for each time step

ds.zonal_sum(by_area=False)

Calculate the zonal sum for each time step

Parameters

by_area
Set to True if you want the cell value to be multiplied by the cell area prior to summing

ds.meridonial_mean()

Calculate the meridonial mean for each year/month combination in files. This applies to each file in an ensemble.

ds.meridonial_min()

Calculate the meridonial minimum for each year/month combination in files. This applies to each file in an ensemble.

ds.meridonial_max()

Calculate the meridonial maximum for each year/month combination in files. This applies to each file in an ensemble.

ds.meridonial_range()

Calculate the meridonial range for each year/month combination in files. This applies to each file in an ensemble.

ds.top()

Extract the top/surface level from a dataset. This extracts the first vertical level from each file in a dataset. This method is most useful for things like oceanic data, where this method will extract the sea surface. You may need to double check that the first vertical level is the surface, as this is not always the case.

ds.bottom(choice='level')

Extract the bottom level or value from a dataset. This extracts the bottom level or value from each netCDF file. Please note that for ensembles, it uses the first file to derive the index of the bottom level.

Parameters

choice
Set to 'level' to extract the bottom level, or 'value' to extract the bottommost non-missing value.

ds.vertical_interp(levels=None, fixed=None, thickness=None, depths=None, surface=None)

Verticaly interpolate a dataset based on given vertical levels. Vertical interpolation is calculated for each time step and grid cell

Parameters

Required:

levelsrequired
list of vertical levels, for example depths for an ocean model, to vertically interpolate to. These must be floats or ints.
fixedrequired
Define whether the vertical levels are the same in all spatial locations. Set to True if they are, e.g. you have z-levels. If you have the likes of sigma-coordinates, set this to False.

Optional:

thicknessif fixed=False
This or depths must be supplied if fixed is False, otherwise vertical thickness/depth cannot be known. Option argument when vertical levels vary in space. One of: a variable, in the dataset, which contains the variable thicknesses; a .nc file which contains the thicknesses; or a Dataset that contains the thicknesses. Note: the .nc file or Dataset must only contain one variable. Thickness should be in metres. Vertical interpolation will take the value from the mid-point of the level.
depthsif fixed=False
This or thickness must be supplied if fixed is False, otherwise vertical thickness/depth cannot be known. Option argument when vertical levels vary in space. One of: a variable, in the dataset, which contains the variable depths; a .nc file which contains the depths; or a Dataset that contains the depths. Note: the .nc file or Dataset must only contain one variable. Depths should be in metres, and be the mid-point of the level.
surfaceif thickness given
If thickness is supplied you must also supply this to identify whether the top or bottom of the level is the surface, i.e. the lowest level. This must be one of 'top' or 'bottom'.

ds.vertical_mean(thickness=None, depth_range=None, fixed=None)

Calculate the depth-averaged mean for each variable. This is calculated for each time step and grid cell.

Parameters

Required:

fixedrequired
Define whether the vertical levels are the same in all spatial locations. Set to True if they are, e.g. you have z-levels. If you have the likes of sigma-coordinates, set this to True.

Optional:

thicknessif fixed=False
This must be supplied when vertical levels vary in space, i.e. fixed=False. One of: a variable, in the dataset, which contains the variable thicknesses; a .nc file which contains the thicknesses; or a Dataset that contains the thicknesses. Note: the .nc file or Dataset must only contain one variable.
depth_range
Only use when vertical levels vary in space. Set a depth range if desired. Should be of the form [min_depth, max_depth].

ds.vertical_min()

Calculate the vertical minimum of variable values. This is calculated for each time step and grid cell.

ds.vertical_max()

Calculate the vertical maximum of variable values. This is calculated for each time step and grid cell.

ds.vertical_range()

Calculate the vertical range of variable values. This is calculated for each time step and grid cell.

ds.vertical_sum()

Calculate the vertical sum of variable values. This is calculated for each time step and grid cell.

ds.vertical_integration(thickness=None, depth_range=None, fixed=None)

Calculate the vertically integrated sum over the water column. This calculates the sum of the variable multiplied by the cell thickness

Parameters

Required:

fixedrequired
Define whether the vertical levels are the same in all spatial locations. Set to True if they are, e.g. you have z-levels. If you have the likes of sigma-coordinates, set this to True.

Optional:

thicknessif fixed=False
This must be supplied when vertical levels vary spatially. One of: a variable, in the dataset, which contains the variable thicknesses; a .nc file which contains the thicknesses; or a Dataset that contains the thicknesses. Note: the .nc file or DataSet must only contain one variable.
depth_range
Set a depth range if desired. Should be of the form [min_depth, max_depth].

ds.vertical_cumsum()

Calculate the vertical sum of variable values. This is calculated for each time step and grid cell.

ds.invert_levels()

Invert the levels of 3D variables. This is calculated for each time step and grid cell.

ds.bottom_mask()

Create a mask identifying the deepest cell without missing values.. This converts a dataset to a mask identifying which cell represents the bottom, for example the seabed. 1 identifies the deepest cell with non-missing values. Everything else is 0, or missing.

ds.regrid(grid=None, method='bil', recycle=False, one_grid=False, **kwargs)

Regrid a dataset to a target grid. Horizontal interpolation

Parameters

Required:

gridrequired
The grid to remap to

Optional:

method
Remapping method. Defaults to "bil". Methods available are: bilinear - "bil"; nearest neighbour - "nn" - "nearest neighbour" bicubic interpolation - "bic". Distance-weighted average - "dis". First order conservative remapping - "con". Second order conservative remapping - "con2". Large area fraction remapping - "laf"
recycle
Set to True if you want to re-use the remapping weights when you are regridding another dataset.
one_grid
Set to True if all files in multi-file dataset have the same grid, to speed things up.
kwargs
Instead of supplying a grid using 'grid', you can supply `lon` and `lat`. These must be equally lengthed lists or arrays that will be used to generate the grid. If you want to regrid to a single location you can just supply a float to lon and lat.

ds.to_latlon(lon=None, lat=None, res=None, method='bil', recycle=False, one_grid=False)

Regrid a dataset to a regular latlon grid

Parameters

Required:

lonrequired
2 element list giving minimum and maximum longitude of target grid
latrequired
2 element list giving minimum and maximum latitude of target grid
resrequired
If float or int given, this will be the horizontal and vertical resolution of the target grid. If 2 element list is given, the first element is the longitudinal resolution and the second is the latitudinal resolution.

Optional:

method
Remapping method. Defaults to "bil". Methods available are: bilinear - "bil"; nearest neighbour - "nn" - "nearest neighbour" bicubic interpolation - "bic". Distance-weighted average - "dis". First order conservative remapping - "con". Second order conservative remapping - "con2". Large area fraction remapping - "laf"
recycle
Do you want the grid and weights to be available for recycling and use in regrid? Defaults to False
one_grid
Set to True if all files in multi-file dataset have the same grid, to speed things up.

ds.resample_grid(factor=None)

Resample the horizontal grid of a dataset

Parameters

factorrequired
The resampling factor. Must be a positive integer. No interpolation occurs. Example: factor of 2 will sample every other grid cell

ds.time_interp(start=None, end=None, resolution='monthly')

Temporally interpolate variables based on date range and time resolution

Parameters

Required:

startrequired
Start date for interpolation. Needs to be of the form YYYY/MM/DD or YYYY-MM-DD.

Optional:

end
End date for interpolation. Needs to be of the form YYYY/MM/DD or YYYY-MM-DD. If end is not given interpolation will be to the final available time in the dataset.
resolution
Time steps used for interpolation. Needs to be "daily", "weekly", "monthly" or "yearly". Defaults to monthly.

ds.timestep_interp(steps=None)

Temporally interpolate a dataset to given number of time steps between existing time steps

Parameters

stepsrequired
Number of time steps to interpolate between existing time steps. For example, if you wanted to go from daily to hourly data you would set steps=24.

ds.fill_na(n=1)

Fill missing values with a distance-weighted average. This carries out infilling for each time step and vertical level. Filling only uses horizontal neighbours, not vertical.

Parameters

n
Number of nearest neighbours to use. Defaults to 1. To

ds.box_mean(x=1, y=1)

Calculate the grid box mean for all variables. This is performed for each time step.

Parameters

x
Number of boxes in the x, e.g. east-west direction
y
Number of boxes in the y, e.g. north-south direction

ds.box_max(x=1, y=1)

Calculate the grid box max for all variables. This is performed for each time step.

Parameters

x
Number of boxes in the x, e.g. east-west direction
y
Number of boxes in the y, e.g. north-south direction

ds.box_min(x=1, y=1)

Calculate the grid box min for all variables. This is performed for each time step.

Parameters

x
Number of boxes in the x, e.g. east-west direction
y
Number of boxes in the y, e.g. north-south direction

ds.box_sum(x=1, y=1)

Calculate the grid box sum for all variables. This is performed for each time step.

Parameters

x
Number of boxes in the x, e.g. east-west direction
y
Number of boxes in the y, e.g. north-south direction

ds.box_range(x=1, y=1)

Calculate the grid box range for all variables. This is performed for each time step.

Parameters

x
Number of boxes in the x, e.g. east-west direction
y
Number of boxes in the y, e.g. north-south direction

ds.mask_box(lon=[-180, 180], lat=[-90, 90])

Mask a lon/lat box

Parameters

lon
Longitude range to mask. Must be of the form: [lon_min, lon_max]
lat
Latitude range to mask. Must be of the form: [lat_min, lat_max]

ds.match_points(df=None, variables=None, depths=None, tmean=False, top=False, nan=None, regrid='bil', max_extrap=5, quiet=False, **kwargs)

Match dataset to a spatiotemporal points dataframe

Parameters

Required:

dfrequired
The column names must be made up of a subset of "lon", "lat", "year", "month", "day" and "depth". Pressure (in dbars), named "pressure", can also be used instead of "depth", which will require the optional dependency seawater to be installed.

Optional:

variables
Str or list of variables. All variables are matched up if this is not supplied. This can include variables generated by assign using kwargs.
depths
If each cell has different vertical levels, this must be provided as a dataset. If each cell has the same vertical levels, provide it as a list. If this is not supplied nctoolkit will try to figure out what they are. Only required if carrying out vertical matchups.
tmean
Set to True or False, depending on whether you want temporal averaging at the temporal resolution given by df. For example, if you only had months in df, but had daily data in ds, you might want to calculate a daily average in the monthly dataset. This is equivalent to apply `ds.tmean(..)` to the dataset.
top
Set to True if you want only the top/surface level of the dataset to be selected for matching.
nan
Value or range of values to set to nan. Defaults to 0. Only required if values in dataset need changed to missing
regrid
Regridding method. Defaults to "bil". Options available are those in nctoolkit regrid method. "nn" for nearest neighbour.
max_extrap
Maximum distance for extrapolation. Defaults to 5.
quiet
Set to True to suppress output
kwargs
Additional arguments to send to assign

ds.phenology(var=None, metric=None, p=None)

Calculate phenologies from a dataset. Each file in an ensemble must only cover a single year, and ideally have all days. The method assumes datasets have daily resolution.

Parameters

Required:

varrequired
Variable to analyze.
metricrequired
Must be peak, middle, start or end. Peak is defined as the day of the maximum value. Middle is the day when the cumulative total of the variable first exceeds the cumulative total for the entire year. Start or end is defined as the first day when the cumulative total exceeds a percentile p of the maximum cumulative total.

Optional:

pfor metric='start'/'end'
Percentile to use for start or end.

nc.merge(*datasets, match=['day', 'year', 'month'])

Merge datasets

Parameters

datasets
Datasets to merge.
match
Temporal matching criteria. This is a list which must be made up of a subset of day, year, month. This checks that the datasets have compatible times. For example, if you want to ensure the datasets have the same years, then use match = ["year"].

ds.append(x=None)

Add new file(s) to a dataset.

Parameters

xrequired
File path(s) to add to the dataset

ds.remove(x=None)

Remove file(s) from a dataset

Parameters

xrequired
File path(s) to remove from a dataset

ds.ensemble_mean(nco=False, ignore_time=False)

Calculate an ensemble mean. This operates on a grid cell by grid cell basis.

Parameters

nco
Do you want to use NCO for the calculation? Default is False, i.e. CDO is used. Modify default if run time is an issue.
ignore_time
If True the mean is calculated over all time steps. If False, the ensemble mean is calculated for each time steps; for example, if the ensemble is made up of monthly files the mean for each month will be calculated.

ds.ensemble_min(nco=False, ignore_time=False)

Calculate an ensemble minimum. This operates on a grid cell by grid cell basis.

Parameters

nco
Do you want to use NCO for the calculation? Default is False, i.e. CDO is used. Modify default if run time is an issue.
ignore_time
If True the min is calculated over all time steps. If False, the ensemble min is calculated for each time steps; for example, if the ensemble is made up of monthly files the min for each month will be calculated.

ds.ensemble_max(nco=False, ignore_time=False)

Calculate an ensemble maximum. This operates on a grid cell by grid cell basis.

Parameters

nco
Do you want to use NCO for the calculation? Default is False, i.e. CDO is used. Modify default if run time is an issue.
ignore_time
If True the max is calculated over all time steps. If False, the ensemble max is calculated for each time steps; for example, if the ensemble is made up of monthly files the max for each month will be calculated.

ds.ensemble_percentile(p=None)

Calculate an ensemble percentile. This will calculate the percentiles for each time step in the files. For example, if you had an ensemble of files where each file included 12 months of data, it would calculate the percentile for each month. This operates on a grid cell by grid cell basis.

Parameters

prequired
percentile to calculate. 0<=p<=100.

ds.ensemble_range()

Calculate an ensemble range. The range is calculated for each time step; for example, if each file in the ensemble has 12 months of data the statistic will be calculated for each month. This operates on a grid cell by grid cell basis.

ds.ensemble_stdev()

Calculate an ensemble standard deviation. The ensemble standard deviation is calculated for each time steps; for example, if the ensemble is made up of monthly files the standard deviation for each month will be calculated. This operates on a grid cell by grid cell basis.

ds.ensemble_sum()

Calculate an ensemble sum. The sum is calculated for each time step; for example, if each file in the ensemble has 12 months of data the statistic will be calculated for each month. This operates on a grid cell by grid cell basis.

ds.ensemble_var()

Calculate an ensemble variance. The ensemble variance is calculated for each time steps; for example, if the ensemble is made up of monthly files the standard deviation for each month will be calculated. This operates on a grid cell by grid cell basis.

ds.merge(join='variables', match=['year', 'month', 'day'], check=True)

Merge a multi-file ensemble into a single file. 2 methods are available. 1) merging files with different variables, but the same time steps. 2) merging files with the same variables, with different times.

Parameters

join
This defines the type of merging to carry out. "variables": this will merge by variable, so that an ensemble with different variables, but the same number of time steps is merged to a single file. "time": this will merge files with the same variables, but different times to a single file, into a single file with ordered times. join defaults to "variables", and uses partial matches, so "var" will give variable based merging.
match
Optional argument when join = 'variables'. A list or str stating what must match in the netCDF files. Defaults to year/month/day. This list must be some combination of year/month/day. An error will be thrown if the elements of time in match do not match across all netCDF files. The only exception is if there is a single date file in the ensemble.
check
By default nctoolkit out checks in case files do not have the same variables etc. Set check to False if you are confident merging will be problem free. If you are unsure if files have the same variables, set check to True to find out. Note: if you do not explicitly provide check and there are more than 30 files in a dataset, checks will be turned off.

ds.split(by=None)

Split the dataset. Each file in the ensemble will be separated into new files based on the splitting argument.

Parameters

byrequired
Available by arguments are 'year', 'month', 'yearmonth', 'season', 'day' 'name', "timestep". year will split files by year, month will split files by month, yearmonth will split files by year and month; season will split files by year, day will split files by day. Using "timestep" will split files by timestep. 'name' will split by variable name

ds.abs()

Method to get the absolute value of variables

ds.add(x=None, var=None)

Add to a dataset. This will add a constant, another dataset or a netCDF file to the dataset. nctoolkit will automatically determine the appropriate comparison required.

Parameters

Required:

xrequired
An int, float, single file dataset or netCDF file to add to the dataset. If a dataset or netCDF file is supplied, this must have only one variable, unless var is provided. The grids must be the same.

Optional:

var
A variable in the x to use for the operation

ds.assign(drop=False, **kwargs)

Create new variables using mathematical operations on existing variables. Existing columns that are re-assigned will be overwritten. This method operators in a similar fashion to the pandas assign method.

Parameters

drop
Set to True if you want existing variables to be removed once the new ones have been created. Defaults to False.
kwargs
New variable names are keywords. All terms in the equation given by the lambda function should evaluate to a numeric. New variables are calculated for each grid cell and time step.

ds.exp()

Method to get the exponential of variables

ds.log()

Method to get the natural log, ln, of variables

ds.log10()

Method to get the base 10 log, log10, of variables

ds.multiply(x=None, var=None)

Multiply a dataset. This will multiply a dataset by a constant, another dataset or a netCDF file.

Parameters

Required:

xrequired
An int, float, single file dataset or netCDF file to multiply the dataset by. If multiplying by a dataset or single file there must only be a single variable in it, unless var is supplied. The grids must be the same.

Optional:

var
A variable in the x to multiply the dataset by

ds.power(x=None)

Powers of variables in dataset

Parameters

xrequired
An int or float to take the variables to the power of

ds.sqrt()

Method to get the square root of variables

ds.square()

Method to get the square of variables

ds.subtract(x=None, var=None)

Subtract from a dataset. This will subtract a constant, another dataset or a netCDF file from the dataset.

Parameters

Required:

xrequired
An int, float, single file dataset or netCDF file to subtract from the dataset. If a dataset or netCDF is supplied this must only have one variable, unless var is provided. The grids must be the same.

Optional:

var
A variable in the x to use for the operation

ds.divide(x=None, var=None)

Divide the data. This will divide the dataset by a constant, another dataset or a netCDF file.

Parameters

Required:

xrequired
An int, float, single file dataset or netCDF file to divide the dataset by. If a dataset or netCDF file is supplied, this must have only one variable, unless var is provided. The grids must be the same.

Optional:

var
A variable in the x to use for the operation

Function	Description	Example
`abs`	Absolute value	`abs(x.sst)`
`ceiling` / `floor`	Ceiling / floor of a variable	`ceiling(x.sst - 1)`
`sqrt` / `exp` / `log` / `log10`	Square root, exponential, natural log, base-10 log	`log10(x.sst + 1)`
`sin` / `cos` / `tan`	Trigonometric functions	`cos(x.var)`
`isnan`	Is the value missing?	`isnan(x.var)`
`cell_area`	Grid-cell area (m²)	`cell_area(x.var)`
`latitude` / `longitude`	Coordinates of the grid cell	`latitude(x.var)`
`level`	Vertical level of the variable	`level(x.var)`
`year` / `month` / `day` / `hour`	Date/time components of the variable	`month(x.var)`
`timestep`	Time step index (Python indexing)	`timestep(x.var)`
`spatial_mean` / `spatial_max` / `spatial_min` / `spatial_sum`	Spatial statistic at each time-step	`spatial_mean(x.var)`
`zonal_mean` / `zonal_max` / `zonal_min` / `zonal_sum`	Zonal statistic at each time-step	`zonal_mean(x.var)`

ds.rename(newnames=None, **kwargs)

Rename variables in a dataset

Parameters

newnames
Dictionary with key-value pairs being original and new variable names * kwargs. Alternative method for renaming

ds.as_missing(value=None)

Change a range or individual value to missing.

Parameters

valuerequired
If int/float is provided, the missing value will be set to that. If a list is provided, values between the two values (inclusive) of the list are set to missing.

ds.missing_as(value=None)

Convert missing values to a constant

Parameters

valuerequired
If int/float is provided, the missing value will be converted to that.

ds.set_fill(value=None)

Set the fill value

Parameters

valuerequired
The fill value to set.

ds.sum_all(drop=True, new_name=None)

Calculate the sum of all variables for each time step

Parameters

drop
Do you want to keep variables?
new_name
If you want to name the output of sum_all to a specific name

ds.set_longnames(name_dict=None, **kwargs)

Set the long names of variables

Parameters

name_dict
Dictionary with key, value pairs representing the variable names and their long names * kwargs. Alternative method for setting units

ds.set_units(unit_dict=None, **kwargs)

Set the units for variables

Parameters

unit_dict
A dictionary where the key-value pairs are the variables and new units respectively. * kwargs. Alternative method for setting units using direct assignment

ds.plot(vars=None, autoscale=True, out=None, coast=None, **kwargs)

Automatically plot a dataset.

Parameters

vars
A string or list of the variables to plot
autoscale
Set to True if you want the colorbar to be scaled to the min/max of the data. Default is True
out
Name of output file if you want to save as html. Defaults to None.
coast
Set to True if you want a coastline to show up on spatial map. Default is True if a working version of cartopy is available. It is False otherwise.
kwargs
Optional args to be sent to hvplot

ds.pub_plot(var=None, extent=None, title=None, legend=None, size='auto', land='auto', colours='auto', norm=None, limits=None, projection='auto', coast='auto', scale='auto', grid=True, grid_colour='auto', legend_position='auto', robust=False, out=None, breaks=None, dpi='figure', font=None, **kwargs)

Static plotting. This requires datasets to have regular latlon grids. Plots a static map, and requires only one variable, time step and vertical level

Parameters

Required:

varrequired
Variable to plot

Optional:

ds
Dataset to plot
extent
List with [lon_min, lon_max, lat_min, lat_max] for plotting extent
title
Character string with plot title
legend
Character string with legend title
size
List with [xsize, ysize] for plotting size
land
Character string with colour required for land. Set to None if you do not want land to show.
colours
Character string with colour map to use. Set to None if you do not want to use a colour map.
norm
Norm to use for colour bar
limits
List with [min, max] for colour bar limits. Please note that if the colour scale passes through zero, the colour scale will be symmetrical around zero. So the limits will be reset to the maximum absolute value of the data.
projection
Cartopy projection to use.
coast
Set to "coarse", "low", "intermediate", "high" or "full" if you want to use GSHHS coastlines
scale
"low", "medium" or "high"
grid
Set to False if you do not want grid lines.
legend_position
= "auto"
robust
Whether to use robust statistics for the colour scale or not
out
Output file name
breaks
List of breaks for the colour bar
dpi
DPI for output file. Default is "figure".
kwargs
kwargs to allow slight misspelling of arguments -------------

ds.to_nc(out, zip=True, overwrite=False, **kwargs)

Save a dataset to a named file. This will only work with single file datasets.

Parameters

Required:

outrequired
Output file name.

Optional:

zip
True/False depending on whether you want to zip the file. Default is True.
overwrite
If out file exists, do you want to overwrite it? Default is False.
kwargs
Optional arguments to be sent to subset.

ds.to_xarray(decode_times=True, **kwargs)

Open a dataset as an xarray object

Parameters

decode_times
Set to False if you do not want xarray to decode the times. Default is True. If xarray cannot decode times, CDO will be used.
kwargs
Optional arguments to be sent to subset.

ds.to_dataframe(decode_times=True, drop_bnds=True, **kwargs)

Convert a dataset to a pandas data frame

Parameters

decode_times
Set to False if you do not want xarray to decode the times prior to conversion to data frame. Default is True.
drop_bnds
Set to False if you do not want to drop the bounds from the data frame. Variables/coordinates with '_bnds' in their names will be dropped. Default is True.
kwargs
Optional arguments to be sent to subset.

ds.zip()

Zip the dataset. This will compress the files within the dataset. This will occur lazily, so will only occur after everything has been evaluated.

ds.format(ext=None)

Change the netCDF format of a dataset. This will compress the files within the dataset. This works lazily.

Parameters

extrequired
New format. Must be one of "nc", "nc1", "nc2", "nc4" and "nc5" . netCDF = nc1 netCDF version 2 (64-bit offset) = nc2/nc netCDF4 (HDF5) = nc4 netCDF4-classi = nc4c netCDF version 5 (64-bit data) = nc5

ds.run()

Run all stored commands in a dataset

nc.cleanup()

Temp file cleaner. Remove all files created during the session that are now out of use

nc.deep_clean()

Deep temp file cleaner. Remove all temporary files ever created by nctoolkit across all previous and current sesions

ds.na_count(over='time', align='right', window=None)

Calculate the number of missing values.

Parameters

over
Time periods to to the count over over. Options are 'time', 'year', 'month', 'day'.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
window
This determines the number of time steps to calculate, on a non-rolling basis. This is useful if you need to calculate the sum every 5 days, for example.

ds.na_frac(over='time', align='right', window=None)

Calculate the fraction of missing values in each grid cell across all time steps.

Parameters

over
Time periods to to the count over over. Options are 'time', 'year', 'month', 'day'.
align
This determines whether the output time is at the left, centre or right hand side of the time window. Options are "left", "centre" and "right"
window
This determines the number of time steps to calculate over, on a non-rolling basis. This is useful if you need to calculate the fraction for every non-overlapping 5 day periods, for example.

ds.distribute(m=1, n=1)

Split the dataset into multiple evenly sized horizontal and vertical new files

Parameters

m
Number of rows
n
Number of columns

ds.collect()

Collect a dataset that has been split using distribute

ds.cell_area(join=True)

Calculate the area of grid cells. Area of grid cells is given in square meters.

Parameters

join
Set to False if you only want the cell areas to be in the output. join=True adds the areas as a variable to the dataset. Defaults to True.

ds.first_above(x=None)

Identify the time step when a value is first above a threshold. This will do the comparison with either a number, a Dataset or a netCDF file.

Parameters

xrequired
An int, float, single file dataset or netCDF file to use for the threshold(s). If comparing with a dataset or single file there must only be a single variable in it. The grids must be the same.

ds.first_below(x=None)

Identify the time step when a value is first below a threshold. This will do the comparison with either a number, a Dataset or a netCDF file.

Parameters

xrequired
An int, float, single file dataset or netCDF file to use for the threshold(s). If comparing with a dataset or single file there must only be a single variable in it. The grids must be the same.

ds.last_above(x=None)

Identify the final time step when a value is above a threshold. This will do the comparison with either a number, a Dataset or a netCDF file.

Parameters

xrequired
An int, float, single file dataset or netCDF file to use for the threshold(s). If comparing with a dataset or single file there must only be a single variable in it. The grids must be the same.

ds.last_below(x=None)

Identify the last time step when a value is below a threshold. This will do the comparison with either a number, a Dataset or a netCDF file.

Parameters

xrequired
An int, float, single file dataset or netCDF file to use for the threshold(s). If comparing with a dataset or single file there must only be a single variable in it. The grids must be the same.

ds.cdo_command(command=None, ensemble=False, check=False)

Apply a cdo command

Parameters

Required:

commandrequired
cdo command to call. This command must be such that "cdo {command} infile outfile" will run.

Optional:

ensemble
Is this an ensemble method command? For example ensmean, mergetime, etc.
check
Check whether the command is valid

ds.nco_command(command=None, ensemble=False)

Apply an nco command

Parameters

Required:

commandrequired
nco command to call. This must be of a form such that "nco {command} infile outfile" will run.

Optional:

ensemble
Set to True if you want the command to take all of the files as input. This is useful for ensemble methods.

ds.compare(expression=None)

Compare all variables to a constant

Parameters

expressionrequired
This a regular comparison such as "<0", ">0", "==0"

ds.gt(x)

Method to calculate if variable in dataset is greater than that in another file or dataset. This currently only works with single file datasets

Parameters

xrequired
File path or nctoolkit dataset

ds.lt(x)

Method to calculate if variable in dataset is less than that in another file or dataset. This currently only works with single file datasets

Parameters

xrequired
File path or nctoolkit dataset

ds.reduce_dims()

Reduce dimensions of data. This will remove any dimensions with only one value. For example, if only selecting one vertical level, the vertical dimension will be removed.

ds.reduce_grid(mask=None)

Reduce the dataset to non-zero locations in a mask

Parameters

maskrequired
single variable dataset or path to .nc file. The mask must have an identical grid to the dataset.

ds.set_precision(x)

Set the precision in a dataset

Parameters

xrequired
The precision. One of 'I8', 'I16', 'I32', 'F32', 'F64'.

ds.check()

Check contents of files for common data problems.

ds.is_corrupt()

Check if files are corrupt

ds.fix_nemo_ersem_grid()

A quick hack to change the grid file in North West European shelf Nemo grids.

ds.set_gridtype(grid)

Set the grid type. Only use this if, for example, the grid is "generic" when it should be lonlat.

Parameters

gridrequired
Grid type. Needs to be one of "curvilinear", "unstructured", "dereference", "regular", "regularnn" or "lonlat".

ds.surface_mask()

Create a mask identifying the shallowest cell without missing values. This converts a dataset to a mask identifying which cell represents top level, for example the sea surface. 1 identifies the shallowest cell with non-missing values. Everything else is 0, or missing. At present this method only uses the first available variable from netCDF files, so it may not be suitable for all data

ds.strip_variables(vars=None)

Remove any variables, such as bnds etc., from variables. This should probably only be done at the end of a processing chain before converting to a dataframe etc., as it is stripping away critical info for netCDF operations.

Parameters

vars
individual or list of variables to select and strip. All variables will be stripped if this is not defined.

ds.no_leaps()

Remove leap years. This uses an undocumented CDO feature to remove Feb 29 and sets the calendar to leap year free

ds.as_double(x)

Set a variable/dimension to double. This is mostly useful for cases when time is stored as an int, but you need a double

Parameters

xrequired
A list of variable/dimensions you want to convert to floats

ds.as_type(x)

Cast one or more variables/dimensions to a specific numeric type.

Parameters

xrequired
A dictionary mapping variables to type. Values in dict must be one of 'int', 'float32' and 'float64'.

ds.reset()

Simple method to fully reset a datset