Functionality¶

The cf package implements the CF data model for its internal data structures and so is able to process any CF-compliant dataset. It is not strict about CF-compliance, however, so that partially conformant datasets may be ingested from existing datasets and written to new datasets.This is so that datasets that are partially conformant may nonetheless be modified in memory.

>>> import cf
>>> f = cf.read('file.nc')
>>> f
[<CF Field: air_temperature(time(12), latitude(64), longitude(128)) K>]
>>> print(f[0])
Field: air_temperature (ncvar%tas)
----------------------------------
Data            : air_temperature(time(12), latitude(64), longitude(128)) K
Cell methods    : time(12): mean (interval: 1.0 month)
Dimension coords: time(12) = [1991-11-16 00:00:00, ..., 1991-10-16 12:00:00] noleap
                : latitude(64) = [-87.8638, ..., 87.8638] degrees_north
                : longitude(128) = [0.0, ..., 357.1875] degrees_east
                : height(1) = [2.0] m

The cf package can:

• read field constructs and domain constructs from netCDF, CDL, Zarr, Kerchunk, PP and UM datasets with a choice of netCDF backends, including from local, HTTP, OPeNDAP and S3 locations,

• be fully flexible with respect to dataset storage chunking,

• create new field and domain constructs in memory,

• write and append field and domain constructs to netCDF and Zarr v3 datasets on disk, with control over HDF5 internal file metadata,

• read, write, and manipulate UGRID mesh topologies,

• read, write, and manipulate HEALPix grids,

• read, write, and create coordinates defined by geometry cells,

• read and write netCDF4 string data-type variables,

• read, write, and create netCDF and CDL datasets containing hierarchical groups,

• read, write, and create data that have been compressed by convention (i.e. ragged or gathered arrays, or coordinate arrays compressed by subsampling), whilst presenting a view of the data in its uncompressed form,

• read and write data that are quantized to eliminate false precision,

• convert field and domain constructs to xarray datasets in memory,

• inspect field constructs,

• test whether two field constructs are the same,

• modify field construct metadata and data,

• create subspaces of field constructs,

• incorporate, and create, metadata stored in external files,

• combine field constructs arithmetically,

• manipulate field construct data by arithmetical and trigonometrical operations,

• perform weighted statistical collapses on field constructs, including those with geometry cells, UGRID mesh topologies, and HEALPix grids,

• perform histogram, percentile and binning operations on field constructs,

• regrid structured grid, UGRID, HEALPix, and DSG field constructs with (multi-)linear, nearest neighbour, first- and second-order conservative and higher order patch recovery methods, including 3-d regridding, and large-grid support,

• apply convolution filters to field constructs,

• create running means from field constructs,

• apply differential operators to field constructs, and

• create derived quantities (such as relative vorticity).

Visualisation¶

Powerful, flexible, and user-friendly visualisations of field constructs are available with the cf-plot package that is installed separately to cf (see the cf-plot documentation for details).

See the cf-plot gallery for the wide range of plotting possibilities with example code.

Training¶

Training material on cf-python and cf-plot i.e. the CF Data Tools, in the form of Jupyter Notebooks, is openly-accessible and available (with instructions for set-up provided) at: https://github.com/NCAS-CMS/cf-tools-training.

Performance¶

The cf package uses Dask for all of its data manipulations, which provides lazy, parallelised, and out-of-core computations of array operations.

Command line utilities¶

During installation the cfa command line utility is also installed, which

• generates text descriptions of field constructs contained in files, and

• creates new datasets aggregated from existing files.

CF data model¶

The CF (Climate and Forecast) metadata conventions (http://cfconventions.org) provide a description of the physical meaning of data and of their spatial and temporal properties and are designed to promote the creation, processing, and sharing of climate and forecasting data using netCDF files and libraries (https://www.unidata.ucar.edu/software/netcdf).

The CF data model identifies the fundamental elements (“constructs”) of the CF conventions and shows how they relate to each other, independently of the netCDF encoding.

The CF data model defines a field construct for storing data with all of its metadata. It is defined in CF-1.13 as follows:

field construct¶

corresponds to a CF-netCDF data variable with all of its metadata. It consists of

• descriptive properties that apply to field construct as a whole (e.g. the standard name),

• a data array,

• a domain construct that describes the locations of each cell of the data array (i.e. the “domain”),

• metadata constructs that describe the physical nature of the data array, defined by

  field ancillary constructs¶

  corresponding to CF-netCDF ancillary variables

  cell method constructs¶

  corresponding to a CF-netCDF cell_methods attribute of data variable

domain construct¶

that describes the locations of each cell of the domain. It may exist independently of a field construct and consists of

• descriptive properties that apply to domain construct as a whole,

• metadata constructs that describe the locations of each cell of the domain, defined by

domain axis constructs¶

corresponding to CF-netCDF dimensions or scalar coordinate variables

dimension coordinate constructs¶

corresponding to CF-netCDF coordinate variables or numeric scalar coordinate variables

auxiliary coordinate constructs¶

corresponding to CF-netCDF auxiliary coordinate variables and non-numeric scalar coordinate variables

coordinate reference constructs¶

corresponding to CF-netCDF grid mapping variables or the formula_terms attribute of a coordinate variable

domain ancillary constructs¶

corresponding to CF-netCDF variables named by the formula_terms attribute of a coordinate variable

cell measure constructs¶

corresponding to CF-netCDF cell measure variables

domain topology constructs¶

corresponding to CF-netCDF UGRID mesh topology variables

cell connectivity constructs¶

corresponding to CF-netCDF UGRID connectivity variables

References¶

Eaton, B., Gregory, J., Drach, B., Taylor, K., Hankin, S., Caron, J.,

Signell, R., et al. NetCDF Climate and Forecast (CF) Metadata Conventions. CF Conventions Committee. https://cfconventions.org/cf-conventions/cf-conventions.html

Hassell, D., and Bartholomew, S. L. (2020). cfdm: A Python reference

implementation of the CF data model. Journal of Open Source Software, 5(54), 2717, https://doi.org/10.21105/joss.02717

Hassell, D., Gregory, J., Blower, J., Lawrence, B. N., and

Taylor, K. E. (2017). A data model of the Climate and Forecast metadata conventions (CF-1.6) with a software implementation (cf-python v2.1), Geosci. Model Dev., 10, 4619-4646, https://doi.org/10.5194/gmd-10-4619-2017

Rew, R., and Davis, G. (1990). NetCDF: An Interface for Scientific

Data Access. IEEE Computer Graphics and Applications, 10(4), 76–82. https://doi.org/10.1109/38.56302

Rew, R., Hartnett, E., and Caron, J. (2006). NetCDF-4: Software

Implementing an Enhanced Data Model for the Geosciences. In 22nd International Conference on Interactive Information Processing Systems for Meteorology, Oceanography, and Hydrology. AMS. Retrieved from https://www.unidata.ucar.edu/software/netcdf/papers/2006-ams.pdf