GeezerVison is showing ncas-cms.github.io — cf.read — Documentation
cf.read¶
-
cf.read(datasets, external=None, verbose=None, warnings=False, aggregate=True, nfields=None, squeeze=False, unsqueeze=False, dataset_type=None, cdl_string=False, select=None, extra=None, recursive=False, followlinks=False, um=None, chunk=True, field=None, height_at_top_of_model=None, select_options=None, follow_symlinks=False, mask=True, unpack=True, warn_valid=False, dask_chunks='storage-aligned', store_dataset_chunks=True, store_dataset_shards=True, domain=False, cfa=None, cfa_write=None, to_memory=None, backend=None, backend_options=None, storage_options=None, cache=True, chunks='auto', ignore_read_error=False, fmt=None, file_type=None, group_dimension_search='closest_ancestor', filesystem=None, legacy_um_backend=False, netcdf_backend=None)[source]¶
Read field or domain constructs from files.
The following file formats are supported: netCDF, CDL, Zarr,
Kerchunk, PP, and UM fields file.
NetCDF and Zarr datasets may be on local disk, on an OPeNDAP
server, or in an S3 object store.
CDL, PP, and UM fields files must be on local disk.
Any amount of files of any combination of file types may be read.
Input datasets are mapped to Field constructs which are returned
as elements of a FieldList, or if the domain parameter is
True, Domain constructs returned as elements of a
DomainList. The returned constructs are sorted by the netCDF
variable names of their corresponding data or domain variables.
NetCDF unlimited dimensions
Domain axis constructs that correspond to NetCDF unlimited
dimensions may be accessed with the
nc_is_unlimited and
nc_set_unlimited methods of a domain axis
construct.
NetCDF hierarchical groups
Hierarchical groups in CF provide a mechanism to structure
variables within netCDF4 datasets. Field constructs are
constructed from grouped datasets by applying the well defined
rules in the CF conventions for resolving references to
out-of-group netCDF variables and dimensions. The group structure
is preserved in the field construct’s netCDF interface. Groups
were incorporated into CF-1.8. For files with groups that state
compliance to earlier versions of the CF conventions, the groups
will be interpreted as per the latest release of CF.
CF-compliance
If the dataset is partially CF-compliant to the extent that it is
not possible to unambiguously map an element of the netCDF dataset
to an element of the CF data model, then a field construct is
still returned, but may be incomplete. This is so that datasets
which are partially conformant may nonetheless be modified in
memory and written to new datasets.
Such “structural” non-compliance would occur, for example, if the
“coordinates” attribute of a CF-netCDF data variable refers to
another variable that does not exist, or refers to a variable that
spans a netCDF dimension that does not apply to the data
variable. Other types of non-compliance are not checked, such
whether or not controlled vocabularies have been adhered to. The
structural compliance of the dataset may be checked with the
dataset_compliance method of the field construct, as
well as optionally displayed when the dataset is read by setting
the warnings parameter.
CDL files
A file is considered to be a CDL representation of a netCDF
dataset if it is a text file whose first non-comment line starts
with the seven characters “netcdf “ (six letters followed by a
space). A comment line is identified as one which starts with any
amount white space (including none) followed by “//” (two
slashes). It is converted to a temporary netCDF4 file using the
external ncgen command, and the temporary file persists until
the end of the Python session, at which time it is automatically
deleted. The CDL file may omit data array values (as would be the
case, for example, if the file was created with the -h or
-c option to ncdump), in which case the relevant
constructs in memory will be created with data with all missing
values.
PP and UM fields files
32-bit and 64-bit Met Office (UK) PP files and Met Office (UK)
fields files of any endian-ness can be read.
2-d “slices” within a single file are combined, where possible,
into field constructs with 3-d, 4-d or 5-d data. This is done
prior to any field construct aggregation (see the aggregate
parameter).
When reading PP and UM fields files, the relaxed_units aggregate
option is set to True by default, because units are not always
available to field constructs derived from UM fields files or PP
files.
Performance
Descriptive properties are always read into memory, but lazy
loading is employed for all data arrays which means that, in
general, data is not read into memory until the data is required
for inspection or to modify the array contents. This maximises the
number of field constructs that may be read within a session, and
makes the read operation fast. The exceptions to the lazy reading
of data arrays are:
Data that define purely structural elements of other data arrays
that are compressed by convention (such as a count variable for
a ragged contiguous array). These are always read from disk. If field or domain aggregation is in use (as it is by default,
see the aggregate parameter), then the data of metadata
constructs may have to be read to determine how the contents of
the input files may be aggregated. This won’t happen for a
particular field or domain’s metadata, though, if it can be
ascertained from descriptive properties alone that it can’t be
aggregated with anything else (as would be the case, for
instance, when a field has a unique standard name).
However, when two or more field or domain constructs are
aggregated to form a single construct then the data arrays of some
metadata constructs (coordinates, cell measures, etc.) must be
compared non-lazily to ascertain if aggregation is possible.
- Parameters:
- datasets:
The dataset, or sequence of datasets, from which to read
field or domain constructs.
Each dataset must be one of:
A string-like path name to the dataset (such as a str
or pathlib.Path instance).
Local names may be relative paths and will have tilde
and shell environment variable expansions applied to
them, followed by the replacement of any UNIX
wildcards (such as *, ?, [a-z], etc.) with
the list of matching names. Remote names (i.e. those
with an http or s3 schema), however, are not
transformed in any way.
- Example:
The local dataset file.nc in the user’s home
directory could be described by any of the
following: '$HOME/file.nc',
'${HOME}/file.nc', '~/file.nc',
'~/tmp/../file.nc'
- Example:
The local datasets file1.nc and file2.nc
could be described by any of the following:
['file1.nc', 'file2.nc'], 'file[12].nc'
A path to a directory will be walked through to find
its contents (recursively if recursive is True),
unless the directory defines a Zarr dataset (which is
ascertained by presence in the directory of
appropriate Zarr metadata files). If a file system has been defined by the filesystem
or storage_options parameters, then the path is
assumed to be in that file system. If a file system
has not been defined, then a remote dataset is
treated, in general, as the file-like object
fsspec.filesystem(protocol).open(dataset, 'rb'),
where protocol is inferred from the start of the
dataset name (e.g. s3`, ``http, etc.). There is a special case for remote s3 datasets for
which a file system has not been provided with the
filesystem parameter. In this case, an
endpoint_url file system storage option is created
(unless it has already been provided), and the
leading scheme and authority are automatically removed
from the dataset name; for instance, the dataset
's3://authority/bucket/file.nc' might be treated
as fsspec.filesystem('s3',
endpoint_url='s3://authority').open('bucket/file.nc',
'rb'). There is a special case for remote http and https
datasets for which a file system has not been provided
with the filesystem parameter. In this case, an
attempt will be made to access the dataset via OPeNDAP
if other methods do not work. If the cdl_string parameter is True, then the
string-like dataset is interpreted as an actual CDL
representation of a dataset, rather than a path name.
A file-like object that accesses the dataset (such as
io.BufferedReader or the result of an fsspec file
system open) A directory-like object that accesses the dataset (such
as fsspec.mapping.FSMap) Any of the following backend objects (see the backend
parameter) that accesses the dataset: pyfive.File,
zarr.Group, umfive.File, netCDF4.Dataset,
scipy.io.netcdf_file, h5py.File, xarray.Dataset,
and xarray.DataTree. An xnetcdf.Dataset object that accesses the dataset. Any object x that accesses the dataset and has the
same API as one of the backend objects, or as
xnetcdf.Dataset. In practice, this means any object
x for which isinstance(x, <backend-object>) is
True for any <backend-object> from the selection
of allowed backend objects. For instance, if you have
created a library called my_pyfive for which
my_pyfive.File is (registered as) a subclass of
pyfive.File, then my_pyfive.File instances can be
passed to Dataset.
- recursive:
bool, optional If True then recursively read sub-directories of any
directories specified with the datasets parameter.
- followlinks:
bool, optional If True, and recursive is True, then also search for
datasets in sub-directories which resolve to symbolic
links. By default directories which resolve to symbolic
links are ignored. Ignored of recursive is
False. Datasets which are symbolic links are always
followed.
Note that setting recursive=True, followlinks=True can
lead to infinite recursion if a symbolic link points to a
parent directory of itself.
- cdl_string:
bool, optional If True and the format to read is CDL then each string
given by the datasets parameter is interpreted as a
string of actual CDL rather than the name of a location
from which field or domain constructs can be read.
Note that when cdl_string is True, the fmt parameter
is ignored as the format is assumed to be CDL, so in this
case it is not necessary to also specify fmt='CDL'.
- dataset_type:
None or (sequence of) str, optional Only read datasets of the given type or types, ignoring
others. If there are no dataset of the given types, or
dataset_type is empty sequence, then an empty list is
returned. If None (the default) all datasets defined by
the dataset parameter are read, and an exception is
raised for any invalid dataset type.
Valid file types are:
dataset_type |
Description |
|---|
'netCDF'
|
A netCDF-3 or netCDF-4 dataset |
'CDL'
|
A text CDL file of a netCDF dataset |
'Zarr'
|
A Zarr v2 (xarray) or Zarr v3 dataset |
'Kerchunk'
|
A Kerchunked dataset |
'UM'
|
A UM fields file or PP dataset |
- external: (sequence of)
str, optional Read external variables (i.e. variables which are named by
attributes, but are not present, in the parent file given
by the filename parameter) from the given external
files. Ignored if the parent file does not contain a
global external_variables attribute. Multiple external
files may be provided, which are searched in random order
for the required external variables.
If an external variable is not found in any external
files, or is found in multiple external files, then the
relevant metadata construct is still created, but without
any metadata or data. In this case the construct’s
is_external method will return True.
- Parameter example:
external='cell_measure.nc'
- Parameter example:
external=['cell_measure.nc']
- Parameter example:
external=('cell_measure_A.nc', 'cell_measure_O.nc')
- extra: (sequence of)
str, optional Create extra, independent fields from netCDF variables
that correspond to particular types of metadata constructs.
Ignored if domain is True.
The extra parameter may be one, or a sequence, of:
extra |
Metadata constructs |
|---|
'field_ancillary'
|
Field ancillary constructs |
'domain_ancillary'
|
Domain ancillary constructs |
'dimension_coordinate'
|
Dimension coordinate constructs |
'auxiliary_coordinate'
|
Auxiliary coordinate constructs |
'cell_measure'
|
Cell measure constructs |
- Parameter example:
To create fields from auxiliary coordinate constructs:
extra='auxiliary_coordinate' or
extra=['auxiliary_coordinate'].
- Parameter example:
To create fields from domain ancillary and cell measure
constructs: extra=['domain_ancillary',
'cell_measure'].
An extra field construct created via the extra parameter
will have a domain limited to that which can be inferred
from the corresponding netCDF variable, but without the
connections that are defined by the parent netCDF data
variable. It is possible to create independent fields from
metadata constructs that do incorporate as much of the
parent field construct’s domain as possible by using the
convert method of a returned field
construct, instead of setting the extra parameter.
- verbose:
int or str or None, optional If an integer from -1 to 3, or an equivalent string
equal ignoring case to one of:
'DISABLE' (0)
'WARNING' (1)
'INFO' (2)
'DETAIL' (3)
'DEBUG' (-1)
set for the duration of the method call only as the
minimum cut-off for the verboseness level of displayed
output (log) messages, regardless of the
globally-configured cf.log_level. Note that
increasing numerical value corresponds to increasing
verbosity, with the exception of -1 as a special case
of maximal and extreme verbosity.
Otherwise, if None (the default value), output messages
will be shown according to the value of the
cf.log_level setting.
Overall, the higher a non-negative integer or equivalent string
that is set (up to a maximum of 3/'DETAIL') for
increasing verbosity, the more description that is printed to
convey how the contents of the netCDF file were parsed and
mapped to CF data model constructs.
- warnings:
bool, optional If True then print warnings when an output field construct
is incomplete due to structural non-compliance of the
dataset. By default such warnings are not displayed.
- um:
dict or None, optional For Met Office (UK) PP files and Met Office (UK) fields
files only, provide extra decoding instructions. This
option is ignored for input files which are not PP or
fields files. In most cases, how to decode a file is
inferrable from the file’s contents via the lookup
headers, but when that is not possible then any of the
following key/value pairs in the dictionary may be used to
guide the decoding:
'um_version': str
The UM version to be used when decoding the
header. Valid versions are, for example, 4.2,
'6.6.3' and '8.2'. In general, a given version
is ignored if it can be inferred from the header (which
is usually the case for files created by the UM at
versions 5.3 and later). The exception to this is when
the given version has a third element (such as the 3 in
6.6.3), in which case any version in the header is
ignored. The default version is 4.5.
'height_at_top_of_model': number
The height in metres of the upper boundary of the top
model level of the atmosphere (TOA). By default the TOA
is taken from the top level’s upper bound defined by
BRSVD1 in the lookup header. If the height can’t be
determined from the header, or the given height is less
than or equal to 0, then a coordinate reference system
will still be created that contains the ‘a’ and ‘b’
formula term values, but without an atmosphere hybrid
height dimension coordinate construct.
Note
A current limitation is that if pseudolevels
and atmosphere hybrid height coordinates are
defined by the same lookup headers then the
height can’t be determined
automatically. In this case the height may
be found after reading as the maximum value of
the bounds of the domain ancillary construct
containing the ‘a’ formula term. The file can
then be re-read with this height as a um
parameter.
- Example:
To specify that the input files are from version 6.6.3 of
the UM: um_config={'um_version': '6.6.3'}
- Example:
To specify that the input files are from version 6.6.3 of
the UM, and the height at top of the model is 85000
metres: um_config={'height_at_top_of_model': 85000,
'um_version': '6.6.3'}
- aggregate:
bool or dict, optional If True (the default) or a dictionary (possibly empty)
then aggregate the field constructs read in from all input
files into as few field constructs as possible by passing
all of the field constructs found the input files to the
cf.aggregate, and returning the output of this function
call.
If aggregate is a dictionary then it is used to
configure the aggregation process passing its contents as
keyword arguments to the cf.aggregate function.
If aggregate is False then the field constructs are not
aggregated.
- squeeze:
bool, optional If True then remove all size 1 dimensions from field
construct data arrays, regardless of how the data are
stored in the dataset. If False (the default) then the
presence or not of size 1 dimensions is determined by how
the data are stored in its dataset.
- unsqueeze:
bool, optional If True then ensure that field construct data arrays span
all of the size 1 dimensions, regardless of how the data
are stored in the dataset. If False (the default) then the
presence or not of size 1 dimensions is determined by how
the data are stored in its dataset.
- select: (sequence of)
str or Query or re.Pattern, optional Only return field constructs whose identities match the
given values(s), i.e. those fields f for which
f.match_by_identity(*select) is True. See
cf.Field.match_by_identity for details.
This is equivalent to, but faster than, not using the
select parameter but applying its value to the returned
field list with its cf.FieldList.select_by_identity
method. For example, fl = cf.read(file,
select='air_temperature') is equivalent to fl =
cf.read(file).select_by_identity('air_temperature').
- warn_valid:
bool, optional If True then print a warning for the presence of
valid_min, valid_max or valid_range properties
on field constructs and metadata constructs that have
data. By default no such warning is issued.
“Out-of-range” data values in the file, as defined by any
of these properties, are automatically masked by default,
which may not be as intended. See the mask parameter for
turning off all automatic masking.
- mask:
bool, optional If True (the default) then mask by convention the data of
field and metadata constructs.
A netCDF array is masked depending on the values of any of
the netCDF attributes _FillValue, missing_value,
_Unsigned, valid_min, valid_max, and
valid_range.
- unpack:
bool If True, the default, then unpack arrays by convention
when the data is read from disk.
Unpacking is determined by netCDF conventions for the
following variable attributes: add_offset,
scale_factor, and _Unsigned.
- domain:
bool, optional If True then return only the domain constructs that are
explicitly defined by CF-netCDF domain variables, ignoring
all CF-netCDF data variables. By default only the field
constructs defined by CF-netCDF data variables are
returned.
CF-netCDF domain variables are only defined from CF-1.9,
so older datasets automatically contain no CF-netCDF
domain variables.
The unique domain constructs of the dataset are
found with the cf.unique_constructs
function. For example:
>>> d = cf.read('file.nc', domain=True)
>>> ud = cf.unique_constructs(d)
>>> f = cf.read('file.nc')
>>> ufd = cf.unique_constructs(x.domain for x in f)
- backend:
None or (sequence of) str, optional Which library to use for opening a string-like, file-like,
or directory-like dataset. An attempt to read the dataset
is made by the given backends in the order in which they
are provided, stopping after the first successful read.
By default backend is None, which is equivalent to
providing the ordered sequence of backends:
('pyfive', 'zarr', 'umfive', 'netCDF4', 'netcdf_file',
'h5py', 'xarray')
If the dataset is given as a (subclass of a) backend
object, then that backend must be one of the backends
identified by the backend parameter.
Performance may be improved by specifying a backend
library, as this reduces or removes unsuccessful dataset
read attempts, which can be expensive, especially for
remote datasets.
The available backends, and the formats they can read,
are:
Backend |
Library |
Dataset formats |
|---|
'pyfive'
|
pyfive
|
netCDF-4 |
'zarr'
|
zarr
|
Zarr, Kerchunk |
'umfive'
|
umfive
|
PP, fields file |
'netCDF4'
|
netCDF4
|
netCDF-4, netCDF-3 |
'netcdf_file'
|
scipy.io.netcdf_file
|
netCDF-3 |
'h5py'
|
h5py
|
netCDF-4 |
'xarray'
|
xarray
|
netCDF-4, netCDF-3,
Zarr, Kerchunk,
GRIB |
Note that the xarray library is itself an interface to
other backends.
- Example:
To only attempt 'netCDF4': 'netCDF4' or
['netCDF4']
- Example:
To only attempt 'netCDF4' or 'pyfive', in that
order: ('netCDF4', 'pyfive')
- backend_options:
None or dict, optional The options to use with each backend when opening a
dataset.
Each backend’s options may be defined with the following
dictionary keys, which take a value of a dictionary of
keyword arguments to the appropriate backend library
object. Each backend has default options, which are used
when the corresponding key has not been set.
- Example:
{'pyfive_options': {'metadata_buffer_size': 2}}
- Example:
{'pyfive_options': {'metadata_buffer_size': 2},
'umfive_options': {'height_at_top_of_model': 85000,
'local_os_cache': False}}
- storage_options:
dict or None, optional Parameters for defining an fsspec filesystem for
accessing a dataset.
If None (the default) and filesystem is also None,
then a dataset is treated as described by the datasets
parameter.
If set to a dictionary of storage options then a dataset
is treated, in general, as the file-like object
fsspec.filesystem(protocol,
**storage_options).open(dataset, 'rb'), where
protocol is inferred from the start of the dataset
name (e.g. s3, http, etc.). S3 is a special case
for which endpoint_url is added to the storage
options, and the leading scheme and authority is
automatically removed from the dataset name passed to the
filesystem open method; for instance, the dataset
's3://authority/bucket/file.nc' is treated as
fsspec.filesystem('s3', endpoint_url='s3://authority',
**storage_options).open('bucket/file.nc').
Added in version (cfdm): 1.13.1.0
- cache:
bool, optional If True, the default, then cache the first and last array
elements of metadata constructs (not field constructs) for
fast future access. In addition, the second and
penultimate array elements will be cached from coordinate
bounds when there are two bounds per cell. For remote
data, setting cache to False may speed up the parsing of
the file.
- dask_chunks:
str, int, None, or dict, optional Specify the Dask chunking for data. May be one of the
following:
'storage-aligned'
This is the default. The Dask chunk size in bytes will
be as close as possible the size given by
cf.chunksize, favouring square-like chunk
shapes, with the added guarantee that the entirety of
each storage chunk lies within exactly one Dask
chunk. This strategy is general the most performant, as
it ensures that when accessing the data, each storage
chunk is read from disk at most once (as opposed to once
per Dask chunk in which it lies).
For instance, consider a file variable that has an array
of 64-bit floats with shape (400, 300, 60) and a storage
chunk shape of (100, 5, 60). This has an overall size of
54.93 MiB, partitioned into 240 storage chunks each of
size 100*5*60*8 bytes = 0.23 MiB. Then:
If cf.chunksize returns 134217728 (i.e. 128
MiB), then the storage-aligned Dask chunks will have
shape (400, 300, 60), giving 1 Dask chunk with size of
54.93 MiB (compare with a Dask chunk shape of (400,
300, 60) and size 54.93 MiB when dask_chunks is
'auto'.) If cf.chunksize returns 33554432 (i.e. 32
MiB), then the storage-aligned Dask chunks will have
shape (200, 260, 60), giving 4 Dask chunks with a
maximum size of 23.80 MiB (compare with a Dask chunk
shape of (264, 264, 60) and maximum size 31.90 MiB
when dask_chunks is 'auto'.) If cf.chunksize returns 4194304 (i.e. 4
MiB), then the storage-aligned Dask chunks will have
shape (100, 85, 60), giving 16 Dask chunks with a
maximum size of 3.89 MiB (compare with a Dask chunk
shape of (93, 93, 60) and maximum size 3.96 MiB when
dask_chunks is 'auto'.)
There are, however, some occasions when, for particular
data arrays in the file, the 'auto' option will
automatically be used instead of storage-aligned Dask
chunks. This occurs when:
The data array in the file is stored contiguously. The data array in the file is compressed by convention
(e.g. ragged array representations, compression by
gathering, subsampled coordinates, etc.). In this case
the Dask chunks are for the uncompressed data, and so
cannot be aligned with the storage chunks of the
compressed array in the file.
'storage-exact'
Each Dask chunk will contain exactly one storage chunk
and each storage chunk will lie entirely within exactly
one Dask chunk.
For instance, consider a file variable that has an array
of 64-bit floats with shape (400, 300, 60) and a storage
chunk shape of (100, 5, 60). This has an overall size of
54.93 MiB, partitioned into 240 storage chunks each of
size 100*5*60*8 bytes = 0.23 MiB. The corresponding
storage-exact Dask chunks will also have shape (100, 5,
60), giving 240 Dask chunks with a maximum size of 0.23
MiB.
There are, however, some occasions when, for particular
data arrays in the file, the 'auto' option will
automatically be used instead of storage-exact Dask
chunks. This occurs when:
The data array in the file is stored contiguously. The data array in the file is compressed by convention
(e.g. ragged array representations, compression by
gathering, subsampled coordinates, etc.). In this case
the Dask chunks are for the uncompressed data, and so
cannot be aligned with the storage chunks of the
compressed array in the file.
auto
The Dask chunk size in bytes will be as close as
possible to the size given by cf.chunksize,
favouring square-like chunk shapes. This may give
similar Dask chunk shapes as the 'storage-aligned'
option, but without the guarantee that each storage
chunk will lie entirely within exactly one Dask chunk.
A byte-size given by a str
The Dask chunk size in bytes will be as close as
possible to the given byte-size, favouring square-like
chunk shapes. Any string value, accepted by the chunks
parameter of the dask.array.from_array function is
permitted. There is no guarantee that a storage chunk
lies entirely within one Dask chunk.
- Example:
A Dask chunksize of 2 MiB may be specified as
'2097152' or '2 MiB'.
-1 or None
There is no Dask chunking, i.e. every data array has one
Dask chunk regardless of its size. In this case each
storage chunk is guaranteed to lie entirely within the
one Dask chunk.
Positive int
Every dimension of all Dask chunks has this number of
elements. There is no guarantee that a storage chunk
lies entirely within one Dask chunk.
- Example:
For 3-dimensional data, dask_chunks of 10 will
give Dask chunks with shape (10, 10, 10).
dict
Each of dictionary key identifies a file dimension, with
a value that defines the Dask chunking for that
dimension whenever it is spanned by a data array. A file
dimension is identified in one of three ways:
the netCDF dimension name, preceded by ncdim%
(e.g. 'ncdim%lat'); the value of the “standard name” attribute of a
CF-netCDF coordinate variable that spans the
dimension (e.g. 'latitude'); the value of the “axis” attribute of a CF-netCDF
coordinate variable that spans the dimension
(e.g. 'Y').
The dictionary values may be a byte-size string,
'auto', int or None, with the same meanings as
those types for the dask_chunks parameter itself, but
applying only to the specified dimension. In addition, a
dictionary value may be a tuple or list of integers
that sum to the dimension size.
Not specifying a file dimension in the dictionary is
equivalent to it being defined with a value of
'auto'.
- Example:
{'T': '0.5 MiB', 'Z': 'auto', 'Y': [36, 37], 'X':
None}
- Example:
If a netCDF file contains dimensions time, z,
lat, and lon, then {'ncdim%time': 12,
'ncdim%lat', None, 'ncdim%lon': None} will ensure
that all time axes have a Dask chunksize of 12;
all lat and lon axes are not Dask chunked; and
all z axes are Dask chunked to comply as closely
as possible with the default Dask chunk size.
If the netCDF file also contains a time coordinate
variable with a “standard_name” attribute of
'time' or “axis” attribute of 'T', then the
same dask chunking could be specified with either
{'time': 12, 'ncdim%lat', None, 'ncdim%lon': None}
or {'T': 12, 'ncdim%lat', None, 'ncdim%lon':
None}.
- store_dataset_chunks:
bool, optional If True (the default) then store the dataset chunking
strategy for each returned data array. The dataset
chunking strategy is then accessible via an object’s
nc_dataset_chunksizes method. When the dataset chunking
strategy is stored, it will be used when the data is
written to a new netCDF file with cf.write
(unless the strategy is modified prior to writing).
If False then no dataset chunking strategy is stored
(i.e. the nc_dataset_chunksizes method will return
None for all returned Data objects). In this case,
when the data is written to a new dataset, the dataset
chunking strategy will be determined by
cf.write.
See the cf.write dataset_chunks parameter for
details on how the dataset chunking strategy is determined
at the time of writing.
- store_dataset_shards:
bool, optional If True (the default) then store the Zarr dataset sharding
strategy for each returned data array. The dataset
sharding strategy is then accessible via an object’s
nc_dataset_shards method. When the dataset sharding
strategy is stored, it will be used when the data is
written to a new Zarr dataset with cf.write
(unless the strategy is modified prior to writing).
If False, or if the dataset being read does not support
sharding (such as a Zarr v2 or netCDF dataset), then no
dataset sharding strategy is stored (i.e. the
nc_dataset_shards method will return None for all
returned Data objects). In this case, when the data is
written to a new Zarr dataset, the dataset sharding
strategy will be determined by cf.write.
- cfa:
dict, optional Configure the reading of CF-netCDF aggregation files.
The dictionary may have any subset of the following
key/value pairs that supplement or override the
information read from the file:
'replace_directory': dict
A dictionary whose key/value pairs define modifications
to be applied to the directories of the fragment file
locations. The dictionary comprises keyword arguments to
the cf.Data.replace_directory` method, which is
used to make the changes. The aggregation file being
read is unaltered. An empty dictionary results in no
modifications.
- Example:
Replace a leading data/model with home,
wherever it occurs: {'replace_directory': {'old':
'data/model', 'new': 'home'}}
- Example:
Normalise all file locations and replace a leading
/archive with /data/obs, wherever it occurs:
{'replace_directory': {'old': '/archive', 'new':
'/data/obs', 'normalise': True}}
- Example:
Normalise all file locations and remove a leading
/data`, wherever it occurs: ``{'replace_directory':
{'old': '/data', 'normalise': True}}.
- cfa_write: (sequence of)
str, optional Register the intention for named construct types to be
subsequently written as CF-netCDF aggregation variables.
This makes no difference to the logical content of any
construct, but ensures that the data of each of specified
construct types will have only one Dask chunk, regardless
of the setting of dask_chunks, which is a requirement
for the creation CF-netCDF aggregation variables.
The cfa_write parameter may be one, or a sequence, of:
cfa_write |
Construct types |
|---|
'field'
|
Field constructs |
'field_ancillary'
|
Field ancillary constructs |
'domain_ancillary'
|
Domain ancillary constructs |
'dimension_coordinate'
|
Dimension coordinate constructs |
'auxiliary_coordinate'
|
Auxiliary coordinate constructs |
'cell_measure'
|
Cell measure constructs |
'domain_topology'
|
Domain topology constructs |
'cell_connectivity'
|
Cell connectivity constructs |
'all'
|
All constructs |
Note
If the dask_chunks parameter is set to None
or -1 then the data of all constructs will
already have only one Dask chunk, so in this
case setting cfa_write will have no further
effect.
- Example:
To register field constructs to be written as CF-netCDF
aggregation variables: cfa_write='field' or
cfa_write=['field'].
- Example:
To register field and auxiliary coordinate constructs to
be written as CF-netCDF aggregation variables:
cfa_write=['field', 'auxiliary_coordinate'].
- cfa_filesystem:
None or filesystem, optional A pre-authenticated filesystem object (for example an
fsspec filesystem instance) to use for opening the
fragment datasets indicated in a CF-netCDF aggregation
file.
If None (the default) then a fragment dataset path is
passed unchanged to the backends defined by the
cfa_backend parameter.
When provided, a fragment dataset path (fragment) is
treated as the file-like object filesystem.open(fragment,
'rb') which is passed to the backends (see the
cfa_backend parameter).
Added in version (cfdm): 1.13.3.0
- cfa_backend:
None or (sequence of) str, optional Which library or libraries to use for reading the dataset
fragments indicated in a CF-netCDF aggregation file. When
the fragments are accessed (which does not occur during
cf.read), an attempt to open each fragment
dataset is made by the given backends in order, stopping
after the first successful read. The available backends
are:
By default cfa_backend is None, which is equivalent to
providing the ordered sequence of backends:
('pyfive', 'zarr', 'netCDF4', 'netcdf_file', 'h5py')
- Example:
To only attempt 'netCDF4': 'netCDF4' or
['netCDF4']
- Example:
To only attempt 'netCDF4' or 'pyfive', in that
order: ('netCDF4', 'pyfive')
{{cfa_backend_options: None or dict, optional}}
- to_memory: (sequence of)
str, optional Read all data arrays of the named construct types into
memory. By default, lazy loading is employed for all data
arrays.
The to_memory parameter may be one, or a sequence, of:
to_memory |
Construct types |
|---|
'all'
|
All constructs |
'metadata'
|
All metadata constructs (i.e.
all constructs except Field
constructs) |
'field'
|
Field constructs |
'field_ancillary'
|
Field ancillary constructs |
'domain_ancillary'
|
Domain ancillary constructs |
'dimension_coordinate'
|
Dimension coordinate constructs |
'auxiliary_coordinate'
|
Auxiliary coordinate constructs |
'cell_measure'
|
Cell measure constructs |
'domain_topology'
|
Domain topology constructs |
'cell_connectivity'
|
Cell connectivity constructs |
- Example:
To read field construct data arrays into memory:
to_memory='field' or to_memory=['field'].
- Example:
To read field and auxiliary coordinate construct data
arrays into memory: to_memory=['field',
'auxiliary_coordinate'].
- group_dimension_search:
str, optional How to interpret a dimension name that contains no
group-separator characters, such as dim (as opposed to
group/dim, /group/dim, ../dim, etc.). The
group_dimension_search parameter must be one of:
'closest_ancestor'
This is the default and is the behaviour defined by the
CF conventions (section 2.7 Groups).
Assume that the sub-group dimension is the same as the
dimension with the same name and size in an ancestor
group, if one exists. If multiple such dimensions exist,
then the correspondence is with the dimension in the
ancestor group that is closest to the sub-group
(i.e. that is furthest away from the root group).
'furthest_ancestor'
This behaviour is different to that defined by the CF
conventions (section 2.7 Groups).
Assume that the sub-group dimension is the same as the
one with the same name and size in an ancestor group, if
one exists. If multiple such dimensions exist, then the
correspondence is with the dimension in the ancestor
group that is furthest away from the sub-group
(i.e. that is closest to the root group).
'local'
This behaviour is different to that defined by the CF
conventions (section 2.7 Groups).
Assume that the Zarr sub-group dimension is different to
any with the same name and size in all ancestor groups.
Note
For a netCDF dataset, for which it is always
well-defined in which group a dimension is
defined, group_dimension_search may only take
the default value of 'closest_ancestor',
which applies the behaviour defined by the CF
conventions (section 2.7 Groups).
For a Zarr dataset, for which there is no means
of indicating whether or not the same dimension
names that appear in different groups correspond
to each other, setting this parameter may be
necessary for the correct interpretation of the
dataset in the event that its dimensions are
named in a manner that is inconsistent with CF
rules defined by the CF conventions (section 2.7
Groups).
- filesystem: optional
A pre-authenticated file system object (for example an
fsspec.filesystem instance) to use for opening the
dataset. When provided, datasets values are treated as
paths understood by filesystem, and local string
pre-processing (tilde/variable expansion, globbing and
directory walking) is bypassed. The file is opened by
calling filesystem.open(dataset, 'rb'), which returns
a file-like object that is passed to the netCDF backend.
If None (the default) and storage_options is also
None, then a dataset is treated as described by the
datasets parameter.
When provided, a dataset is treated as the file-like
object filesystem.open(dataset, 'rb') which is passed
to the backends (see the backend parameter).
Added in version (cfdm): 1.13.1.0
{{read _noncompliance_report: bool, optional}}
- umversion: deprecated at version 3.0.0
Use the um parameter instead.
- height_at_top_of_model: deprecated at version 3.0.0
Use the um parameter instead.
- field: deprecated at version 3.0.0
Use the extra parameter instead.
- follow_symlinks: deprecated at version 3.0.0
Use the followlinks parameter instead.
- select_options: deprecated at version 3.0.0
Use methods on the returned FieldList instead.
- chunk: deprecated at version 3.14.0
Use the dask_chunks parameter instead.
- chunks: deprecated at version 3.17.0
Use the dask_chunks parameter instead.
- fmt: deprecated at version 3.17.0
Use the dataset_type* parameter instead.
- ignore_read_error: deprecated at version 3.17.0
Use the dataset_type parameter instead.
- file_type: deprecated at version 3.18.0
Use the dataset_type parameter instead.
- netcdf_backend: Deprecated at version 3.21.0
Use backend instead.
- Returns:
FieldList or DomainListThe field or domain constructs found in the input
dataset(s). The list may be empty.
Examples
>>> x = cf.read('file.nc')
Read a file and create field constructs from CF-netCDF data
variables as well as from the netCDF variables that correspond to
particular types metadata constructs:
>>> f = cf.read('file.nc', extra='domain_ancillary')
>>> g = cf.read('file.nc', extra=['dimension_coordinate',
... 'auxiliary_coordinate'])
Read a file that contains external variables:
>>> h = cf.read('parent.nc')
>>> i = cf.read('parent.nc', external='external.nc')
>>> j = cf.read('parent.nc', external=['external1.nc', 'external2.nc'])
>>> f = cf.read('file*.nc')
>>> f
[<CF Field: pmsl(30, 24)>,
<CF Field: z-squared(17, 30, 24)>,
<CF Field: temperature(17, 30, 24)>,
<CF Field: temperature_wind(17, 29, 24)>]
>>> cf.read('file*.nc')[0:2]
[<CF Field: pmsl(30, 24)>,
<CF Field: z-squared(17, 30, 24)>]
>>> cf.read('file*.nc')[-1]
<CF Field: temperature_wind(17, 29, 24)>
>>> cf.read('file*.nc', select='units=K')
[<CF Field: temperature(17, 30, 24)>,
<CF Field: temperature_wind(17, 29, 24)>]
>>> cf.read('file*.nc', select='ncvar%ta')
<CF Field: temperature(17, 30, 24)>
|