Official websites use .gov
A
.gov website belongs to an official government
organization in the United States.
Secure .gov websites use HTTPS
A
lock (
) or https:// means you’ve safely connected to
the .gov website. Share sensitive information only on official,
secure websites.
Common Data Format (CDF) is a conceptual data abstraction for storing, manipulating, and accessing multidimensional datasets. CDF is referred to as a data abstraction because the actual physical format in which datasets are stored is not discussed. Instead, the form of the datasets and the means (interface) by which they may be manipulated are described. The CDF distribution is packaged in a self-extracting installer. Once the installer is downloaded and run, all distributed files, i.e., application programmming interfaces (API), test programs, batch files, help information and the documents, will be placed into a directory of choice. The CDF home page contains examples, documentation, FAQs, and tools. The CDF User Guide and CDF Language Reference Manuals can be found here.
If the CDF files of interest are in the SPDF archive and supported by the Coordinated Data Analysis Web (CDAWeb) system, the files can be read, plotted, listed, and downloaded using the browser based system. For users writing their own reading/analysis software, the CDF software distribution supports reading of files in many programming languages from many hardware platforms, e.g. Python, C, Fortran, Perl, Java, IDL®, and MATLAB. The CDF library, for each programming language, provides the routines to read attributes and variables in a variety of ways; see the information on all of the CDF language guides.
Command line tools and data systems will display data from and allow users to read CDF files. The CDAWeb data system provides access to this data through its web-based user interface and the Coordinated Data Analysis System (CDAS) Web Services Application Programming Interface (API) allow software developers to use CDAS services in their own software applications. For software developers who simply want to use the existing web (HTML) interface to CDAS, use the CDAWeb system. For those developing an application that requires science data from any of the datasets held by CDAS, the CDAS Web Services will provide an interface to the data for the application. They provide the full capability of CDAWeb and are implemented as Representational State Transfer (REST) style Web Services. More information can be found in the CDAWeb Quick Start Guide.
CDFDump is a command line program that dumps the metadata/data in a CDF. It displays or extracts the contents of a CDF file to a screen (default) or text file. This program extracts and displays one variable at a time while the CDFexport command line program extracts and displays the contents of a CDF file in a table format, each column representing a variable. CDFdump extracts the value of the variable attributes, but CDFexport does not extract the value of the variable attributes.
The CDFDump program and SKTEditor Java application provide ways to quickly view the structure of a CDF. The SKTEditor is a graphical user interface (GUI) application for creating and editing International Solar Terrestrial Physics (ISTP) compliant CDF files. Please consult the CDF distributed utility guide to learn more about the programs and to download the software.
The Heliophysics Application Programming Interface (HAPI), to the CDAWeb supported data, provides a standard way to access Heliophysics data and is similar to the API provided by the CDAS Web Services described above. See the HAPI Specification for more details on the concept. Client software to access a HAPI server is available here. The official, HAPI-compliant implementation for CDAWeb data is available here.
CDFX is a Graphical User Interface (GUI) IDL® based software tool, included in the CDAWlib library, that reads, edits, and displays the contents of a CDF file and can also be used for time subsetting, plotting, listing variable data and fine tuning the metadata. It requires a local installation of IDL®. It displays image plots, a variety of time-series plot types, listings of variables along with their metadata and it will save subsets of data to new CDF files. Familiarity with CDF and ISTP CDF/metadata conventions will be helpful when using CDFx.
The main CDF distribution provides software packages for each supported programming language to read and manipulate CDFs. There are descriptions and examples below for three popular languages, but please also consult the quick start guides for all programming languages supported by CDF for further information and detailed examples.
For Python, IDL®, and MATLAB®: How do I load a CDF into a data structure?
Python:: There are currently two Python libraries available to read CDF files: (1) The Pure Python (CDFlib) package, which does not depend on the CDF core library, and (2) the SpacePy package, which does depend on the CDF core library. The CDFlib Package contains the basic CDF functionality; it can load data into Xarray objects and does not require installation of the NASA CDF software. The SpacePy package contains more advanced CDF functionality, loads data into a SpacePy data model, is more efficient than CDFlib because of the included CDF C libraries, and has the optional use of locally installed NASA CDF software. More information and coding examples can be found below and in the Python Quick Start Guide-CDF.
IDL®: There are four primary mechanisms for working with CDF files in IDL®: built-in CDF routines, SPDF/CDF provided routines (refered to as “the patch”) the SPDF/CDAWlib Package, mentioned above, provide access to locally stored CDF files, and the CDAS Web Service Library for remotely accessing CDAWeb CDF data files. See the examples below and the CDF IDL® Quickstart Guide for an introduction. Also, within IDL®, users can use the internal Help system to find detailed descriptions of all CDF routines (they begin with prefix “CDF_”).
MATLAB®: is the language of technical computing. It is matrix-based and allows for the most natural expression of computational mathematics. Its high-level functions make it easy to read CDF files. Its low-level functions provide direct access to routines in the CDF C API library. The low-level functions should be utilized if users require more control over the import operation, such as data subsetting for large datasets, or to export data to a CDF file. To use the MATLAB® CDF low-level functions effectively, the user must be familiar with the CDF C interface.
In addition to Python, IDL® and Matlab, the CDF distribution provides libraries to read CDF files in C, C#, Java, Perl, see the programming [quickstart guides] CDF language guides
Users can read the CDF into an Xarray structure.
$ pip install xarray
$ python3
>>> # Import required module
>>> import cdflib
>>>
>>> # Read CDF file into Xarray Dataset object and display variables and global attributes
>>> data1 = cdflib.cdf_to_xarray("soho_celias-pm_30s_20200101_v02.cdf", to_unixtime=True, fillval_to_nan=True)
>>> data1
<xarray.Dataset>
Dimensions: (Epoch: 2112, dim0: 3)
Coordinates:
* Epoch (Epoch) float64 1.578e+09 1.578e+09 ... 1.578e+09 1.578e+09
Dimensions without coordinates: dim0
Data variables:
V_p (Epoch) float32 320.0 315.0 323.0 326.0 ... 332.0 327.0 331.0
N_p (Epoch) float32 5.32 5.62 4.99 4.78 6.09 ... 5.0 4.88 5.08 4.93
Vth_p (Epoch) float32 19.0 19.0 19.0 19.0 18.0 ... 19.0 19.0 19.0 19.0
NS_angle (Epoch) float32 0.5 0.9 1.4 1.8 1.2 0.4 ... -0.0 -0.0 0.3 0.3 0.2
V_He (Epoch) float32 321.0 317.0 324.0 328.0 ... 334.0 328.0 333.0
CRN (Epoch) uint16 2225 2225 2225 2225 2225 ... 2225 2225 2225 2225
GSE_POS (Epoch, dim0) float32 199.1 -79.5 8.0 199.1 ... 198.4 -77.0 8.5
label_GSE (dim0) <U7 'X (GSE)' 'Y (GSE)' 'Z (GSE)'
HC_RANGE (Epoch) float32 145.8 145.8 145.8 145.8 ... 145.8 145.8 145.8
HG_LAT (Epoch) float32 -2.9 -2.9 -2.9 -2.9 -2.9 ... -3.0 -3.0 -3.0 -3.0
HG_LONG (Epoch) float32 71.0 71.0 71.0 71.0 71.0 ... 57.9 57.9 57.9 57.9
Attributes: (12/22)
Project: ISTP>International Solar-Terrestrial Physics
Source_name: SOHO>Solar Heliospheric Observatory
Discipline: Solar Physics>Heliospheric Physics
Data_type: 30S>30 second resolution
Descriptor: CELIAS-PM>Proton Monitor
Data_version: 2
... ...
Instrument_type: Plasma and Solar Wind
LINK_TEXT: SOHO CELIAS-PM 30 second data available at
LINK_TITLE: the SOHO Archive
HTTP_LINK: https://soho.nascom.nasa.gov/data/archive.html
>>>
>>> # Access global attributes
>>> # data1.attrs['Project']
>>> data1.Project
'ISTP>International Solar-Terrestrial Physics'
>>>
>>> # Display variable and its attributes
>>> # data1.data_vars['V_p']
>>> data1.V_p
<xarray.DataArray 'V_p' (Epoch: 2112)>
array([320., 315., 323., ..., 332., 327., 331.], dtype=float32)
Coordinates:
* Epoch (Epoch) float64 1.578e+09 1.578e+09 ... 1.578e+09 1.578e+09
Attributes: (12/14)
CATDESC: Proton speed, scalar
FIELDNAM: Proton speed
LABLAXIS: Proton V
FILLVAL: [-1.e+31]
VALIDMIN: [0.]
VALIDMAX: [10000.]
... ...
DEPEND_0: Epoch
FORMAT: F7.0
DISPLAY_TYPE: time_series
standard_name: Proton speed
long_name: Proton V
units: km/s
>>>
>>> # Access variable data
>>> # data1.data_vars['V_p'].data
>>> data1.V_p.data
array([320., 315., 323., ..., 332., 327., 331.], dtype=float32)
>>>
>>> # Access variable attributes
>>> # data1.V_p.attrs['CATDESC']
>>> data1.V_p.CATDESC
'Proton speed, scalar'More detailed SpacePy information.
IDL® provides a complete set of built-in routines (functions and procedures) for working with locally stored CDF files. See CDF Overview for a quick introduction and IDL® Help for detailed description of all CDF routines (CDF routines begin with prefix “CDF_”). There are two approaches to using built-in CDF routines, reading an entire CDF or reading/writing data from/to individual variables and metadata from/to individual attributes.
Quickly read a whole CDF located on your machine with CDF_READCDF (part of the CDF’s IDL® DLM). The retrieved information is presented in an IDL® hash form. See CDF_READCDF.txt for a detailed description of the function. Below are the function syntax and example code:
cdf = CDF_READCDF ( Id, [, /DATAONLY] [, /ENCODING] [, /GLOBALS] [, /STRING]
[,/INFO] [, /METADATA] [, /NOENTRY] [, /VARATTRS]
[, /TO_COLUMN_MAJOR] [, VARS=…]
[, REC_COUNT=…] [, REC_START=…]
IDL> file ='/data/ace/mag/level_2_cdaweb/mfi_k2/2022/ac_k2_mfi_20221115_v03.cdf'
IDL> id = CDF_OPEN(file)
IDL> cdf = CDF_READCDF (id, /string, /encoding)
IDL> ;to see the variables in the cdf
IDL> cdf['Variables'].keys()
IDL> ; to print the Epoch values
IDL> cdf['Variables', 'Epoch', 'VarData']The following example demonstrates opening a CDF file and reading data from a variable and metadata from global and variable attributes.
IDL> ; Open CDF file
IDL> cdf_id = CDF_OPEN('example.cdf', /READONLY)
IDL> ; Read CDF global attribute called Project into IDL® variable called project
IDL> CDF_ATTGET, cdf_id, 'Project', 0, project
IDL> ; Read CDF variable attribute called CATDESC for CDF variable called Epoch into IDL® variable called epoch_catdesc
IDL> CDF_ATTGET, cdf_id, 'CATDESC', 'Epoch', epoch_catdesc
IDL> ; Read all records from CDF variable called Epoch into IDL® variable called epoch
IDL> CDF_VARGET, cdf_id, 'Epoch', epoch , REC_COUNT=0
IDL> ; Close CDF file
IDL> CDF_CLOSE, cdf_idThe next example demonstrates creating a CDF file and writing data and metadata, including creating a variable storing data and global and variable attributes storing metadata.
IDL> ; Create new CDF file, erase if the file already exists
IDL> cdf_id = CDF_CREATE('example.cdf', /CLOBBER)
IDL> ; Create CDF global attribute called Project, and write a string
IDL> att1_id = CDF_ATTCREATE(cdf_id, 'Project', /GLOBAL_SCOPE)
IDL> CDF_ATTPUT, cdf_id, 'Project', 0, 'ISTP>International Solar-Terrestrial Physics', /CDF_CHAR
IDL> ; Create CDF zVariable called Epoch of type CDF_EPOCH, and write data from IDL® variable called epoch
IDL> var1_id = CDF_VARCREATE(cdf_id, 'Epoch', /REC_VARY, ALLOCATE=n_epoch, /CDF_EPOCH, /ZVARIABLE)
IDL> CDF_VARPUT, cdf_id, 'Epoch', epoch
IDL> ; Create CDF variable attribute called CATDESC, and write a string for CDF variable Epoch
IDL> att2_id = CDF_ATTCREATE(cdf_id, 'CATDESC', /VARIABLE_SCOPE)
IDL> CDF_ATTPUT, cdf_id, 'CATDESC', 'Epoch', 'Time, number of milliseconds since 01-Jan-0000 00:00:00.000', /CDF_CHAR
IDL> ; Close CDF file
IDL> CDF_CLOSE, cdf_idMATLAB® supports CDF reading and writing with two groups of modules, cdflib and cdfread/cdfinfo/cdfwrite. The module cdflib enables creating/reading/writing portions of CDF variables, while cdfread/cdfwrite reads/writes whole variables. Each release of MATLAB® is based on a particular version of CDF; for instance MATLAB® version 2018a is based on CDF V3.6.1. While this CDF version includes the newer types: CDF_INT8 and CDF_TIME_TT2000, however, the MATLAB® modules still do not support them. The CDF patch provides modified and expanded capabilities of the original cdfread/cdfinfo/cdfwrite from MATLAB®. The list of MATLAB® functions are listed here, please also reference the quickstart guide.
Times in CDF are generally stored in one of three forms: CDF_EPOCH (8-byte floating milliseconds from 0AD), CDF_EPOCH16 (two 8-byte floats of seconds from 0AD and picoseconds of that second), and CDF_TIME_TT2000 (8 byte integer of nanoseconds from 2000 AD). SPDF’s spdfcdfread can handle all time types, and optionally can convert their values to MATLAB®‘s datenum. SPDF’s spdfcdfread also provides cdfepoch and cdftt2000 objects (more accurate but not as efficient as MATLAB®‘s datenum).
If users get error messages reading a CDF file while using MATALB distributed CDF package/module they should try the patch from the CDF home page. The patch includes many user suggested enhancements and features. This patch works only for MATLAB® version R2007a and later.
Data can be displayed as plots, listings, and outputs in CDF, netCDF, CSV, JSON, and audio files. Note, the early ISTP data use the term key parameters (KP, K0) to describe one minute quick-look data that is not as fully calibrated as higher-resolution (H0, H1, etc.) data. The interface to CDAWeb progresses through a series of pages (Home Page, Data Selector, Data Explorer).
· Variables/Parameters for each dataset are displayed on the data explorer form.
· Select start and stop time, which is pre-populated with the latest available time.
· Select activity (Download, List, or Plot) to perform the appropriate operation.
· Select variables only to List or Plot (download option lists all CDF data files for selected times including all variables).
· Press “submit” button to perform the action.
The data are then processed and displayed (as plots or listings), or you can download the CDF data files themselves.
· Plotting
The plots are in fact a set of PNG image(s) that can be easily downloaded and saved locally if desired. (Note that the powerful underlying IDL® plotting software is available for download to further analyze the datasets.)
· Listing
The listings are displayed as hyperlinks that can be displayed on the browser and/or saved directly as an ASCII file.
· Download CDFs
Selecting this option will generate a list of CDF datasets for the particular time period. Each file can be downloaded by clicking on the appropriate hyperlink.
Autoplot is an interactive browser for data on the web; give it a URL or the name of a file on your computer and it tries to create a sensible plot of the contents in the file. Autoplot was developed to allow quick and interactive browsing of data and metadata files that are often encountered on the web. Autoplot was developed under the NASA Heliophysics Data & Model Consortium (HDMC) for Heliophysics program in a collaborative effort among several institutions, including support or code contributions from PDS-PPI Node, RBSP-ECT, and the Radio and Plasma Wave Group at The University of Iowa.