How to use CDF in IDL® Quick Start Guide

1.Introduction

Common Data Format (CDF) is a conceptual data abstraction for storing, manipulating, and accessing multidimensional datasets. CDF is referred to as a data abstraction because the actual physical format in which datasets are stored is not discussed. Instead, the form of the datasets and the means (interface) by which they may be manipulated are described. CDF is supported by commercial and open source data analysis/visualization software such as the Interactive Data Language (IDL).
The CDF home page contains examples, documentation, FAQs, and tools - https://cdf.gsfc.nasa.gov/.
The CDF User Guide is available at: https://cdf.gsfc.nasa.gov/html/cdf_docs.html.

There are four primary mechanisms for working with CDF files in IDL: IDL® built-in CDF routines, SPDF/CDF provided routines (described below as “the patch”) and the SPDF/CDAWlib Package provide access to locally stored CDF files, while the Coordinated Data Analysis System (CDAS) Web Service Library is used for remotely accessing CDAWeb data.

2.Installation

Though IDL® comes with the CDF library, it is strongly recommended that users install the IDL® CDF patch to get the latest version and functionality. The IDL® CDF patch consists of two major files: shared CDF library and platform-independent Dynamically Loadable Module (DLM) definition file. For non-Windows systems, make sure the downloaded dynamic library (the .so file) has executable permission on the user’s machine.

2.1 Windows:

Script installation (need to run as the administrator) to place the patch into an existing IDL® package in the c:\program files directory.
Download Gnu’s wget tool: wget.exe https://spdf.gsfc.nasa.gov/pub/software/cdf/dist/latest-release/idl/windows/wget.exe
Download the script: install_idl_win.bat from a terminal window.
Run the script from an administrator terminal window. Make sure wget.exe is accessible from PATH variable. Verify the test run shows the correct CDF library version and the last updated leap second date.

install_idl_win.bat

2.2 Mac OS X (Intel Mac or Apple Silicon):

Script installation (may need to have the root account access) to place the patch into an existing IDL® package in the /Applications or other.

Download the script: install_idl_mac.sh from a terminal window.
Make the script’s permission to executable:

chmod +x install_idl_mac.sh
Run the script. It must be run from an id that has the root permission if the installation goes into /Applications folder. Verify the test run shows the correct CDF library version and the last updated leap second date.

./install_idl_mac.sh

2.3 Linux:

Script installation (need to have the root account access) to place the patch into an existing IDL® package in the /usr/local system directory.
Download the script: install_idl_unix.sh from a terminal window.
Make the script’s permission to executable:

chmod +x install_idl_unix.sh
Run the script from an id that has the root permission. Verify the test run shows the correct CDF library version and the last updated leap second date.

./install_idl_unix.sh

3.IDL® Built-In CDF Routines

IDL® provides a complete set of built-in routines (functions and procedures) for working with locally stored CDF files. See CDF Overview (CDF Overview) for a quick introduction and IDL® Help for detailed description of all CDF routines (CDF routines begin with prefix “CDF_”). There are two approaches to using built-in CDF routines, reading an entire CDF or reading/writing data from/to individual variables and metadata from/to individual attributes.

Quickly read a whole CDF located on your machine with CDF_READCDF (part of the CDF’s IDL® DLM). The retrieved information is presented in an IDL® hash form. See CDF_READCDF.txt for detailed description of the function. Below are the function syntax and example code:

cdf = CDF_READCDF ( Id, [, /DATAONLY] [, /ENCODING] [, /GLOBALS] [, /STRING]

[,/INFO] [, /METADATA] [, /NOENTRY] [, /VARATTRS]

[, /TO_COLUMN_MAJOR] [, VARS=…]

[, REC_COUNT=…] [, REC_START=…]

IDL> file ='/data/ace/mag/level_2_cdaweb/mfi_k2/2022/ac_k2_mfi_20221115_v03.cdf'
IDL> id = CDF_OPEN(file)
IDL> cdf = CDF_READCDF (id, /string, /encoding)
IDL> ;to see the variables in the cdf
IDL> cdf['Variables'].keys()
IDL> ; to print the Epoch values
IDL> cdf['Variables', 'Epoch', 'VarData']

The following example demonstrates opening a CDF file and reading data from a variable and metadata from global and variable attributes.

IDL> ; Open CDF file
IDL> cdf_id = CDF_OPEN('example.cdf', /READONLY)
IDL> ; Read CDF global attribute called Project into IDL&reg; variable called project
IDL> CDF_ATTGET, cdf_id, 'Project', 0, project
IDL> ; Read CDF variable attribute called CATDESC for CDF variable called Epoch into IDL&reg; variable called epoch_catdesc
IDL> CDF_ATTGET, cdf_id, 'CATDESC', 'Epoch', epoch_catdesc
IDL> ; Read all records from CDF variable called Epoch into IDL&reg; variable called epoch
IDL> CDF_VARGET, cdf_id, 'Epoch', epoch , REC_COUNT=0
IDL> ; Close CDF file
IDL> CDF_CLOSE, cdf_id

The next example demonstrates creating a CDF file and writing data and metadata, including creating a variable storing data and global and variable attributes storing metadata.

IDL> ; Create new CDF file, erase if the file already exists
IDL> cdf_id = CDF_CREATE('example.cdf', /CLOBBER)
IDL> ; Create CDF global attribute called Project, and write a string
IDL> att1_id = CDF_ATTCREATE(cdf_id, 'Project', /GLOBAL_SCOPE)
IDL> CDF_ATTPUT, cdf_id, 'Project', 0, 'ISTP>International Solar-Terrestrial Physics', /CDF_CHAR 
IDL> ; Create CDF zVariable called Epoch of type CDF_EPOCH, and write data from IDL&reg; variable called epoch
IDL> var1_id = CDF_VARCREATE(cdf_id, 'Epoch', /REC_VARY, ALLOCATE=n_epoch, /CDF_EPOCH, /ZVARIABLE)
IDL> CDF_VARPUT, cdf_id, 'Epoch', epoch
IDL> ; Create CDF variable attribute called CATDESC, and write a string for CDF variable Epoch
IDL> att2_id = CDF_ATTCREATE(cdf_id, 'CATDESC', /VARIABLE_SCOPE)
IDL> CDF_ATTPUT,  cdf_id, 'CATDESC', 'Epoch', 'Time, number of milliseconds since 01-Jan-0000 00:00:00.000', /CDF_CHAR
IDL> ; Close CDF file
IDL> CDF_CLOSE, cdf_id

4.More Ways to Read ISTP Compliant Files

In addition to the CDF IDL® interface routines discussed in section 3, there are more ways to read ISTP compliant CDF data files. Depending on what you want to do with the data/metadata that has been read in, will determine which of the options below you will want to take. The option you choose will also depend on whether the data files are located on your local machine or available via the CDAWeb system (SPDF archive https://spdf.gsfc.nasa.gov/pub/data).

4.1 CDAWeb IDL® Library

The CDAS Web Service Library (IDL® CDAWeb Access) provides easy access to CDAWeb data (SPDF - Coordinated Data Analysis Web (CDAWeb) from an IDL® program without need for storing CDF files locally. The library can be accessed by downloading and restoring an IDL® SAVE file (https://cdaweb.gsfc.nasa.gov/WebServices/REST/spdfcdas.sav). Visit CDAS Web Service Library webpage (IDL® CDAWeb Access) for detailed installation instructions and functionality description. This is further illustrated with an IDL® Jupyter notebook example (CdasWsIdlExample).

With this library, an IDL® user can retrieve data from CDAWeb by entering a single IDL® command (spdfgetdata). The data is automatically downloaded and read into an IDL® structure (with its associated metadata/documentation) in the user’s local IDL® environment. The user can then write their own IDL® code to analyze or visualize the data. There is also a GUI program to make discovering and retrieving CDAWeb data even easier. Finally, there is an example program that demonstrates many of the lower-level calls for those who want to incorporate CDAWeb data access into a larger IDL® program.

Determine if you need the CDF patch for IDL.
Download spdfcdas.sav
Start IDL® 8.4 or higher.
IDL> restore, ‘spdfcdas.sav’, /skip_existing

One-line Data Access: To retrieve data from the AC_H2_MFI dataset into the IDL® variable “d”, do the following in an IDL® session:

IDL> d = spdfgetdata(‘AC_H2_MFI’, [‘Magnitude’ , ‘BGSEc’], [‘2009-06-01T00:00:00.000Z’, ‘2009-06-03T00:00:00.000Z’])

Notes:

Epoch values are returned in d.epoch.dat
Magnitude values are in d.magnitude.dat
BGSEc values are in d.bgsec.dat

4.2 Use the CDAWlib Package of routines to read CDFs located on your machine

The CDAWlib package (CDAWlib) includes IDL® functions and procedures used to read CDFs and NetCDFs, to plot data from data files, to list data (to ascii files) and to create subset/superset CDFs. It is the software that underlies the CDAWeb data system (SPDF - Coordinated Data Analysis Web (CDAWeb). The easiest way to access the package is by downloading and restoring an IDL® SAVE file (https://cdaweb.gsfc.nasa.gov/WebServices/REST/spdfcdas.sav). Visit the CDAWlib webpage (CDAWlib) for detailed installation instructions and the descriptions of the functions included in the package.

The CDAWlib functions include:

READ_MYCDF reads variables from locally stored CDF data files and returns all data and metadata for these variables in a single structure (it follows pointer attribute values, such as DEPEND_0, Labl_ptr_1, etc. and will return these variables as well).
PLOTMASTER plots the data returned by READ_MYCDF.
LIST_mystruct generates ASCII listing files of ISTP-compliant CDF files in formatted text, csv and json formats.
Write_mycdf creates a cdf file, containing the data and metadata returned by the read_myCDF function.
The IDLmakecdf software creates ISTP-compliant CDF files using metadata from a Master CDF file (created with SPDF - ISTP CDF Skeleton Editor) and user-provided data.
The package also includes a CDF viewing and editing GUI program called CDFx.

Depending on what you want to do with the data/metadata that you have read in, will determine which approach you want to take.

4.2.1 Quickly reading ISTP Compliant CDF files located on your machine with CDAWlib’s read_myCDF

The function READ_MYCDF reads from one to many variables from one to many CDF files, and returns all data and metadata for these variables in a single structure of the form: structure_name.variable_name.attribute_name.value. Note, there are several additional keywords and options to the function, such as tstart/tstop (time range). Please see the code header at https://cdaweb.gsfc.nasa.gov/pub/software/cdawlib/source/read_myCDF.pro for the most recent documention.

Calling Sequence

Result = READ_MYCDF(varnames, cdfnames) ; to read specific variables

Result = READ_MYCDF(”,/all, cdfnames) ; to read ALL variables

Example code:

IDL> ;Specify two data files to read
IDL> a=strarr(2)
IDL> a[0]='i8_k0_mag_19951008_v01.cdf'
IDL> a[1]='i8_k0_mag_19951013_v01.cdf'
IDL> b=read_mycdf('RMS,RMS_p',a) ; read the RMS and RMS_p variables from these two files`
IDL> help,/struct,b   ; show the variables read in
**Structure < 4006fca8 >, 4 tags, length=296936, refs=1:
 RMS            STRUCT    -> < Anonymous > Array(1)
 RMS_P          STRUCT    -> < Anonymous > Array(1)
 EPOCH          STRUCT    -> < Anonymous > Array(1)
 CARTESIAN       STRUCT    ->< Anonymous > Array(1)
IDL> help,/struct,b.RMS                      ; show the structure of the RMS variables structure`
**Structure < 400cd808 >, 32 tags, length=147488, refs=2:
 VARNAME        STRING    'RMS'
 TITLE          STRING    'IMP-8 Magnetic Field'
 PROJECT        STRING    'ISTP>International Solar-Terrestrial Physics'
Etc.
 DEPEND_1       STRING    'cartesian'
 CDFTYPE        STRING    'CDF_REAL4'
 DAT         FLOAT     Array(3, 12206) ;  the .dat structure element contains the actual data

5.Writing ISTP Compliant CDF Files

There are two ways to write ISTP compliant CDF files.

5.1 Quickly Create a subset or superset CDF by reading one or many ISTP Compliant CDF files using CDAWlib’s read_mycdf and write_mycdf

Example code:

IDL> ;Specify four data files to read
IDL> start='1995/10/08 10:00:00.000'
IDL> stop='1995/10/10 10:00:00.000'
IDL> a=strarr(4)
IDL> a[0]='/home/cdaweb/data/0MASTERS/i8_15sec_mag_00000000_v01.cdf'
IDL> a[1]='/home/cdaweb/data/imp/imp8/mag/mag_15sec_cdaweb/1995/i8_15sec_mag_19951008_v03.cdf'
IDL> a[2]='/home/cdaweb/data/imp/imp8/mag/mag_15sec_cdaweb/1995/i8_15sec_mag_19951009_v03.cdf'
IDL> a[3]='/home/cdaweb/data/imp/imp8/mag/mag_15sec_cdaweb/1995/i8_15sec_mag_19951010_v03.cdf'
IDL> ; read just 2 variables from these four files for the specified time range
IDL> b=read_mycdf(['SC_Pos_GSE','F1_Average_B_15s'],a, TSTART=start, TSTOP=stop) 
IDL> ;write the requested data variables and associated variable(s) data out to another cdf
IDL> s = write_mycdf(b,/autoname) ; write_mycdf will take up to 35 structures returned by read_mycdf calls and output each to a CDF file (each with their own appropriate file name – based on the dataset  name).  This output filename would be i8_15secs_mag_19951008100003_19951010095948.cdf

5.2 Create a CDF data file with your own data using the IDLmakecdf package

The IDLMakecdf program file contains two functions: read_master_cdf and write_data_to_cdf which are intended to be used together. The general purpose is to allow someone who has their own data, to use the routines to setup the “structure” of the cdf with a skeleton cdf. Then add their data to the cdf variables by simply setting the pointer variables in the structure returned by read_master_cdf and then write the data out to the desired cdf data file with the write_data_to_cdf routine.

Example code:

IDL> out_cdf = 'hk_test.cdf'                                                                                                                                                                                                                                                                                                                                                         
IDL> ; Read the master cdf (copy its contents to our out_cdf) and define a structure,                                                                                                         
IDL> ; in this case "buf1" so that I may add the data to the variables.                                                                                                                                                                                                                                                                                                              
IDL> buf1 = read_master_cdf('hk_h0_vlf_00000000_v01.cdf',out_cdf)   
IDL> ;  this master cdf can be found at: [ https://cdaweb.gsfc.nasa.gov/pub/software/cdawlib/0MASTERS/hk_h0_vlf_00000000_v01.cdf ]()
IDL> ;  the ascii version of this cdf can be seen at: https://cdaweb.gsfc.nasa.gov/pub/software/cdawlib/0SKELTABLES/hk_h0_vlf_00000000_v01.skt
IDL> ;buf1, would look like (a tag for each variable):
IDL> ;EPOCH          STRUCT    -> <Anonymous>Array[1]
IDL> ;E_FREQ         STRUCT    -> <Anonymous>Array[1]
IDL> ;E_SPD          STRUCT    -> <Anonymous>Array[1]
;
IDL> help, /struct, buf1.e_spd would reveal the structure for the e_spd variable
IDL> ; VARNAME        STRING   'e_spd'
IDL> ; DATA           POINTER <PtrHeapVar1>
IDL> ; to put values in the data pointer for the e_spd variable do something like this:
IDL> espd = fltarr(16,num_rec) ; fill the array with the necessary numbers
IDL> *buf1.E_SPD.data = espd  ; load them into the buf1 data pointer for this variable
IDL> ; to write the data out to the specified output cdf file call:
IDL> stat2 = write_data_to_cdf(out_cdf, buf1)

6.Notes for Older Versions of IDL

CDFs created by IDL® 6.3 or newer are version 3.x CDFs that are not readable by the built-in CDF 2.7 routines in IDL® 6.2 or earlier. One way around this is to create CDF 2.7 files using the CDF_SET_CDF27_BACKWARD_COMPATIBLE routine. The better method is to install the CDF IDL® patch for the latest version of CDF, as noted in section 2.