Quick Start Guide - CDF with Python

This is intended as a basic guide for installing the software needed to read and write Common Data Format (CDF) files. We include three recommended packages that provide the necessary capabilities for working with CDF science data files in Python.

Guide Contents

• Quick Start Guide - CDF with Python
  • Guide Contents
  • CDF Resources
  • Prerequisite Software [If Needed]
    • Install Python
    • NASA CDF Software
  • CDFlib Package
    • Install CDFlib
  • SpacePy Package
    • Install SpacePy
  • CDAS Web Services Package
    • Install CDASWS

CDF Resources

Common Data Format (CDF) is a conceptual data abstraction for storing, manipulating, and accessing multidimensional datasets. CDF is referred to as a data abstraction because the actual physical format in which datasets are stored is opaque. Instead, the structure of the datasets and the means (interface) by which they may be manipulated are described here. NASA’s CDF Homepage contains detailed examples, documentation, FAQs, and installation tools.

Prerequisite Software [If Needed]

Install Python

Users can install Python or one of its various distributions (Anaconda, Miniconda, Conda-Forge, etc.). Python >= 2.6 is required to read CDF files, but Python 3 is the recommended install. It is also recommended to include a package manager (pip, conda, mamba, etc.) in your installation.

Please discuss with your system administrator on which Python distribution to install. This guide provides instructions for the installation of the Miniforge distribution on a Unix-based system. Thus, any commands used for the remainder of this guide will be in reference to this system environment.

Miniforge Install

• Download Installer
  • Miniforge Downloads
  • Choose OS Type & CPU Architecture
• Run Installer

  $ cd <Download_Directory>
  $ sh Miniforge3-<OS_Type>-<CPU_Arch>.sh

  Follow on-screen instructions.

• Verify Python Install
  • Should See (base) in Terminal Prompt
  • Check Initialization

    $ cat ~/.<shell_profile>

    Should be a >>> conda initialize >>> section added to the user's shell profile. If not, run $ /conda init <shell>.

  • Restart Terminal/Session
  • Check Command Paths

    $ which python
    $ which python3

    All results should point to the Miniforge installation path.

NASA CDF Software

Users may need to locally install the NASA CDF software. This software package either comes pre-compiled or can be compiled by the user. For ease of installation, we recommend using the pre-compiled libraries. The process for installing the pre-compiled software is shown below:

CDF Software Install

• Download Installer
  • NASA CDF Software
  • Choose OS
  • Reference AAREADME*.txt
  • Download Installer File
• Install Software
  • Open Installer File
  • Follow On-Screen Instructions
  • Reference AAREADME*.txt [If Needed]
• Add to Shell Profile
  • Open Shell Profile

    $ nano ~/.<shell_profile>

  • Modify the Shell Profile
    • Add Following to File

      # Added for NASA CDF Library
      source <install_path>/bin/definitions.<shell>
      export PATH="<install_path>/bin:$PATH"
      

      Please reference the notes in the definitions.<shell> files that correlate with your specific shell environment.

    • Save Changes
      • WriteOut:
        • control + o
        • enter
    • Close Editor
      • Exit:
        • control + x

CDFlib Package

CDFlib is a pure Python implementation of the CDF libraries. It is capable of reading & writing CDF files and does not require the installation of NASA’s CDF software. CDFlib requires Python >= 3.8 and attempts to replicate the syntax structure of the CDF functions used in IDL®.

Install CDFlib

• Create Virtual Environment

  Here, we utilize the conda manager to build a virtual environment, but venv is another option. Please see the venv documenation, if needed.

  $ conda create -n cdf_cdflib python=3
  $ conda activate cdf_cdflib

  Must run $ conda activate <vir_env> only once per session to switch out from the (base) environment.

• Install Package

  $ conda install cdflib

  Alternatively, CDFlib can be installed via PyPI by using pip instead of conda.

• Verify Install

  • Launch iPython

    $ python

  • Verify CDFlib

    >>> import cdflib
    >>> cdflib.__version__

• Read & Write Example

  • CDFlib Example

SpacePy Package

SpacePy is a heliophysics Python package for basic data analysis, modeling, and visualization. This package includes a set of pre-compiled CDF C libraries but also searches for any locally installed NASA CDF software. SpacePy utilizes these C libraries for better performance and has a Pythonic-style interface for CDFs called pyCDF. CDF files are treated as context managers in pyCDF and behave like a Python dictionary, holding CDF variables that are keyed by each variable’s name. Data pulled from CDFs are loaded into dictionary-like objects called datamodels. More information can be found in the SpacePy Quick Start Guide.

In addition to a CDF interface, SpacePy also includes a suite of other space science libraries that include superposed epoch classes, drift shell tracing, access to magnetic field models, streamline tracing, bootstrap confidence limits, and time & coordinate conversion methods.

Install SpacePy

• Create Virtual Environment

  Here, we utilize the conda manager to build a virtual environment, but venv is another option. Please see the venv documenation, if needed.

  $ conda create -n cdf_spacepy python=3
  $ conda activate cdf_spacepy

  Must run $ conda activate <vir_env> only once per session to switch out from the (base) environment.

• Install Package

  $ pip install spacepy

• Verify SpacePy Install

  • Launch iPython

    $ python

  • Check PyCDF

    >>> from spacepy import pycdf
    >>> pycdf.lib.version
    >>> pycdf.lib.libpath

• Read & Write Example

  • SpacePy Example

CDAS Web Services Package

The Coordinated Data Analysis System (CDAS) is a RESTful web service that is used to access NASA’s non-solar heliophysics data archive, CDAWeb. The CDAS web services package (CDASWS) is an API used to extent the connection to CDAS through Python.

The CDAS Web Services allows software developers to use CDAS in their own applications. Selected variables and time ranges can be easily loaded into Python structures using the CDASWS API. This package is ideal for users that specifically require CDF or netCDF files that are archived at NASA’s Space Physics Data Facility and available in CDAWeb. This API hides the HTTP, JSON/XML, and CDF details of the CDAS Web Services.

Install CDASWS

• Create Virtual Environment

  Here, we utilize the conda manager to build a virtual environment, but venv is another option. Please see the venv documenation, if needed.

  $ conda create -n cdf_cdasws python=3
  $ conda activate cdf_cdasws

  Must run $ conda activate <vir_env> only once per session to switch out from the (base) environment.

• Install CDF Dependencies

  • Install CDFlib

    or

  • Install SpacePy

• Install CDASWS Package

  $ conda install cdasws

  Alternatively, CDASWS can be installed via PyPI by using pip instead of conda.

• Verify Install via Examples

  • Python package data retrieval examples with various data models
  • More CDASWS Examples