2011 June 2 (RMC) - updated to reflect comments by Steve Allen
2011 May 26 (original) Robert M. Candey robert.m.candey@nasa.gov
Current CDF_Epoch time scheme is nominally continuous Gregorian time from 0AD with no leap seconds or defined coordinate system. In practice, it generally holds UTC times, but leap seconds are overloaded onto the first second of the next day. Also mission data generally have ill-defined times, sometimes including leap seconds and sometimes only UTC at start of mission with no leap seconds afterwards. Science data should have well-defined times to enable more accurate cross-comparison between missions and as documentation for future archive use. The leap second issue does not generally affect data before 1972.
The CDF project currently supports specialized time variables: CDF_EPOCH (8-byte float) and CDF_EPOCH16 (two 8-byte floats). CDF_EPOCH is milliseconds from 0AD (a leap year) in an undetermined coordinate system; and CDF_EPOCH16 is seconds from 0AD and picoseconds within that second. At present, the CDF library routines for converting between Epoch and YMD HMS is reversible in the sense that they will return the same HMS as was used on input, but don't allow entering an actual leap second. Simple subtraction of Epochs does not include leap seconds in between. Strictly speaking, CDF Epoch (as it stands) is time from 0AD with no leap seconds, but in practice time values are usually entered in UTC, whether they included leap seconds during the mission or not. Epoch assumes the Gregorian calendar leap year scheme into the past (even though they should probably be Julian days before 1582). Since Epoch conversion routines ignore leap seconds, a leap second and the first second of the next day are set to the same EPOCH value and converting back returns the first second of the next day. It also ignores the varying lengths of seconds in UTC from 1961 to 1972 (both UTC second longer than SI second and leaps added in sub-second units).
GPS, SOHO, THEMIS, C/NOFS and other missions don't add leap seconds, while WMAP, Wind, Geotail, Polar, TRACE, RXTE do. To further complicate things, missions update their spacecraft clocks over a period of time around the leap seconds, as much as days off, and some slew the spacecraft clock over time and some increment at once but not always on time. Missions carry time in CDFs in EPOCH/EPOCH16, ASCII variables as strings, ISTP-like PB5 values, or in double-precision variables for seconds since some base time (such as THEMIS Unix time). These various schemes greatly hinder cross-comparison of data with other missions and hinder use by general data analysis tools (CDAWeb, Autoplot, etc.), requiring custom software development and attendant errors.
CDF will add a new CDF data type, CDF_TIME_TT2000, defined as an 8-byte signed integer with a fixed Time_Base=J2000 (Julian date 2451545.0 TT or 2000 January 1, 12h TT), Resolution=nanoseconds, Time_Scale=Terrestrial Time (TT), Units=nanoseconds, Reference_Position=rotating Earth Geoid. Given a current list of leap seconds, conversion between TT and UTC is straightforward (TT = TAI + 32.184s; TT = UTC + deltaAT + 32.184s, where deltaAT is the sum of the leap seconds since 1960; for example, for 2009, deltaAT = 34s). Accurate conversions from other time scales may require that data providers use the JPL NAIF SPICE or similar library, but data users will easily be able to convert to UTC time. Use of an 8-byte integer provides time with nanosecond resolution for the next 280 years, so data providers will no longer need 16-byte CDF_EPOCH16 variables to carry their highest time resolution (and in half the storage space).
CDF_EPOCH/CDF_EPOCH16 and their conversion routines will continue to be supported (including for times outside the range of CDF_TIME_TT2000) and will have variable attributes for defining the Time_Scale, Reference_Position, Resolution, and Leap_Seconds_Included (as well as Epoch's fixed Time_Base=0AD and Units=milliseconds). These attributes may also be applied to other time variables such as THEMIS times and for returning to mission times (which may lack some leap seconds).
Time_Base: fixed (0AD, 1900, 1970 (POSIX), J2000 (used by CDF_TIME_TT2000), 4714 BC (Julian)) or flexible (provider-defined)
Time_Scale: TT (same as TDT, used by CDF_TIME_TT2000), TAI (same as IAT, TT-32.184s), UTC (includes leap seconds), TDB (same as SPICE ET), EME1950 [default: UTC]
Reference_Position [optional]: Topocenter (local), Geocenter , rotating Earth geoid (used by CDF_TIME_TT2000). Reference_Position is optional metadata to account for time variance with position in the gravity wells and with relative velocity. While we could use a combined TimeSystem attribute that defines mission-specific time scales where needed, such as UTC-at-STEREO-B, it's cleaner to keep them separate as Time_Scale=UTC and Reference_Position=STEREO-B.
Leap_Seconds_Included [required for UTC only]: comma-delimited list (within brackets) of leap seconds included in the form of a lists of ISO8601 times when each leap second was added, appended with the size of the leap second in ISO8601 relative time (+/- time, most commonly: "+1s") [default: standard list of leap seconds up to time of data]. Leap_Seconds_Included is needed to account for time scales that don't have all 34 (in 2009) leap seconds and for the clocks in various countries that started using leap seconds at different times. The full list is required to handle the equally or more common case where a time scale starts at a specific UTC but continues on without leap seconds in TAI mode; this is basically what missions that don't add leap seconds are doing.
$ cat tai-utc.dat | awk 'ORS="," { val = $7 - prev } {prev = $7} { print $1$2"01+" val "s" }'
Leap_Seconds_Included="1961JAN01+1.42282s,1961AUG01-0.05s,1962JAN01+0.47304s,1963NOV01+0.1s,1964JAN01+1.29427s,1964APR01+0.1s,1964SEP01+0.1s,1965JAN01+0.1s,1965MAR01+0.1s,1965JUL01+0.1s,1965SEP01+0.1s,1966JAN01+0.47304s,1968FEB01-0.1s,1972JAN01+5.78683s,1972JUL01+1s,1973JAN01+1s,1974JAN01+1s,1975JAN01+1s,1976JAN01+1s,1977JAN01+1s,1978JAN01+1s,1979JAN01+1s,1980JAN01+1s,1981JUL01+1s,1982JUL01+1s,1983JUL01+1s,1985JUL01+1s,1988JAN01+1s,1990JAN01+1s,1991JAN01+1s,1992JUL01+1s,1993JUL01+1s,1994JUL01+1s,1996JAN01+1s,1997JUL01+1s,1999JAN01+1s,2006JAN01+1s,2009JAN01+1s"
Units [optional]: SI measurement unit: s, ms, ns (10^-9 seconds for 8-byte CDF_TIME_TT2000), ps
Resolution [optional]: using ISO8601 relative time format, for example: 1s. Resolution provides the smallest change in time that is measured.
Absolute_Error [optional]: Absolute or systematic error in same units as Units attribute.
Relative_Error [optional]: Relative or random error in same units as Units attribute to specify the accuracy of the time stamps relative to each other. This is usually much smaller than Absolute_Error.
Bin_Location [optional]: relative position of time stamp to the data measurement bin, with 0.0 at beginning of time bin and 1.0 at end Default is 0.5 for the time at the center of the data measurement. Since clock readings are usually truncated, the real value may be closer to 0.0.
Use of CDF_TIME_TT2000 is substantially the same as for EPOCH. Data providers will add their time variables in SKTeditor or text skeleton file with the new variable type before creating the data CDFs. Data providers are responsible for accurately converting their data times to CDF_TIME_TT2000 time format, via routines such as the SPICE library or using the UTC-to-TT2000 conversion routines provided with the CDF library. The resulting 8-byte integers will be written to the CDFs with the same calls as for other variables. Data users will read the time variables as they do for other variables and then use the CDF library routines for conversion to UTC. The main change will be that the conversion routines will need an up-to-date list of leap seconds; scripts will be provided for updating the list included with the CDF software.
Add data type CDF_TIME_TT2000 read/write to CDF API and code; add routines for converting between TT2000 and UTC times (as strings and time parts); add interfaces for Matlab, IDL, Java and Perl; add to SKTeditor
The recommended FILLVAL is -9223372036854775808LL (hex '8000000000000000'XLL), which is more conveniently entered as 9999-12-31T23:59:59.999999999. Although primarily used internally to the CDF library, the PADVALUE is set to -9223372036854775807LL, which is more conveniently entered as 0000-01-01T00:00:00.000000000. Valid values range from 1707-09-22T12:13:15.145224194 to 2292-04-11T11:46:08.670775807.
Change User Guide section 2.5.4 to "Time Data Types" and add:
"CDF_TIME_TT2000 8-byte signed integer
The CDF_TIME_TT2000 data type is used to store date and time values as a signed integer number of nanoseconds in Terrestrial Time (TT) on the rotating Earth Geoid from J2000 (Julian date 2451545.0 TT or 2000 January 1, 12h TT). The standard format used to display a CDF_TIME_TT2000 value is yyyy-mm-ddThh:mm:ss.ccccccccc, with subseconds down to nanosecond resolution."
Add CDF_INT8 and CDF_TIME_TT2000 to "2.5.5 Equivalent Data Types"
Add CDF_TIME_TT2000 and EPOCH16 to other references of CDF_EPOCH throughout the document.
Besides adding the new data types, CDF_INT8 and CDF_TIME_TT2000, the main changes will be to create new versions of the various time-handling routines, and add conversions between CDF_EPOCH* and CDF_TIME variables. The UTC-related routines require an up-to-date list of leap seconds. This list will be read in on the first call to the routine and stored in a common block or global variable for performance reasons. The read routine could check for an updated list at the standard URLs and compare it to the cached list, although this is best done periodically through a separate script or cron job. An environment variable will be used to point to the list when in a non-standard place and can be used to over-ride the standard list if desired (perhaps to handle conversion back to mission time that only used a subset of leap seconds).
generic time conversion routines for C, Fortran, IDL, Perl, Java, Python, Matlab
CDF_time_to_UTC_parts returns arrays of time parts in UTC from an input array of CDF_TIME_TT2000 values. Fortran and C version has scalar parameters only (no arrays).
CDF_time_to_UTC_parts(cdf_time_var, year, [month, day, hour, minute, second, millisecond, microsecond, nanosecond])
with input parameter time as CDF_int8[ ] and all output parameters in CDF_REAL4[ ] (should there be a version for CDF_REAL8?).
The various output parameters are almost all optional with the last parameter receiving the remainder as fractional units [Mike is this doable?]. No easy way to get day-of-year rather than day-of-month? Fractional month is not allowed as ill-defined. Output parameters (when provided) have the usual constraints (year[1707-2292], month[1-12], day[1-31 or 1-366], hour[0-23]. minute[0-59], second[0-60], millisecond[0-999], microsecond[0-999], nanosecond[0-999]).
CDF_time_from_UTC_parts returns an array of CDF_TIME_TT2000 values computed from input arrays of time parts in UTC. Fortran and C version has scalar parameters only (no arrays).
CDF_int8[ ] CDF_time_from_UTC_parts(year, [month, day, hour, minute, second, millisecond, microsecond, nanosecond])
with input parameters all in CDF_REAL4[ ] (should there be a version for CDF_REAL8?) as fractional units (for instance year can include fractional years as well). Fractional month is not allowed as ill-defined.
The various input parameters are almost all optional and accumulate appropriately. In particular, subsecond variables (millisecond, microsecond, nanosecond) are optional and usually second with fractional seconds is used if the variable has enough resolution. Basically the input values would sum up into nanoseconds (adding leap seconds as needed) after multiplying by the appropriate constants. For the special case of month=0, day becomes day-of-year rather than day-of-month.
Example calls:
time = CDF_time_from_UTC_parts(1970., 1., 1., 0., 0., seconds_input) # allows seconds from Unix beginning
time = CDF_time_from_UTC_parts(2000., 0., modified_julianday-0.5) # use modified Julian day times
We could have a CDF_time_from_UTC_parts_strict routine that has usual constraints (year[1707-2292], month[1-12], day[1-31 or 1-366], hour[0-23]. minute[0-59], second[0-60], millisecond[0-999], microsecond[0-999], nanosecond[0-999]), possibly all in integers; but probably better to leave to users to check unless this is too hard to code.
CDF_time_to_UTC_string returns arrays of time strings in UTC from an input array of CDF_TIME_TT2000 values. Fortran and C version has scalar parameters only (no arrays).
CDF_char[ ] CDF_time_to_UTC_string(cdf_time_var, [time_format, time_resolution])
Same as CDF_time_to_UTC_parts but returning time as a string in a standard format or user specified time_format (if we use a standard library). If provided, Time_resolution provides how much resolution to compute, specified using ISO8601 relative time format (for example: 1s). If an illegal field is detected, the value returned will be ILLEGAL_EPOCH_VALUE.
time_format values (default is 3)
0 DD-Mon-YYYY
hh:mm:ss.ccccccccc
1 YYYYMMDD.ttttttt
2 YYYYMMDDhhmmss
3 YYYY-MM-DDThh:mm:ss.cccccccccZ
(The characters T and Z are the CDF_EPOCH type 3 place holders)
where:
DD the
day of the month (1-31)
Mon the abbreviated month name: (Jan, Feb, Mar,
Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, or Dec)
MM the month number (1-12)
YYYY the year (A.D.)
hh the hour (0-23)
mm the minute (0-59)
ss the second (0-59)
cccccccccc fractional seconds
The SPICE library handles more time string formats <https://naif.jpl.nasa.gov/pub/naif/toolkit_docs/C/req/time.html> <https://Naif.jpl.nasa.gov/pub/naif>.
CDF_time_from_UTC_string returns arrays of CDF_TIME_TT2000 values computed from arrays of time strings in UTC. Fortran and C version has scalar parameters only (no arrays).
CDF_int8[ ] CDF_time_from_UTC_string(time_string, [time_format])
Same as CDF_time_from_UTC_parts but computing from time strings in a standard format or user specified time_format (if we use a standard library). If an illegal field is detected, the value returned will be ILLEGAL_EPOCH_VALUE. time_format is same as for CDF_time_to_UTC_string.
CDF_time_from_UTC_EPOCH returns arrays of CDF_TIME_TT2000 values computed from arrays of CDF_EPOCH values in UTC. Fortran and C version has scalar parameters only (no arrays).
CDF_int8[ ] CDF_time_from_UTC_EPOCH(CDF_EPOCH[ ] epoch_time)
Same as CDF_time_from_UTC_parts but computing from CDF_EPOCH values, assumed to be in UTC.
perhaps provide similar routine for CDF_EPOCH16:
CDF_int8[ ] CDF_time_from_UTC_EPOCH16(CDF_EPOCH16[2, ] epoch_time)
CDF_time_to_UTC_EPOCH returns arrays of CDF_EPOCH values in UTC computed from arrays of CDF_TIME_TT2000 values. Fortran and C version has scalar parameters only (no arrays).
CDF_EPOCH[ ] CDF_time_to_UTC_EPOCH(CDF_int8[ ] cdf_time_var)
Same as CDF_time_to_UTC_parts but computing to CDF_EPOCH values in UTC.
perhaps provide similar routine for CDF_EPOCH16
Compare two arrays of CDF_TIME_TT2000 values and return array of integers, where 1 if the value of time1 is greater (a later date and time) than time2, 0 if same, and -1 if time1 is less than time2.
CDF_int2[ ] Result = CDF_time_compare(time1, time2)
Return difference in time between two input arrays of CDF_TIME_TT2000 values, time1 - time2 in nanoseconds.
CDF_int8[ ] Result = CDF_time_difference(time1, time2)
Perhaps add optional resolution parameter to specify rounding to seconds, milliseconds or microseconds.
Ideas for a generic CDF time object scheme that perhaps could subsume EPOCH and EPOCH16 also, and perhaps use SPICE library to handle many time schemes.
new (arraySize): create new CDF_TIME object with arraySize values (default to 1)
read (fileID, [varName]): read CDF_TIME variable, varName (default to "time") from CDF file opened with fileID
write (fileID, [varName]): write CDF_TIME variable, varName (default to "time") from CDF file opened with fileID
compareTo(cdf_time2): return array of -1 (less), 0 (same), +1 (more) values for self object compared to cdf_time2 object
difference(cdf_time2, [/seconds, /milliseconds, /microseconds, /nanoseconds]): return array of (self - cdf_time2) in seconds, milliseconds, microseconds, nanoseconds (default)
toString([timeFormat, Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base]): convert times to string of timeFormat (defaults to ISO8601 form) in Time_Scale (defaults to UTC) from Time_Base with Leap_Seconds_Included (defaults to standard list from NIST) with Resolution (defaults to "+1s")
fromString: same as above in reverse.
toBreakdown(year, month, day, hour, minute, second, millisecond, microsecond, nanosecond, Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base): same as toString but returns arrays of time parts. Last given part contains fractional values for remaining Resolution.
fromBreakdown: same as toBreakdown in reverse; parts can be zero or large and time will sum appropriately.
toTime(Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base): returns one-dimensional array of times in Time_Scale (useful for J2000, POSIX time, etc.)
fromTime: same as toTime in reverse
block older CDF software with new magic number if needed, else CDF reading routines will not recognize the new data type
encode checks second=60 against the leap seconds list
how to cache leap second list (common block)
how to update leap second list regularly (scripts, check via URL when connected to network, perhaps warn user to run small utility to update)
CDF version 3.3.2 will include the basic support and routines for CDF_TIME_TT2000 and CDF_INT8 variable types, for testing and feedback; expected in mid 2011.
CDF version 3.4 will have full support for the CDF_TIME_TT2000 and CDF_INT8 variable types; expected late 2011 or early 2012.
*************************************************************************
There are more ways of measuring time (calendars and time scales) than you might think. Basically UT1 follows the Earth's rotation and UTC stays within 0.9 seconds using leap seconds, and TAI is atomic time with no leap seconds. Countries use various versions of UTC and GMT as their time basis <http://www.ucolick.org/~sla/leapsecs/epochtime.html>. The US proposes to stop adding leap seconds <http://www.ucolick.org/~sla/leapsecs/USSG7.pdf>. There are many time systems in use and it's not clear what exact time system is used on each mission. GPS, SOHO, THEMIS, C/NOFS and other missions don't add leap seconds, while WMAP, Wind, Geotail, Polar, TRACE, RXTE do. Some software packages do and some don't. Strictly speaking, CDF Epoch (as it stands) has no leap seconds from 0 AD, but in practice time values were basically entered in UTC, whether they included leap seconds during the mission or not.
Mostly, people assume they know what is meant by a specific time measurement, when in fact time scales and units have varied greatly over the years and even now still come in many flavors. Heliophysics missions are similarly inconsistent, with some using the spacecraft counter, others use a running clock, and other upload corrections to time including leap seconds. To further complicate things, missions update their spacecraft clocks over a period of time around the leap seconds, as much as days off, and some slew the spacecraft clock over time and some increment at once but not always on time. These complications are too confusing to document in a standard way inside the CDFs but should be covered in mission documentation. For better accuracy, the Leap_Seconds_Included list could have the times when each leap second was fully added to the spacecraft clock.
Because missions use different time bases, with and without leap seconds, we could either document what time systems are used where possible (and enable entering leap seconds) (with their attendant complexities), or develop a fixed time system suitable for archiving and easy reading, as proposed here.
At least with most metric units, there is a standard specification for each, but not for time. Even ignoring many local times and summer time shifts, time scales include:
Universal Time (UT): based on the rotation of the Earth on its axis (mean solar day)
Ephemeris Time (ET): based on the revolution of the Earth in its orbit around the Sun
Atomic Time (AT): based on the quantum mechanics of the atom
UT1: true measure of the Earth's rotation, corrected for polar motion and used in celestial navigation
UT2: UT1 corrected for the seasonal variation due to irregularities in the Earth's rotation
International Atomic Time (TAI): time scale maintained by the BIPM
Coordinated Universal Time (UTC): atomic time scale at rate of TAI but kept within 0.9 s of UT1 by leap seconds as decided by International Earth Rotation Service (IERS) (current civil time)
Terrestrial Dynamical Time (TDT) or Terrestrial Time (TT): like ET with origin on the Earth geoid, (TAI + 32.184 s) <http://en.wikipedia.org/wiki/Terrestrial_Time>
GPS Time (GPST): origin at midnight of 1980 January 5/6 so TAI is constantly ahead of GPS Time by 19 s, and UTC currently behind by 15s. GPS time has an extra field for the current leap second count since 1980 (when there were only 9 leap seconds) and a field for future leap seconds. GPS time accumulates seconds since 1980 in the form of truncated weeks (week mod 1024) and time of week.
Temps International (TI): proposed new time system without leap seconds
EME1950 - seconds since 1950 Jan 1 00:00:00 in Earth Mean Equator.
J2000 - seconds since 2000 Jan 1 12:00:00 in Earth Mean Equator
Julian Date (JD) - days and fractions since noon January 1, 4714 BC (Julian calendar)
Network Time Protocol (NTP) time is a 32-bit count of seconds since 1900 (with a 32-bit subsecond part) with 86400 logical seconds per day. On the leap second, NTP time subtracts 1 second. NTP time tracks GPS time minus the leap seconds.
POSIX time is seconds since UTC 1970 January 1, not counting leap seconds. The signed 32bit implementation will overflow on 2038-01-19T3:14:08Z. Unix time is the realization of Posix time on Unix (and similar) computers. It has 86400 logical seconds per day. It tracks NTP time (NTP time minus 2208988800) except on the leap second, Unix time often counts then uncounts the leap seconds, stops for a second, or slews the computer clock for a bit.
CDF_EPOCH: time from 01-Jan-0000 00:00:00.000, no leap seconds, so 19s behind GPS, 34s behind TAI, 12s? behind Tplot epoch
Barycentric Coordinate Time (TCB) at the geocenter
Barycentric Dynamical Time (TDB): origin at center of the solar system (often used for ephemeris). TDB is defined as the following linear transformation of TCB: TDB = TCB − LB x (JDTCB − T0) x 86400 + TDB0, where T0 = 2443144.5003725, and LB = 1.550519768x10−8 and TDB0 = − 6.55 x 10−5 s are defining constants. <http://www.iau.org/static/resolutions/IAU2006_Resol3.pdf>
Olson right time is the other time scale in the Olson library (other being the commonly used Posix time). Olson right time provides a realization of UTC, with a SI second count since 1970 as a 32/64-bit value, that be converted to/from UTC even on a leap second. Olson right time is GPS time plus 9 seconds.
UTC-SLS is Unix time without a 1 second pause. Instead, time adjusts with a well-defined gradient over a course of time near the leap second.
Many more Time Scales at <http://www.ucolick.org/~sla/leapsecs/timescales.html>.
In addition, time varies with gravity and speed relativisticly, and so properly should include documenting location and velocity, but the relativistic effects are small for most current missions.
The CDF project supports specialized time variables: CDF_EPOCH (8-byte float) and CDF_EPOCH16 (two 8-byte floats). CDF_EPOCH is milliseconds from 0AD (a leap year) in an undetermined coordinate system and CDF_EPOCH16 is seconds from 0AD and picoseconds within that second. The following routines are available for conversion:
computeEPOCH[16] calculates a CDF_EPOCH[16] value given the time individual components
EPOCH[16]breakdown decomposes a CDF_EPOCH[16] value into the time individual components
encodeEPOCH[16]* encodes a CDF_EPOCH[16] value into various date/time character string formats
parseEPOCH[16]* parses various date/time character formats and returns a CDF_EPOCH value
CDF routines in IDL:
CDF_EPOCH[16] procedure computes or breaks down CDF_EPOCH[16] values
CDF_ENCODE_EPOCH[16] function encodes a CDF_EPOCH[16] variable into a string
CDF_EPOCH_COMPARE function compares two epoch (date and time) values
CDF_EPOCH_DIFF function compares two epoch (date and time) values and returns the difference in milliseconds (default) or microseconds
CDF_PARSE_EPOCH[16] function parses a properly-formatted input string into a double-precision value
Matlab
?
Each mission has unique constraints on their internal time keeping due to onboard hardware and processes. Some spacecraft have clocks set to UTC and others have free-running timers that must be calibrated periodically to UTC. Missions adjust for leap seconds in various ways. The resulting data are generally tagged with UTC times eventually. Following are notes on internal time handling for some heliophysics missions.
WMAP onboard UTC clock, leap second field updated as single stored cmd, clock error and drift adjusted weekly (+- 15ms)
Wind (+- 10 ms), leap second adjusted in ten 100ms steps over 2 hours to minimize impact on attitude sensors and stored commands
SOHO no leap seconds, uses TAI (actually 6-byte onboard clock)
TRACE leap second adjustment
RXTE leap second
ACE no onboard UTC clock, just counter, leap seconds added in gnd processing
FDF uses Timing coefficients file; used predicted leap seconds from 1998-2002 so off by 1-2 sec
Geotail divides data files at leap seconds. That is, one file ends at XX:59:60, and the other file starts at XX:00:00
THEMIS going with non-leap seconds across the board
C/NOFS no leap seconds
STEREO uses UTC
/* CCSDS epoch is 1 Jan 1958 */
/* PB5J epoch is 24 May 1968 */
/* time_t epoch is 1 Jan 1970 */
/* HST epoch is 25 Apr 1990 */
/* TRMM epoch is 1 Jan 1994 */
/* XTE epoch is 1 Jan 1994 (for B2 patch) */
/* TRACE epoch is 24 May 1968 */
/* SWAS epoch is 24 May 1968 */
Two-Line orbital Elements (TLE) use time since the epoch (TSE) of the file, so orbit propagation software should add any leap second that occurred since the epoch of the TLE file. Sardi's models and SSCweb internally have no leap seconds; FDF presumably accounts for leap seconds. STK uses leap seconds.
Space Physics Coordinate Transformations: transforming from inertial coordinates (as used for spacecraft orbits) to coordinate systems spinning with the Earth requires UTI as a measure of the spin phase of the Earth. Leap seconds are used to keep UTC within 0.9 seconds of UT1 (a spin phase error of less than 6.5e-5 radians). Do coordinate transformation software automatically assume leap seconds? If so, dropping leap seconds (as recently proposed to the ITU) will require obtaining the difference between true UT1 and whatever time system is used. Mike Hapgood on coordinate transforms <http://www.mssl.ucl.ac.uk/grid/iau/extra/local_copy/SP_coords/leap_secs_in_sp.htm> from <http://www.mssl.ucl.ac.uk/grid/iau/extra/local_copy/SP_coords/ct_home.htm>
Perhaps the best source code for time conversions is in the JPL SPICE system <http://naif.jpl.nasa.gov/naif/>
<http://naif.jpl.nasa.gov/pub/naif/toolkit_docs/C/req/time.html> (also nice string conversion routine)
command line tool chronos at <https://naif.jpl.nasa.gov/pub/naif/utilities>
<http://www.cv.nrao.edu/~rfisher/Ephemerides/computer_code.html>
<http://www.lsc-group.phys.uwm.edu/~ballen/grasp-distribution/GRASP/src/utility/utctime.c>
NOVAS library is unencumbered <http://aa.usno.navy.mil/software/novas/novas_info.php>
IAU Standards of Fundamental Astronomy (SOFA) <http://www.iausofa.org/>
<http://www.iau-sofa.rl.ac.uk/2001_0331/Timescales.html>
<http://hpiers.obspm.fr/eop-pc/earthor/utc/TAI-UTC_tab.html> <http://hpiers.obspm.fr/eop-pc/earthor/utc/UTC-offsets_tab.html>
<http://maia.usno.navy.mil/ser7/tai-utc.dat>
<https://naif.jpl.nasa.gov/pub/naif/generic_kernels/lsk/naif0012.tls>
A historical tabulation of leap-seconds: <http://hpiers.obspm.fr/eoppc/bul/bulc/UTC-TAI.history>
IERS Conventions <http://www.iers.org/nn_11216/IERS/EN/Publications/TechnicalNotes/>
TEMPO 2 reducing pulsar timing observations <http://www.atnf.csiro.au/research/pulsar/tempo2/>
<http://tycho.usno.navy.mil/systime.html>
<http://tycho.usno.navy.mil/leapsec.html>
<http://www.cv.nrao.edu/~rfisher/Ephemerides/times.html>
<http://www.stjarnhimlen.se/comp/time.html>
<http://www.mssl.ucl.ac.uk/grid/iau/extra/local_copy/SP_coords/leap_secs_in_sp.htm>
<http://www.ucolick.org/~sla/leapsecs>
<http://www.ucolick.org/~sla/leapsecs/timescales.html>
<http://en.wikipedia.org/wiki/Leap_second>
proposed POSIX time API <http://www.cl.cam.ac.uk/~mgk25/time/c/>
<http://leapsecond.com/java/gpsclock.htm>
NVO Time coordinate metadata <http://www.ivoa.net/Documents/WD/STC/STC-20040723.html>
<https://heasarc.gsfc.nasa.gov/docs/xte/abc/time_tutorial.html>
Standish, E. M., "Time scales in the JPL and CfA ephemerides", Astronomy and Astrophysics, v.336, p.381-384 (1998) <http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1998A%26A...336..381S&defaultprint=YES&filetype=.pdf>
Fukushima, T., "Time ephemeris", Astronomy and Astrophysics, vol. 294, no. 3, p. 895-906, 02/1995 <http://adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=1995A%26A...294..895F&link_type=ARTICLE&db_key=AST&high=>
USNO <http://aa.usno.navy.mil/publications/docs/Circular_179.php>
Space-Time Coordinate Metadata for the Virtual Observatory <http://www.ivoa.net/Documents/latest/STC.html>
<http://www.merlyn.demon.co.uk/datelinx.htm>
<http://www.merlyn.demon.co.uk/leapsecs.htm>
IAU Time Commission 31 <http://www.atnf.csiro.au/iau-comm31/activities.php>
"The Debate over UTC and Leap Seconds" <http://www.agi.com/downloads/resources/user-resources/downloads/whitepapers/DebateOverUTCandLeapSeconds.pdf>
IAU Nomenclature for Fundamental Astronomy (NFA) <http://syrte.obspm.fr/iauWGnfa/>
R A Nelson , D D McCarthy , S Malys , J Levine , B Guinot , H F Fliegel , R L Beard and T R Bartholomew, "The leap second: its history and possible future", Metrologia Vol. 38, #6, p. 509 (2001), doi:10.1088/0026-1394/38/6/6<http://www.atnf.csiro.au/iau-comm31/pdf/Leap%20Second.pdf>
Mailing list on dropping leap seconds: <http://six.pairlist.net/mailman/listinfo/leapsecs>
NASA has proposed a fall back recommendation making GPS time (with no leap seconds) a standard interval time scale for precision timekeeping projects, while leaving UTC alone (proposal no longer online).
A history of the proposal to eliminate leap-seconds oriented against the proposal: <http://www.ucolick.org/~sla/leapsecs/nc1985wp7a.html>
VALUES FOR TIME_SYSTEM AND REFERENCE_FRAME (NORMATIVE)
The values in this annex represent the set of acceptable values for the TIME_SYSTEM and REFERENCE_FRAME keywords in the OPM, OMM, and OEM. (For details and description of these time systems, see reference [G1].) If exchange partners wish to use different settings, the settings should be documented in the ICD.
A1 TIME_SYSTEM METADATA KEYWORD
If MET or MRT is chosen as the TIME_SYSTEM, then the epoch of either the start of the mission for MRT, or of the event for MET, should either be given in a comment in the message or provided in an ICD. The time system for the start of the mission or the event should also be provided in the comment or the ICD. If these values are used for the TIME_SYSTEM, then the times given in the file denote a duration from the mission start or event. However, for clarity, an ICD should be used to fully specify the interpretation of the times if these values are to be used. The time format should only utilize three digit days from the MET or MRT epoch, not months and days of the months.
GMST Greenwich Mean Sidereal Time
GPS Global Positioning System
MET Mission Elapsed Time (note)
MRT Mission Relative Time (note)
SCLK Spacecraft Clock (receiver) (requires rules for interpretation in ICD)
TAI International Atomic Time
TCB Barycentric Coordinate Time
TDB Barycentric Dynamical Time
TCG Geocentric Coordinate Time
TT Terrestrial Time
UT1 Universal Time
UTC Coordinated Universal Time
TDB or TT can be approximated by TDB=UTC+32.184s+<leapseconds>; TT and TDB differ by <2ms over the year
ETR time (Sysplex Timer time) based on running counter of 1/4096 microseconds from 1900 (originally 64bit (expires 2041) and later 128bit), UTC, Local Time
leap seconds p134 "IBM′s recommendation is to set the leap seconds to zero unless you require a leap second specification"
Date class with milliseconds from 1970; allows 60 and 61 seconds, but leap seconds seem to ignored otherwise <http://download.oracle.com/javase/6/docs/api/java/util/Date.html>
compareTo(Date anotherDate) Compares two Dates for ordering
Calendar <http://download.oracle.com/javase/6/docs/api/java/util/Calendar.html>
When a Calendar is in lenient mode, it accepts a wider range of calendar field values than it produces. When a Calendar is in non-lenient mode, it throws an exception if there is any inconsistency in its calendar fields.
Calendar getTime, getTimeInMillis, setTime, setTimeInMillis, add, setLenient
DateFormat <http://download.oracle.com/javase/6/docs/api/java/text/DateFormat.html>
Proposed JavaX.Time <http://threeten.sourceforge.net/apidocs/javax/time/package-summary.html>
Uses the UTC-SLS mapping from UTC to guarantee 86400 seconds per day(stretching over 1000 seconds), leap seconds only after 1972 (prior assumes 10s offset)
Duration class <http://threeten.sourceforge.net/apidocs/javax/time/Duration.html>
nanoseconds between two Instants (seconds in long and ns in int for 96bits total)
UTCRules <http://threeten.sourceforge.net/apidocs/javax/time/UTCRules.html>
Handles leap seconds: convertToTAI, convertToUTC, getLeapSecondDates, getLeapSecondAdjustment, get TAIoffset, registerLeapSecond (add to table)
License: "As is" with credit encouraged <http://naif.jpl.nasa.gov/naif/rules.html>
Time stored in floating double seconds from J2000 (2000 JAN 01 12:00:00 or 2000 JAN 1.5) in Barycentric Dynamical Time (TDB) or Ephemeris Time (ET).
Chronos command line utility <http://naif.jpl.nasa.gov/pub/naif/toolkit_docs/IDL/ug/chronos.html>
% CHRONOS -SETUP <setup file name OR kernel file name(s)> -FROM <"from" time system> [-FROMTYPE <"from" time type>]
-TO <"to" time system> [-TOTYPE <"to" time type>] [-FORMAT <output time format picture>] -TIME <input time> | -BATCH
[-SC <sc ID>] [-CENTER <central body ID>] [-LANDINGTIME <UTC time of landing>] [-SOL1INDEX <index of first SOL>] [-NOLABEL] [-TRACE]
Time conversion routines <http://naif.jpl.nasa.gov/pub/naif/toolkit_docs/IDL/ug/tictoc.html>
<http://naif.jpl.nasa.gov/pub/naif/toolkit_docs/IDL/icy/cspice_str2et.html>
<http://naif.jpl.nasa.gov/pub/naif/toolkit_docs/IDL/cspice/et2utc_c.html>
"This routine does not attempt to account for variations in the length of the second that were in effect prior to Jan 1, 1972. For days prior to that date, we assume there are exactly 86400 ephemeris seconds. Consequently the UTC Gregorian calendar strings produced for epochs prior to Jan 1, 1972 differ from the corresponding TDB calendar strings by approximately 41.18 seconds. (TDB Gregorian calendar strings are produced by the routine ETCAL)."
Valid EPOCHs are Y1966 (Cline Time) and Y2000 (Ephemeris Time). Time, a real*8 value that is the number of seconds since Jan 1, 1966 at 00:00:00.000 for Cline Time or Jan 1, 2000 at 00:00:00.000 for Ephemeris Time.
<http://www-ssc.igpp.ucla.edu/~leel/cassini/doc/flatfiles/programmerguide/libc/time_igpp.doc>
"Important!!! The cline time system does not account for leap seconds! However, Ephemeris Time Does. There should be 32 second difference between Cline Time And Ephemeris Time at Jan, 01, 2000."
IAU generally use J2000, defined as (TT - 2000 January ld 12h TT) in days/36525 defined at the geocenter and at the date 2000.0 January 1.5 TT = Julian Date 2451545.0 TT.
<http://starlink.jach.hawaii.edu/docs/sun211.htx/node657.html#xref_TimeFrame>
FITS generally use Modified Julian Date, in a variety of formats. Strictly, the Epoch value should be supplied in the TDB timescale. Elsewhere it says: if the current TimeFrame system is "Besselian epoch" the default is "TT", otherwise it is "TAI".
Provider-defined Time_Base allows matching mission schemes (such as THEMIS) and potentially allows using smaller variable size for high time resolution, although adds complexity in merging files within a dataset (if Time_Base is short-term, such as daily) and definitely between datasets (have to compute common Time_Base). Using a standard fixed Time_Base greatly simplifies software and user understanding, plus doesn't require passing the attribute to conversion routines. Choice of fixed Time_Base is mostly arbitrary with 0AD being current Epoch scheme, so may want a different Time_Base to distinguish if Epoch variable type is used. For more limited variable size, Time_Base has to be closer to times of data.
Bobby's assertions: A1: Time_Base should be fixed unless there's real need to save storage size by using 4 or 8 byte variable as offset from Time_Base. Actual value of Time_Base is not important if using 16-byte time or even 8 bytes for nanosecond or courser resolution.
Provider-defined Time_Scale allows matching mission schemes and captures this in metadata, but greatly complicates user understanding and conversion software. Generic reading code would need to accurately convert many different Time_Scales with their attendant nuances and inconsistencies, which is particularly difficult to do in the number of commonly-used languages (Java, C, IDL, etc.). Choice of which fixed Time_Scale to use is a more difficult decision. UTC is used by some missions but not all, and requires passing lists of leap seconds used to conversion routines. Tracking leap seconds in future data and computations is especially tricky. TT/TDT is the most straightforward and doesn't require passing as an attribute to other routines. NAIF SPICE and other libraries use TDB/ET internally, but not all missions are currently supported. TT and TDB also greatly simplify calculations such as time differences and dataset comparisons (no leap seconds to track separately). TDB requires complex code to accurately convert to UTC, while UTC and TT are related by a fixed amount plus leap seconds. If Time_Scale=UTC then always need Leap_Seconds_Included list.
A2: Time_Scale should be fixed for simplicity and to reduce software development and testing. Should be TT for simplicity (no list of leap seconds required and easy to compute differences), best long term archive understandability, and ease of converting to UTC (given list of leap seconds). Data providers may need SPICE routines to accurately their mission times convert to TT, but need only the list of leap seconds for converting from UTC times.
If Time_Base=UTC, then Leap_Seconds_Included is required to accurately account for the leap seconds actually used in the times. For instance, some missions used UTC at start of mission but add no more leap seconds. Predictive (future) data is written without future leap seconds since they aren't defined at time of writing (more than 6 months ahead), but later users need to know how to correct those times to true UTC afterwards.
Provider-defined Resolution allows optimizing resolution to variable size, but complicates software. We could continue with two resolutions for 8 and 16 byte variables. Fractional seconds since Time_Base simplifies software and allowing easier tradeoff of resolution between 8 and 16 byte versions, but at loss of range due to 11 bits used for the exponent. Using whole units enables using all bits but 1 (for sign) for range, and 8bytes is sufficient for nanoseconds over 584 years. Nanoseconds are sufficient for recording any practical time measurement, since even the best clocks have accuracy on order of a few nanoseconds. High-resolution examples include Polar PWI CDF data recorded at under 2 microseconds and sounding rocket data with 100 nanosecond resolution. The SPICE library uses ET in 8 byte float of fractional seconds from J2000, so the range providing microsecond resolution is ±143 years. Conversion from integer nanoseconds to SPICE-like floating point will lose some resolution (11 bits), but is only significant for resolutions finer than microseconds. Time_Base should be selected to maximize spacecraft data range of 1957 onwards. Older data can use the existing Epoch scheme with millisecond resolution, especially as leap seconds are not defined before 1972 and times are less and less accurate further into the past.
Range from Time_Base in days/years:
|
Resolution |
4byte Integer |
8byte Integer |
16byte Integer |
8byte float (EPOCH16 and SPICE ET) |
|
days |
±2.1e9 days / 5.9e6 years |
±9.2e18 / 2.5e16 |
±1.7e38 / 4.6e35 |
|
|
seconds |
±24855 / 68 |
±1.1e14 / 2.9e11 |
±2.0e33 / 5.4e30 |
±5.2e10 days / 1.4e8 years |
|
milliseconds |
±24 days |
±1.1e11 / 2.9e8 |
±2.0e30 / 5.4e27 |
±5.2e7 / 142710 |
|
microseconds |
|
±1.1e8 / 292271 |
±2.0e27 / 5.4e24 |
±52125 / 143 |
|
nanoseconds |
|
±106752 / 292 |
±2.0e24 / 5.4e21 |
±52 days |
|
picoseconds |
|
±107 days |
±2.0e21 / 5.4e18 |
|
A3: Use 8 bytes of fixed nanoseconds as Resolution for a range of 584 years. Time_Base of J2100 provides a time range between 1808 and 2392, and J2000 between 1708 and 2292. J2000 is consistent with the SPICE ET definition (although more resolution than ET).
Choice of variable size is only important to conserve storage space and memory. Compression (for large blocks) reduces this effect and having a single size simplifies CDF and user codes. Unsigned integer might be clearer (time only forward from Time_Base) but may not be supported on all operating systems and compilers.
A4: Use 8 byte signed integer as sufficient for nanosecond resolution on a reasonable time range.
Overloading the current Epoch variable type with attributes reduces CDF coding somewhat, but risks confusing users and providing wrong values when using the old routines or with attributes not passed properly. Storing data with times in a new variable type (CDF_TIME) forces users to use the updated CDF library and conversion routines, but its use is optional for data providers satisfied with the existing Epoch scheme.
A5: Should declare new time variable type to clearly distinguish from old scheme.
Bobby's assertions A1-A5 leads to proposed solution at top (same as Solution S11 below).
Time data and associated software should meet the following requirements and considerations, as required by these communities: data providers (scientists and supporting staff), current data users, future users of archived CDFs, SPDF developers (CDF and CDAWeb), and outside developers (commercial and individual display and analysis packages).
R1: Record times accurately in CDFs (including needed metadata and nano or picosecond precision possible), accounting for leap seconds properly [software run by data provider]. If needed, software needs to update when leap seconds are added (perhaps check standard lists online when opening a CDF for input).
Affects: Time_Base needs to be well defined for providers and users; provider-defined Time_Scale greatly complicates time accuracy and understanding; Leap_Seconds_Included is required if Time_Scale=UTC and at least required for time conversions; Resolution, variable size, and time range need to be sufficient to capture times accurately.
R2: Unambiguously record the time scale and time base with the CDF times [added by data provider if not defined by this standard].
Affects: need clear definition of Time_Scale, Leap_Seconds_Included and Time_Base and inclusion of metadata where not fixed in the time standard.
R3: Record uncertainties in mission time measurements and time scale [added by data provider, similar requirement for all variables].
Affects: add metadata, including Time_Resolution, Time_Scale and Leap_Seconds_Included.
R4: Convert times in various time scales to the CDF time system, with leap seconds as needed (including mission times with no leap seconds since launch) [software run by data provider].
Affects: provide time conversion routines based on SPICE library and document time issues well for data provider education.
R5: Output times in various time scales accurately, with leap seconds as needed [software run by data users].
Affects: provide time conversion routines based on SPICE library.
R6: Record times in CDFs in storage-efficient manner (including compression on 16-byte time arrays) [CDF library issue].
Affects: use compression and 8 byte integers or allow multiple Time_Base and smaller? variables.
R7: Read/Write times in CDFs in computation- and memory-efficient manner [CDF library issue].
Affects: test and improve CDF library and time conversion code; discourage use of 16 byte variables.
R8: Store each time variable as stand-alone object with its own metadata, to enable multiple time variables per file and merging or splitting files, including between different datasets [CDF library issue]. This excludes storing metadata in CDF structural info; not the same as array size and compression info used by CDF internals.
Affects: encourages use of fixed scheme for internal time storage and allow use of extra metadata for documentation.
R9: Use well-tested software routines for time conversion and storage (due to many time nuances, including leap seconds and various time scales) [CDF library issue].
Affects: encourage use of existing well-tested time routines, such as SPICE library.
R10: Minimize software development and testing for CDF team and for all users.
Affects: encourage straightforward and simple definitions, and use of existing well-tested time routines, such as SPICE library.
R11: Develop clear time scheme that can be correctly implemented in various software languages [CDF-internal, data provider software, and long-term data user software and interpretation].
Affects: encourages use of fixed scheme for internal time storage.
R12: Minimize changes for data providers and especially data users, from current time system and ISTP/SPDF guidelines. Preserve existing Epoch interpretation and software API.
Affects: encourages reuse of existing Epoch scheme, but practically not possible. New time format requires new CDF library at minimum (including updates to IDL and Matlab).
R13: Clearly record future times (as in predictive and model data), distinguishing where possible leap seconds are unknown at time of writing into the CDF (future times are indeterminate with respect to leap seconds), to prevent errors in later interpretation.
Affects: encourages use of non-leapsecond time scale or requires Leap_Seconds_Included metadata be carried and used.
R14: Clearly distinguish between current Epoch usage and the proposed time system, to prevent errors in later interpretation.
Affects: encourages use of new time variable or at least changing the Time_Base from 0AD, where times in new format will have wrong century.
R15: Store table of leap seconds with each time variable for self-sufficiency and to define unambiguous times when and if leap seconds were used [CDF library issue].
Affects: requires Leap_Seconds_Included metadata be carried and used or use non-leapsecond time scale.
R16: Times should be easy to merge between CDF files and between datasets.
Affects: discourages complex time schemes and custom Time_Bases.
R17: Object-oriented APIs can hide complexities of time schemes.
Affects: development of new libraries in C, Fortran, IDL, Perl, Python, Matlab, etc.
how to handle predict data and missing or offset leap seconds?
how to handle leap seconds coming at different times for each mission in conversion?
improved handling of sparse data or high precision large runs (such as for 1000 values/second)?
|
|
Time_Base |
Time_Scale |
Leap_Seconds_Included |
Resolution |
VarSize |
NewVarType |
Other |
|
S1 |
0AD |
UTC |
included |
ms (ps for Epoch16) |
8(16) |
no |
|
|
S2 |
1900 |
UTC |
included |
ms (ps for Epoch16) |
8(16) |
no |
|
|
S3 |
1970 |
TDT |
none |
ms (ps for Epoch16) |
8(16) |
no |
|
|
S4 |
1900 |
TDT |
none |
ms (ps for Epoch16) |
8(16) |
no |
|
|
S5 |
0AD |
? |
if needed |
ms (ps for Epoch16) |
8(16) |
yes |
|
|
S6 |
0AD |
fixed (UTC or TDT) |
if UTC |
ms (ps for Epoch16) |
8(16) |
yes |
|
|
S7 |
provider-defined |
provider-defined |
if needed |
provider-defined |
provider-defined |
yes |
|
|
S8 |
4147BC |
Julian |
none |
fractional day |
8(16) |
yes |
|
|
S9 |
0AD |
TDT/UTC |
included |
seconds/picoseconds+ leapseconds |
8(16) |
redefine EPOCH16 |
|
|
S10 |
provider-defined |
fixed (UTC or TDT) |
if UTC |
fractional seconds |
8 float |
yes |
|
|
S11 |
J2000 or J2100 |
TT |
none needed |
nanoseconds |
8 integer |
yes |
|
|
S12 |
J2000 |
TDB/ET |
none |
fractional seconds |
8 float |
yes |
|
|
S13 |
provider-defined |
TT |
none |
fractional seconds |
8 float |
yes |
|
S1: Time_Base=0AD, Time_Scale=UTC, Leap_Seconds_Included=included, Resolution=ms (ps for Epoch16), VarSize=8(16), NewVarType=no
This redefines existing Epoch system to include leap seconds and makes Time_Scale explicit; breaks reversibility and existing times (old B1)
S2: Time_Base=1900, Time_Scale=UTC, Leap_Seconds_Included=included, Resolution=ms (ps for Epoch16), VarSize=8(16), NewVarType=no
This clearly distinguishes old times from new, since century wrong if used with old routines (old B2)
S3: Time_Base=1970, Time_Scale=TDT, Leap_Seconds_Included=none, Resolution=ms (ps for Epoch16), VarSize=8(16), NewVarType=no
This provides POSIX Unix-like time with no leap seconds (old B3), easy to confuse with POSIX time as actually implemented (usually reset to UTC regularly)
S4: Time_Base=1900, Time_Scale=TDT, Leap_Seconds_Included=none, Resolution=ms (ps for Epoch16), VarSize=8(16), NewVarType=no
Bobby's current favorite (old B4), although nothing magical about BaseTime, so S3 might work
S5: Time_Base=0AD, Time_Scale=?, Leap_Seconds_Included=if needed, Resolution=ms (ps for Epoch16), VarSize=8(16), NewVarType=yes
New time variable type but otherwise similar to existing scheme with added attributes (old C)
S6: Time_Base=0AD, Time_Scale=fixed (UTC or TDT), Leap_Seconds_Included=if UTC, Resolution=ms (ps for Epoch16), VarSize=8(16), NewVarType=yes
New time variable type and defined fixed Time_Scale, but otherwise similar to existing scheme with added attributes (old D)
S7: Time_Base=provider-defined, Time_Scale=provider-defined, Leap_Seconds_Included=if needed, Resolution=provider-defined VarSize=provider-defined, NewVarType=yes
New time variable type with maximum flexibility; requires extensive code to cover all possibilities (old E)
S8: Time_Base=4147BC, Time_Scale=Julian, Leap_Seconds_Included=none, Resolution=fractional day, VarSize=8(16), NewVarType=yes
Julian day scheme, where sum of two 8-byte floats provide Julian day, with resolution apportioned between the two.
S9: Time_Base=0AD, Time_Scale=TDT/UTC, Leap_Seconds_Included=included, Resolution=seconds/picoseconds+leapseconds, VarSize=8(16), NewVarType=redefineEPOCH16, Unit=ps
Nand's idea of first 8byte for seconds in TDT and second 8-byte in picoseconds in UTC, including all leap seconds (old G), needs some flag to indicate redefinition of Epoch16
S10: Time_Base=provider-defined, Time_Scale= fixed (UTC or TDT), Leap_Seconds_Included=if UTC, Resolution=fractional seconds VarSize=8, NewVarType=yes
New time variable type with flexibility of Time_Base to allow flexibility in setting resolution of 8 byte float; smaller file size than high resolution 16-byte variable; complicates merging time between datasets but straightforward to compute. Could allow various Time_Base written in ISO-8601 format with a single call to convert to an internal number to add to Epoch. This would allow 8 bytes to hold picoseconds if Time_Base=doy.
S11: Time_Base=J2000, Time_Scale=TT, Leap_Seconds_Included=none, Resolution=nanoseconds VarSize=8 integer, NewVarType=yes
Final proposal; see beginning.
S12: Time_Base=J2000, Time_Scale=TDB, Leap_Seconds_Included=none, Resolution=fractional seconds VarSize=8 float, NewVarType=yes
SPICE fomat
S13: Time_Base=provider-defined, Time_Scale= TT, Leap_Seconds_Included=none, Resolution=fractional seconds VarSize=8 float, NewVarType=yes
Allow moving Time_Base to get full resolution where needed.
A. (NOT VIABLE) Keep existing Epoch[16] scheme, but redefine explicitly to UTC and include leap seconds (and change conversion routines). Although most space physics data are recent and there have been few leap seconds, this leads to incorrect times for some missions and breaks reversibility of the Epoch routines.
Pros: straightforward
Cons: same time in old Epoch data are different, doesn't clarify time bases for existing mission data, older CDF version conversion routines and user custom code will compute wrong times for new CDFs with no way to distinguish, overall bad idea. At present, the CDF library routines for converting between Epoch and YMD HMS is reversible in the sense that they will return the same HMS as was used on input, but doesn't allow entering an actual leap second. To allow input of a leap second (similar to days=366 in a leap year), the routines would need only a small change (allow seconds=60), but would no longer be reversible without further modification and other input parameters to say what time system to use on retrieval. Simple subtraction of Epochs would no longer be accurate if a leap second is in between.
B1. (NOT VIABLE) Add attributes to existing Epoch[16] variables to specify time scale and leap seconds (otherwise similar to A); Time_Base=0AD and Leap_Seconds_Included="" default to current scheme with Time_Scale still ambiguous/unknown. Attributes need to be attached to individual time variables, since any variation allowed needs to be allowed per time variable (not file-based), to allow multiple time variables and merging between files (even of different datasets).
Pros: finally distinguishes time scales if attributes are checked.
Cons: users could use old code to convert incorrectly, must pass attributes to new code to work correctly.
B2. Same as B1 with fixed Time_Base="1900" and Leap_Seconds_Included defined as the default, will clearly distinguish times computed with old codes as bad.
Pros: finally distinguishes time scales if attributes are checked.
Cons: must pass attributes to new code to work correctly.
B3. Same as B1 with fixed Time_Base="1970", Leap_Seconds_Included="", Time_Scale="TDT" will provide POSIX time.
Pros: finally distinguishes time scales if attributes are checked.
Cons: must pass attributes to new code to work correctly,
B4. Same as B1 with fixed Time_Base="1900", Leap_Seconds_Included="", Time_Scale="TDT"
C. (NOT VIABLE) Create a new variable type, CDF_TIME, and add variable attributes for various time scales and leap second array. Probably fixed resolution (16 bytes for picoseconds), although perhaps optional 8 bytes for milliseconds only.
Pros: handles all time scales and combinations of leap seconds, prevents confusion with old Epoch[16] variables, handles predictive data and mission times where leap seconds not added.
Cons: requires new version of CDF, complex choices of time scales and code development, comparison between datasets with different time scales requires conversion.
D. Create a new variable type, CDF_TIME_TDT or CDF_TIME_UTC, with fixed time scale (either ET/TT/TDT or UTC) and limited optional other attributes. Probably fixed resolution (16 bytes for picoseconds), although perhaps optional 8 bytes for milliseconds only.
Pros: fixed time base, good for long term archive if well-defined and times properly converted on entry, prevents confusion with old Epoch[16] variables.
Cons: requires new version of CDF, possibly loses original times and time base and scale info, future leap seconds not defined for predictive data (if UTC),
E. (NOT VIABLE) [Time_Base+offset] Create a new variable type, CDF_FLEX_TIME, with user-set time scales, Time_Base and variable size (2, 4, 8, 16 byte) as specified in attributes, such as "seconds from 1970-01-01 in TDT" or "microseconds from 2008-08-23T12:45:33.65 (launch) in UTC", probably broken out into attributes: Time_Base, Units (days, hours, seconds), Time_Scale. Time_Scale="POSIX" would be Units=seconds from Time_Base=1970 and Time_Scale="ACE" would be Units=seconds from Time_Base="<launch of ACE>". This scheme is used by NetCDF UDUNITS, although deprecated now.
Pros: greatest flexibility, most compact, easiest for missions.
Cons: complex, assorted software routines required, tricky testing, difficult to convert to other scales for comparison, not archival, complicates merging CDFs
F. (NOT VIABLE) Create a new variable type, CDF_MIX_TIME, with two 8-byte floats both in Julian day units with Time_Scale="TDB". The time is the sum of the two floats and can be apportioned as needed for resolution. So the time JD(TDB)=2450123.7 could be expressed as [2450123.7D0, 0D0] (JD method), [2451545D0, -1421.3D0] (J2000 method), [2440587.5D0, 9536.2D0] (POSIX time), [2400000.5D0, 50123.2D0] (MJD method), or [2450123.5D0, 0.2D0] (date & time method). Non-changing bytes can be zero or not included and instead carried in Time_Base.
Pros: allows easy input and computation in various time bases (non-leap second)
Cons:
G. (from Nand) Epoch16 where first 8 bytes are seconds from Time_Base in Time_Scale=ET and second 8 bytes are fractional seconds with all leap seconds included (as needed).
Alternative 1: Default the existing time base to UTC and change Epoch routines to add leap seconds, since most space physics data is recent and there have been few leap seconds. This leads to incorrect times for some missions and breaks reversibility of the Epoch routines.
At present, the CDF library routines for converting between Epoch and YMD HMS is reversible in the sense that they will return the same HMS as was used on input, but doesn't allow entering an actual leap second. To allow input of a leap second (similar to days=366 in a leap year), the routines would need only a small change (allow seconds=60), but would no longer be reversible without further modification and other input parameters to say what time system to use on retrieval. Simple subtraction of Epochs would no longer be accurate if a leap second is in between.
Alternative 2: have time data types (4, 8, 16 bytes) to be time from the Time_Base only (with or without leap seconds), so Time_Scale="POSIX" would be Units=seconds from Time_Base=1970 and Time_Scale="ACE" would be Units=seconds from Time_Base="<launch of ACE>".
One specific application of this scheme is in NetCDF UDUNITS, along the lines of
time:units = "hours since 1990-11-25 12:00 UTC";
or
long base_time;
base_time:units = "sec";
base_time:origin = "Epoch"; where Epoch is 1970 Jan 1 00:00:00 UTC
Some netCDF forum posts greatly discourage continued use of this scheme. Adding arbitrary units is likely to lead to inconsistent conversions among various user routines attempting to process the data. The hybrid Gregorian/Julian calendar used by the UDUNITS-2 package cannot be changed. Dates on or after 1582-10-15 are assumed to be Gregorian dates; dates before that are assumed to be Julian dates. In particular, the year 1 BCE is immediately followed by the year 1 CE.
UDUNITS routines:
Function: void ut_decode_time (double time, int* year, int* month, int* day, int* hour, int* minute, double* second, double* resolution)
Decodes a time from a double-precision value into its individual components. The variable referenced by resolution will be set to the resolution (i.e., uncertainty) of the time in seconds.
Function: ut_unit* ut_offset_by_time (const ut_unit* const unit, const double origin)
Returns a timestamp-unit equivalent to the time unit unit referenced to the time-origin origin (as returned by ut_encode_time()). For example:
const ut_unit* second = ...
const ut_unit* secondsSinceTheEpoch =
ut_offset_by_time(second, ut_encode_time(1970, 1, 1, 0, 0, 0.0));
Leap seconds are not taken into account. You should pass the returned pointer to ut_free() when you no longer need the unit. If an error occurs, then this function returns NULL and ut_get_status() will return one of the following:
Besides adding the new data type, CDF_TIME_TT2000, with its attributes, the main changes will be to create new versions of the various time-handling routines to use these new attributes when available and to add them as parameters (both From and To) on conversions and differencing. Routines also needed for conversion between CDF_EPOCH* and CDF_TIME variables. All routines will need parameters to receive the attributes, although they may use the default values if no attributes are provided. The JPL SPICE library has the best-tested set of time conversion routines and even includes corrections for relativistic effects. Useful if the routines can receive and return time structures with the attributes included.
CDF_time, cdf_time_var, year, month, day, hour, minute, second, millisecond, microsecond, nanosecond, picosecond, Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base, /COMPUTE_TIME, /BREAKDOWN_TIME
For flexibility, cdf_time,/compute routine would accepts arrays of floats (up to 64 bits) in all input values (and return array of times). The various input parameters are almost all optional and would accumulate appropriately, so we can use this call:
CDF_time, time, 1970,1,1,0,0, seconds_input,0,0,0,0,/COMPUTE_EPOCH,/Time_Scale='TDT'
(allows seconds from Unix beginning)
CDF_time, epoch, 2000,0,modified_julianday-0.5,0,0,0,0,0,/COMPUTE_EPOCH
or any other combination. In particular, subsecond variables are optional and usually decimal second is used if it has enough resolution. Basically the input values would sum up into psec or seconds. For the special case of month=0, day becomes day_of_year rather than day_of_month. Fractional year and month might be complicated but fractional or multiple day, hours, minutes, etc. should be summable after multiplying by the appropriate constants (adding leap seconds if needed). User must pass Time attributes explicitly except for object-oriented languages that can carry them with the cdf_time_var object.
CDF_time_string, cdf_time_var, time_string, time_format, Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base, /COMPUTE_TIME, /BREAKDOWN_TIME
Same as CDF_time but using time as a string in a standard format or user specified time_format. If provided, Resolution provides how much resolution to compute.
CDF_time_base, cdf_time_var, alt_time_var, /COMPUTE_TDT_TIME, /COMPUTE_ALT_TIME, Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base
Convert between CDF_TIME_TT2000 variable and alternative time variable, where attributes describe the scale and base of the alternate time variable. Time_Base might be POSIX or JulianDay or other schemes.
Result=CDF_time_compare(time1, time2)
Returns 1 if the value of time1 is greater (a later date and time) than time2, 0 if same, and -1 if time2 is greater than time1.
If we use time attributes, these will need to be passed in for both times so proper comparison can be made. If no attributes passed in, then assume they are the same (although leap seconds will need to be handled, so may require Leap_Seconds_Included if Time_Scale="UTC").
Result=CDF_time_difference(time1, time2, /seconds, /milliseconds, /microseconds, /nanoseconds, /picoseconds)
Return time1 - time2 in seconds, millseconds or picoseconds. Similar to CDF_time_compare, will require attributes for both times or at least Leap_Seconds_Included if Time_Scale="UTC".
The SPICE C-based library handles more time string formats <https://naif.jpl.nasa.gov/pub/naif/toolkit_docs/C> <https://Naif.jpl.nasa.gov/pub/naif>.
new (arraySize): create new CDF_TIME object with arraySize values (default to 1)
read (fileID, [varName]): read CDF_TIME variable, varName (default to "time") from CDF file opened with fileID
write (fileID, [varName]): write CDF_TIME variable, varName (default to "time") from CDF file opened with fileID
compareTo(cdf_time2): return array of -1 (less), 0 (same), +1 (more) values for self object compared to cdf_time2 object
difference(cdf_time2, [/seconds, /milliseconds, /microseconds, /nanoseconds]): return array of (self - cdf_time2) in seconds, milliseconds, microseconds, nanoseconds (default)
toString([timeFormat, Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base]): convert times to string of timeFormat (defaults to ISO8601 form) in Time_Scale (defaults to UTC) from Time_Base with Leap_Seconds_Included (defaults to standard list from NIST) with Resolution (defaults to "+1s")
fromString: same as above in reverse.
toBreakdown(year, month, day, hour, minute, second, millisecond, microsecond, nanosecond, picosecond, Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base): same as toString but returns arrays of time parts. Last given part contains fractional values for remaining Resolution.
fromBreakdown: same as toBreakdown in reverse; parts can be zero or large and time will sum appropriately.
toTime(Time_Scale=Time_Scale, Reference_Position=Reference_Position, Leap_Seconds_Included=Leap_Seconds_Included, Resolution=Resolution, Units=Units, Time_Base=Time_Base): returns one-dimensional array of times in Time_Scale (useful for J2000, POSIX time, etc.)
fromTime: same as toTime in reverse