JCDF

Overview

JCDF is a pure java library capable of reading files in the Common Data Format defined by NASA. It runs within the J2SE1.5 (or later), but other than that has no dependencies, neither the official CDF C library nor any other java class libraries.

Documentation

The classes are provided with comprehensive javadocs. Start reading at the CdfContent class for high-level access to CDF data and metadata, or CdfReader for low-level access to the CDF internal records.

Comparison with the official CDF library

JCDF is a completely separate implementation from the Java interface to the official CDF library, which uses native code via JNI. It was written mainly with reference to the CDF Internal Format Description document (v3.4). Minor updates at version JCDF 1.1 are believed to bring it into line with CDF v3.6.

The main benefit of using JCDF, and the reason for developing it, is that it's pure java, so it can be deployed using only the JCDF jar file. There is no need to install the system-dependent official CDF library.

The API is very different from that of the official CDF library. JCDF gives you a simple view of the CDF file, in terms of its global attributes, variable attributes and variables. This is fairly easy to use, but may or may not suit your purposes. It's also possible to get a low-level view of the CDF file as a sequence of CDF records.

JCDF offers no capabilities for writing or editing CDF files, it only reads them.

Implementation Notes

JCDF is based on NIO mapped ByteBuffers, it's expected to be reasonably fast, but I haven't done any benchmarking.

Use of mapped buffers leads to efficient serial and random I/O, but does have some associated problems concerning release of allocated resources. For reasons that are quite complicated, it is hard to release mapped buffers after use, which means that for instance files you have read using JCDF may be hard to delete, and that reading large files may leak significant amounts of virtual memory. There are some more or less nasty ways to work round this in user code or in the library. If these issues cause practical problems for library users, you are encouraged to contact me by email or open a github issue.

Implementation Status

Support for the CDF format is almost, but not totally, complete. In particular:

• Versions: The code was written with reference to version 3.4 (updated to 3.6) of the CDF Internal Format Description document. Following comments in that document, it is believed that versions 2.6, 2.7 and 3.* of the CDF format are supported.
• Large files: The library imposes no restriction on file size, so >2Gb files should be OK for v3 CDF files (CDF v2 did not support 64-bit addressing) as long as a 64-bit JVM is in use. At JCDF v1.2 I didn't have any large files to test on, and there was a bug which caused a failure. Jeremy Faden kindly pointed me at some large CDFs which allowed me to identify and fix the bug, so at v1.2 and beyond this is tested and working.
• Compression: All formats supported (GZIP, HUFF, AHUFF, RLE).
• Numeric encodings: Normal big- and little-endian encodings supported, but VMS D_FLOAT and G_FLOAT encodings are not supported.
• Layout: Single-file CDF files are supported, but multiple-file CDF files are not. This could be added fairly easily if necessary.
• I/O: Access is read-only, there is no attempt or intention to support write access.
• Data types: All CDF data types are supported, more or less. Unsigned integer types are transformed to larger signed types on read, because of the difficulty of handling unsigned integers in java, so for instance a CDF_UINT1 is read as a java short (16-bit) integer.
• Record data access: For array-valued variables you can currently only read a whole record at a time, not just part of an array-valued record. You can either get the raw elements or a shaped version. This is considerably less flexible than a hyper-read.

Utilities

The library comes with a couple of simple utilities for examining CDF files:

CdfList:

displays the metadata and data from a CDF file, along the lines of the cdfdump command in the official CDF distribution. If the -data flag is supplied, record data as well as metadata is shown. See CdfList examples.

CdfDump:

displays a dump of the sequence of low-level CDF records found in the CDF file, along the lines of the cdfirsdump command in the official CDF distribution. If the -fields flag is supplied, field information from each record is shown. If the -html flag is supplied, the output is in HTML with files offsets displayed as hyperlinks, which is nice for chasing pointers. See CdfDump examples.

Downloads

The source code is hosted on github at https://github.com/mbtaylor/jcdf. It comes with a makefile that can be used to build the jar file, javadocs, and this documentation, and to run some tests.

Pre-built copies of the jar file and documentation for the current version (v1.2-4) can be found here:

• jcdf.jar
• javadocs
• Maven central repository

Previous versions may be available at https://www.star.bristol.ac.uk/mbt/releases/jcdf/.

History

Version 0.1 (28 Jun 2013)

Initial release. Tested, documented and believed working, though could use some more testing and perhaps functionality related to time-related DataTypes. Support for some time types not complete.

Version 1.0 (13 Aug 2013)

• More extensive tests added.
• Fix failure when reading non-sparse variables with zero records.
• Fix bug: pad values not explicitly defined are given default values rather than causing an error.
• Fix EPOCH16 bug.
• Add EPOCH16 formatting.
• Modify CdfList data output: better formatting, and distinguish NOVARY values from virtual ones.
• TIME_TT2000 values now handled correctly, including leap seconds, and optional leap seconds table referenced by CDF_LEAPSECONDSTABLE environment variable as for NASA library. Internal leap seconds table is updated until 2012-07-01.

Version 1.1 (23 Apr 2015)

• 2015-07-01 leap second added to internal leap second table.
• Updated to match v3.6 of the CDF library/format. The GDR field rfuD is now renamed as leapSecondLastUpdated. It is also used when formatting TIME_TT2000 data values for output; if the library leap second table is out of date with respect to the data a warning is issued for information, and if the time values are known to have leap seconds applied invalidly, an error is thrown or a severe log message is issued. This behaviour follows that of the official CDF library.

Version 1.2 (9 Sep 2015)

• Fix a bug that caused a failure when accessing large (>2Gb) files.
• Update tests to use NASA's CDF library v3.6.0.4; this fixes a missing leap second bug, which simplifies the tests somewhat.

Version 1.2-1 (25 Sep 2015)

• Fix a bug in leap second handling - it was sensitive to the JVM's default time zone, and only ran correctly if the default time zone matched UTC. The error in formatting TIME_TT2000 data values was previously a second out within up to a day of the occurrence of a leap second.
• Add Variable.getDescriptor method.
• Improve the build (mainly javadocs) a bit to reduce some warnings that showed up in JDKs later than Java 5. Also add a switch in the makefile to reduce noise when building under JDK8.

Version 1.2-2 (4 Jan 2017)

• 2017-01-01 leap second added to internal leap second table.

Version 1.2-3 (16 Nov 2017)

• Fix bugs in low-level reading code (BankBuf): unsigned bytes could be read wrong in some cases, and data could be read wrong near the boundaries of multi-buffer files (only likely to show up for files >2Gbyte). Thanks to Lukas Kvasnica (Brno) for identifying and fixing these.
• Add unit tests to test the supplied Buf implementations.
• Some minor adjustments to the build/test framework to accommodate Debian packaging (mostly replacement of test data files encumbered by NASA copyright statement). No change to distributed library code.

Version 1.2-4 (15 Dec 2021)

• Some runtime log warnings about unexpected field values removed. These generally concerned fields marked "Reserved for future use" in the CDF Internal Format Description, and reported on deviations from a particular CDF version (currently 3.4), but which may have legitimate use in later versions of the CDF format.
• Fix an error in interpreting the leapSecondLastUpdated field from the Global Descriptor Record; this could cause a RuntimeException with a message about the library leap second table being out of date during TT2000 date formatting.

Context

This software was written by Mark Taylor at the University of Bristol at the request of, and funded by, the science archive group at ESA's European Space Astronomy Centre (ESAC).

It is used within TOPCAT/STILTS/STIL to enable access to CDF tables alongside other tabular data formats by that software. It is believed to be in some use also at ESAC and elsewhere. If anybody out there is using it and is willing to be listed here, let me know.

It is licenced under the LGPL, though if you need a different licence I can probably fix it.

My thanks to several people who have helped with development and testing, including Christophe Arviset, Robert Candey, Michael Liu, Jeremy Faden and Bogdan Nicula.

Bugs, questions, feedback, enhancement requests welcome to m.b.taylor@bristol.ac.uk.