It is always possible to access a table's data sequentially, that is starting with the first row and reading forward a row at a time to the last row; it may or may not be possible to tell in advance (using ) how many rows there are. To perform sequential access, use the method to get a object, which is an iterator over the rows in the table. The RowSequence's method moves forward a row without returning any data; to obtain the data use either or ; the relative efficiencies of these depend on the implementation, but in general if you want all or nearly all of the values in a row it is a good idea to use getRow, if you just want one or two use getCell. You cannot move the iterator backwards. When obtained, a RowSequence is positioned before the first row in the table, so (unlike an ) it is necessary to call next before the first row is accessed.

Here is an example of how to sum the values in one of the numeric columns of a table. Since only one value is required from each row, getCell is used: The next example prints out every cell value. Since it needs all the values in each cell, it uses getRow: In this case a tidier representation of the values might be given by replacing the print call with: void dummy5( StarTable table, Object[] row, int icol ) { System.out.print( table.getColumnInfo( icol ) .formatValue( row[ icol ], 20 ) + "\t" ); }

If a table's method returns true, then it is possible to access the cells of a table in any order. This is done using the or methods directly on the table itself (not on a RowSequence). Similar comments about whether to use getCell or getRow apply as in the previous section.

If an attempt is made to call these random access methods on a non-random table (one for which isRandom() returns false), an will be thrown.

What do you do if you have a sequential-only table and you need to perform random access on it? The utility method takes any table and returns one which is guaranteed to provide random access. If the original one is random, it just returns it unchanged, otherwise it returns a table which contains the same data as the submitted one, but for which isRandom is guaranteed to return true. It effectively does this by taking out a RowSequence and reading all the data sequentially into some kind of (memory- or disk-based) data structure which can provide random access, returning a new StarTable object based on that data structure. Exactly what kind of data structure is used for caching the data for later use is determined by the StoragePolicy currently in effect - this is described in .

Clearly, this might be an expensive process. For this reason if you have an application in which random access will be required at various points, it is usually a good idea to ensure you have a random-access table at the application's top level, and for general-purpose utility methods to require random-access tables (throwing an exception if they get a sequential-only one). The alternative practice of utility methods converting argument tables to random-access when they are called might result in this expensive process happening multiple times.

The class can read FITS binary (BINTABLE) and ASCII (TABLE) table extensions. Unless told otherwise the StarTableFactory.makeStarTable method will find the first table extension in the named FITS file. If the name supplied to the StarTableFactory ends in a # sign followed by a number however, it means that the requested table is in the indicated extension of a multi-extension FITS file. Hence 'spec23.fits#3' refers to the 3rd extension (4th HDU) in the file spec23.fits. The suffix '#0' is never used in this context for a legal FITS file, since the primary HDU cannot contain a table.

To retrieve all the tables in a multi-extension FITS files, use one of the makeStarTables methods of StarTableFactory instead.

If the table is stored in a FITS binary table extension in a file on local disk in uncompressed form, then the file will be mapped rather than read when the StarTable is constructed, which means that constructing the StarTable is very fast. If you want to inhibit this behaviour, you can refer to the file as a URL, for instance using the designation 'file:spec23.fits' rather than 'spec23.fits'; this fools the handler into thinking that the file cannot be mapped.

Header cards in the table's HDU header will be made available as table parameters (see ). Only header cards which are not used to specify the table format itself are visible as parameters (e.g. NAXIS, TTYPE* etc cards are not). HISTORY and COMMENT cards are run together as one multi-line value.

Currently, binary tables are read rather more efficiently than ASCII ones.

The handler also reads a variant of the FITS format - see .

As well as normal binary and ASCII FITS tables, STIL supports FITS files which contain tabular data stored in column-oriented format. This means that the table is stored in a BINTABLE extension HDU, but that BINTABLE has a single row, with each cell of that row holding a whole column's worth of data. The final (slowest-varying) dimension of each of these cells (declared via the TDIMn header cards) is the same, namely, the number of rows in the table that is represented. The point of this is that all the cells of a given column are stored contiguously, which for very large, and especially very wide tables means that certain access patterns (basically, ones which access only a small proportion of the columns in a table) can be much more efficient since they require less I/O overhead in reading data blocks.

Such tables are perfectly legal FITS files, but most non-STIL software will probably not recognise them as tables in the usual way. This format is mostly intended for the case where you have a large table in some other format (possibly the result of an SQL query) and you wish to cache it in a way which can be read efficiently by a STIL-based application.

Like normal FITS, there are two handlers for this format: (like FITS-plus) can read a VOTable as metadata from the primary HDU, and does not.

Colfits format is only available for data which stored as an uncompressed file in the file system (not, for instance, from a URL).

The class reads VOTables; it should handle any table which conforms to the VOTable 1.0, 1.1 or 1.2 specifications (as well as quite a few which violate them). In particular, it can deal with a DATA element whose content is encoded using any of the formats TABLEDATA, BINARY or FITS.

While VOTable documents often contain a single table, the format describes a hierarchical structure which can contain zero or more TABLE elements. By default, the StarTableFactory.makeStarTable method will find the first one in the document for you, which in the (common) case that the document contains only one table is just what you want. If you're after one of the others, identify it with a zero-based index number after a '#' sign at the end of the table designation. So if the following document is called 'cats.xml': ... ... ]]> then 'cats.xml' or 'cats.xml#0' refers to the "Star Catalogue" and 'cats.xml#1' refers to the "Galaxy Catalogue".

To retrieve all of the tables from a given VOTable document, use one of the makeStarTables methods of StarTableFactory instead.

Much more detailed information about the VOTable I/O facilities, which can be used independently of the generic I/O described in this section, is given in .

In many cases tables are stored in some sort of unstructured plain text format, with cells separated by spaces or some other delimiters. The class attempts to read these and interpret what's there in sensible ways, but since there are so many possibilities of different delimiters and formats for exactly how values are specified, it won't always succeed.

Here are the rules for how the ASCII-format table handler reads tables:

• Bytes in the file are interpreted as ASCII characters
• Each table row is represented by a single line of text
• Lines are terminated by one or more contiguous line termination characters: line feed (0x0A) or carriage return (0x0D)
• Within a line, fields are separated by one or more whitespace characters: space (" ") or tab (0x09)
• A field is either an unquoted sequence of non-whitespace characters, or a sequence of non-newline characters between matching single (') or double (") quote characters - spaces are therefore allowed in quoted fields
• Within a quoted field, whitespace characters are permitted and are treated literally
• Within a quoted field, any character preceded by a backslash character ("\") is treated literally. This allows quote characters to appear within a quoted string.
• An empty quoted string (two adjacent quotes) or the string "null (unquoted) represents the null value
• All data lines must contain the same number of fields (this is the number of columns in the table)
• The data type of a column is guessed according to the fields that appear in the table. If all the fields in one column can be parsed as integers (or null values), then that column will turn into an integer-type column. The types that are tried, in order of preference, are: Boolean, Short Integer, Long, Float, Double, String
• Empty lines are ignored
• Anything after a hash character "#" (except one in a quoted string) on a line is ignored as far as table data goes; any line which starts with a "!" is also ignored. However, lines which start with a "#" or "!" at the start of the table (before any data lines) will be interpreted as metadata as follows:
  • The last "#"/"!"-starting line before the first data line may contain the column names. If it has the same number of fields as there are columns in the table, each field will be taken to be the title of the corresponding column. Otherwise, it will be taken as a normal comment line.
  • Any comment lines before the first data line not covered by the above will be concatenated to form the "description" parameter of the table.

If the list of rules above looks frightening, don't worry, in many cases it ought to make sense of a table without you having to read the small print. Here is an example of a suitable ASCII-format table: # # Here is a list of some animals. # # RECNO SPECIES NAME LEGS HEIGHT/m 1 pig "Pigling Bland" 4 0.8 2 cow Daisy 4 2 3 goldfish Dobbin "" 0.05 4 ant "" 6 0.001 5 ant "" 6 0.001 6 ant '' 6 0.001 7 "queen ant" 'Ma\'am' 6 2e-3 8 human "Mark" 2 1.8 In this case it will identify the following columns: Name Type ---- ---- RECNO Integer SPECIES String NAME String LEGS Integer HEIGHT/m Float It will also use the text "Here is a list of some animals" as the Description parameter of the table. Without any of the comment lines, it would still interpret the table, but the columns would be given the names col1..col5.

If you understand the format of your files but they don't exactly match the criteria above, the best thing is probably to write a simple free-standing program or script which will convert them into the format described here. You may find Perl, awk or sed suitable languages for this sort of thing. Alternatively, you could write a new input handler as explained in - you may find it easiest to subclass the class in this case.

The handler can read data in the semi-standard CSV format. The intention is that it understands the version of that format spoken by MS Excel amongst others, though the documentation on which it is based was not obtained directly from Microsoft.

The rules for data which it understands are as follows:

• Each row must have the same number of comma-separated fields.
• Whitespace (space or tab) adjacent to a comma is ignored.
• Adjacent commas, or a comma at the start or end of a line (whitespace apart) indicates a null field.
• Lines are terminated by any sequence of carriage-return or newline characters ('\r' or '\n') (a corollary of this is that blank lines are ignored).
• Cells may be enclosed in double quotes; quoted values may contain linebreaks (or any other character); a double quote character within a quoted value is represented by two adjacent double quotes.
• The first line may be a header line containing column names rather than a row of data. Exactly the same syntactic rules are followed for such a row as for data rows.

The class reads the Tab-Separated Table (TST) format, which is used by a number of astronomical software packages including Starlink's GAIA, and ESO's SkyCat on which it is based. A definition of the format can be found in Starlink Software Note 75. The implementation here ignores all comment lines: special comments such as the "#column-units:" are not processed.

An example looks like this: decVB_VU_B ------------ 5:09:08.7 -8:45:15 4.27 -0.19 -0.90 5:07:50.9 -5:05:11 2.79 +0.13 +0.10 5:01:26.3 -7:10:26 4.81 -0.19 -0.74 5:17:36.3 -6:50:40 3.60 -0.11 -0.47 [EOD] ]]>

The class reads tables in the text-based format used by CalTech's Infrared Processing and Analysis Center, which is defined at . Tables can store column name, type, units and null values, as well as table parameters. They typically have a filename extension ".tbl" and are used for Spitzer data amongst other things. An example looks like this: \title='Animals' \ This is a table with some animals in it. | RECNO | SPECIES | NAME | LEGS | HEIGHT | | char | char | char | int | double | | | | | | m | | | | null | | | 1 pig Pigling Bland 4 0.8 2 cow Daisy 4 2 3 goldfish Dobbin 0 0.05 4 ant null 6 0.001

Some support is provided for files produced by the World Data Centre for Solar Terrestrial Physics. The format itself apparently has no name, but files in this format look something like the following: Column formats and units - (Fixed format columns which are single space seperated.) ------------------------ Datetime (YYYY mm dd HHMMSS) %4d %2d %2d %6d - %1s aa index - 3-HOURLY (Provisional) %3d nT 2000 01 01 000000 67 2000 01 01 030000 32 ...

The handler class is experimental; it was reverse-engineered from looking at a couple of data files in the target format, and may not be very robust.

The FITS handler, , will output a FITS file with N+1 HDUs for N tables; the first (primary) HDU has no interesting content, and subsequent ones (the extensions) are of type BINTABLE, one for each output table.

A variant handler, is also provided. This behaves in much the same way, but in the case of columns which contain variable-shaped arrays (ones for which the last element of ColumnInfo.getShape() is negative) it will store the array data in the 'heap' following the table data in the BINTABLE HDU, using the 'P' or 'Q' data type specifiers in the relevant TFORMn header cards.

To write the FITS header for the table extension, certain things need to be known which may not be available from the StarTable object being written; in particular the number of rows and the size of any variable-sized arrays (including variable-length strings) in the table. This may necessitate two passes through the data to do the write.

See the "Binary Table Extension" section of the FITS standard for more details of the FITS BINTABLE format.

StarTableOutput will write in FITS format (without variable length array-valued columns) if a format string "fits" is used, or the format string is null and the destination string ends in ".fits".

A variant form of FITS file is written by the handler. This is the same as the basic form, except that the primary HDU (HDU#0) contains a 1-d array of characters which form the text of a DATA-less VOTable. The FITS tables in the subsequent extensions are understood to contain the data. The point of this is that the VOTable can contain all the rich metadata about the table(s), but the bulk data are in a form which can be read efficiently. Crucially, the resulting FITS file is a perfectly good FITS table on its own, so non-VOTable-aware readers can read it in just the usual way, though of course they do not benefit from the additional metadata stored in the VOTable header.

While the normal VOTable/FITS encoding has some of these advantages, it is inconvenient in that either (for in-line data) the FITS file is base64-encoded and so hard to read efficiently, in particular for random access or (for referenced data) the table is split across two files.

As described in , STIL supports FITS files in which each column is stored as a single cell of a one-row BINTABLE extension. Two writers are provided, which writes VOTable-format metadata as a byte array in the primary HDU as for FITS-plus, and which writes a blank primary HDU.

The VOTable handler, , can write VOTables in a variety of flavours (see ). In all cases, a StarTableOutput will write a well-formed VOTable document with a single RESOURCE element holding one or more TABLE elements. The different output formats (TABLEDATA/FITS/BINARY, inline/href) are determined by configuration options on the handler instance. The default handler writes to inline TABLEDATA format.

The href-type formats write a (short) XML file and FITS or binary files with a similar name into the same directory, holding the metadata and bulk data respectively. The reference from the one to the other is a relative URL, so if one is moved, they both should be.

For more control over writing VOTables, consult .

The class writes to a simple text format which is intended to be machine readable (and fairly human readable as well). It can be read in by the ASCII input handler, and is described in more detail in .

The class writes to the semi-standard CSV format, which may optionally including an initial line containing column names. It can be read by the CSV input handler, and is described in more detail in .

The class writes to the text-based Tab-Separated Table format. It can be read in by the TST input handler, and is described in more detail in .

The class writes to a simple text-based format which is designed to be read by humans. According to configuration, this may or may not output table parameters as name:value pairs at before the table data themselves.

Here is an example of a short table written in this format: +-------+----------+--------+------+--------+--------+ | index | Species | Name | Legs | Height | Mammal | +-------+----------+--------+------+--------+--------+ | 1 | pig | Bland | 4 | 0.8 | true | | 2 | cow | Daisy | 4 | 2.0 | true | | 3 | goldfish | Dobbin | 0 | 0.05 | false | | 4 | ant | | 6 | 0.0010 | false | | 5 | ant | | 6 | 0.0010 | false | | 6 | human | Mark | 2 | 1.9 | true | +-------+----------+--------+------+--------+--------+

The class writes tables as HTML 3.2 TABLE elements. According to configuration this may be a freestanding HTML document or the TABLE element on its own (suitable for incorporation into larger HTML documents).

The class writes tables as LaTeX tabular environments, either on their own or wrapped in a LaTeX document. For obvious reasons, this isn't too suitable for tables with very many columns.

Mirage is a powerful standalone tool developed at Bell Labs for interactive analysis of multidimensional data. It has not however been developed in recent years. It uses its own file format for input. The class can write tables in this format.

Java/STIL does not come with the facility to use any particular SQL database "out of the box"; some additional configuration must be done before it can work. This is standard JDBC practice, as explained in the documentation of the class. In short, what you need to do is define the "jdbc.drivers" system property to include the name(s) of the JDBC driver(s) which you wish to use. For instance to enable use of MySQL with the Connector/J database you might start up java with a command line like this: java -classpath /my/jars/mysql-connector-java-3.0.8-stable-bin.jar:myapp.jar -Djdbc.drivers=com.mysql.jdbc.Driver my.path.MyApplication One gotcha to note is that an invocation like this will not work if you are using 'java -jar' to invoke your application; if the -jar flag is used then any class path set on the command line or in the CLASSPATH environment variable or elsewhere is completely ignored. This is a consequence of Java's security model.

For both the reader and the writer described below, the string passed to specify the database query/table may or may not require additional authentication before the read/write can be carried out. The general rule is that an attempt will be made to connect with the database without asking the user for authentication, but if this fails the user will be queried for username and password, following which a second attempt will be made. If username/password has already been solicited, this will be used on subsequent connection attempts. How the user is queried (e.g. whether it's done graphically or on the command line) is controlled by the 's object, which can be set by application code if required. If generic I/O is being used, you can use the get/setJDBCHandler methods of the or being used.

To the author's knowledge, STIL has so far been used with the following RDBMSs and drivers:

MySQL

MySQL has been tested on Linux with the Connector/J driver and seems to work; tested versions are server 3.23.55 with driver 3.0.8 and server 4.1.20 with driver 5.0.4. Sometimes tables with very many (hundreds of) columns cannot be written owing to SQL statement length restrictions. Note there is known to be a column metadata bug in version 3.0.6 of the driver which can cause a ClassCastException error when tables are written.

PostgreSQL

PostgreSQL 7.4.1 apparently works with its own JDBC driver. Note the performance of this driver appears to be rather poor, at least for writing tables.

Oracle

You can use Oracle with the JDBC driver that comes as part of its Basic Instant Client Package. URLs look something like "jdbc:oracle:thin:@//hostname:1521/database#SELECT ...".

SQL Server

There is more than one JDBC driver known to work with SQL Server, including jTDS and the Microsoft JDBC driver. Some evidence suggests that jTDS may be the better choice, but your mileage may vary.

Sybase ASE

There has been a successful use of Sybase 12.5.2 and jConnect (jconn3.jar) using a JDBC URL like "jdbc:sybase:Tds:hostname:port/dbname?user=XXX&password=XXX#SELECT...". An earlier attempt using Sybase ASE 11.9.2 failed.

You can view the result of an SQL query on a relational database as a table. This can be done either by passing the query string directly to a or by passing it to the generic method (any string starting 'jdbc:' in the latter case is assumed to be an SQL query string). The form of this query string is as follows: # ]]> The exact form is dependent on the JDBC driver which is installed. Here is an example for MySQL: jdbc:mysql://localhost/astro1?user=mbt#SELECT ra, dec FROM swaa WHERE vmag<18 If the username and/or password are required for the query but are not specified in the query string, they will be prompted for.

Note that the StarTable does not represent the JDBC table itself, but a query on table. You can get a StarTable representing the whole JDBC table with a query like SELECT * from table-name, but this may be expensive for large tables.

You can write out a StarTable as a new table in an SQL-compatible RDBMS. Note this will require appropriate access privileges and may overwrite any existing table of the same name. The general form of the string which specifies the destination of the table being written is: # ]]>

Here is an example for MySQL with Connector/J: jdbc:mysql://localhost/astro1?user=mbt#newtab which would write a new table called "newtab" in the MySQL database "astro1" on the local host with the access privileges of user mbt.

This example shows how you can wrap a table to provide a sorted view of it. It subclasses , which is a wrapper that presents its base table with the rows in a different order.

Suppose you have three arrays representing a set of points on the plane, giving an index number and an x and y coordinate, and you would like to manipulate them as a StarTable. One way is to use the class, which gives you a table of a specified number of rows but initially no columns, to which you can add data a column at a time. Each added column is an instance of ; the class provides a convenient implementation which wraps an array of objects or primitives (one element per row). StarTable makeTable( int[] index, double[] x, double[] y ) { int nRow = index.length; ColumnStarTable table = ColumnStarTable.makeTableWithRows( nRow ); table.addColumn( ArrayColumn.makeColumn( "Index", index ) ); table.addColumn( ArrayColumn.makeColumn( "x", x ) ); table.addColumn( ArrayColumn.makeColumn( "y", y ) ); return table; }

A more general way to approach this is to write a new implementation of ; this is like what happens in Swing if you write your own to provide data for a . In order to do this you will usually want to subclass one of the existing implementations, probably , or . Here is how it can be done: class PointsStarTable extends RandomStarTable { // Define the metadata object for each of the columns. ColumnInfo[] colInfos_ = new ColumnInfo[] { new ColumnInfo( "Index", Integer.class, "point index" ), new ColumnInfo( "X", Double.class, "x co-ordinate" ), new ColumnInfo( "Y", Double.class, "y co-ordinate" ), }; // Member variables are arrays holding the actual data. int[] index_; double[] x_; double[] y_; long nRow_; public PointsStarTable( int[] index, double[] x, double[] y ) { index_ = index; x_ = x; y_ = y; nRow_ = (long) index_.length; } public int getColumnCount() { return 3; } public long getRowCount() { return nRow_; } public ColumnInfo getColumnInfo( int icol ) { return colInfos_[ icol ]; } public Object getCell( long lrow, int icol ) { int irow = checkedLongToInt( lrow ); switch ( icol ) { case 0: return new Integer( index_[ irow ] ); case 1: return new Double( x_[ irow ] ); case 2: return new Double( y_[ irow ] ); default: throw new IllegalArgumentException(); } } } In this case it is only necessary to implement the getCell method; implements the other data access methods (getRow, getRowSequence) in terms of this.

In this example we will append to a table a new column in which each cell contains the sum of all the other numeric cells in that row.

First, we define a wrapper table class which contains only a single column, the one which we want to add. We subclass , implementing its abstract methods as well as the getCell method which may be required if the base table is random-access. We could use this class on its own if we just wanted a 1-column table containing summed values. The following snippet however combines an instance of this class with the table that it is summing from, resulting in an n+1 column table in which the last column is the sum of the others:

It is important to understand that when STIL reads in a a VOTable document, it creates one or more StarTables from one or all of the TABLE elements and then discards the document. This means that information in the document's structure which does not map naturally onto the StarTable model may be lost. Such information currently includes COOSYS elements, GROUPing of PARAMETERs and FIELDs, and the hierarchical relationship between tables arranged in RESOURCE elements. It is possible that some of these will be stored in some way in VOTable-type StarTables in the future, but some loss of information is an inevitable consequence of the fact that STIL's model of a table is designed to provide a generic rather than a VOTable-specific way of describing tabular data.

If you want to avoid this kind of data loss, you should use the custom VOTable document parser described in , which retains the entire structure of the document.

When a StarTable is created by reading a TABLE element, its parameter list (as accessed using ) is assembled by collecting all the PARAM elements in the TABLE element and all the PARAM and INFO elements in its parent RESOURCE. When a VOTable is written, all the parameters are written as PARAMs in the TABLE.

There is a one-to-one correspondence between a StarTable's ColumnInfo objects (accessed using ) and the FIELD elements contained in the corresponding TABLE. The attributes of each fields are interpreted (for reading) or determined (for writing) in a number of different ways:

• datatype and arraysize values depend on the class and shape of objects held in the column.
• name, unit ucd and Utype values can be accessed using the corresponding methods on the ColumnInfo object (get/set Name(), UnitString(), UCD() and Utype() respectively).
• ID width, precision and type are held as String-type auxiliary metadata items in the ColumnInfo object, keyed by constants defined by the VOStarTable class (, , and respectively).
• LINK elements are represented by URL-type auxiliary metadata items in the ColumnInfo object, keyed by their title or, if it doesn't have one, ID attribute.

So if you have read a VOTable and want to determine the name, ucd and ID attributes of the first column, you can do it like this: private StarTable readVOTable(){return null;};void dummy7() { StarTable table = readVOTable(); ColumnInfo col0 = table.getColumnInfo(0); String name0 = col0.getName(); String ucd0 = col0.getUCD(); String id0 = (String) col0.getAuxDatumValue(VOStarTable.ID_INFO, String.class); }

And if you are preparing a table to be written as a VOTable and want to set the name, ucd and ID attributes of a certain column, and have it contain an element <LINK title='docs' href='...'>" you can set its ColumnInfo up like this: ColumnInfo configureColumn(String name, String ucd, String id, URL docURL) { ColumnInfo info = new ColumnInfo(name); info.setUCD(ucd); info.setAuxDatum(new DescribedValue(VOStarTable.ID_INFO, id)); info.setAuxDatum(new DescribedValue(new URLValueInfo("docs",null), docURL)); return info; }

The class and shape of each column in a StarTable (accessed using the get/setContentClass() and get/setShape() methods of ) correspond to the datatype and arraysize attributes of the corresponding FIELD element in the VOTable. You are not expected to access the datatype and arraysize elements directly.

How Java classes map to VOTable data types for the content of columns is similar to elsewhere in STIL. In general, scalars are represented by the corresponding primitive wrapper class (Integer, Double, Boolean etc), and arrays are represented by an array of primitives of the corresponding type (int[], double[], boolean[]). Arrays are only ever one-dimensional - information about any multidimensional shape they may have is supplied separately (use the getShape method on the corresponding ). There are a couple of exceptions to this: arrays with datatype="char" or "unicodeChar" are represented by String objects since that is almost always what is intended (n-dimensional arrays of char are treated as if they were (n-1)-dimensional arrays of Strings), and unsignedByte types are represented as if they were shorts, since in Java bytes are always signed. Complex values are represented as if they were an array of the corresponding type but with an extra dimension of size two (the most rapidly varying).

The following table summarises how all VOTable datatypes are represented: 1 -------- ---------------- --------------------- boolean Boolean boolean[] bit boolean[] boolean[] unsignedByte Short short[] short Short short[] int Integer int[] long Long long[] char Char String or String[] unicodeChar Char String or String[] float Float float[] double Double double[] floatComplex float[] float[] doubleComplex double[] double[] ]]>

The simplest way to read tables from a VOTable document is to use the generic table reading method described in (or for streaming) in which you just submit the location of a document to a , and get back one or more objects. If you're after one of several TABLE elements in a document, you can specify this by giving its number as the URL's fragment ID (the bit after the '#' sign, or the third argument of for streaming).

The following code would give you StarTables read from the first and fourth TABLE elements in the file "tabledoc.xml": void dummy4() throws IOException { StarTableFactory factory = new StarTableFactory(); StarTable tableA = factory.makeStarTable( "tabledoc.xml", "votable" ); StarTable tableB = factory.makeStarTable( "tabledoc.xml#3", "votable" ); } or equivalently void dummy4a() throws IOException { VOTableBuilder votBuilder = new VOTableBuilder(); boolean wantRandom = false; StoragePolicy policy = StoragePolicy.getDefaultPolicy(); StarTable tableA = votBuilder.makeStarTable( DataSource.makeDataSource( "tabledoc.xml" ), wantRandom, policy ); StarTable tableB = votBuilder.makeStarTable( DataSource.makeDataSource( "tabledoc.xml#3" ), wantRandom, policy ); } Note this will perform two separate parses of the document, one for each table built.

If you want all the tables in the document, do this: void dummy4b() throws IOException { VOTableBuilder votBuilder = new VOTableBuilder(); DataSource datsrc = DataSource.makeDataSource( "tabledoc.xml" ); StoragePolicy policy = StoragePolicy.getDefaultPolicy(); TableSequence tseq = votBuilder.makeStarTables( datsrc, policy ); List tList = new ArrayList(); for ( StarTable table; ( table = tseq.nextTable() ) != null; ) { tList.add( table ); } } which only performs a single pass and so is more efficient.

All the data and metadata from the TABLEs in the VOTable document are available from the resulting StarTable objects, as table parameters, s or the data themselves. If you are just trying to extract the data and metadata from a single TABLE element somewhere in a VOTable document, this procedure is probably all you need.

VOTable documents consist of a hierarchy of RESOURCE, DEFINITIONS, COOSYS, TABLE elements and so on. The methods described in the previous subsection effectively approximate this as a flat list of TABLE elements. If you are interested in the structure of the VOTable document in more detail than the table items that can be extracted from it, you will need to examine it in a different way, based on the XML. The usual way of doing this for an XML document in Java is to obtain a DOM (Document Object Model) based on the XML - this is an API defined by the W3C representing a tree-like structure of elements and attributes which can be navigated by using methods like and .

STIL provides you with a DOM which can be viewed exactly like a standard one (it implements the DOM API) but has some special features.

• All elements in it are instances of the class (which itself implements the DOM interface). This provides a few convenience methods such as which can be useful but don't do anything that you couldn't do with the Element interface alone.
• Some of the elements, according to their name, are instances of specialised subclasses of VOElement which provide methods specific to their rôle in a VOTable document. For instance every GROUP element in the tree is represented by a ; this class has a method which returns all the FIELD elements associated with that group (this method examines its FIELDref children and locates their FIELD elements elsewhere in the DOM). The various specific element types are not considered in detail here - see the javadocs for the subclasses of .
• The most important of these special element subclasses is . A TableElement can provide the table data stored within it; to access these data you don't need to know whether it is stored in TABLEDATA, FITS or BINARY form etc.
• Full ID/ref cross-referencing is supported for elements which have ID attributes in the VOTable specification - this is required so that for instance FIELDref elements can access their FIELDs, and TABLE elements can define their structure by reference to previously defined ones. If you need to locate cross-references by hand you can use the method.
• In most cases, the DOM you acquire will not contain the bulk data in the VOTable XML. Specifically, the children of TABLEDATA elements (a lot of TR and TDs) and of STREAM elements (long Base64-encoded strings containing FITS/binary data) will be absent. User code inspecting the DOM is rarely interested in these elements, only in the table data they represent, and this can be obtained from the corresponding TABLE element.
• The DOM is modifiable - that is you can add, remove and relocate nodes within it in the standard ways permitted by the DOM API.

To acquire this DOM you will use a , usually feeding a File, URL or InputStream to one of its makeVOElement methods. The bulk data-less DOM mentioned above is possible because the VOElementFactory processes the XML document using SAX, building a DOM as it goes along, but when it gets to the bulk data-bearing elements it interprets their data on the fly and stores it in a form which can be accessed efficiently later rather than inserting the elements into the DOM. SAX (Simple API for XML) is an event driven processing model which, unlike DOM, does not imply memory usage that scales with the size of the document. In this way any but the weirdest VOTable documents can be turned into a DOM of very modest size. This means you can have all the benefits of a DOM (full access to the hierarchical structure) without the disadvantages usually associated with DOM-based VOTable processing (potentially huge memory footprint). Of course in order to be accessed later, the data extracted from a stream of TR elements or from the inline content of a STREAM element has to get stored somewhere. Where it gets put is determined by the VOElementFactory's StoragePolicy (see ).

If for some reason you want to work with a full DOM containing the TABLEDATA or STREAM children, you can parse the document to produce a DOM Document or Element as usual (e.g. using a ) and feed that to one of the the 's makeVOElement methods instead.

Having obtained your DOM, the easiest way to access the data of a TABLE element is to locate the relevant in the tree and turn it into a using the adapter class. You can interrogate the resulting object for its data and metadata in the usual way as described in . This StarTable may or may not provide random access ( may or may not return true), according to how the data were obtained. If it's a binary stream from a remote URL it may only be possible to read rows from start to finish a row at a time, but if it was in TABLEDATA form it will be possible to access cells in any order. If you need random access for a table and you don't have it (or don't know if you do) then use the methods described in .

It is possible to access the table data directly (without making it into a StarTable) by using the method of the TableElement, but in this case you need to work a bit harder to extract some of the data and metadata in useful forms. See the documentation for details.

One point to note about VOElementFactory's parsing is that it is not restricted to elements named in the VOTable standard, so a document which does not conform to the standard can still be processed as a VOTable if parts of it contain VOTable-like structures.

Here is an example of using this approach to read the structure of a, possibly complex, VOTable document. This program locates the third TABLE child of the first RESOURCE element and prints out its column titles and table data.

Versions of STIL prior to V2.0 worked somewhat differently to this - they produced a tree structure representing the VOTable document which resembled, but wasn't, a DOM (it didn't implement the W3C DOM API). The current approach is more powerful and in some cases less fiddly to use.

SAX (Simple API for XML) is an event-based model for processing XML streams, defined in the package. While generally a bit more effort to use than DOM, it provides more flexibility and possibilities for efficiency, since you can decide what to do with each element rather than always store it in memory. Although a DOM built using the mechanism described in the previous section will usually itself be pretty small, it will normally have to store table data somewhere in memory or on disk, so if you don't need, and wish to avoid, this overhead, you'd better use event-based processing directly. This section describes how to do that.

The basic tool to use for VOTable-aware SAX-based processing is a , which is a SAX implementation that monitors all the SAX events and when it comes across a TABLE element containing DATA it passes SAX-like messages to a user-supplied which can do what it likes with them. TableHandler is a callback interface for dealing with table metadata and data events defined by STIL in the spirit of the existing SAX callback interfaces such as ContentHandler, LexicalHandler etc. You define a TableHandler by implementing the methods , and .

For full details of how to use this, see the appropriate javadocs, but here is a simple example which counts the rows in each TABLE in a VOTable stream. import javax.xml.parsers.SAXParserFactory; import org.xml.sax.ContentHandler; import org.xml.sax.InputSource; import org.xml.sax.XMLReader; import uk.ac.starlink.table.StarTable; import uk.ac.starlink.votable.TableContentHandler; import uk.ac.starlink.votable.TableHandler;

The VOTable parser provided is believed to be able to parse correctly any VOTable document which conforms to the 1.0, 1.1 or 1.2 VOTable recommendations. In addition, it will happily cope with documents which violate the standard in that they contain extra elements or attributes; such elements or attributes will be inserted into the resulting DOM but ignored as far as producing StarTables goes. In general, if there is something obvious that the parser can do to make sense of a document outside of the letter of the standard, then it tries to do that.

There is currently one instance in which it can be useful for the parser deliberately to violate the standard, as a workaround for an error commonly encountered in VOTable documents. According to the standard, if a FIELD (or PARAM) element is declared like this: (1) ]]> it is considered equivalent to (2) ]]> that is, it describes a column in which each cell contains a single character (the same remarks apply to datatype="unicodeChar"). In fact, when people (or machines) write (1) above, what they often mean to say is (3) ]]> that is, it describes a column in which each cell contains a variable length string. In particular, some tables returned from the service have contained this defect. Working to the letter of the standard, this can lead to columns in which only the first character of the string in cell is visible. By default STIL interprets (1) above, in accordance with the standard, to mean (2). However, if you want to work around the problem and interpret (1) to mean (3), by using VOElementFactory's method or variable, or from outside the program by setting the system property votable.strict="false".

Depending on your application, you may wish to provide the option of output to tables in a range of different formats including VOTable. This can be easily done using the generic output facilities described in .

The simplest way to output a table in VOTable format is to use a , which will output a VOTable document with the simplest structure capable of holding a TABLE element, namely: ]]> The writer can be configured/constructed to write its output in any of the formats described in (TABLEDATA, inline FITS etc) by using its and methods. In the case of streamed output which is not inline, the streamed (BINARY or FITS) data will be written to a with a name similar to that of the main XML output file.

Assuming that you have your StarTable ready to output, here is how you could write it out in two of the possible formats: void outputAllFormats( StarTable table ) throws IOException { // Create a default StarTableOutput, used for turning location // strings into output streams. StarTableOutput sto = new StarTableOutput(); // Obtain a writer for inline TABLEDATA output. VOTableWriter voWriter = new VOTableWriter( DataFormat.TABLEDATA, true ); // Use it to write the table to a named file. voWriter.writeStarTable( table, "tabledata-inline.xml", sto ); // Modify the writer's characteristics to use it for referenced FITS output. voWriter.setDataFormat( DataFormat.FITS ); voWriter.setInline( false ); // Use it to write the table to a different named file. // The writer will choose a name like "fits-href-data.fits" for the // actual FITS file referenced in the XML. voWriter.writeStarTable( table, "fits-href.xml", sto ); }

You may wish for more flexibility, such as the possibility to write a VOTable document with a more complicated structure than a simple VOTABLE/RESOURCE/TABLE one, or to have more control over the output destination for referenced STREAM data. In this case you can use the class which handles only the output of TABLE elements themselves (the hard part), leaving you free to embed these in whatever XML superstructure you wish.

Once you have obtained your VOSerializer by specifying the table it will serialize and the data format it will use, you should invoke its method followed by either or . For inline output, the output should be sent to the same stream to which the XML itself is written. In the latter case however, you can decide where the streamed data go, allowing possibilities such as sending them to a separate file in a location of your choosing, creating a new MIME attachment to a message, or sending it down a separate channel to a client. In this case you will need to ensure that the href associated with it (written into the STREAM element's href attribute) will direct a reader to the right place.

Here is an example of how you could write a number of inline tables in TABLEDATA format in the same RESOURCE element: \n" ); out.write( "\n" ); out.write( "Some tables\n" ); for ( int i = 0; i < tables.length; i++ ) { VOSerializer.makeSerializer( DataFormat.TABLEDATA, tables[ i ] ) .writeInlineTableElement( out ); } out.write( "\n" ); out.write( "\n" ); out.flush(); } ]]> and here is how you could write a table with its data streamed to a binary file with a given name (rather than the automatically chosen one selected by VOTableWriter): \n" ); out.write( "\n" ); out.write( "\n" ); DataOutputStream binOut = new DataOutputStream( new FileOutputStream( binaryFile ) ); VOSerializer.makeSerializer( DataFormat.BINARY, table ) .writeHrefTableElement( out, "file:" + binaryFile, binOut ); binOut.close(); out.write( "\n" ); out.write( "\n" ); out.write( "\n" ); out.flush(); } ]]>

VOSerializer contains some more fine-grained methods too which can be used if you want still further control over the output, for instance to insert some GROUP elements after the FIELDs in a table. Here is an example of that: void dummy8(StarTable table) throws IOException {\n" ); out.write( "\n" ); out.write( "\n" ); VOSerializer ser = VOSerializer .makeSerializer( DataFormat.TABLEDATA, table ); ser.writeFields( out ); out.write( "" ); ser.writeInlineTableElement( out ); out.write( "\n" ); out.write( "\n" ); out.write( "" ); ]]>}