B.41.2 Examples

Here are some examples of tpipe in use with explanations of what's going on. For simplicity these examples assume that you have the stilts script installed and are using a Unix-like shell; see Section 3 for an explanation of how to invoke the command if you just have the Java classes.

stilts tpipe cat.fits

Writes a FITS table to standard output in human-readable form. Since no mode specifier is given, omode=out is assumed, and output is to standard output in text format.

stilts tpipe cmd='head 5' cat.fits.gz

Does the same as the last example, but with one processing step: only the first five rows of the table are output. In this case, the input file is compressed using gzip - this is automatically detected.

stilts tpipe ifmt=csv xxx.csv \
             cmd='keepcols "index ra dec"' \
             omode=out ofmt=fits xxx.fits

Reads from a comma-separated values file, writes to a FITS file, and discards all columns in the input table apart from INDEX, RA and DEC. Note the quoting in the cmd argument: the outer quotes are so that the argument of the cmd parameter itself (keepcols "index ra dec") is not split up by spaces (to protect it from the shell), and the inner quotes are to keep the colid-list argument of the keepcols command together.

stilts tpipe ifmt=votable \
             cmd='addcol IV_SUM "(IMAG+VMAG)"' \
             cmd='addcol IV_DIFF "(IMAG-VMAG)"' \
             cmd='delcols "IMAG VMAG"' \
             omode=out ofmt=votable \
       < tab1.vot \
       > tab2.vot

Replaces two columns by their sum and difference in a VOTable. Since neither the in nor out parameters have been specified, the input and output are actually byte streams on standard input and standard output of the tpipe command in this case. The processing steps first add a column representing the sum, then add a column representing the difference, then delete the original columns.

stilts tpipe cmd='addskycoords -inunit sex fk5 gal \
                               RA2000 DEC2000 GAL_LONG GAL_LAT' \
             6dfgs.fits 6dfgs+gal.fits

Adds columns giving galactic coordinates to a table. Both input and output tables are FITS files. The galactic coordinates, stored in new columns named GAL_LONG and GAL_LAT, are calculated from FK5 J2000.0 coordinates given in the existing columns named RA2000 and DEC2000. The input (FK5) coordinates are represented as sexagesimal strings (hh:mm:ss, dd:mm:ss), and the output ones are numeric degrees.

stilts -disk tpipe 2dfgrs_ngp.fits \
                   cmd='keepcols "SEQNUM AREA ECCENT"' \
                   cmd='sort -down AREA' \
                   cmd='head 20'

Displays selected columns for the 20 rows with largest values in the AREA column of a FITS table. First the columns of interest are selected, then the rows are sorted into descending order by the value of the AREA column, then the first 20 rows of the resulting table are selected, and the result is written to standard output. Since a sort is being performed here, it's not possible to do all the processing a row at a time, since all the AREA values must be available for comparison during the sort. Two things are done here to accommodate this fact: first the column selection is done before the sort, so that it's only a 3-column table which needs to be available for random access, reducing the temporary storage required. Secondly the -disk flag is supplied, which means that temporary disk files rather than memory will be used for caching table data.

stilts tpipe 2dfgrs_ngp.fits \
             cmd='keepcols "SEQNUM AREA ECCENT"' \
             cmd='sorthead -down 20 AREA'

Has exactly the same effect as the previous example. However, the algorithm used by the sorthead filter is in most cases faster and cheaper on memory (only 20 rows ever have to be stored in this case), so this is generally a better approach than combining the sort and head filters.

stilts tpipe omode=meta cmd=@commands.lis http://archive.org/data/survey.vot.Z

Outputs column and table metadata about a table. In this case the table is a compressed VOTable at the end of a URL. Processing is performed according to the commands contained in a file named "commands.lis" in the current directory.

stilts tpipe in=survey.fits 
             cmd='select "skyDistanceDegrees(hmsToDegrees(RA),dmsToDegrees(DEC), \
                                             hmsToDegrees(2,28,11),dmsToDegrees(-6,49,45)) \
                          < 5./60."' \
             omode=count

Counts the number of rows within a given 5 arcmin cone of sky in a FITS table. The skyDistanceDegrees function is an expression which calculates the distance between the position specified in a row (as given by its RA and DEC columns) and a given point on the sky (here, 02:28:11,-06:49:45). Since skyDistanceDegrees's arguments and return value are in decimal degrees, some conversions are required: the RA and DEC columns are sexagesimal strings which are converted using the hmsToDegrees and dmsToDegrees functions respectively. Different versions of these functions (ones which take numeric arguments) are used to convert the coordinates of the fixed point to degrees. The result is compared to a constant expression representing 5 arcminutes in degrees. Any rows of the input table for which this comparison is true are included in the output. An alternative function, skyDistanceRadians which works in radians, is also available. These functions and constants used here are described in detail in Section 10.6.5 and Section 10.6.6.

stilts tpipe ifmt=ascii survey.txt \
             cmd='select "OBJTYPE == 3 && Z > 0.15"' \
             cmd='keepcols "IMAG JMAG KMAG"' \
             omode=stats

Calculate statistics on the I, J and K magnitudes of selected objects from a catalogue. Only those rows with the given OBJTYPE and in the given Z range are included. The minimum, maximum, mean, standard deviation etc of the IMAG, JMAG and KMAG columns will be written to standard output.

stilts -classpath lib/drivers/mysql-connector-java.jar \
       -Djdbc.drivers=com.mysql.jdbc.Driver \
       tpipe in=x.fits cmd="explodeall" omode=tosql \
             protocol=mysql host=localhost db=ASTRO1 dbtable=TABLEX \
             write=dropcreate user=mbt

Writes a FITS table to an SQL table, converting array-valued columns to scalar ones. To make the SQL connection work properly, the classpath is augmented to include the path of the MySQL JDBC driver and the jdbc.drivers system property is set to the JDBC driver class name. The output will be written as a new table named TABLEX in the MySQL database named ASTRO1 on a MySQL server on the local host. The password, if required, will be prompted for, as would any of the other required parameters if they had not been given on the command line. Any existing table in ASTRO1 with the name TABLEX is overwritten. The only processing done here is by the explodeall command, which takes any columns which have fixed-size array values and replaces them in the output with multiple scalar columns.

java -classpath stilts.jar:lib/drivers/mysql-connector-java.jar
     -Djdbc.drivers=com.mysql.jdbc.Driver \
     uk.ac.starlink.ttools.Stilts \
     tpipe in=x.fits \
           cmd=explodeall \
           omode=out \
           out="jdbc:mysql://localhost/ASTRO1?user=mbt#TABLEX"

This does exactly the same as the previous example, but achieves it in a slightly different way. In the first place, java is invoked directly with the necessary flags rather than getting the stilts script to do it. Note that you cannot use java's -jar flag in this case, because doing it like that would not permit access to the additional classes that contain the JDBC driver. In the second place we use omode=out rather than omode=tosql. For this we need to supply an out value which encodes the information about the SQL connection and table in a special URL-like format. As you can see, this is a bit arcane, which is why the omode=tosql mode can be a help.

stilts tpipe USNOB.FITS cmd='every 1000000' omode=stats

Calculates statistics on a selection of the rows in a catalogue, and writes the result to the terminal. In this example, every millionth row is sampled.