Next Previous Up Contents
Up: Release Notes
Previous: Acknowledgements


C.2 Version History

Releases to date have been as follows:

Version 0.1b (29 April 2005)
First public release
Version 0.2b (30 June 2005)
Version 1.0b (30 September 2005)
This is the first non-experimental release of STILTS, and it incorporates major changes and backward incompatibilities since version 0.2b.

Parameter system
The parameter system has undergone a complete rewrite; there is now only a single command "stilts", invoked using the stilts script or the stilts.jar jar file, and the various tasks are named as subsequent arguments on the command line. Command arguments are supplied after that. The new invocation syntax is described in detail elsewhere in this document. As well as invocation features such as improved on-line help, optional prompting, parameter defaulting, and more uniform access to common features, this will make it more straightforward to wrap these tasks for use in non-command-line environments, such as behind a SOAP or CORBA interface, or in a CEA-like execution environment.
Crossmatching
A new command tmatch2 has been introduced. This provides flexible and efficient crossmatching between two input tables. Future releases will provide commands for intra-table and multi-table matching.
Concatentation
A new command tcat has been introduced, which allows two tables to be glued together top-to-bottom. This is currently working but very rudimentary - improvements will be forthcoming in future releases.
Calculator
A new utility command calc has been introduced, which performs one-line expression evaluations from the command line.
Pipeline filters
The following new filter commands for use in tpipe and other commands have been introduced:
  • addskycoords: calculates new celestial coordinate pair from existing ones (FK4, FK5, ecliptic, galactic, supergalactic)
  • replacecol: replaces column data, using existing metadata
  • badval: replaces given 'magic' value with null
  • replaceval: replaces given 'magic' value with any specified value
  • tablename: edits table name
  • explodecols and explodecols commands replace explode

The new stream parameter of tpipe now allows you to write filter commands in an external file, to facilitate more manageable command lines.

Wildarding for column specification is now allowed for some filter commands.

Algebraic functions
  • New functions for converting time values between different coordinate systems (Modified Julian Date, ISO-8601, Julian Epoch and Besselian Epoch).
  • New RANDOM special function.
Documentation
SUN/256 has undergone many changes. Much of the tool documentation is now automatically generated from the code itself, which goes a long way to ensuring that the documentation is correct with respect to the current state of the code.

Version 1.0-1b (7 October 2005)
Fixed jar file manifest bug which prevented working on Java 1.5
Version 1.1 (10 May 2006)
A number of new features and capabilities have been introduced:
tcube Command
The new tcube command calculates N-dimensional histograms (density maps) from N columns of an input table and writes the result to a FITS file.
Processing Filters
The following new filters have been added:
  • stats filter provides the same information as the old stats output mode, but allows much more flexible use of the results. It can also calculate many new quantities, including quantiles, skew and kurtosis.
  • meta filter provides the same information as the old meta output mode, but allows much more flexible use of the results.
  • assert filter provides in-pipeline logical assertions.
  • uniq filter collapses multiple adjacent identical or similar rows.
  • sorthead filter provides a (usually) more efficient method of doing what you could previously do by combining sort and head filters.
  • colmeta filter adds/modifies metadata for selected columns.
  • check filter checks table in stream - for debugging purposes only.

Additionally usage of the sort filter has been changed so that it can now do everything that sortexpr used to be able to do; sortexpr is now withdrawn.

Output Modes
The following new output modes have been introduced:
  • plastic mode broadcasts the table to one or all registered PLASTIC listeners.
  • cgi mode writes the table to standard output in a form suitable for output from a CGI script.
  • discard mode throws away the table.
and usage of the following has been modified:
  • topcat mode now attempts to use PLASTIC (amongst other methods) to contact TOPCAT.
  • stats and meta modes are mildly deprecated in favour of the corresponding new filters (see above).
Other new features
  • New IPAC table format input handler added.
  • New csv-noheader format variant output handler added.
  • roundDecimal and formatDecimal functions introduced for more control over visual appearance of numeric values.
  • Experimental facilities for automatically generating a CEA application description file.
Bug fixes and minor improvements
  • Now copes with 'K'-format FITS binary table columns (64-bit integers).
  • Improved, though still imperfect, retention of table-wide metadata in VOTables.
  • Distinctions between null and false values in boolean columns are handled more carefully for FITS and VOTable files.
  • Efficiency improvement when writing FITS-plus format (now only requires a maximum of two passes rather than three of the input rows).
  • Added the mark.workaround system property which can optionally work around a bug in some input streams ("Resetting to invalid mark" errors).
  • Fixed a bug in Cartesian matching which failed to match if the required error in any dimension was zero.
  • Fixed erroneous reports about unknown ucd and utype attributes of TABLE element in votlint.
  • When joining tables, column name comparison to determine whether deduplication is required is now case-insensitive.
  • Error message improved when no automatic format detection is attempted for streamed tables.
  • Setting istream=true is now less likely to cause a "Can't re-read stream" error.
Version 1.2 (7 July 2006)
Column-oriented Storage
New features for permitting column-oriented storage (colfits format, new startable.storage policy "sideways") have been introduced. These can provide considerable efficiency improvements for certain tasks when working with very large (and especially wide) tables.
New VO commands
Added two new commands for querying Virtual Observatory services:
  • multicone - Makes multiple cone search queries to the same service
  • regquery - Queries the VO registry
These tasks are experimental and may be modified or renamed in future releases.
Other items
  • transpose filter added.
  • Added flux conversion functions (Jansky<->magnitude).
  • ISO-8601 strings now permit times of 24:00:00 as they should.
Version 1.2-1 (3 August 2006)
Version 1.3 (5 October 2006)
Table Concatenation
The old tcat command has been replaced by more capable tcat and tcatn commands. Between them these provide concatenation of an unlimited number of homogeneous or heterogeneous input tables. Additional columns may be added to indicate which of the input tables given output rows originated from.
Parameter value indirection
Certain parameters (in in tcat, cmd and friends) may now be specified in the form "@filename". This indicates that the value for the parameter is to be obtained by reading it from the named file. This is useful if a very long value is required for the parameter in question. The script parameter of tpipe has therefore been withdrawn, since it did just the same thing.
MySpace access
Direct access to the MySpace virtual file system is now provided by use of ivo:- or myspace:-type URLs.
Conversion functions
  • Time conversion functions between MJD and Decimal Year have been added (Section 10.7.20).
  • toHex and fromHex numeric conversion functions have been added (Section 10.7.4).
Documentation improvements
  • The HTML version of SUN/256 now uses CSS to provide better highlighting of commands etc.
  • The Output Modes and Processing Filter sections are now split into subsections to make the table of contents clearer.
  • The Command Reference section now has only one level of subsection listed in the table of contents to make it clearer.
Other new features and improvements
  • Added -J flag to stilts script for passing flags directly to Java.
  • Added new out parameter to votlint.
  • Added -ifndim and -ifshape flags to explodeall filter.
  • The exact match mode in tmatch2 now copes with array-valued columns.
  • Added force parameter to multicone task as a workaround for some broken services.
  • Added Sample (as opposed to Population) Standard Deviation/Variance calculation options to the stats filter.
  • Improved CEA description file output - now contains details of all tasks rather than just a few, as well as various improvements in documentation etc.
Bug fixes
  • Fixed erroneous complaints from votlint about utype attribute on RESOURCE elements.
  • Fixed a couple of minor crossmatching bugs (which wouldn't have affected results).
Version 1.3-1 (Starlink Hokulei release)
Version 1.3-2 (6 July 2007)
Version 1.3-3 (4 Sep 2007)
Version 1.3-4 (10 Sep 2007)
Version 1.3-5 (30 Oct 2007)
Version 1.4 (6 December 2007)
Table joins
This version provides more cross matching functionality. Added to the existing tmatch2 command are new tasks:
  • tskymatch2: stripped down version of tmatch2 for ease of use when matching with sky coordinates.
  • tmatch1: internal matcher, finds groups of objects within a table.
  • tmatchn: finds group or multiple-pair matches between multiple (>2) tables.

Two tasks have been renamed for improved clarity and consistency:

There has also been some enhancement and rationalisation of parameters for all table join tools (tmatch* as well as tjoin, coneskymatch and sqlskymatch):

  • All table join commands now use similar fixcols and suffix* parameters to control renaming of duplicated columns in output tables (note this replaces the old duptag* parameters in tmatch2).
  • Crossmatching tasks have a new progress parameter which allows you to configure whether progress is reported to the console.
  • The copycols parameter of coneskymatch and sqlskymatch now defaults to "*" (include all columns from input table in the output).

Section 7 of the manual has been somewhat rearranged and improved.

Other enhancements
  • FITS reader now imports table HDU header cards as table parameters.
  • CeaWriter can now output CEA service definition XML config file as well as app-description file (experimental - may be withdrawn).
Bug fixes
  • Embedded spaces in output ASCII format table column names are now substituted with underscores.
  • Fix a bug which caused an infinite number of dots to be printed when attempting a crossmatch with an empty input table.
  • Corrected votlint handling of TABLEDATA-type multi-dimensional char/unicodeChar arrays. These are now split up into strings by counting characters rather than using whitespace delimiters. I think it's doing the right thing now.
Version 1.4-1 (28 January 2008)
New RDBMS-related features
  • New command sqlclient, which is a general JDBC-based SQL command-line client.
  • New command sqlupdate, which allows updates to existing rows in SQL tables.
  • Some changes to tosql output mode:
    • choice of options for how to write to the database output table, controlled by new associated parameter write (can be create, dropcreate or append)
    • associated parameter newtable renamed dbtable
    • associated parameter database renamed db for consistency with other commands
Local and service-based matching command enhancements
  • New parameter scorecol added to tmatch2, coneskymatch and sqlskymatch commands, which controls adding a new column to match output tables containing a goodness-of-match value.
  • New parameter parallel added to coneskymatch task which allows multiple cone searches to be carried out in parallel.
  • New parameter erract added to coneskymatch which controls response to isolated failures in individual cone search queries.
General improvements
  • Improved error reporting (reasons for errors are now reported even without the -debug flag).
  • Add new help option help='*' which prints help for all parameters of a task at once.
  • Added (mostly undocumented) +verbose flag for reducing verbosity level.
  • Minor improvements to CEA app-description.
  • Downgraded from WARNING to INFO log messages about the (extremely common) VOTable syntax error of omitting a FIELD/PARAM element's datatype attribute.
Version 1.4-2 (26 March 2008)
Minor enhancements:
  • Add progress parameter to tmatchn.
  • Add emptyok parameter to coneskymatch.
Bugfixes:
  • Fixed pair matching performance bug (slower if tables were not given in the right order) introduced at v1.4.
  • Fixed null handling error in calc task.
  • Fixed error in stats filter cardinality value calculation.
  • Fixed minor bugs in suffix addition for matching commands fixcols.
  • Removed unformatted XML output in stats filter usage message.
  • Try to avoid exponential format in cone search URLs (some endpoints seem to require fixed point format).
  • Minor CEA fixes.
Version 2.0b (23 October 2008)
This version contains two new major items, plotting and server mode. Both work, but are missing desirable features and have not had extensive testing in the field, so should be considered experimental at this stage.
Plotting
Two table plotting commands are now provided: See also the new Plotting section in the manual.
Server/Servlet Mode
A new command server is provided which allows STILTS commands to be executed via HTTP. One purpose of this is to facilitate server-side use of the plotting commands co-located with data to generate on-the-fly graphical summaries of server-held datasets.
Smaller enhancements and bugfixes
  • Efficiency improvements (~25%? in both CPU time and memory usage) for HEALPix-based sky crossmatching (thanks to Nikolay Kouropatkine at Fermilab for a new version of the PixTools library).
  • New class Arrays added to algebraic functions.
  • New Appendix Commands by Category added to manual.
  • Add minReal and maxReal functions (max/min ignoring blank values) in class Arithmetic.
  • Sexagesimal field identification for ASCII input files is less stringent (now permits minutes or seconds equal to 60).
  • Minor CEA fixes.
  • HEALPix bug fix (PixTools bug fix update).
  • Fix bug in use of tcat's loccol parameter.
Version 2.0-1 (23 December 2008)
Version 2.0-2 (9 January 2009)
Version 2.0-3 (27 March 2009)
Version 2.0-4 (17 July 2009)
Version 2.0-5 (2 Oct 2009)
Version 2.1 (6 November 2009)
Note that the STIL API used by this release has changed in some backwardly incompatible ways, and may change further. If you're using STILTS as a library rather than an application you might want to wait for a later release when the API has settled down.
Version 2.1-1 (21 December 2009)
Version 2.1-2 (24 March 2010)
Version 2.2 (6 August 2010)
New capabilities for multi-table I/O have been introduced: There are some additional enhancements: And a number of bug fixes:

Finally, from this release STILTS requires version 1.5 (a.k.a. 5.0) of the Java J2SE Runtime Environment; it will no longer run on version 1.4, which is now very old. I don't expect this to cause compatibility issues for anyone, but I'm interested to hear if that's not the case.

Version 2.2-1 (23 December 2010)
Version 2.3 (9 May 2011)
TAP
The new commands tapquery and tapresume have been introduced. These provide support for the Table Access Protocol (TAP), and allow freeform queries in an SQL-like language to be made to remote databases.
Minor enhancements
  • Random Groups HDUs are now tolerated, though not interpreted, within FITS files.
  • Added soapout parameter to regquery command.
  • Added count, variance and stdev functions to Arrays.
  • Upgrade to JSAMP v1.2.
  • Improve text rendering in funcs window display.
  • Attempt case-sensitive matching before case-insensitive for column names.
  • Fix replaceval filter to work with Infinities.
Bug fixes and workarounds
  • JDBC table input handler now effectively downcasts BigInteger/BigDecimal types to Long/Double. The PostgreSQL JDBC driver seems to use the Big* types routinely for numeric values (which I don't think it used to do).
  • Add workaround for J2SE bug #4795134, which could cause errors when reading compressed FITS files.
  • Fix FITS character handling bug which could cause corrupted FITS files on output in presence of non-ASCII characters.
  • Fix (some) JDBC connection leaks.
  • Add missing parameters dashNS and linewidthNS to plot2d task.
Version 2.3-1 (30 June 2011)
Version 2.4 (27 October 2011)
Crossmatching:
  • Two new asymmetric match options best1 and best2 have been added for the find parameter in the pair matching commands tmatch2 and tskymatch2. They correspond to finding the best match in table B for each row in table A, and in crowded fields often provide more intuitive semantics than the previous symmetric best option (in non-crowded fields there is generally no difference). This replicates the matching performed by some other tools, including Aladin.
  • New matchers have been added to permit matching of general elliptical, rather than just circular, regions in both planar and sky coordinates; see 2d_ellipse, and skyellipse.
  • Another new matcher is available for dealing with per-object errors in Cartesian coordinates (previously per-object errors could only be handled in sky coords); see Nd_err.
  • Semantics of the skyerr matcher have changed slightly.
Expression language functions:
  • Algebraic functions involving angles are now mostly available using degrees as well as radians. The Coords class has been replaced by CoordsDegrees and CoordsRadians classes providing sky coordinate functions, and a new class TrigDegrees provides normal degree-based trigonometric functions alongside the radian-based versions in Maths. Some of the old function names have changed to make clear that they use radians and not degrees. This change should be much more convenient in most cases; sorry it's taken so long to get round to.
  • Add new join function is added to the Arrays class to combine all the elements of an array into a string.
taplint:
There are several bugfixes and changes related to the TAP validator tool taplint, mostly thanks to bug reports etc from the TAP community:
  • Improve test logic for record limiting queries.
  • Errors no longer reported (e.g. E-Qxx-CNAM) for unexpected TAP_SCHEMA table column ordering (when running query stage but no metadata acquisition stages).
  • Add new stage MDQ, which checks query result columns for all tables against declared metadata.
  • Add check of versioned and unversioned LANG variants.
  • Now uses corrected upload ID (ivo://ivoa.net/std/TAPRegExt#upload-*) as per most recent TAPRegExt draft.
Bug fixes and minor enhancements:
  • Add parameter parse to tapquery command, allowing pre-send syntax checking of submitted ADQL.
  • Add experimental system properties star.basicauth.user and star.basicauth.password.
  • Improve resilience of coneskymatch in the presence of unreliable or inconsistent DAL services.
  • A PARAMref element with no referent in a VOTable no longer causes an uncaught NullPointerException.
Version 2.5 (28 March 2013)
New coverage-related functionality:
  • Add new command pixsample which can sample pixel data from HEALPix table files (useful for things like Schlegel dust extinction). Also addpixsample filter, which does the same job.
  • Add new command pixfoot which can generate MOC (Multi-Order Coverage) maps.
  • Add MOC-based coverage filter to coneskymatch when using some Cone Search services (mostly VizieR). This uses the Multi-Order Coverage map service operated by CDS. It can make VizieR multi-cone queries much faster by not doing cone searches that are outside the coverage region of the catalogue in question.
  • Add new class Coverage to the expression language containing MOC-related functions (currently, just inMoc).
Other new capabilities:
  • Add IPAC table output format.
  • Add new class KCorrections to the expression language, containing a method for calculating K-corrections following the method of Chilingarian and Zolotukhin.
  • You can now reference tables in multi-extension FITS files by name (EXTNAME or EXTNAME-EXTVER) as an alternative to by HDU index.
VOTable enhancements:
  • VOTable input, output and validation are now supported for version 1.3 of the VOTable standard.
  • The version of the VOTable format used for VOTable output can now be selected, by using the system property votable.version. Output version is VOTable 1.2 by default.
  • votlint has been changed so that it handles different VOTable versions more capably. Versions 1.1+ are now validated against a schema (which is how those versions are defined) rather than against a DTD hacked to do the same job as the schema. VOTable 1.3 validation is now provided.
  • The votcopy command has a new version parameter to control output version, and a new nomagic parameter to control whether VALUES/null attributes are removed where appropriate.
  • Infinite floating point values are now correctly encoded in VOTable output ("+Inf"/"-Inf", not "Infinity"/"-Infinity" as in previous versions).
  • votlint is now stricter about floating point TD element contents.
  • VOTable output no longer writes the schemaLocation attribute by default.
Other enhancements:
  • Add new function hypot (=sqrt(x*x+y*y)) to the Maths class in expression language.
  • Add new split functions for string splitting to the Strings class in expression language.
  • Add -utype flags for addcol, replacecol, colmeta and setparam filters, and utype option for meta filter.
  • Some changes to the toString function: it now works on non-numeric values, gives the right answer for Long integers and character values, and returns a blank value rather than the string "null" or "NaN" for blank inputs.
  • Sexagesimal to numeric angle conversion functions now permit the seconds part of the sexagesimal string to be missing.
  • Changes to the IPAC format definition are accommodated: the "long"/"l" type, which is apparently now official, no longer generates a warning, and headers may now use minus signs instead of whitespace.
  • Add OBS stage (ObsTAP validation) to taplint.
  • Add more checks to CAP stage of taplint. Declared languages (including features) and output formats are now checked.
  • Tidy up error reporting a bit (fewer duplicate nested messages reported).
  • PNG graphics output no longer has transparent background.
  • Issue a warning for high values of coneskymatch parallel parameter.
  • Upgrade JSAMP library to version 1.3-3.
  • Upgrade Grégory Mantelet's ADQL library to version 1.1.
Bug fixes:
  • Fix serious and long-standing bug (bad TZERO header, causes subsequent reads to fail) for FITS output of boolean array columns.
  • Fix small but genuine sky matching bug. The effect was that near the poles matches near the specified threshold could be missed. The bug was in the PixTools library, fixed at the 2012-07-28 release.
  • Fix bug in tmatchn group mode which could result in output rows with columns from only a single table, i.e. not representing an inter-table match, even when join*=default.
  • Fix bug which failed when attempting to read FITS files with complex array columns (TFORMn=rC/rM).
  • Fix failure when caching very large sequential tables.
  • Fix bug in replacecol and replaceval filters which could cause truncation of strings in FITS and possibly VOTable output when the new value was longer than the previously declared maximum length.
  • Fix tcat, tcatn so that in most cases output column metadata is compatible with all input tables, not just the first one in terms of nullability, array shape etc.
  • Adjust SQL writer to avoid a type error for MySQL.
  • Fix bug in HMS sexagesimal formatting: minus sign was omitted from negative angles. Now the output is forced positive.
  • Cope with 1-column CSV files.
  • Use the correct form "rows"/"bytes" rather than "row"/"byte" for TAP capability unit values.
  • Fix error bar rendering bug which could result in diagonal lines being offset near the edge of plots.
Version 2.5-1 (1 July 2013)
New functionality
  • Add read-only support for CDF (NASA Common Data Format) files.
  • Add Median Absolute Deviation calculation (MedAbsDev and ScMedAbsDev) options to stats filter.
  • Improved handling of HTTP basic authorization. 401s now generate a useful message about the star.basicauth.* system properties if they have not been set up.
Bug fixes and minor enhancements
  • Fix CSV regression bug introduced at v2.5 - CSV files now work again with MSDOS-style line breaks.
  • Fixed FITS output bug which could result in badly-formed string-valued header cards (no closing quote).
  • Source code is now managed by git and not subversion. The format of the "Starjava revision" string reported by the -version flag has changed accordingly.
  • Output mode meta now copes better with array-valued table parameters.
  • Implemented fixes to reduce the chance of users inadvertently overloading external Cone/SIA/SSA services with multicone-like queries. First, fix it so that abandoned queries are properly terminated, rather than continuing to hit the server until completion or JVM shutdown. Second, implement a sensible default maximum value for the parallel parameter of skyconematch (though this may be adjusted with a system property).
  • Quoting behaviour has changed when generating SQL to write to RDBMS tables. This ought to reduce problems related to mixed-case identifiers. However, it is possible that it could lead to unforseen new anomalies.
  • More toString overloads - now works for byte and boolean values too.
Version 2.5-2 (7 March 2014)
Version 2.5-3 (4 July 2014)
New and improved functionality:
  • Add new command cdsskymatch. In most cases (for querying tables that can be found in VizieR) this can and should be used instead of coneskymatch - it's much faster.
  • Commands coneskymatch, sqlskymatch and pixfoot will now guess RA/Dec columns if relevant parameters are left blank.
  • Added new graphics output format png-transp to generate PNG files with transparent backgrounds.
  • Upgraded Gregory Mantelet's ADQL library to version 1.2. Better ADQL parsing.
Improvements and adjustments to taplint:
  • Rework taplint API to facilitate static acquisition of report codes during programmatic use. A few error codes have changed.
  • Add new "duff query" test to taplint.
  • Avoid taplint MDQ stage data type mismatch error report for BOOLEAN/boolean declared/returned data.
  • taplint now takes steps to ensure that TAP_SCHEMA column list query is not truncated.
  • taplint now flags absence of ObsCore table with I[nfo] not F[ailure] status.
  • Change the implementation of taplint stages which perform validation against XSD schemas. Schemas from external namespaces may now be imported and used. The CPV stage, which was previously broken and disabled by default, is now fixed and enabled by default. Known/expected schemas are stored locally, and a warning is reported if external ones are used. Schema validation seems remarkably complicated, so it's possible there are still errors in this implementation - if you suspect so, please report it.
  • Add missing geometric reserved words to ADQL reserved word list. This fixes some problems with column names like "DISTANCE" in taplint tests.
  • Fixed some bugs related to TAP table uploads. In particular these could cause incorrect table upload error reports in taplint.
Version 3.0 (3 October 2014)
New plotting commands:
A set of new plotting commands are provided which give comprehensive access to all the new-style visualisation capabilities available in TOPCAT v4. These commands are documented in Section 8. These commands, and the underlying visualisation facilities, are considerably more capable than the, now deprecated, old-style plot commands plot2d, plot3d and plothist.
Programmatic invocation:
Programmatic invocation of STILTS tasks from third-party java code is now officially sanctioned and documented in the new Section 12. To support this changes have been made to the parameter system (Parameter class now supports generics) and there are some visible changes to the user documentation as well (parameters now report their data type, and tasks report their classname). Normal (e.g. command-line) usage should not undergo any changes, but a fair bit of UI code has changed, so unexpected problems are possible.
Other items:
  • Add new output mode gui, which displays the table data in a scrollable window on the screen.
  • Add new -allowunused flag to the stilts command. If this is set, then unused parameter settings on the command line just result in a warning, not failure of the command.
  • Attempting to write FITS tables with >999 columns now fails with a more helpful error message.
  • Improved Unicode handling in VOTables. Fixed a serious bug in votcopy that generated unreadable output to BINARY or BINARY2 serialization if any non-empty column had datatype="unicodeChar". Also improved behaviour when copying between tables with unicodeChar columns; these are usually preserved now, rather than being squashed to datatype char. Some lurking Unicode-related issues remain.
  • The TAP client now tolerates whitespace around UWS status codes.
  • taplint: downgrade unknown post-table QUERY_STATUS value message from Error to Warning.
Version 3.0-1 (13 November 2014)
New functionality:
  • Add (experimental) read-only support for Gaia/DPAC GBIN format.
  • Add new task tapskymatch.
  • Functions in class Coverage adjusted: new function nearMoc, and MOC can be identified by VizieR table IDs as well as by filename/URL.
  • For repeat filter, add -row|-table flags to control sequence of output rows.
  • For setparam and repeat filters, allow use of an algebraic expression for values, not just a literal value.
  • Add special values $ncol and $nrow to the expression language to refer to the column and row counts in a table. The special variable index is also deprecated in favour of $index or $0.
Bugfixes and minor improvements:
  • Add some more colour maps for aux/density shading.
  • Fix stilts invocation script to pick up classes from stilts.jar in script directory in preference to other places (e.g. topcat-full.jar).
  • Fix taplint to permit application/xml not just text/xml content-type where appropriate (UWS stage).
  • Fix taplint so it doesn't warn (W-TMV-UNSC) about unknown VOSITables schema.
  • Fix taplint so that unicodeChar matches CHAR/VARCHAR in the same way as char for column type declaration purposes.
  • Fix taplint so that capabilities document can have TAPRegExt dataModel ivo-id elements with xs:anyURI rather than vr:IdentifierURI (only a warning is issued in the latter case), in anticipation of TAPRegExt-1.0 Erratum #1.
  • Adjust taplint to handle adql:TIMESTAMP columns more carefully on upload and retrieval.
  • Update JSAMP to v1.3.5.
Version 3.0-2 (6 February 2015)
Plotting enhancements:
  • Linear fitting of points is now available using the linearfit layer type for plot2plane. Points may be weighted.
  • You can add titles to plots using the new title parameter.
  • New plot layer type sizexy allows plotting (optionally autoscaled) markers with horizontal and vertical extents independently determined by input data.
  • More flexibility when assigning colour maps, in aux and density shading modes, and spectrogram layer. New parameters *func allow assignment of different data->ramp mapping functions (sqrt and square as well as linear and logarithmic), and new parameters *quant allow quantisation of the colour map to discrete levels.
  • Replace maxsizeN parameter with autoscaleN for size plot layer type. You can now optionally turn off autoscaling and specify marker size in pixels instead.
  • Add auxcrowd parameter to plot2 tasks to influence tick crowding on aux axis colour ramp. Also adjust default to use fewer ticks.
  • Add some "dart" options (fixed-base open or filled triangles) for plotting vectors (see arrowN parameter in layers like xyvector).
  • Add some "triangle" options (variable-base open or filled triangles) for plotting ellipses (see ellipseN parameter in layers like xyellipse).
  • Histogram normalisation option adjusted so that total area under bars, rather than total height of bars, is fixed.
  • The PlotDisplay class that forms the result of plot2 commands can now have PointSelectionListeners registered on it. This lets you determine what point a user has clicked on if you're using the plotting classes from third party java code.
FITS I/O:
  • Reworked part of the FITS table input implementation, in particular adjusting the way memory mapping is done to reduce resource requirements on some platforms. If you notice any difference, it should be reduced virtual and perhaps resident memory usage, and some (~10%?) performance improvements, when reading large FITS/colfits files. If you were previously having problems with large memory allocations leading to disk thrashing and system lockup when scanning files larger than RAM (this didn't happen on all OSes), these will hopefully have gone away. However, please report anything that appears to be working worse than before, or continued memory usage issues.
  • Colfits files can now be accessed from streams, not just uncompressed disk files (though that's not necessarily a good idea).
Bugfixes and workarounds:
  • Fixed a query bug (missing REQUEST=queryData parameter) in the multi-SSA mode (servicetype=ssa) of coneskymatch. This long-standing bug would have stopped this command working at all with well-behaved SSA services.
  • Fixed error in fits-var output (PCOUNT header card did not include block alignment gap).
  • Graphics coordinates are now calculated in floating point rather than as integers. This fixes problems that could cause scaled vectors, ellipses etc to be drawn with shapes or orientations badly wrong due to rounding errors. It also improves plotting of analytic functions, especially to vector contexts (PDF/EPS).
  • Fix some problems to do with zooming to very large/small plot axis ranges.
  • Hide error bars (etc) that would extend to negative values on logarithmic axes; previously they were being drawn in anomalous places.
  • Fix NullPointerException bug when null value was supplied to multi-word parameter (e.g. tcube).
  • Fix Aux axis positioning for 3D plots so that the numeric labels don't get snipped off at top and bottom.
  • Add a hack that allows LDAC FITS tables to be treated sensibly in auto-format-detection mode.
  • Make VOTable handling more robust against unknown (illegal) datatypes.
  • Add missing parameters auxmax, auxmin to plotting task documentation.
Version 3.0-3 (14 April 2015)
Version 3.0-4 (17 August 2015)
Bugfixes (some significant):
  • Fix a serious bug in processing of FITS bit vector (TFORMn='rX') columns. Values read from these columns are presented as a boolean[] array. In all previous versions of STIL the bits have appeared in that array in the wrong sequence (LSB..MSB per byte rather than the other way round). Apologies to anyone who may have got incorrect science results from this error in the past, and thanks to Paul Price for helping to diagnose it.
  • Fix a less serious bug with TFORMn='rX' processing; attempting to read a single-element bit vector column (TFORMn=1X or X) previously resulted in an error making the file unreadable. Values read from such columns are now presented as Boolean scalars.
  • Fix a VOTable reading bug relating to bit vector data (datatype="bit") appearing in BINARY/BINARY2 serializations. This one was more obvious, it would usually generate an error when attempting to read the file.
  • Fix serious bug in time conversion for CDF TIME_TT2000 data types.
  • Fix a bug in votcopy that converted columns from datatype unsignedByte to short when transforming. Since v3.0-3 this is no longer necessary. In the case of converting to a binary serialization, since v3.0-3 this was causing it to generate unreadable VOTable output.
  • Fix a bug in votcopy that failed to handle columns with datatype bit. In the case of converting to a binary serialization, these were in all previous versions generating unreadable VOTable output. Now they convert them to columns with datatype boolean (not perfect, but better).
  • Fix skyvector bug: dlat and dlon values were being used the wrong way round.
  • Upgrade JEL to v2.0.2. Fixes problem with evaluating void-typed expressions, and possibly some other obscure bugs.
  • Some taplint bug fixes.
Behaviour changes:
  • Changes to the way that TAP service table/column name reports are interpreted (to conform to original intention of TAP standard). Taplint now checks that table/column names from TAP_SCHEMA and /tables endpoint are regular-or-delimited-identifiers, but no longer submits example queries using supplied column names wrapped in additional quotes.
  • Modify the heuristics that determine whether the first row of a CSV file is a header.
Enhancements (mostly minor):
  • Added some functions to the Arrays class that return array-valued results from array-valued parameters: add, subtract, multiply, divide, reciprocal, condition.
  • Improve error reporting in the face of non-VOTable TAP error responses. In many cases this makes it much easier to see what's going wrong with a TAP query.
  • As a diagnostic tool, when making TAP queries, a log message giving a roughly equivalent curl(1) command is now issued at the CONFIG level (visible using flags -verbose -verbose).
  • New taplint parameter maxtable limits the number of tables tested in the stage that queries data from each individual table (MDQ). May be useful for very large services.
  • New tapquery parameter upvotformat to determine what VOTable serialization variant is used to transmit uploaded tables to the TAP server. Previously uploads were always BINARY which ought to work, but the parameter now defaults to TABLEDATA, since some services (e.g. CADC) currently fail with binary uploads.
  • Minor improvement to version reporting (reports java specification version, no longer issues warning for absent revision string).
  • Update JCDF library to v1.1 (minor changes to do with leap seconds).
Version 3.0-5 (22 October 2015)
Version 3.0-6 (27 November 2015)
Crossmatching bug fix
Fix a long-standing crossmatch bug relating to range restriction during pre-processing. This could have caused missed associations (but not false positives) near the edge of coverage regions when using per-row errors, if the scale of the errors differed (especially differed significantly) between the matched tables. It affected matcher values of <n>-d_err, skyerr, 2d_ellipse and skyellipse only. Thanks to Grant Kennedy (IoA) for reporting this bug.
Density plots
Some more options for making weighted density plots have been added. Since v3.0 the Density shading mode has let you see the density of plotted points, but this lacked some features. Three new ways to do density plots are added; these all give you the option of weighting by an additional coordinate (like the Aux mode), choosing the combination method (mean, median, sum, max, ...), and displaying the quantitative value-colour mapping on the shared colour ramp (previously aux axis) beside the plot. The new density plots are:
  • Weighted shading mode, using shaped marker kernels on the screen pixel grid, available for all plot types
  • SkyDensity layer, using HEALPix bins on the celestial sphere, for the Sky plot
  • Density layer (later obsoleted by Grid layer), using square N*N screen pixel bins, for the Plane plot
The details are somewhat experimental and may undergo some adjustments in future releases (feedback welcome).
Colour maps
There are various changes affecting selection and display of colour maps used for density and aux axis shading:
  • The default colour map for Aux mode, and other layers using the shared colour map, is no longer Rainbow! It's Inferno. Rainbow colour maps are much hated by visualisation experts. Of course you can still choose Rainbow if you like.
  • Add some new colour maps: Viridis, Inferno, Magma and Plasma from Matplotlib 1.5, the SRON rainbow variant developed by Paul Tol, some diverging maps (HotCold, RdBu, PiYG, BrBG) and a qualitative constant chroma/luminance map HueCL.
  • The options for Density and Aux shading are now mostly the same as each other except where there's good reason to differ. Previously they were different in haphazard ways.
  • An attempt is made to give the default form of each colour map a sensible name, without leading minus signs.
  • Fix it so that the whole range of each map is distinguishable from white. This is a good idea when you're plotting symbols on a white background, which is common in stilts. Perhaps there are cases it's not such a good idea; if you think so, complain and I may change it back.
  • Try to fix it so that all the colour maps go in the same direction (light->dark) where applicable.
  • Throw out a couple of particularly useless colour maps.
  • Colour map ramp display is now different for non-absolute maps; their effect is shown on a selection of base colours, not just for one base colour.
Minor items
  • Try harder to identify epoch columns (suitable for time plot), in particular look for VOTable xtype of JD or MJD, and units of year.
  • Add some functions to the Tilings class to do with solid angles (healpixSqdeg, healpixSteradians, steradiansToSqdeg, sqdegToSteradians, SQDEG).
  • Fix plot bug; titles were painted in white for pixel output formats.
  • Rationalise plot report logging. Some more diagnostic information about plots is now logged at the INFO level (visible if topcat is run with the -verbose flag).
Version 3.0-7 (10 June 2016)
Expression Language
The JEL library underlying the expression language parser has been upgraded to v2.1.1 (thanks to Konstantin Metlov), now supporting variable-length argument lists among other things. This allows the following improvements:
  • New functions that support any number of arguments are provided: array, intArray, stringArray in class Arrays; concat, join in class Strings; and sum, mean, variance, stdev, min, max, median, countTrue in new class Lists.
  • Some old lists of similarly-named functions with fixed numbers of arguments have been replaced by single functions that take an arbitrary number of arguments (e.g. array(x1), array(x1,x2), ... array(x1,x2,x3,x4,x5,x6,x7,x8) replaced by array(values...)).
  • The 2-argument min/max functions in class Arithmetic have been renamed minNaN/maxNaN to avoid confusion, but in most cases existing expressions involving min/max will work as before.
  • Some functions that used to require string arguments will now auto-convert numeric types (e.g. concat(toString(RA),";",toString(DEC)) can now be written concat(RA,";",DEC)).
  • You can now implement user-defined functions with variable numbers of arguments.
  • Writing large (>=2**31) literal integers used to fail with an inscrutable error message. Now the message tells you to append the "L" character.
Time plots
The plot2time command has been enhanced so that it can make multi-zone plots - multiple plots stacked vertically that share the same horizontal (time) axis but have independent vertical axes. The time plot itself and this multi-zone feature are currently experimental; in future versions they may be improved or changed, and the multi-zone feature may be extended to other plot types. Some other changes and fixes have gone along with this:
  • A few API changes have been made to support multi-zone plots, including generalising the NavigationListener interface and rearranging some PlotDisplay and AbstractPlot2Task constructor/factory method arguments. For single-zone plots the changes are not very substantial. This only affects you if you are using the STILTS classes as a java plotting library. If that applies to you and you have trouble upgrading, I'm happy to provide assistance.
  • plot2time now supports shading modes (shadingN and associated parameters).
  • The spectrogram layer now uses the (per-plot, or more precisely now per-zone) Aux colour map rather than a layer-specific colour map. This means that the colour ramp is displayed alongside the plot, but also that some parameters have been renamed (e.g. auxmapZ replaces spectromapN, where Z is an optional zone suffix and N is an optional layer suffix).
  • The function layer type now works in the time plot, rather than throwing an error. However, it doesn't work very well, since the time coordinate is in unix seconds rather than something more user-friendly.
  • Fixed a serious bug in ISO-8601 axis labelling. In some cases axis labels were being drawn at positions badly different from the correct position.
Miscellaneous enhancements and changes
  • This and subsequent releases target Java SE 6, so will no longer run under the (now very ancient) Java 5 runtime.
  • Provide more careful documentation of licensing arrangements. The distributed LICENSE.txt file notes that the starjava code is LGPL, and documents licenses for each third-party dependency.
  • Add Fill layer type for Plane and Time plots.
  • tapquery and tapresume now use blocking HTTP requests rather than repeated polls to wait for asynchronous TAP job completion from services that declare themselves UWS 1.1 compliant.
  • Add new parameters executionduration and destruction to tapquery command. These let you request resource limit adjustments when submitting an asynchronous TAP job.
  • Improve sky plot border painting.
  • Clean up noisy Cubehelix colour map.
  • New function countTrue in class Arrays.
  • New stage EXA for taplint checks TAP /examples endpoint. Note the details of the examples format are still under discussion, (this version targets WD-DALI-1.1-20160415 & WD-TAP-1.1-20160428, somewhat informed by TAPNotes-2013-12-13), so the details may change in future.
  • taplint now validates ObsCore 1.1 where declared alongside ObsCore 1.0. Currently uses PR-ObsCore-v1.1-20160330.
  • The taplint API has changed slightly: the class that used to be Reporter is now called TextOutputReporter. If you are using taplint programmatically you may need to make small changes. This results from some refactoring that makes it easier to customise taplint output.
  • Replaced opaque config option with transparency for plane and sky density plots.
  • Changed implementation of GIF exporter for plots, from Acme to ImageIO. Shouldn't be any noticable difference. Acme encoding dependency removed.
Bug fixes
  • Fix bug in cumulative histogram calculation.
  • Fix read failure for FITS files with non-blank TDIM for zero-length columns.
  • Fix bugs that led to timezone-dependent results when reading ISO-8601 or decimal year time columns.
  • Fix numeric field truncation bug in LaTeX table output.
  • Fix some parameter handling errors in coneskymatch.
  • Fix NullPointerException bug for disjoint regions in some cases in tmatchn.
Version 3.0-8 (13 September 2016)
Version 3.0-9 (23 September 2016)
Version 3.1 (8 March 2017)
Plotting improvements
  • Improved documentation of plot layer types and shading modes in the user document - each option now has an example graphic and the text of the command that generates it.
  • New layer type grid, to plot a 2-d weighted histogram. This replaces the density layer, which has been withdrawn (grid can do all the same things and more, except specify bin size in screen pixels).
  • New plot layer type quantile, which can (e.g.) plot median lines through noisy data.
  • New plot layer type gaussian, for Gaussian fits to histograms.
  • Histogram-like layer types are now available from the plot2time command as well as from plot2plane, so you can now plot histograms with a temporal horizontal axis.
  • New normalisation (scaling) option unit for histogram, KDE, and KNN plots.
  • New normalisation (scaling) options per_day etc for histogram, KDE and KNN layers when used from the plot2time command.
  • Colour names recognised by the plot2* command *color parameters now include the (140) CSS-like colours alongside the dozen standard plotting colours.
  • Plot commands insets parameter now lets you specify margins round plots using any combination of <top>,<left>,<bottom>,<right>, rather than requiring all values or none.
  • Add new parameter auxwidth to plot2 commands, to control colour ramp lateral dimension in pixels.
  • Modify plan caching arrangements for STILTS plots. This results in much better interactive performance (navigation etc) for some plot layer types plotted to the screen. There are also some minor API changes to a few of the plot2 utility classes, which now provide more flexibility for caching.
  • Various tweaks to the details of how plots are positioned on the screen or in output graphics files.
Miscellaneous enhancements
  • Fixed the match score (distance measure) for combined matchers. Previously, the score was a linear sum of the unscaled distances for the constituent matchers, which meant a Best match was pretty meaningless. Now, it adds scaled distances in quadrature, so Best matching should give you a somewhat sensible result.
  • The skyerr and nd_err matchers now report output separations as scaled (dimensionless) values rather than in physical units; this means the results are more comparable, so Best matches will make more sense.
  • taplint can now optionally write its output in JSON format (see format parameter).
  • Update taplint OBS stage for PR-ObsCore-v1.1-20160923.
  • The Maths function hypot now takes an arbitrary number of arguments (instead of exactly two).
  • Add blockmaxrec parameter to tapskymatch, and improve warning text in case of result truncations.
  • Improve tapquery/tapresume behaviour when monitoring async jobs; if the service goes down, the application will wait for it to come back rather than bailing out.
  • Options for MOC output format (mocfmt parameter in pixfoot command) have changed; option ascii is replaced by json. The old ASCII format was slightly broken JSON in any case.
  • MOC library upgraded to v4.6 (from v3.3). Improved MOC output.
  • Update JCDF library to v1.2-2 (2017-01-01 leap second).
Bug fixes
  • Fix KDE/KNN plotting bug that could get scaling badly wrong for normalised cumulative plots.
  • Fix some regression bugs relating to plot2 command insets parameter, present since v3.0-6.
  • Remove spurious padding from EPS graphics output.
  • Fix small bug in plot2plane/plot2time label painting; horizontal axis label was sometimes off the bottom of the plot by a few pixels.
  • Fix subpixel offset of colour ramp frame in PDF/PostScript graphics output.
Version 3.1-1 (29 September 2017)
New functionality
  • New plot layer types xycorr and skycorr for error ellipses rotated as specified by Gaia-style correlation values.
  • New plot layer type skygrid can draw multiple sky system coordinate axis grids on sky plot.
  • Colour map parameters (auxmap, densemapN etc) for the plotting commands will now accept custom colour maps that interpolate between a list of named colours, e.g. "HotPink-yellow-SkyBlue".
  • New position angle calculation functions posAngRadians and posAngDegrees added to expression language.
Table I/O changes
  • It is now possible to write and re-read tables with >999 columns to FITS or colfits format. The new limit is 2^31 columns. This uses a non-standard convention; software that is not aware of the convention (e.g. CFITSIO or earlier STILTS versions) will only be able to use the first 998 columns of tables written in this way.
  • For VOTable columns that reference COOSYS elements, the relevant information is now accessible as column auxiliary metadata (CoosysSystem, CoosysEpoch, CoosysEquinox) e.g. using the meta filter.
  • Any columns referencing COOSYS elements read from VOTable-based formats (VOTable or FITS-plus) will now be written out to VOTable-based formats with equivalent COOSYS references included. Currently not table parameters though.
  • The default version for output VOTables is now VOTable 1.3. New output formats votable-binary2-inline and votable-binary2-href are now offered alongside the five previously available VOTable variants.
  • Slight changes to the FITS-plus output handler VOTable formatting in the primary HDU; now uses default output VOTable version rather than VOTable 1.1.
  • FITS keywords using the ESO HIERARCH convention can now be read as table parameters rather than ignored when reading FITS tables.
Minor behaviour changes
  • The charset parameter of votcopy now defaults to UTF-8 rather than the system default.
  • Replace parameter autoscale with unit in skyvector and skyellipse plot layers.
  • Change autoscaling defaults for plot layer types xyvector, xyzvector, skyvector, xyellipse, skyellipse; autoscaling is now turned off by default. Apologies for this change to behaviour, but it's better for consistency with new layer types xycorr and skycorr, and presents less danger of misinterpreted plots.
  • Taplint: downgrade type mismatch error E_QTYP to warning W_QTYP in view of TAP-1.0 Erratum #3.
  • The COOSYS element is now passed without a deprecation warning in VOTable 1.3 documents by the taplint and votlint validators, in view of VOTable-1.3 Erratum #1. The warning is still issued for VOTable 1.2 documents, but the text is toned down.
  • Taplint no longer issues a warning about DataModel references that do not match the vr:IdentifierURI type; this follows TAPRegExt Erratum #1.
  • Modify taplint Examples document @vocab attribute testing in the light of DALI 1.1.
Bug fixes
  • Update PixTools (HEALPix) library to 2017-09-06 version (https://github.com/kuropat/eag-HEALPix, 447a7be073876dba32). This fixes a bug in healpixRingIndex that could give the wrong value for small values of longitude near zero in the equatorial region. It seems possible that this might have led to very infrequent missed associations when crossmatching in these regions, but tests appear to indicate that no such errors would actually have resulted.
  • Long fields (>10240 characters) in output CSV files are no longer truncated.
  • Fix misfeature in skyvector and skyellipse plot layers; these now preserve orientation on the sky even when viewsys differs from datasys. Previously the dlat/dlon/posang parameters were always interpreted in the view, rather than the data, sky coordinate system.
  • Fix bug in plot title placement.
  • Fix filename generation error for plotting command animation output (wrong number of digits if frame count was an exact power of 10).
Version 3.1-2 (7 November 2017)
New expression language functions
  • New functions desigTo* for extracting positions from IAU-style object designations (like 2MASS J04355524+1630331) - use with care.
  • New functions polarDistanceDegrees and polarDistanceRadians; these calculate the distance in 3d space between two positions specified in spherical polar coordinates.
  • New functions phase to help with phase folding given a known period. A new modulus function mod, whose output is always positive (unlike the % operator) is also added.
Bug fixes and workarounds
  • Upgrade JEL to v2.1.2: now you can use functions in the expression language that have the same name as table columns.
  • Add fix to discard on read any VOTable/FITS-plus table parameters with names "uk.ac.starlink.topcat.plot2.TopcatLayer*". These useless items were added in potentially large numbers when saving plotted tables from TOPCAT v4.5 (TOPCAT bug). A null tpipe operation at this version will therefore purge these items.
  • Fix plot bug that sometimes caused error bars to come out very small when exporting to a graphics file. This was a regression bug (present in v3.1-1 but not v3.1).
  • Fix another tiny error bar plotting bug too.
  • Small taplint update; required UCD for obs_publisher_did/publisher_id changed to match REC-ObsCore-1.1 (even though it's an illegal UCD1+).
Version 3.1-3 (24 April 2018)
New Functionality
  • New DataLink validator task datalinklint.
  • Add class Gaia to the expression language. This contains functions for estimating distances from parallaxes and propagating astrometric parameters and errors to different epochs. The functions are not specific to data from the Gaia astrometry satellite, but are presented in a form convenient for use with the Gaia DR2 source catalogue.
  • New array manipulation functions slice and pick added to class Arrays.
  • New utility function square added to class Maths.
  • The addcol and colmeta filters now accept -shape and -elsize flags for defining array and string shape/extent column metadata.
  • DataLink-style service descriptor RESOURCEs in input VOTables (and FITS-plus tables) now appear as table parameters.
Plotting Enhancements
  • Various improvements to the Contour plotter. You can now contour quantities weighted by a given coordinate, rather than just point density. The smoothing from weighted point samples to the grid being contoured can be performed using sum, mean and other combination methods, it now uses a Gaussian kernel rather than a square top hat, and performance is considerably improved especially at large smoothing widths. The contour levels used are now reported, so can be seen by specifying the -verbose flag. The contours are now plotted correctly right up to the edge of the visible plot, and various bugs have been fixed.
  • The histogram plotter now has a combine parameter (sum, mean, median, min, max, stdev, etc) for weighted histograms. This means that you can plot e.g. the mean value of a given quantity per interval on the X axis rather than just the summed quantity. A corresponding (though somewhat less well-defined) option is also provided for the kde and densogram plotters.
  • The combine parameter that configures how values are binned in various histogram-like plots (skydensity, healpix) grid, and since this version also contour, histogram, kde) now has two new options, sum-per-unit and count-per-unit. These work like the existing sum and count options, but scale the combined values by the relevant unit (e.g. X axis unit or solid angle). Where these units are physical, a perunit parameter is also provided for scaling in convenient units: second, day, year etc for time (plot2time), and square degree, arcminute, arcecond etc for solid angle (skydensity, healpix). In some cases, the combine default values have changed.
  • The same new combine/perunit options are also added to the tskymap command.
  • The linearfit layer type is now available for plot2time as well as plot2plane.
Minor behaviour changes
  • votlint no longer considers an integer- or array-typed PARAM element with attribute value="" to be an error.
  • votlint no longer Warns about PARAM/@ref->FIELD/@ID references, since this usage pattern is now found in DataLink Service Descriptors.
  • The IPAC table reader now matches data type specifications case-insensitively.
  • Improvements in documentation of the expression language functions: in the Function documentation section classes are now listed in alphabetical order, and examples are included in some cases. Some readability improvements have also been made in the function browser displayed by the funcs command.
Bugfixes
  • taplint's VOTable validation now includes XML schema/DTD validation as well as the custom VOTable checks. It was always supposed to do this, but it seems that it didn't, at least in recent versions.
  • Update JCDF to v1.2-3; fixes some CDF reading bugs.
  • Prevent coneskymatch from sending cone search queries with sr=NaN.
  • Minor fix in taplint OBS stage; ObsCore v1.1 s_region UCD is pos.outline;obs.field, not phys.outline;obs.field (a relic from PR-ObsCore-1.1-20161004).
Version 3.1-4 (18 May 2018)
Bugfixes and minor enhancements:
Version 3.1-5 (2 November 2018)
Plotting:
  • New spheregrid plot layer, for plotting a spherical net around the origin, added to plot2cube and plot2sphere.
  • New line3d plot layer, for plotting lines joining points in 3d, added to plot2cube and plot2sphere.
  • New sortaxis parameter, for joining out-of-sequence points, added to line plot layer.
  • Replace skygrid plot layer parameter crowd with two parameters, loncrowd and latcrowd, so you can control grid line spacing for meridians and parallels independently. Note this is a backwardly-incompatible change, so could break existing scripts. This change may have introduced slight changes to sky axis grid line spacing at low crowding levels.
  • Contour level calculations improved; in some cases this previously didn't work well at low point density, resulting in missing contours. The meaning of the zero parameter has also changed slightly, it now defaults to 1 and is not phase folded.
  • Sphere plot now centers axes on zero if the center would otherwise be near zero.
  • Address long-standing plot auto-ranging issue; when auto-ranging resulted in the bottom X limit or right-hand Y limit being exactly equal to zero, the corresponding zero-valued results were not plotted. Auto-ranging has been slightly adjusted to avoid that. This results in pixel-level changes to plot appearance in some cases; these can be avoided by explicitly setting xmax/ymin etc.
Expression Language:
  • New functions in class Gaia: polarXYZ, astromXYZ, astromUVW, icrsToGal, galToIcrs, icrsToEcl, eclToIcrs. These can calculate Cartesian position and velocity components from (e.g. Gaia) astrometric parameters.
  • New Shapes class added for working with polygons in the X,Y plane; has functions isInside and polyLine.
  • Rearrange expression language documentation slightly: there is a new section listing Special Tokens. This includes "$random", which was previously undocumented and named "RANDOM".
  • New functions urlEncode and urlDecode added to class Strings.
  • New convenience function exp10 in class Maths.
Validators:
  • Improvements to taplint UWS stage: now validates job documents against UWS schema, tests uws:job/@version attribute, and does improved and version-sensitive (UWS v1.0/v1.1) validation of UWS timestamps. Also does more complete deletion of submitted jobs.
  • taplint schema validation now takes account of VOTable 1.3 Erratum #2 - FIELD/@precision attribute values "F0"/"E0" are now permitted.
  • The VOTable validator in votlint and taplint no longer complains about multiple INFO elements in the same scope sharing the same name attribute.
  • Minor improvements to datalinklint error reporting.
Miscellaneous Enhancements:
  • New collapsecols filter added, to convert a run of adjacent scalar columns into an array column (the opposite of explodecols).
Minor enhancements and behaviour changes:
  • GBIN read fix to work around changed behaviour in recent GaiaTools (versions 19.4.*, >=20.1.0 and >=21.0.0) that caused GBIN table reading to fail.
  • Fix some additional commands (votcopy, votlint, datalinklint) so they follow HTTP 3xx redirects automatically in more cases when given a URL for input. Most STILTS commands already do this.
  • Introduce a couple of measures to reduce the likelihood of unintentional service overload from coneskymatch and friends: add a progressively increasing delay for erract=retry* error handling modes, and decrease the soft maximum for the parallel parameter to 5 (from 10).
  • Slight improvements to the JDBC Configuration section of this manual.
  • Small change to MOC handling that might possibly avoid some network-related performance issues.
Bugfixes:
  • Fix function plot layer so that NaN values are omitted rather than interpolated.
  • The stilts startup script now correctly follows symlinks on some OSes where it didn't work before (including OSX). It also looks for jar files to use in slightly different places on OS X.
Version 3.1-6 (9 May 2019)
Plotting enhancements
  • Introduced new colour scaling options histogram and histolog for the parameters auxfunc, densefunc etc in plot2plane and the other plotting commands. This can make it much easier to see structure in quantities that do not vary smoothly over their min-max range.
  • New plot layer types poly4, mark4, polygon introduced for drawing outline or filled quadrilaterals and other polygons in all plot types.
  • The line plot layer now has an aux parameter that can vary the colour of the line along its length according to some third quantity.
  • plot2sky now by default draws a small scale bar at the bottom left corner of the plot, indicating the scale in degrees, minutes or seconds. It can be switched off using the new scalebar parameter.
  • New option Car0 for the plot2sky projection parameter; this is like Car (Plate Carrée) but has longitude=0 at the left/right edge rather than the center of the plot.
  • The healpix plot layer can now plot HEALPix levels up to 20 (previously the maximum was 13). Also, autoranging now works, so if a plot is made of a HEALPix file covering part of the sky, the sky view is centered on it in the same way as for marker plots. Memory management is improved for fairly large maps.
HEALPix-FITS support
Various changes introduced to support the semi-standard HEALPix-FITS serialization convention. Available information about HEALPix encoding (level, index column, ordering scheme, coord sys) can now be stored in custom table parameters (of the form STIL_HPX_*), and is used by FITS output handlers to insert the relevant FITS headers. The existing FITS handlers do this where it's not disruptive, and the new fits-healpix output handler will additionally move and rename columns if required. This metadata is round-tripped by FITS and VOTable I/O handlers. It is added automatically by the tskymap command, and can be manipulated by hand using the new healpixmeta filter. FITS support is not perfect: the BAD_DATA FITS keyword is ignored, and the 1024-element array-valued column variant is not understood.
VOTable 1.4 support
Support has been introduced for version 1.4 of the VOTable format and its new TIMESYS element.
  • votlint (and hence taplint) now supports Version 1.4 VOTables: the 1.4 schema is used for XSD validation, and the TIMESYS element is checked for attribute content and suitable referencing.
  • For VOTable columns that reference TIMESYS elements, the relevant information is now accessible as column auxiliary metadata (TimesysTimeorigin, TimesysTimescale, TimesysRefposition) e.g. using the meta filter.
  • Any columns referencing TIMESYS elements read from VOTable-based formats (VOTable or FITS-plus) can now be written out to VOTable-based formats with equivalent TIMESYS references included, so TIMESYS round-tripping for columns works; however this will only be done if the VOTable output format is set to version 1.4. By default (at least as long as 1.4 is not finalised) the output version is 1.3. To enable this TIMESYS output, set the system property -Dvotable.version=1.4. Currently this TIMESYS output works only for table columns (FIELDs) not parameters (PARAMs).
  • The timeoffset attribute of a TIMESYS element referenced by a VOTable column is used to make sense of column data when interpreting it as an absolute time. Currently, the only use of this is in time plots.
This VOTable 1.4 support has resulted in some minor related behaviour changes:
  • FIELD/@ref attributes are no longer imported as "VOTable ref" column aux metadata items, since they often interfere with TIMESYS references. Doing this was probably always a bad idea since the referencing is not kept track of within the application, so withdrawing this functionality makes sense, but beware that it might change or break some existing behaviour.
  • votlint modifications above may now interrogate external vocabulary resources during validation, meaning that external network connections may be made during validation, which didn't happen in previous versions.
Other enhancements
  • New functions indexOf in class Arrays to find position of a given value in an array.
  • New functions parseDoubles and parseInts in class Conversions for extracting array values from strings (experimental).
  • Fix expression compilation so that $ID column references referring to nonexistent columns are rejected at compile time rather than causing trouble during evaluation.
  • The validator tasks taplint and datalinklint now insert the token "(IVOA-validate)" into the User-Agent header of all HTTP requests they make, for convenience of services that want to identify validators. This is a convention discussed within the IVOA Operations IG.
Bug and misfeature fixes
  • Improve colour ramp quantisation (plot parameters auxquant etc); the full color range is now included.
  • Use period not comma as decimal separator for non-sexagesimal plot2sky axis labels regardless of Locale; also avoid trailing comma sometimes erroneously present.
  • Fix bug/misfeature in CDF table parameter construction: CDF global attributes were ignored (with a "WARNING: Omitting complicated global attribute" message) if they contained any null entries. Now such entries are just ignored and the table parameter is constructed from the global attribute using the non-null entries.
  • Fix some bugs relating to plotting values close to the limits of the double precision range.
  • Fix a problem with the link2 plot layers on sky plots with Aitoff/Car projections that caused short lines that should span the antimeridian to appear as long lines crossing the whole sky. Such links are now just not drawn.
  • Fix misleading error message about seq parameter for underconfigured plot layers in plot2* commands.
Version 3.2 (18 November 2019)
Runtime environment: Java 8
From this release, STILTS requires Java 8 (a.k.a. Java 1.8) or greater to run, rather than Java 6 as for previous releases. Java 8 has been around since 2014, so it should be available on all but very ancient platforms. If execution fails with a java.lang.UnsupportedClassVersionError then you need to upgrade.
New or enhanced functionality:
  • New cone command to execute simple spatial DAL queries (Cone Search, SIA, SSA).
  • Most plot layer types, though not the intial data preparation, will now run in parallel for large datasets. When using one of the plot2* commands interactively (omode=swing), this should make interacting with slow plots faster on multi-core machines. (In rare cases this multi-threading might cause problems with memory usage; it can be effectively turned off if required by using the system property java.util.concurrent.ForkJoinPool.common.parallelism.)
  • New class URLs contains expression language utility functions for constructing certain service URLs: hips2fitsUrl, bibcodeUrl, doiUrl, arxivUrl, simbadUrl and nedUrl. Existing functions urlEncode and urlDecode have been moved to URLs from class Strings.
  • New functions parseBigInteger and parseBigDecimal in class Conversions.
  • Add new parameter usepos to polygon plot layer, to toggle inclusion of reference position in polygon vertex list.
  • Add parameter interface added to TAP query tasks tapquery, tapskymatch and taplint. This can control how TAP version is determined.
  • Modified behaviour for offset (e.g. unsigned) longs in FITS files. 64-bit integer columns (TFORMn='K') with non-zero integer offsets (TSCALn=1, TZEROn<>0) are now represented internally as Strings; previously they were represented as Long integers, but values out of the possible range appeared as null (with a warning written through the logging system). Such columns are most commonly seen representing unsigned long values. If written back out to FITS, the offset long value will be reinstated, but other output formats cannot encode unsigned longs, so they will stay as strings.
Minor enhancements and behaviour changes:
  • VOTable output now writes VOTable version 1.4 by default (was 1.3).
  • Modified plot legend display so that small markers are displayed a bit bigger in the legend than on the plot for readability.
  • Add new colour map Cividis.
  • Permit FITS and VOTable files with zero-length string columns. Previously all-null or zero-length string columns were sometimes forced to single-character values.
  • Update mapped file unmapping implementation to work for java9+.
  • Updated taplint OBS stage according to ObsCore-1.1 Erratum #1 (corrected UCDs). No code changes required for ObsCore-1.1 Erratum #2.
  • More taplint tests added for TAP 1.1-style capabilities document.
  • Taplint now recognises ADQL 2.1 language features (as listed in PR-ADQL-2.1-20180112), avoiding some E_KEYX errors.
  • VOResource validation in taplint now uses VOResource v1.1 not v1.0. Erratum #1 (multiple security methods) has been applied.
  • Some taplint enhancements: add some new general tests, and some only used when testing TAP 1.1 services.
  • Adopt VOTable-1.3 Erratum #3 in votlint; arraysize="1" now provokes a Warning.
  • Updates to meta filter documentation.
  • Minor changes to behaviour when querying TAP 1.1 services (REQUEST parameter is omitted).
  • JSAMP to version 1.3.7.
Bugfixes:
  • Fix regression issues introduced at v3.1-6 relating to pixsample command; it now reads HEALPix metadata from headers correctly again.
  • Avoid sometimes losing precision when reading ASCII/CSV values in the range +/-(1e-45..1e-38).
  • Fix plot axis ranging bug: padding was not always applied properly for logarithmic axes.
  • JEL bug fix update, to avoid unwanted debugging output for String function null returns.
Version 3.2-1 (5 June 2020)
File Formats:
  • The ECSV (Enhanced Character Separated Values) storage format is now supported for input and output.
  • The Feather storage format is now supported for input and output.
Performance:
A number of implementation changes have been made which may improve performance, particularly for crossmatching (typical improvements for large sky crossmatches are a factor of 2, though YMMV). These should have no effect on the results, but if anybody notices crossmatching behaviour which is changed since previous versions or otherwise suspicious, please report it.
  • The HEALPix implementation has been replaced; all HEALPix manipulation is now done using the excellent cds-healpix-java library written by François-Xavier Pineau from CDS, which speeds up sky crossmatching considerably. Many thanks to François-Xavier and to CDS for providing this library and for assistance with its use; thanks also to Nikolay Kuropatkin from FermiLab whose PixTools library served this purpose in STILTS up till now.
  • Rows are now binned during crossmatches using a HashSet rather than a TreeSet.
  • Evaluations of the arcsin function in Sky matches now use the (Apache via cds-healpix) FastMath implementation rather than the standard J2SE version.
  • Performance is improved when reading long String values from FITS files.
Plot commands:
  • The new layer types area, central and arealabel can be used for plotting region data supplied as area coordinates in the form of STC-S (e.g. from ObsCore/EPN-TAP s_region), DALI polygon/circle/point, or (ASCII) MOC columns. These layers are available from the plot2plane, plot2sky and plot2sphere commands.
  • The plot2time command has been improved, and is no longer considered experimental. A new ttypeN parameter is added, which allows you to explicitly define (as MJD, JD, ISO-8601 etc) how input values are mapped to time. If no such mapping is specified or can be guessed from the input metadata, time values are now interpreted by default as MJD rather than (as previously) Unix seconds.
  • Provide more options for the shapeN parameter of the plot mark layer type: fat_circle, fat_cross etc using thicker lines.
  • Add line thickness parameter thick to contour plot.
  • Plot storage parameter has new options disk, policy and persistent. These allow off-heap plot data caching, so that interactive plots for very large data sets may be made without running of of memory. The persistent option additionally means that results of expensive data read operations can be cached between invocations.
  • The line layer's sortaxisN parameter now accepts the option time rather than X (or Y) when used with plot2time.
  • Improved plot axis labelling in LaTeX mode, e.g. in LaTeX write "3x106" not "3e6".
  • Add new option anchorN=center to the label plot layer.
  • Add parameters minsize and minshape to poly4 plot layer type.
  • Improve accuracy when drawing large HEALPix tile boundaries in some cases for skydensity and healpix sky plot layers.
Other minor enhancements:
  • Basic support for SIA version 2 as well as version 1 in cone and coneskymatch commands (new option servicetype=sia2).
  • Add new conversion functions *ToUnixSec to Times class.
  • Add new functions midLon and midLat to Sky class.
  • Slightly improve sampling accuracy of pixsample output.
Bug and misfeature fixes:
  • Fix bug in SkyGrid plot layer that made it ignore the viewsys parameter, always assuming equatorial so plotting the wrong grid lines for non-equatorial view sky systems. Note this bug applied only to STILTS, not TOPCAT.
  • Fix regression bug since last release that refused to make dataless plots.
  • Fix polygon plots in plot2sphere - this combination was failing to plot anything, now it works.
  • FITS ASCII table extensions with TFORM values of In are now treated as 64-bit integers for n>=10 rather than n>10.
  • Modify taplint processing of QUERY_STATUS declarations; overflow may now be declared before, as well as after, the table.
  • Slightly improve plot axis labelling for small numbers; remove unnecessary decimal point in scientific notation in some cases.
Version 3.2-2 (24 August 2020)
New Functionality:
Bugfixes and minor enhancements:
  • Fix Healpix trouble at large angles, e.g. sky crossmatch failure with match radius >6 degrees.
  • Fix ECSV output bug: encoding was incorrect for metadata scalars with certain non-alphanumeric first characters, leading to invalid YAML.
  • Votlint now warns about use of vocabulary terms marked as preliminary or deprecated as well as unknown ones (although currently no such terms exist in relevant vocabularies).
  • Remove some unhelpful per-column metadata items from ECSV output.
  • Improve seeding of the expression language $random special value; it should now be less dependent on JVM details.
  • TAP curl(1) equivalent logging flag error fixed (write --compressed not --compress).
Version 3.3 (23 October 2020)
Plot server
A plot server has been added that can serve interactive plots in web applications for server-side tables. Documentation for the server command has been mostly moved into new Server Mode section.
Bug fixes
  • Upgrade cds-healpix-java library to v0.28_1; avoid occasional sky crossmatch failures (termination with error).
  • Fix minor plotting bug that could cause white points to be invisible.
Version 3.4 (11 January 2021)
In this version, the table handling library STIL has been upgraded to v4.0, which enables some enhancements including multithreading and new I/O handler features.
Notable new functionality:
  • Auto file format detection now examines filenames to help guess format; this means that e.g. for CSV files named with the ".csv" extension, it is no longer necessary to specify ifmt=csv.
  • Several operations can now run in parallel, so should execute faster for large datasets on multi-core machines. Parallelised operations are the output modes stats and count, the filters stats and sort, and the task tskymap. The stats and sort filters have -[no]parallel flags that can be used to select between sequential and parallel execution. Parallel data caching for plots is also available using the storage=parallel option, but this is currently experimental and not generally recommended. Future releases will parallelise other operations, including matching.
  • I/O handlers can now in some cases take parenthesised options, e.g. ofmt=votable(version=V12,format=BINARY). For examples see individual handler documentation, e.g. the VOTable output handler and ECSV input handler.
  • Scheme specifiers can now be used to specify input tables, to read tables not based on external files. Options currently provided include simple sequence, simulated sky and strange attractor data.
  • Improved documentation of I/O formats in this document: full docs are now provided in Section 5.1 for all formats rather than summaries with pointers to SUN/253. The Table I/O section has been somewhat reorganised and tidied up.
Minor enhancements:
  • When the stats filter calculates quantiles, the -qapprox flag can be supplied to use an (approximate) algorithm that will not run out of memory.
  • Add combine parameter to tcube, so it can produce generalised weighted maps.
  • Add new options acos and cos for colour scaling options auxfunc, densefunc etc; these provide linear-like stretch functions with steeper/flatter ends, which may be useful for shading by quantities with most variation near to/far from the middle of the range.
  • The progress filter no longer slows down pipelines to any noticeable extent.
  • COOSYS and TIMESYS attributes are now preserved during VOTable I/O for table PARAMs (as well as for FIELDs, which was already the case).
  • Tidy up open files in the table concatenation command tcat; with lazy=true it should no longer fail with "Too many open files".
  • The every filter has a new -exact|-approx flag; -approx may be required for certain operations to run in parallel.
  • Slightly improved documentation for cdsskymatch.
Bugfixes:
  • Fix bug that could give unhelpful table load error message for very short non-FITS files.
  • Fix broken example in tmatchn documentation.
  • Fix taplint to cope with VOSI1.1 detail=min-style response from /tables endpoint; avoid spurious E-TMC-CM21/E-TMC-FM21 errors.
  • Cope better with infinite values in aux plot coordinates.
  • Fix histogram ranging bug (plot failure under certain circumstances).
  • Upgrade JEL to v2.1.2: bugfix plus some efficiency improvements in object creation.
  • Avoid inserting NULL_VALUE custom metadata entries into ECSV output.
  • Make taplint changes for later edits to ObsCore-1.1 Erratum #1 (dataproduct_type and dataproduct_subtype UCDs are now meta.code.class).
Version 3.4-1 (10 June 2021)
File formats:
  • Apache Parquet format is now supported for input and output (note not available in all configurations).
  • AAS Machine-Readable Table (MRT) format is now supported for input.
  • ECSV format input and output handlers are upgraded to version 1.0 of the ECSV format, meaning they can now read and write array-valued columns.
  • Add configuration option header for CSV input handler, to indicate whether header line is present.
  • Add configuration option maxSample for CSV and ASCII input handlers to reduce 2-pass read time.
  • Variable-length array-valued columns in FITS tables (P/Q descriptors) can now be read even in compressed or streamed input.
Other new functionality:
  • Add new plot layers to plot2plane for plotting data (such as spectra or time series) in array-valued columns: lines, marks, yerrors and xyerrors.
  • Add new functions to class Arrays for working with array values: arrayFunc, intArrayFunc, sequence, constant.
  • Add new table scheme test.
  • The cumulative option in histogram-like plots can now take the values none/forward/reverse, not just true/false.
  • Introduce the aitoff0 option alongside aitoff as a value for the plot2sky projection parameter. In the same way as car/car0, this gives an Aitoff projection but with the longitude=0 line at the left/right edge instead of the center.
  • Modify tcube so that it accepts expression language expressions rather than just column name/ID values for the cols and scale parameters.
Taplint changes:
  • Add new taplint stage LOC, for ObsLocTAP validation. This release validates against the draft standard version PR-ObsLocTAP-20210609.
  • Taplint stage TMV now validates against (a pre-REC version of) version 1.2 of the VODataService schema instead of version 1.1.
  • Downgrade taplint report for accepting unknown LANG from E_DSUC to W_OLNG.
  • Relax taplint timestamp upload round-tripping requirements; folding adql:TIMESTAMP->timestamp no longer provokes error E-UPL-TMCX.
  • Relax some taplint type checking for TAP_SCHEMA columns of TAP 1.0 services (see TAP 1.0 Erratum #3); report E-TMS-TSCT is downgraded to W-TMS-TSCT.
  • Taplint W_EXVC TOPCAT compatibility warnings are downgraded to I_EXVT.
Minor enhancements and workarounds:
  • The fits-var output handler now avoids use of the THEAP keyword (no pre-heap gap is written). Heap padding is legal FITS, but bugs in other FITS software mean that some third party components (including fverify in FTOOLS e.g. v3.14-3.50) have problems with such files.
  • Fix server to work with scheme-specified tables having a non-default tablefactory parameter.
  • JDBC output no longer attempts to create VARCHAR(0) columns.
  • Output of cache filter now retains no reference to input table, so it can be used to manage deeply nested tables in (Jy)STILTS pipelines.
  • JyStilts now has access to the same table schemes as STILTS.
  • Collapse whitespace in some table metadata description items in output form meta filter and output mode. This prevents unwanted linebreaks as well as tidying whitespace-formatted values.
  • Space-delimited ECSV files now write empty fields quoted.
  • Unknown or unsupported column datatype values in ECSV files are now treated like string rather than causing table read failure.
  • Empty strings in FITS 1-character columns are now returned as blank values rather than ASCII NUL ('\0').
  • Undersized, including zero-length, strings written to FITS columns are now by default terminated with an ASCII NUL rather than in some cases padded with spaces.
Bugfixes:
  • Fix regression bug in previous release (v3.4) that broke use of jdbc: URLs.
  • Fix regression bug in previous release (v3.4) affecting text, ascii and ipac output formats, that could cause cached table output (e.g. from filters cache or random) to contain many repeated copies of the same row.
  • Upgrade cds-healpix-java library to v0.29.3; this avoids occasional sky crossmatch failures (termination with error) introduced at STILTS v3.2-1.
  • Fix bug in addcol filter that could evaluate the row index ($index/$0) incorrectly during parallel processing.
  • Fix plot2sky bug that got clon/clat positioning wrong for some projections such as CAR0.
  • Fix issue with cumulative histograms; bars beyond the last sample are now displayed with total value not zero.
  • Fix bugs that meant writing long (>2Gb) fits-var files could output incorrect/corrupted FITS.
  • Fix long-standing file caching bug; mostly seemed to affect large (>2Gb) streams.
  • Fix documentation bug in cone; output parameters out, ofmt, ocmd and omode are now listed.
  • Fix optional argument processing bug in JyStilts (optional named arguments like start in tloop were being ignored).
  • Fix tcube bug that caused problems with some input filter (icmd) processing.
  • Fix long-standing logic error in ASCII/CSV input handler that could misidentify column types and cause read failures.
  • Fix failure when trying to plot with auxfunc=histogram and explicitly set auxmin/auxmax bounds outside of data range.
  • FITS TZERO headers are now written correctly with numeric values rather than string values.
Version 3.4-2 (15 October 2021)
Taplint and Votlint changes:
  • New stage UUC added to taplint. This checks units (against the VOUnits standard) and UCDs (against the UCD1+ standard and legacy UCD1 list).
  • New stage EPN added to taplint. This checks EPN-TAP (*.epn_core) tables against some version of EPN-TAP 2.0 (currently PR-EPNTAP-2.0-20211012). Since EPN-TAP is still in draft at time of writing, this stage is not currently run by default.
  • Modify taplint stages parameter usage; you can now add and remove stages individually using +XXX/-XXX syntax.
  • Taplint now does a better job of categorising VOTable parse errors, so multiple repeats of similar messages can be avoided. Ad-hoc VOTable error codes have changed from "VOxx" to "Yxxx".
  • Votlint, hence also taplint, now checks values in FIELD/PARAMs marked with xtype attributes for conformance to rules from DALI 1.1 sec 3.3.
  • Taplint MDQ stage now reports mismatches between declared and result Units, UCDs, Xtypes, and Utypes as well as datatypes.
  • Taplint and datalinklint now identify themselves as validators using the User-Agent token "(IVOA-test)" rather than "(IVOA-validate)" as before; this follows the published content of the SoftID-1.0 IVOA Note.
  • Taplint now examines the service Server header and tries to check it for VO component identification as recommended by SoftID.
  • Votlint now has a maxrepeat parameter like taplint (previously the repeat count was hard-wired to 4).
  • Votlint now does a better job of identifying similar errors, so output suppressing repeated messages can be more compact. Suppression of repeated messages is reported.
  • Votlint now optionally checks the content of unit and ucd attributes, checking against the VOUnits and UCD1+/UCD1 standards respectively.
  • Distinguish votlint reports for recognised and unrecognised (e.g. VODML) elements in foreign XML namespaces.
  • Add timestamp line at start of taplint output.
Other new functionality:
Minor enhancements and behaviour changes:
  • The fast parameter has been withdrawn from polygon-plotting plot types area, polygon, poly4; the shape-filling algorithm has been improved, and it's no longer necessary to choose between speed and accuracy.
  • Replace the julianToUnixSec function in class Times, which didn't do what it said it did, with jdToUnixSec.
  • The mjdToIso and mjdToDate functions in class Times now prepend the string "(BCE)" to dates before the Common Era.
  • Guess meaning for some non-standard COORDSYS values in HEALPix-FITS files, e.g. allow "GALACTIC" instead of "G".
  • Revert to sequential processing in some cases for HEALPix plot layer to reduce resource usage.
  • Columns that are all blank in ASCII-like tables (CSV, ASCII, TST) are now interpreted as type String not boolean.
  • Files compressed using multi-stream bzip2 compression (e.g. pbzip2 output) are now supported alongside single-stream bzip2.
Bugfixes:
  • Fix serious threading bug that could return nonsense values from fixed-length string fields during parallel processing, for instance statistics calculation, of large cached or randomised tables.
  • Upgrade of cds-healpix-java library to v0.30.2; this fixes some HEALPix cell plotting bugs.
  • Fix aux ranging bug that meant auxmax without auxmin was not honoured in plot commands. This was a regression bug introduced at v3.1-5.
Version 3.4-3 (31 January 2022)
New functionality:
  • The PDS4 (NASA's Planetary Data System v4) file format is now supported for input tables.
  • New task tgridmap writes flexible N-dimensional density maps/histograms as tables.
  • The cols parameter in tskymap is generalised to allow more flexible calculation of gridded quantities, specifically calculations using different aggregation methods for different quantities at the same time.
  • New plot2plane layers statline and statmark, to plot quantities like the mean of array-valued input columns.
  • New class Randoms contains pseudo-random number generation functions. Special token $random is now deprecated.
  • New quantities ArrayNGood, ArraySum, ArrayMean, ArrayStDev added to stats filter, allowing per-element statistics calculations for fixed-length numeric array data.
  • New taplint parameter tables to restrict which tables are tested.
  • Provide quantile options for the combiner parameters of some commands.
Minor enhancements and behaviour changes:
  • Add new colour map Painbow.
  • Improve identification of TIME_TT2000 columns as time values in certain CDF files.
  • Improved autoranging for gaussian plots.
  • Minor improvements to aggregated column output metadata.
  • JyStilts is now intended to be run using Jython 2.7.2, which means it works under Java 17, unlike Jython 2.5.3 which was the previous target version. The JyStilts classes haven't changed, but the distributed jystilts.jar file now comes with Jython 2.7.2.
  • In the starjava distribution, the jystilts script now uses an internal copy of Jython 2.7.2, it no longer requires an external Jython installation.
  • Taplint UUC stage now reports UCD word list versions.
  • Adjustments to taplint EPN-TAP validation: track draft standard changes, fix bugs, add UCD check for measurement_type column content. This version currently corresponds to PR-EPNTAP-2.0-20211022.
  • Make a couple of adjustments to SVG graphics output: output is now to a bare svg element (no XML or DOCTYPE declaration), and a viewBox attribute is included which may improve scaling behaviour in some contexts.
  • The monolithic stilts.jar file and stilts_jars.zip archive now contain the stilts.version file at the top level, rather than just buried in uk/ac/starlink/ttools/.
  • The seqcol column in output from the tcat and tcatn commands is now of type (32-bit) integer not (16-bit) short.
  • Permit leading 's' in ASCII-MOC area specifications, following MOC 2.0.
Bug fixes:
  • Fix bugs in parsing MOC-ASCII strings in area plot; trailing depth specifier was interpreted as cell index and some cells near end of MOC were omitted/misshapen.
  • Fix failure when attempting to read unsigned 32-bit integer values from parquet files.
  • Bugfix update of JCDF to v1.2-4.
Version 3.4-4 (6 April 2022)
Performance
  • Substantial I/O performance improvements, mainly for FITS and VOTable formats e.g.: writing to FITS 2x, reading FITS from a stream 2x, reading VOTable with inline BINARY/BINARY2 4x, writing VOTable with inline BINARY/BINARY2 2x, writing VOTable with TABLEDATA 1.5x.
  • New parameter runner in matching commands tmatch2 etc which can be set to parallel for multi-threaded execution. This should speed up large matches on multi-core machines. This is believed to work correctly, but since it is less well tested than the legacy sequential mode, at this release sequential processing is still the default.
  • Other matching performance improvements.
  • FITS I/O is now all done internally, there is no longer a dependency on the nom.tam.fits package.
New functionality
  • New command xsdvalidate added for validating against XML schemas.
  • New class Bits with bit manipulation functions bitCount, hasBit, toBinary, fromBinary.
  • Add configuration options compact and encoding to VOTable output handler. By default thin (<=4 column) TABLEDATA VOTables are now written in "compact" mode, using reduced whitespace.
  • FITS BINTABLE headers used as table parameters now support FITS 4.0 long-string syntax (CONTINUE records).
  • FITS header values of the form "(a,b,c,...)" are now interpreted where possible as numeric arrays; this works for long-string values (CONTINUE records) as well.
Bug Fixes and workarounds
  • Upgrade JEL to v2.1.3-pre1. This fixes a bug that caused evaluation failure when comparing a String against null.
  • Taplint now reports absent TAP service with an error code rather than failing with an exception.
  • Upgrade Unity library to 1.1 pre-release and improve VOUnits validation reporting (new status "GUESSED_UNIT").
  • Fix taplint bug: EPN validation failed for spatial_frame_type=none.
  • ECSV format now preserves table name.
  • FITS BINTABLE reader now copes with (illegal?) embedded spaces in TDIMn headers.
  • Adjust MRT null handling; "-" in a single-character field no longer interpreted as null.
Version 3.4-5 (10 June 2022)
New Functionality
  • New plot2plane layer arrayquantile, that can plot e.g. medians of multiple per-row spectra with arbitrary wavelength coordinates.
  • X/Y array-valued plot layer types lines, marks, yerrors, xyerrors, statline and statmark will now accept a missing xs or ys array coordinate, and assume a suitable linear sequence.
  • The <colid-list> syntax used by keepcols, delcols etc now allows column ranges to be specified with the syntax <first-colid>-<last-colid>.
  • The assert filter now takes optional <msg-expr> argument.
  • Add offset options xoff and yoff for label plot layer type.
  • Minor taplint EPN stage updates corresponding to latest EPN-TAP draft; now matches PR-EPNTAP-20220420.
Performance
  • Sky crossmatching performance improvements.
  • Reduce number of file mapping calls by FITS readers.
Bug Fixes
  • Fix FITS parsing issue that could result in StackOverflowError for long array-valued headers.
  • Fix bugs in statline/statmark layers that incorrectly treated blank array values and negative array values on logarithmic axes.
  • Fix bug in multi-threaded read of string columns from colfits files.
  • Fix NullPointerException failure in taplint UUC stage or votlint for unknown units recognised only in non-VOUnit syntaxes (e.g. "Crab").
  • Fix legend positioning bug for label layer type.
  • Aitoff sky projections are now more correctly documented as Hammer-Aitoff (though not renamed in the UI).
Version 3.4-6 (8 July 2022)
Enhancements
  • New command arrayjoin added.
  • Modify functions add, subtract, multiply, divide in class Arrays; these now all take either two array arguments or an array and a scalar in either order.
  • New function dotProduct in class Arrays.
  • Provide value* functions for column references to strangely-named columns.
  • Minor taplint EPN stage update; now corresponds to REC-EPNTAP-2.0.
Bug Fix
  • Fix FITS output so it doesn't fail when attempting to write metadata with non-ASCII Unicode characters.
Version 3.4-7 (5 October 2022)
Performance:
  • Crossmatching performance improvements: sky matching does better pre-selection of potential matches based on sky region, post-processing row sorting etc accelerated, various other steps parallelised. Large matches might typically be about twice as fast as before.
  • By default, matching commands like tmatch2 now run in parallel.
  • The usage of the runner parameter, used to control parallel execution of matching commands like tmatch2, has changed slightly: options parallel<n> and parallel-all are added, while option parallel, the new default, is now limited to a fixed number of processors (currently 6).
  • Add new parameter runner to tskymap and tgridmap; this allows to control parallel execution, since sequential may be faster than the default parallel option for large files on spinning disks.
  • Add new option "time" to the progress parameter of matching commands like tmatch2.
New functionality:
  • Add new plot layer type handles to mark a reference position for X/Y array data.
  • New function sequence(n,start,step) in class Arrays.
  • New functions tfcatStatus and tfcatMessage in class VO for validating instances of the Time-Frequency Radio Catalogue format.
  • Provide new option TFCAT for areatype parameter in area-like plots, giving partial support for TFCat shape descriptions.
  • Filter fixcolnames now deduplicates column names as well as fixing up their syntax.
  • Fix relevant filters to issue a WARNING if new column names clash with existing ones.
  • Add new section listing Colour Maps to this document.
  • Add some new colour maps from CMasher: cosmic, ember, gothic, rainforest, voltage, bubblegum, gem, chroma, neon, tropical (sequential); guppy, iceburn, redshift, pride (diverging); infinity (cyclic).
Bug fixes and workarounds:
  • Fix VOTable reader and votlint so that BINARY/2 VOTables with no columns don't read forever.
  • Fix PDS4 reader to accept columns of type ASCII_Numeric_Base16 without the read operation failing.
  • Fix bug in skygrid layer that failed to inherit configuration items from base grid.
Version 3.4-8 (20 April 2023)
Axis drawing improvements and changes:
  • Secondary X and Y axes can now be plotted in plot2plane; see new parameters x2func, y2func, x2label, y2label, and example plot.
  • Secondary T and Y axes can now be plotted in plot2time; see new parameters t2func, y2func, t2label, y2label, and example plot.
  • Matching Ticks are now plotted by default on all four sides of plots drawn by plot2plane and plot2time, rather than just on primary (left/bottom) axes. This can be configured with the new shadow parameter.
  • Reduce frequency of minor ticks.
  • Major tick marks on axes now extend only inside the plot bounds not outside.
  • Grid lines plotted by plot2plane, plot2time and plot2sky are now all partially transparent and plotted over the plot content (previously they were opaque and the sky grid was drawn over, while the plane and time lines were drawn under, the plot content). Grid line transparency is controlled by the new gridtrans parameter.
  • Add new options ExternalSys/InternalSys for plot2sky labelpos parameter, to display lon/lat axis names alongside axis values.
Other new functionality:
  • New task tgroup and filter group to calculate aggregate functions on groups of rows (like ADQL/SQL GROUP BY).
  • New function inSkyEllipse.
  • New filter constcol for identifying and removing constant-valued table columns.
  • Add keepall parameter to the arrayjoin command, to configure whether rows with no arrays appear in the output.
  • Add aparams parameter to arrayjoin command, to turn table parameters of loaded tables into scalar-valued columns.
  • Add sortaxis parameter to lines layer, to cope with unsorted array data.
  • Add configuration options readMeta and hierarchicalNames to the GBIN input handler.
  • VOTable service descriptor I/O now preserves contentType and exampleURL PARAMs, introduced in DataLink 1.1
  • Improve datalinklint validation of semantics terms, now checks online vocabularies not just hard-coded ones.
  • The datalinklint validator is upgraded to PR-DataLink-1.1-20230413. It now has a version parameter.
  • Update taplint in accordance with DALI 1.1 Erratum #1; example continuation elements without an empty resource attribute no longer provoke an Error.
Workarounds and minor behaviour improvements:
  • Modify column width determination in text-like output formats (text, ascii, ipac) to avoid occasional unwanted truncation of formatted values. Tables are now read in two passes, the first to establish column widths and the second to write the data. By default all rows are sampled, but the sampledRows option can be configured so that only some rows are sampled, which is more like the old behaviour.
  • Update Ucidy library to v1.3; UCD validation is now done based on v1.5 of the UCD1+ list of terms.
  • If only one single point is plotted on the (default) sin projection in plot2sky without explicit clon/clat/radius specification, the default view is now zoomed out to the whole sky rather than zoomed in to a few milliarcsec.
  • Improve ECSV reader performance, especially for Gaia DR3 bulk download files (which use semi-standard "null" token).
  • Write empty string not semi-standard "nan" token for NaN in ECSV writer.
  • The JDBC input scheme should now read columns that are array-valued in the database as array values that can be used in STILTS; previously they were read as opaque Array objects.
  • Make FITS and VOTable output handlers robust against input tables that declare incorrect row counts.
  • The PDS4 reader now reads Ascii_Numeric_Base16/8/2 fields as numeric not string (updated pds4-jparser library code).
  • Slight change to RESOURCE structure of Primary HDU metadata in multi-table FITS-plus output. This fixes a problem in which saved Service Descriptors could end up associated with the wrong tables.
  • HTTP redirects with response code 308 (Permanent Redirect) are now handled in the same way as 307 (Temporary Redirect).
  • Withdraw antialias parameter from lines layer, since it didn't really work.
  • Improve handling of illegal sky coordinates (latitude out of range) in sky matching; in v3.4-7 only, bad positions caused match failure, now they are just ignored.
  • Improve transparency rendering in skygrid layer.
Bug fixes:
  • Fix a nasty bug that could plot skyvector, skyellipse and skycorr shapes incorrectly. The numeric value of the shape size coordinates (ellipse radii, vector extents) was interpreted in degrees before being rescaled as appropriate, so that values of a few tens or larger resulted in signficant distortions. Now fixed.
  • Fix stability issue when using find=best (which should be used with caution) in tmatch2 and tskymatch2. Results may be different in crowded regions, and may also have differed between v3.4-6 and v3.4-7, but are not obviously more or less correct.
  • Fix Cartesian matcher overflow issue; for very large ratio between coordinate extent and match radius this could cause pathologically slow, though not incorrect, crossmatching.
  • Fixed bug which failed to plot some markers of multi-marker shapes like mark2 when multithreaded (i.e. for large tables). Regression bug since introduction of multithreaded plotting in STILTS v3.2.
  • Fix line drawing bug that meant dotted and dashed lines were often not displayed correctly. This was a regression bug introduced in STILTS 3.1-6.
  • Fix auto-range and export bugs that ignored final bar for forward cumulative histograms.
  • Fix occasional NullPointerException in taplint EPN stage.
  • Avoid some plot failures related to oversized legends and undersized plot windows.
  • Adjust colour ramp (aux axis) painting to avoid faint white stripes seen when vector graphics (e.g. PDF) output was rendered in some external viewers.
  • Fail with an error rather than silently reading a broken table when encountering GaiaTools/zStd-jni bug during GBIN input.
  • Empty/invalid fields encountered by the PDS4 reader no longer cause the table read to fail.
Version 3.4-9 (1 November 2023)
New functionality:
  • New plot command plot2corner.
  • The authentication handling has changed, and authenticated access can now be made to services that comply with the (draft) "SSO_next" proposal for advertising authentication in the VO. The star.basicauth.user/star.basicauth.password system properties can no longer be used to set authentication information globally for the application, but the auth.username/auth.password system properties can be used in a similar way (though this should be done with caution).
  • Introduced alternative syntax ("*" separator) for matcher combinations to restrict match separations to the scaled unit sphere.
Minor enhancements and behaviour changes:
  • Taplint EXA stage now checks RDFa property attributes against the http://www.ivoa.net/rdf/examples vocabulary.
  • Votlint is aware of VOTable 1.5 and can check against some features from WD-VOTable-1.5-20230913 (currently just vocabularised system and refposition attributes of COOSYS).
  • Downgrade taplint warning about non-standard language features from Warning to Info (W-CAP-CULF -> I-CAP-CULF).
Bugfixes:
  • Fix some IVOID case-sensitivity issues that could result in spurious taplint error reports.
  • Fix bug which could ignore some points in multi-threaded multi-dataset plots. This bug was introduced in v3.2, and was supposed to be fixed at v3.4-8 but wasn't.
  • Fix fencepost error in reverse cumulative histogram plotting.
  • Fix missing secondary X axis bug in stacked Time plots.
  • Fix bug in histogram plotting that could cause crashes for small ranges far from the origin.
  • Fix aux axis/legend positioning issue in sky plot (regression bug introduced at v3.4-8).
  • Fix rendering bug for omode=gui with huge (>130 million row) tables.
  • Fix datalinklint bug that reported a W-SDND warning for all tables having multiple service descriptors.
Version 3.4-10 (29 February 2024)
New functionality:
  • Add support for HAPI time series services: input handler and scheme.
  • Parameter geomN in plot2cube can now take value "vector" to use xyz 3-element array values as coordinate specifiers.
  • The spectrogram plot layer now tries to plot spectra on a spectral axis, as controlled by the new scalespecN option.
  • New plot2time parameter cellgap configures gap between stacked plots.
  • New array functions loop with arguments (start,end) or (start,end,step).
  • The first argument of the inMoc and nearMoc Coverage functions can now be an ASCII MOC string as well as a MOC file location or VizieR table identifier.
  • Add -xtype flag to filters addcol, colmeta, replacecol and setparam.
Minor enhancements and behaviour changes:
  • Add a Strings and Quoting section to this document.
  • The CoosysRefposition metadata item (VOTable 1.5) is now listed if present by the meta filter. This information is also propagated to VOTable output if the output VOTable version is set to 1.5 (currently not the default).
  • Allow use of trailing backslash to specify multiline filters in '@'-indirection files for cmd-type parameters.
  • ISO-8601 conversion functions in Times now accept a trailing "Z" even for date-only specifications, and now accept YYYY-DDD format for dates as well as YYYY-MM-DD.
  • Add new colour map Sunset.
  • datalinklint now checks and uses INFO/@name="standardID" version declarations in the links document being validated. DataLink handling and datalinklint are now believed to be up to date w.r.t. REC-DataLink-1.1.
  • Some fixes/improvements to the stilts startup script. It now works better in the MacOS DMG image, it's more robust against pathnames with embedded spaces, and it will use topcat-extra.jar if present in preference to topcat-full.jar.
  • Parameters datasysN in plot2sky and geomN in plot2cube are now included in the documentation. The previously undocumented option to supply plot2cube coordinates in spherical polar form is now documented.
  • Improve error reporting for corrupted/truncated FITS files.
  • Use authentication and HTTP-level compression for external (href-referenced) VOTable STREAM data.
  • Update VOTable 1.5 schema from WD to REC-VOTable-1.5 (affects votlint).
  • Upgrade Unity to v1.1-b1, which corresponds to REC-VOUnits-1.1. Only minor functional differences expected since previous version (1.1-b1).
  • Upgrade VOLLT adqlLib to official 2.0-beta release.
  • Upgrade snakeyaml library (used for ECSV headers) from 1.25 to 2.2. No change in behaviour (or security) expected, but prevents vulnerability warnings in some circumstances.
Bugfixes:
  • Fix sky match failure in case of very large error radius (>22 deg).
  • Fix votlint to permit empty TD elements in all cases for VOTable versions >=1.3. Previously empty cells for integer and array fields provoked an error report (which is correct for VOTable 1.0-1.2, but not for 1.3+).
  • Fix tcat/tcatn bug that sometimes resulted in no rows included from an input table.
  • Cope better with out of range pixel indices in healpix plots.
  • Fix issue with fat_triangle_up and fat_triangle_down marker shape names.
  • Fix FITS multi-table read bug that sometimes generated a non-fatal error message when reading basic FITS files using tmulti.
  • Some spectrogram bug fixes.
  • Fix plot2time so that zone-specific text style configuration works correctly for legends and colour ramp annotations.


Next Previous Up Contents
Up: Release Notes
Previous: Acknowledgements

STILTS - Starlink Tables Infrastructure Library Tool Set
Starlink User Note256
STILTS web page: http://www.starlink.ac.uk/stilts/
Author email: m.b.taylor@bristol.ac.uk
Mailing list: topcat-user@jiscmail.ac.uk