From m.b.taylor@bristol.XXX Fri Feb 26 09:36:57 2010 Date: Fri, 26 Feb 2010 09:36:47 +0000 (GMT) From: Mark Taylor To: topcat-user@sympa.XXX Subject: Test message Test message. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From marquett@iap.XXX Fri Feb 26 18:03:51 2010 Date: Fri, 26 Feb 2010 18:03:43 +0000 From: Jean-Baptiste Marquette To: topcat-user@sympa.XXX Subject: A missing string function in TOPCAT/STILTS ? Dear all, I have a table containing a column of strings such as (0.216637, 0.287873) The index of the comma is not constant as I can have (0.1455, 0.142567) My purpose is to extract both numerical values to two new columns but I didn't find any string function able to separate the two blocks using the comma as separator, as the strtok function does in C. I wonder if TOPCAT/STILTS contains presently enough functionalities to do the job. Any help welcome, Cheers from Chile, JB Marquette From m.b.taylor@bristol.XXX Fri Feb 26 18:19:14 2010 Date: Fri, 26 Feb 2010 18:19:13 +0000 (GMT) From: Mark Taylor To: Jean-Baptiste Marquette Cc: topcat-user@sympa.XXX Subject: Re: A missing string function in TOPCAT/STILTS ? On Fri, 26 Feb 2010, Jean-Baptiste Marquette wrote: > Dear all, > > I have a table containing a column of strings such as > > (0.216637, 0.287873) > > The index of the comma is not constant as I can have > > (0.1455, 0.142567) > > My purpose is to extract both numerical values to two new columns but I didn't > find any string function able to separate the two blocks using the comma as > separator, as the strtok function does in C. I wonder if TOPCAT/STILTS contains > presently enough functionalities to do the job. Hi Jean-Baptiste, Congratulations on posting the first question to the list! You can do this using regular expressions - one of the String functions replaceAll, replaceFirst or matchGroup could be used. If the column is called "pair", the following two expressions would pull out the numeric parts of the first and second numbers respectively: matchGroup(pair, "([0-9.]*),") matchGroup(pair, ", ([0-9.]*)") If the numbers might contain minus signs you'd have to adjust the regular expressions accordingly. You then probably want to apply toDouble() to the result. Note these regex-based functions might be a bit slow, so if you have a large table it might be worth writing it out and reloading it after adding these columns. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From marquett@iap.XXX Fri Feb 26 18:31:21 2010 Date: Fri, 26 Feb 2010 15:31:09 -0300 From: Jean-Baptiste Marquette To: Mark Taylor Cc: topcat-user@mlist2.XXX Subject: Re: A missing string function in TOPCAT/STILTS ? Le 26 févr. 2010 à 15:19, Mark Taylor a écrit : > Congratulations on posting the first question to the list! > > You can do this using regular expressions - one of the String functions > replaceAll, replaceFirst or matchGroup could be used. > > If the column is called "pair", the following two expressions would > pull out the numeric parts of the first and second numbers respectively: > > matchGroup(pair, "([0-9.]*),") > matchGroup(pair, ", ([0-9.]*)") > > If the numbers might contain minus signs you'd have to adjust the > regular expressions accordingly. > > You then probably want to apply toDouble() to the result. > > Note these regex-based functions might be a bit slow, so if you have > a large table it might be worth writing it out and reloading it > after adding these columns. Hi Mark, Thanks for your quick answer. I have an error message : stilts tpipe ifmt=ascii in=all_r.dat cmd='addcol test matchGroup(AstromSigma_Reference, "([0-9.]*),")' Unused arguments: '([0-9.]*),' ')' Command was: addcol test matchGroup(AstromSigma_Reference, "([0-9.]*),") A mismatch in parentheses ? ================================================ Bien cordialement/Very truly yours/Mit freundlichen Gruessen, Jean-Baptiste Marquette Institut d'Astrophysique de Paris CNRS - UMR 7095 Université Pierre & Marie Curie 98bis Bd Arago 75014 Paris - France Tel : +33 (0) 1 4432 8196 Fax : +33 (0) 1 4432 8001 ================================================ ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________ From marquett@iap.XXX Fri Feb 26 18:31:22 2010 Date: Fri, 26 Feb 2010 15:31:09 -0300 From: Jean-Baptiste Marquette To: Mark Taylor Cc: topcat-user@mlist2.XXX Subject: Re: A missing string function in TOPCAT/STILTS ? Le 26 févr. 2010 à 15:19, Mark Taylor a écrit : > Congratulations on posting the first question to the list! > > You can do this using regular expressions - one of the String functions > replaceAll, replaceFirst or matchGroup could be used. > > If the column is called "pair", the following two expressions would > pull out the numeric parts of the first and second numbers respectively: > > matchGroup(pair, "([0-9.]*),") > matchGroup(pair, ", ([0-9.]*)") > > If the numbers might contain minus signs you'd have to adjust the > regular expressions accordingly. > > You then probably want to apply toDouble() to the result. > > Note these regex-based functions might be a bit slow, so if you have > a large table it might be worth writing it out and reloading it > after adding these columns. Hi Mark, Thanks for your quick answer. I have an error message : stilts tpipe ifmt=ascii in=all_r.dat cmd='addcol test matchGroup(AstromSigma_Reference, "([0-9.]*),")' Unused arguments: '([0-9.]*),' ')' Command was: addcol test matchGroup(AstromSigma_Reference, "([0-9.]*),") A mismatch in parentheses ? ================================================ Bien cordialement/Very truly yours/Mit freundlichen Gruessen, Jean-Baptiste Marquette Institut d'Astrophysique de Paris CNRS - UMR 7095 Université Pierre & Marie Curie 98bis Bd Arago 75014 Paris - France Tel : +33 (0) 1 4432 8196 Fax : +33 (0) 1 4432 8001 ================================================ ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________ From m.b.taylor@bristol.XXX Sat Feb 27 11:14:12 2010 Date: Sat, 27 Feb 2010 11:14:12 +0000 (GMT) From: Mark Taylor To: topcat-user@sympa.XXX Subject: Re: A missing string function in TOPCAT/STILTS ? On Fri, 26 Feb 2010, Jean-Baptiste Marquette wrote: > Le 26 févr. 2010 à 15:19, Mark Taylor a écrit : > > > Congratulations on posting the first question to the list! > > > > You can do this using regular expressions - one of the String functions > > replaceAll, replaceFirst or matchGroup could be used. > > > > If the column is called "pair", the following two expressions would > > pull out the numeric parts of the first and second numbers respectively: > > > > matchGroup(pair, "([0-9.]*),") > > matchGroup(pair, ", ([0-9.]*)") > > > > If the numbers might contain minus signs you'd have to adjust the > > regular expressions accordingly. > > > > You then probably want to apply toDouble() to the result. > > > > Note these regex-based functions might be a bit slow, so if you have > > a large table it might be worth writing it out and reloading it > > after adding these columns. > > > Hi Mark, > > Thanks for your quick answer. > > I have an error message : > > stilts tpipe ifmt=ascii in=all_r.dat cmd='addcol test matchGroup(AstromSigma_Reference, "([0-9.]*),")' > Unused arguments: '([0-9.]*),' ')' > > Command was: addcol test matchGroup(AstromSigma_Reference, "([0-9.]*),") > > A mismatch in parentheses ? Yuk. I'm afraid this is a problem with the command line parsing that STILTS does, which really doesn't work well when the expressions need to contain quote characters and parentheses. It can be fixed in this case (for me using csh anyway) by avoiding unnecessary spaces and using some backslashes like this: stilts tpipe ifmt=ascii in=all_r.dat cmd='addcol p1 matchGroup(AstromSigma_Reference,\"([0-9.]*),\")' cmd='addcol p2 matchGroup(AstromSigma_Reference,\",\\\\s*([0-9.]*)\")' (the \\\\s* part matches spaces after the comma if any are present), but I agree this is really a mess. If you add the column in TOPCAT rather than doing it in STILTS you can do it with fewer problems, since there it doesn't need to worry about command line parsing. By the way there was a mistake in my earlier reply: > > You then probably want to apply toDouble() to the result. I meant parseDouble() not toDouble(). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From thomas.boch@astro.XXX Mon Mar 1 08:22:15 2010 Date: Mon, 01 Mar 2010 09:21:47 +0100 From: Thomas Boch To: topcat-user@mlist2.XXX Subject: Tip to retrieve all columns when multicone-querying a VizieR catalogue Hi, This tip is intended for Topcat users who query a VizieR catalogue through the multiple cone search, and end up frustrated because they don't get back all the columns of the requested catalogue. The solution is to edit the Cone Search URL and to add the parameter 'VERB=3' , eg http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=II/246&VERB=3 This should also work in STILTS. Mark : maybe Topcat could have an explicit checkbox to query the service with the maximum verbosity ? Cheers, Thomas -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From m.b.taylor@bristol.XXX Mon Mar 1 15:53:59 2010 Date: Mon, 1 Mar 2010 15:53:59 +0000 (GMT) From: Mark Taylor To: topcat-user@sympa.XXX Subject: Re: Tip to retrieve all columns when multicone-querying a VizieR catalogue On Mon, 1 Mar 2010, Thomas Boch wrote: > Hi, > > This tip is intended for Topcat users who query a VizieR catalogue through the > multiple cone search, and end up frustrated because they don't get back all > the columns of the requested catalogue. > > The solution is to edit the Cone Search URL and to add the parameter 'VERB=3' > , eg http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=II/246&VERB=3 > This should also work in STILTS. Thanks Thomas, > Mark : maybe Topcat could have an explicit checkbox to query the service with > the maximum verbosity ? OK, the next release will contain a Verbosity selector in the multi-cone window. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Mar 3 17:47:28 2010 Date: Wed, 3 Mar 2010 17:47:28 +0000 (GMT) From: Mark Taylor To: topcat-user@sympa.XXX Cc: Simon Murphy , Eduardo Gonzalez-Solares Subject: JyStilts Dear TOPCAT users, prompted by a query from Anthony Smith at Sussex,, who uses STILTS with the Jython-based HIPE (Herschel Interactive Processing Environment), I've decided to add a Jython interface to STILTS. STILTS, if you don't know, is the command-line counterpart to TOPCAT. Jython is a pure-java implementation of Python. Python is a scripting language becoming increasingly popular with astronomers. The upshot of this is that it will be possible to perform STILTS scripting from within a Python-like environment, including your own python code and third-party python modules, with the exception that only pure-python (or jython) modules, and not ones based on C code, will be usable. In many cases this should end up being more convenient, less painful, and more efficient than the current approach of doing it all from a unix (or similar) command line. This is not yet in public release, but for anyone who would like to take a look there is a pre-release version at: ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ If you unpack stilts_doc.zip, you'll find HTML/PDF copies of the manual, which has a new section (currently sec 4) on JyStilts, including background, installation instructions (easy), reference material, and some tutorial and examples. If you have any comments, suggestions bug reports, illuminating experiences etc, please pass them on. I would particularly welcome insights from native python speakers, (I've only learnt the language in the last couple of weeks) about things which look like they could or should be done differently to be more pythonic. Also feedback about whether the documentation is adequate would be useful. Thanks, Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From molinaro@oats.XXX Thu Mar 4 11:08:13 2010 Date: Thu, 04 Mar 2010 11:08:10 +0000 From: molinaro@oats.XXX To: topcat-user@sympa.XXX Subject: Topcat command line load multiple tables Hi everybody, I'm trying to load multiple tables at topcat startup using the -f format table1 table2 flag but it seems not to work. Can anyone help me? I'm able to load a single table but gives me an "index of of bounds" error when trying to load the second. bye Marco From m.b.taylor@bristol.XXX Thu Mar 4 11:15:14 2010 Date: Thu, 4 Mar 2010 11:15:14 +0000 (GMT) From: Mark Taylor To: topcat-user@sympa.XXX Subject: Re: Topcat command line load multiple tables On Thu, 4 Mar 2010, molinaro@oats.XXX wrote: > Hi everybody, > I'm trying to load multiple tables at topcat startup using the -f format table1 > table2 flag but it seems not to work. > Can anyone help me? > I'm able to load a single table but gives me an "index of of bounds" error when > trying to load the second. Marco, something like topcat -f csv t1.csv t2.csv works for me. Maybe there's a problem with the second table itself? Can you tell me (a) exactly what the command line is and (b) the error stack trace (either from the terminal, or "Show Details" from the error window and cut and paste the result). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Thu Mar 4 13:20:18 2010 Date: Thu, 4 Mar 2010 13:20:18 +0000 (GMT) From: Mark Taylor To: topcat-user@sympa.XXX Subject: Re: Topcat command line load multiple tables On Thu, 4 Mar 2010, molinaro@oats.XXX wrote: > Hi everybody, > I'm trying to load multiple tables at topcat startup using the -f format table1 > table2 flag but it seems not to work. > Can anyone help me? > I'm able to load a single table but gives me an "index of of bounds" error when > trying to load the second. Marco, thanks for the additional information. It turns out this was a stupid bug in version 3.5 that I fixed (and then forgot about) a few months ago. If you upgrade to the current version. TOPCAT 3.5-1, the problem should go away. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From murphysj@mso.XXX Sat Mar 6 03:00:34 2010 Date: Sat, 6 Mar 2010 14:00:04 +1100 From: Simon Murphy To: topcat-user@mlist2.XXX Subject: VOTable NULL values Hi Mark et al, Does TOPCAT fully support the VOTable NULL value descriptor? When I load a table with null values defined in the header it recognises them in the column metadata display, but when I plot things the nasty 999 null values give some horrible scaling. When parsing the table is there a way to replace null values with the TOPCAT null value (empty cell)? Cheers, Simon Simon J. Murphy e: murphysj@mso.XXX Ph.D. Student w: www.mso.anu.edu.au/~murphysj Mount Stromlo Observatory p: (+61 2) 6125 0254 Australian National Uni. m: 0422 981 531 From m.b.taylor@bristol.XXX Mon Mar 8 08:18:30 2010 Date: Mon, 8 Mar 2010 08:18:30 +0000 (GMT) From: Mark Taylor To: topcat-user@sympa.XXX Subject: Re: VOTable NULL values On Sat, 6 Mar 2010, Simon Murphy wrote: > Hi Mark et al, > > Does TOPCAT fully support the VOTable NULL value descriptor? When I load a > table with null values defined in the header it recognises them in the column > metadata display, but when I plot things the nasty 999 null values give some > horrible scaling. Hi Simon, yes TOPCAT should do all the right things with VOTable nulls. If you can send me the table in question (or at least a bit of it) I'll try to see what's up. > When parsing the table is there a way to replace null values with the TOPCAT > null value (empty cell)? If you have a table that uses a 'magic' value for whatever reason you can turn it into nulls from within TOPCAT by adding a new or replacement column with an expression like "colname == 999 ? NULL : colname". However, it's easier to do by using the STILTS badval filter: http://www.starlink.ac.uk/stilts/sun256/badval.html Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From sylvain.lafrasse@obs.XXX Tue Mar 9 13:45:14 2010 Date: Tue, 9 Mar 2010 14:41:56 +0100 From: Sylvain LAFRASSE To: topcat-user@mlist2.XXX Subject: Losing Formats while converting from VOTable to FITS Dear all, Do you if STILTS use VOTABLE 'width' and 'precision' attributes to fulfill FITS TDISP keywords (as proposed in section 2.3 of VOTable 1.1 specification) ? I did not manage to get it working... Am I missing something ? Neither 'meta' nor 'setparam' functions seem to provide something like this... Best regards, ------------------------------------------------------------------------------------------------------------------------ Sylvain LAFRASSE Tel : +33 (0)4 76 63 55 30 Fax : +33 (0)4 76 44 88 21 Laboratoire d'Astrophysique de Grenoble (LAOG) 414, rue de la Piscine Domaine Universitaire de Saint-Martin-d'Hères BP 53 France 38041 Grenoble cedex 9 ------------------------------------------------------------------------------------------------------------------------ From m.b.taylor@bristol.XXX Tue Mar 9 13:59:39 2010 Date: Tue, 9 Mar 2010 13:59:38 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Losing Formats while converting from VOTable to FITS On Tue, 9 Mar 2010, Sylvain LAFRASSE wrote: > Dear all, > > Do you if STILTS use VOTABLE 'width' and 'precision' attributes to fulfill > FITS TDISP keywords (as proposed in section 2.3 of VOTable 1.1 specification) > ? > I did not manage to get it working... Am I missing something ? > Neither 'meta' nor 'setparam' functions seem to provide something like this... Hi Sylvain, I'm afraid the answer is that STIL/TS pretty much ignores both the FITS TDISP cards and VOTable width/precision attributes. If this is bad news for you, I'd be interested to hear what it is you need it/them for. It might be possible to do something about it in the future (though no promises). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Sylvain.Lafrasse@bvra.XXX Tue Mar 9 14:39:51 2010 Date: Tue, 9 Mar 2010 15:38:18 +0100 From: Sylvain LAFRASSE To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Losing Formats while converting from VOTable to FITS On 9 mars 10, at 14:59, Mark Taylor wrote: > On Tue, 9 Mar 2010, Sylvain LAFRASSE wrote: > >> Dear all, >> >> Do you if STILTS use VOTABLE 'width' and 'precision' attributes to >> fulfill >> FITS TDISP keywords (as proposed in section 2.3 of VOTable 1.1 >> specification) >> ? >> I did not manage to get it working... Am I missing something ? >> Neither 'meta' nor 'setparam' functions seem to provide something >> like this... > > Hi Sylvain, > > I'm afraid the answer is that STIL/TS pretty much ignores both the > FITS TDISP cards and VOTable width/precision attributes. > > If this is bad news for you, I'd be interested to hear what it is you > need it/them for. It might be possible to do something about it in > the > future (though no promises). Hello Mark, I'm using STILTS to (amongst other filtering concerns) convert a huge VOtable to FITS to publish it in Vizier at CDS (one of their preferred file format). And François told me that my catalog would look better with explicit format description in the form of TDISPxxx keywords. So I added width and precision in my VOTable, hoping that STILTS will seamlessly handle it during the conversion. Too bad. But we will look at other ways to circumvent this for the time being. Again thank you for your prompt answer. From sylvain.lafrasse@obs.XXX Wed Mar 10 10:37:29 2010 Date: Wed, 10 Mar 2010 11:31:45 +0100 From: Sylvain LAFRASSE To: topcat-user@bristol.XXX Subject: Column names with dashes Dear all, Do you know a way to test for non-blank value on columns with names containing dashes, like "f-Rmag" ? I tried the expected without success: stilts tpipe in=$PREVIOUSCATALOG cmd="progress ; select !NULL_$ {COLUMN_NAME}" out=$CATALOG ; Best regards, ------------------------------------------------------------------------------------------------------------------------ Sylvain LAFRASSE Tel : +33 (0)4 76 63 55 30 Fax : +33 (0)4 76 44 88 21 Laboratoire d'Astrophysique de Grenoble (LAOG) 414, rue de la Piscine Domaine Universitaire de Saint-Martin-d'Hères BP 53 France 38041 Grenoble cedex 9 ------------------------------------------------------------------------------------------------------------------------ From m.b.taylor@bristol.XXX Wed Mar 10 10:58:34 2010 Date: Wed, 10 Mar 2010 10:58:34 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Column names with dashes On Wed, 10 Mar 2010, Sylvain LAFRASSE wrote: > Dear all, > > Do you know a way to test for non-blank value on columns with names containing > dashes, like "f-Rmag" ? > I tried the expected without success: > stilts tpipe in=$PREVIOUSCATALOG cmd="progress ; select !NULL_${COLUMN_NAME}" > out=$CATALOG ; Hi Sylvain, to use columns with un-identifier-like names in algebraic expressions, you have to use the $ID form, as described here: http://www.starlink.ac.uk/stilts/sun256/jel-colref.html so if f-Rmag is the 8th column, write cmd='select !NULL_$8' aLTERNATIvely you can rename the column on the fly before the select command is executed. You can do this by hand using the colmeta filter: cmd='colmeta -name fRmag f-Rmag; select !NULL_fRmag' or, using a new convenience filter 'fixcolnames' I introduced only in the most recent release (2.1-1): cmd='fixcolnames; select !NULL_f_Rmag' Fixcolnames (http://www.starlink.ac.uk/stilts/sun256/fixcolnames.html) replaces any unhelpful characters in column names with underscores. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Mar 10 15:55:14 2010 Date: Wed, 10 Mar 2010 15:55:14 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Large-scale crossmatching Hi all, One of the things I'm currently working on is improving the scalability of crossmatching within TOPCAT/STILTS. This is limited by available memory (not so much by CPU time), and currently starts to fail for table sizes larger than around a million rows or so. This figure is very approximate, since it depends on the details of the match in question as well as how much heap memory is available to your Java Virtual Machine. I have made recent improvements to raise this limit significantly - by something like a factor of five for some common matching scenarios, with speed increases of a factor of two or so. These improvements are not yet in public release, but they are available in pre-release versions for the adventurous. One thing that's become clear to me while working on it is that the performance and scalability of the algorithm depends a lot on the details of the crossmatch required. The changes I've made so far improve performance for some regimes, but leave others unaffected, or possibly even worse than before. I don't at present plan to improve it to such an extent that all possible match regimes are maximally scalable, since that would involve quite a lot of work, and some of it might not end up being actually used by anybody. Also, it's difficult to do benchmarking and optimisation without real-life, or at least realistic, large data sets and match output requirements. But, if there are real-life matching requirements that people have, for which either TOPCAT/STILTS's existing matching facilities seem to be underpowered, or which resemble larger requirements that they're expecting to have in the future, I'd be keen to talk about the details and maybe experiment with datasets to see if performance is good enough or could be improved. So, if you've got tables that you'd like to crossmatch with TOPCAT/STILTS, but that seem to be too big, or think you might do soon, I'd be grateful if you could get in touch with me either on or off list. Note, I don't expect that these tools are ever going to be suitable for truly enormous matches (e.g. complete 2MASS vs SDSS); I think problems of that scale (1e9 rows) will for the forseeable future have to be tackled using more specialized tools. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From pkim.visual@gmail.XXX Tue Mar 16 13:41:09 2010 Date: Tue, 16 Mar 2010 13:40:58 +0000 From: pkim.visual@gmail.XXX To: topcat-user@sympa.XXX Subject: Greek letters or any special characters I search through the manual but I have not found. How to write Greek letters or any special characters to labels and other places in the plot?? Kamil From m.b.taylor@bristol.XXX Tue Mar 16 16:46:35 2010 Date: Tue, 16 Mar 2010 16:46:35 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Greek letters or any special characters On Tue, 16 Mar 2010, pkim.visual@gmail.XXX wrote: > I search through the manual but I have not found. > > How to write Greek letters or any special characters to labels and other places > in the plot?? Sorry, Kamil, this can't currently be done. Maybe one day. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Thu Mar 25 10:24:00 2010 Date: Thu, 25 Mar 2010 13:23:45 +0300 From: Ivan Zolotukhin To: topcat-user@sympa.XXX Subject: TOPCAT's echo of row.highlight Hello, Seems that TOPCAT is the only sampified VO application that transmits a row if it is activated upon received SAMP row.highlight message. This echo (not caused by user actions in TOPCAT) potentially creates bad effects, some sort of infinite loops. Here's the example with browser SAMP plugin where highlight event is caused by mouse hover (changing that to mouse click is not an option by a number of reasons). 1. User points mouse over rows 1, 2, 3, stopping at row 3; three row.highlight messages sent. 2. TOPCAT resends them but by some occasion (due to delays in a way) their order is changed to 1, 3, 2. 3. Browser receives TOPCAT's row.highlight messages and performs visual highlighting while user is not moving mouse. 4. Browser notices that since mouse is over row 3 and highlight is on row 2 after TOPCAT, it should send one more row.highlight message with row 3. 5. The situation can easily be extrapolated to a bigger number of events. Does this TOPCAT's echo really contain some use cases beneath? I'm almost sure what you Mark will answer :) but at least you know about the problem. -- Regards, Ivan From m.b.taylor@bristol.XXX Thu Mar 25 14:51:06 2010 Date: Thu, 25 Mar 2010 14:51:05 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: TOPCAT's echo of row.highlight On Thu, 25 Mar 2010, Ivan Zolotukhin wrote: > Hello, > > Seems that TOPCAT is the only sampified VO application that transmits > a row if it is activated upon received SAMP row.highlight message. > This echo (not caused by user actions in TOPCAT) potentially creates > bad effects, some sort of infinite loops. Here's the example with > browser SAMP plugin where highlight event is caused by mouse hover > (changing that to mouse click is not an option by a number of > reasons). > > 1. User points mouse over rows 1, 2, 3, stopping at row 3; three > row.highlight messages sent. > 2. TOPCAT resends them but by some occasion (due to delays in a way) > their order is changed to 1, 3, 2. > 3. Browser receives TOPCAT's row.highlight messages and performs > visual highlighting while user is not moving mouse. > 4. Browser notices that since mouse is over row 3 and highlight is on > row 2 after TOPCAT, it should send one more row.highlight message with > row 3. > 5. The situation can easily be extrapolated to a bigger number of events. > > Does this TOPCAT's echo really contain some use cases beneath? > > I'm almost sure what you Mark will answer :) but at least you know > about the problem. Ivan, thanks for the report. Not sure if your prediction of my response is correct, but I am aware of this, and I regard it as a misfeature ... all right, a bug. Unfortunately it's hard to fix because of details of the implementation, so until now I've kept quiet and hoped that nobody notices. Although I know it can in principle cause trouble (it's not too hard to get two instances of TOPCAT to play infinite ping-pong) I don't think I've so far seen it causing problems in a real user situation. I can't quite tell from your description above - is this behaviour a serious issue for your usage, or is it a minor annoyance? Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Thu Mar 25 16:10:38 2010 Date: Thu, 25 Mar 2010 19:10:21 +0300 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: TOPCAT's echo of row.highlight Mark, Thanks for you comment. > Although I know it can in principle cause trouble (it's not too hard > to get two instances of TOPCAT to play infinite ping-pong) I don't > think I've so far seen it causing problems in a real user situation. > I can't quite tell from your description above - is this behaviour > a serious issue for your usage, or is it a minor annoyance? In my SAMP client, upon receiving of row.highlight message, I should scroll the web page so that the row is visible if it is not in the moment. This sort of loop described above leads to serious browser malfunction when user moves a mouse over HTML table in the least bit fast. Returning echoes from TOPCAT cause strange errors when a browser (it varies between them) mistakes if a row received is visible or not, and tries to scroll the web page upon late row.highlight messages caused by previous mouse movement. This leads to annoying web page twitching. I would rate the importance of the problem as 7 out of 10 points. I'm not insisting on you to fix this immediately, but just let me know your plans (e.g. will wait for the next complaint or something), so that I can decide what to do here, because I don't want to leave it as it is now. -- Regards, Ivan From m.b.taylor@bristol.XXX Thu Mar 25 16:53:33 2010 Date: Thu, 25 Mar 2010 16:53:33 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: TOPCAT's echo of row.highlight On Thu, 25 Mar 2010, Ivan Zolotukhin wrote: > Mark, > > Thanks for you comment. > > > Although I know it can in principle cause trouble (it's not too hard > > to get two instances of TOPCAT to play infinite ping-pong) I don't > > think I've so far seen it causing problems in a real user situation. > > I can't quite tell from your description above - is this behaviour > > a serious issue for your usage, or is it a minor annoyance? > > In my SAMP client, upon receiving of row.highlight message, I should > scroll the web page so that the row is visible if it is not in the > moment. This sort of loop described above leads to serious browser > malfunction when user moves a mouse over HTML table in the least bit > fast. Returning echoes from TOPCAT cause strange errors when a browser > (it varies between them) mistakes if a row received is visible or not, > and tries to scroll the web page upon late row.highlight messages > caused by previous mouse movement. This leads to annoying web page > twitching. I would rate the importance of the problem as 7 out of 10 > points. > > I'm not insisting on you to fix this immediately, but just let me know > your plans (e.g. will wait for the next complaint or something), so > that I can decide what to do here, because I don't want to leave it as > it is now. Yes, OK. I will try to have a look at this in the next few days. If it's not as difficult as I remember, I may be able to fix it. In the best case I might be able to sort it out and let you have a fixed version soon, but even then it would probably not make its way into a public release immediately (I've only just done one), and even if it does, people will probably still encounter your web page using older versions. So even in the best case, this could be an issue that you need to work around. Nag me again if I don't come up with something more definite by the end of next week. Apologies for this bad behaviour (I mean TOPCAT's, not mine). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From august.fly@gmail.XXX Tue Mar 30 21:43:09 2010 Date: Tue, 30 Mar 2010 20:43:00 +0100 From: august.fly@gmail.XXX To: topcat-user@sympa.XXX Subject: Table Column Referencing Hi, I am experiencing unexpected behavior of column name referencing in TOPCAT. Starting from a single input table I am performing a series of two further multicone searches where each new table is a "New Joined Table, one row per input row" from the former. Say generically, I have 3 tables: Original, Original+GALEX, Original+GALEX+2MASS. After finishing both multicone searches I went back and replaced a column in the "Original+GALEX" match with a new one based on it. So the 2nd table now has a column named, "name" and another column (now hidden) named "name_old." Unexpectedly, the column "name" in the third table, "Original+GALEX+2MASS" changed to "name_old." The column name changes in the Column Info window, and for all other references to that column (such as plot windows) but it does NOT change for the Table Data window, where the column is still called "name." Attempts to use the "name" column fail in operator expressions in the third table. No data in the 3rd table is changed. The expected behavior is that all tables/columns are completely distinct and operations do not affect parent, sibling or child tables. I do not find any reference to table column SQL like aliasing in the documentation and the Table parameters for these successive joins do not log parent/child/sibling relationships. I suspect that there is a bug somewhere (the fact that the column name in the Table Data and Column Info windows do not match is almost certainly a bug) but look forward to clarification on table column referencing wrt my expectations. - Gus Full details of my workflow are shared at http://www.evernote.com/pub/august2fly/astrosoftware See entry: "TOPCAT Table Column Referencing" From kurtcovive@gmail.XXX Wed Mar 31 11:28:42 2010 Date: Tue, 30 Mar 2010 23:53:45 +0100 From: kurtcovive@gmail.XXX To: topcat-user@sympa.XXX Subject: Join many tables in a big one Hi Mark. I have many tables and I want to join all of them in an big one. There's any way to save these tables in an only file which, when you open it, you'll have all the tables open in topcat? In the same way, can I concatenate the diferent tables in an only table but with out losing columns of information? Thanks and sorry for my poor English From kurtcovive@gmail.XXX Wed Mar 31 01:08:29 2010 Date: Wed, 31 Mar 2010 00:08:21 +0100 From: Jorge To: topcat-user@sympa.XXX Subject: Join many tables in a big one Hi Mark. I've got about 50 separate tables and I need to join all of them in a big one. I thought two different solutions to do that but, Idon´t know how to make them. 1: Save the 50 separate tables in an only file which, when I open it, Topcat opens the 50 separate tables at once. 2: In the case of concatenate the tables, I need to keep all the information possible. This is so difficult whith the topcat task. Is possible to join the tables in a big one with out losing columns of information? Thanks From sylvain.lafrasse@obs.XXX Wed Mar 31 11:28:42 2010 Date: Wed, 31 Mar 2010 10:42:19 +0200 From: Sylvain LAFRASSE To: topcat-user@bristol.XXX Subject: Official acknowledgement ? Dear all, I can't seem to find the official sentence to acknowledge use of TOPCAT/STILTS/STIL for my forthcoming catalog... Am I missing something on the website ? Best regards, Sylvain From m.b.taylor@bristol.XXX Wed Mar 31 10:37:46 2010 Date: Wed, 31 Mar 2010 10:37:46 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Cc: Sylvain LAFRASSE Subject: Re: Official acknowledgement ? On Wed, 31 Mar 2010, Sylvain LAFRASSE wrote: > I can't seem to find the official sentence to acknowledge use of > TOPCAT/STILTS/STIL for my forthcoming catalog... > Am I missing something on the website ? Sylvain thanks for asking! There is no official form of words (do you think I should have one?) but it is much appreciated if you can acknowledge TOPCAT (and/or STILTS and/or STIL as appropriate) by name. In this case you can either cite one of the URLs: http://www.starlink.ac.uk/topcat/ http://www.starlink.ac.uk/stilts/ http://www.starlink.ac.uk/stil/ or one of the papers with the following bibcodes: 2005ASPC..347...29T (TOPCAT) 2006ASPC..351..666T (STILTS) 2009ASPC..411..510T (STILTS plotting) This information is summarized in the FAQ: http://www.starlink.ac.uk/topcat/faq.html#acknowledgement (admittedly not so easy to find). I'd be happy if you could tell me what the forthcoming catalogue is when it's available too. Best wishes Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Mar 31 10:55:11 2010 Date: Wed, 31 Mar 2010 10:55:11 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Join many tables in a big one On Wed, 31 Mar 2010, Jorge wrote: > Hi Mark. > > I've got about 50 separate tables and I need to join all of them in a big one. > I thought two different solutions to do that but, Idon´t know how to make > them. > > 1: Save the 50 separate tables in an only file which, when I open it, Topcat > opens the 50 separate tables at once. > > 2: In the case of concatenate the tables, I need to keep all the information > possible. This is so difficult whith the topcat task. Is possible to join the > tables in a big one with out losing columns of information? Jorge, do all the tables have a similar form (all have the same columns as each other), or are they different? Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From sylvain.lafrasse@obs.XXX Wed Mar 31 12:50:00 2010 Date: Wed, 31 Mar 2010 13:49:47 +0200 From: Sylvain LAFRASSE To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Official acknowledgement ? Hi Mark, By the way, here's what I wrote, insperied by Vizier and Simbad CDS webservices (and I think an official sentence would be useful): This research has made use of the TOPCAT (http://www.starlink.ac.uk/topcat/ ) and STILTS (http://www.starlink.ac.uk/stilts/) software, provided by Mark Taylor of Bristol University, England. Amicalement, Sylvain On 31 mars 10, at 11:37, Mark Taylor wrote: > On Wed, 31 Mar 2010, Sylvain LAFRASSE wrote: > >> I can't seem to find the official sentence to acknowledge use of >> TOPCAT/STILTS/STIL for my forthcoming catalog... >> Am I missing something on the website ? > > Sylvain > > thanks for asking! There is no official form of words (do you think > I should have one?) but it is much appreciated if you can acknowledge > TOPCAT (and/or STILTS and/or STIL as appropriate) by name. > In this case you can either cite one of the URLs: > > http://www.starlink.ac.uk/topcat/ > http://www.starlink.ac.uk/stilts/ > http://www.starlink.ac.uk/stil/ > > or one of the papers with the following bibcodes: > > 2005ASPC..347...29T (TOPCAT) > 2006ASPC..351..666T (STILTS) > 2009ASPC..411..510T (STILTS plotting) > > This information is summarized in the FAQ: > > http://www.starlink.ac.uk/topcat/faq.html#acknowledgement > > (admittedly not so easy to find). Oups, I missed it. Sorry ! > > I'd be happy if you could tell me what the forthcoming catalogue > is when it's available too. I will keep you updated when it gets public. > Best wishes > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, > UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/ > ~mbt/ > From m.b.taylor@bristol.XXX Wed Mar 31 15:06:03 2010 Date: Wed, 31 Mar 2010 15:06:03 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Join many tables in a big one On Wed, 31 Mar 2010, Jorge wrote: > Hi Mark. > > I've got about 50 separate tables and I need to join all of them in a big one. > I thought two different solutions to do that but, Idon´t know how to make > them. > > 1: Save the 50 separate tables in an only file which, when I open it, Topcat > opens the 50 separate tables at once. > > 2: In the case of concatenate the tables, I need to keep all the information > possible. This is so difficult whith the topcat task. Is possible to join the > tables in a big one with out losing columns of information? Jorge, it is possible to prepare a single file which contains many different tables, which will all be loaded into TOPCAT when the file is loaded. There are two possibilities: 1. a VOTable with one TABLE element for each table you want 2. a FITS file with one BINTABLE (or TABLE) HDU for each table you want However, you can't currently prepare either of these formats using TOPCAT or STILTS :-(. I am considering adding that possibility in a future release. In the mean time, you can prepare a suitable FITS file using FTOOLS (http://heasarc.nasa.gov/docs/software/ftools/ftools_menu.html). First, make sure that your individual tables are in FITS format, with an empty or innocuous primary HDU and the table as the first extension HDU (saving them as "fits-basic" from TOPCAT will achieve this if they are not already in that format; actually only the first one has to be "fits-basic", the others can be either "fits" or "fits-basic"). Then, use the FTOOLS fappend or ftappend commands to copy table HDUs from the second and subsequent files to the first one: ftappend 't2.fits[1]' t1.fits ftappend 't3.fits[1]' t1.fits ftappend 't4.fits[1]' t1.fits .. ftappend 't50.fits[1]' t1.fits Then loading t1.fits into TOPCAT will make all your tables show up. Note however: when they are loaded in, their names are currently assigned like t1.fits-1 t1.fits-2 t1.fits-3 t1.fits-4 .. t1.fits-5 (is there a standard FITS header in a BINTABLE HDU which it would be sensible to use as a table name?) Disclaimer: I don't have much experience with FTOOLS, but I think the above should work. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Mar 31 17:57:14 2010 Date: Wed, 31 Mar 2010 17:57:14 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Official acknowledgement ? Ivan, On Wed, 31 Mar 2010, Ivan Zolotukhin wrote: > You should probably collect the papers acknowledging TOPCAT usage in a > special section of your website as well, something like Science with > TOPCAT. It's not that easy to find them because of the mentioned lack > of the official acknowledgement, but if you ask the community, it will > help to find most prominent ones. In my mind, that may simplify your > next funding iteration. thanks, a "Science with TOPCAT" section on the website is not a bad idea, I may compile one (I have a partial list I've compiled myself which I am indeed using for my current funding iteration). > As for official words or citation, IMO there'll be only benefits to have them. On reflection, I don't think I will provide one, for this reason: such an official form of words really ought to reference the projects and funding agencies which have supported the work. Since TOPCAT's funding history is quite long and disparate (see http://www.starlink.ac.uk/topcat/faq.html#Q1.20), it could get rather verbose. As long as a website or (preferably from the point of literature searches) publication citation is given, it should be OK. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bianca@lambrate.XXX Thu Apr 1 09:05:31 2010 Date: Thu, 1 Apr 2010 09:04:30 +0200 From: Bianca Garilli To: topcat-user@mlist2.XXX Subject: Re: Join many tables in a big one Hi Mark, I just tried your usefull suggestion given to Jorge, but it does not work exactly as you said I have a fits file with an empty primary header, and 2 binary tables in it. I load it into topcat as FITS, but the only table I see is the first one. Still, using ftools, I correctly see the 2 tables in the file (i.e. they ARE there) I'm using topcat 3.4-1, might that be the reason? Also, you might consider to use the EXTNAME k/w in the fits header to name tables in topcat. I'm not sure it is a compulsory k/w, but IF it is present, it can be used Cheers Bianca -- ************** NOT IN MY NAME ******************************** -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 ------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Thu Apr 1 10:26:30 2010 Date: Thu, 1 Apr 2010 10:26:30 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Join many tables in a big one On Thu, 1 Apr 2010, Bianca Garilli wrote: > Hi Mark, I just tried your usefull suggestion given to Jorge, > but it does not work exactly as you said > I have a fits file with an empty primary header, and 2 binary tables > in it. I load it into topcat as FITS, but the only table I see is the first > one. Still, using ftools, I correctly see the 2 tables in the file (i.e. they > ARE there) > I'm using topcat 3.4-1, might that be the reason? yes, multiple table load has only recently been introduced. You will need TOPCAT v3.5 or later for this to work. > Also, you might consider to use the EXTNAME k/w in the fits header > to name tables in topcat. I'm not sure it is a compulsory k/w, but > IF it is present, it can be used Yes, that looks appropriate, thanks for the suggestion. I will see about using EXTNAME header cards as table names in future releases. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Apr 7 11:06:22 2010 Date: Wed, 7 Apr 2010 11:06:22 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Table Column Referencing Gus, thanks again for this detailed report. I've made a change which fixes the particular problem that you describe. I think it should also improve behaviour for other similar cases; in particular in future changing the name, units, and other metadata of one table should not affect those items in any other derived table. This change will be in the next public release; if you want to try it out before then you can look at the pre-release version in ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Mark On Tue, 30 Mar 2010, august.fly@gmail.XXX wrote: > Hi, > > > I am experiencing unexpected behavior of column name referencing in TOPCAT. > > > Starting from a single input table I am performing a series of two further > multicone searches where each new table is a "New Joined Table, one row per > input row" from the former. Say generically, I have 3 tables: Original, > Original+GALEX, Original+GALEX+2MASS. After finishing both multicone searches > I went back and replaced a column in the "Original+GALEX" match with a new one > based on it. So the 2nd table now has a column named, "name" and another column > (now hidden) named "name_old." > > > Unexpectedly, the column "name" in the third table, "Original+GALEX+2MASS" > changed to "name_old." The column name changes in the Column Info window, and > for all other references to that column (such as plot windows) but it does NOT > change for the Table Data window, where the column is still called "name." > Attempts to use the "name" column fail in operator expressions in the third > table. No data in the 3rd table is changed. > > > The expected behavior is that all tables/columns are completely distinct and > operations do not affect parent, sibling or child tables. I do not find any > reference to table column SQL like aliasing in the documentation and the Table > parameters for these successive joins do not log parent/child/sibling > relationships. > > > I suspect that there is a bug somewhere (the fact that the column name in the > Table Data and Column Info windows do not match is almost certainly a bug) but > look forward to clarification on table column referencing wrt my expectations. > > > - Gus > > > > Full details of my workflow are shared at > > http://www.evernote.com/pub/august2fly/astrosoftware > > See entry: "TOPCAT Table Column Referencing" -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Apr 7 15:56:42 2010 Date: Wed, 7 Apr 2010 15:56:42 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: TOPCAT's echo of row.highlight Ivan, On Thu, 25 Mar 2010, Ivan Zolotukhin wrote: > Hello, > > Seems that TOPCAT is the only sampified VO application that transmits > a row if it is activated upon received SAMP row.highlight message. > This echo (not caused by user actions in TOPCAT) potentially creates > bad effects, some sort of infinite loops. Here's the example with > browser SAMP plugin where highlight event is caused by mouse hover > (changing that to mouse click is not an option by a number of > reasons). I believe this is now fixed. Receipt of a table.highlight.row SAMP message should not now cause TOPCAT to send a similar such message. This fix will be in the next public release; there's a pre-release version at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ which you're welcome to try. If you do, and you still think it's behaving suboptimally in this respect, let me know. Thanks for the report. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From august.fly@gmail.XXX Wed Apr 7 17:30:19 2010 Date: Wed, 07 Apr 2010 16:30:12 +0100 From: august.fly@gmail.XXX To: topcat-user@sympa.XXX Subject: Table IO/ Session Saving hi, it may be entirely a product of my particular use case(s) but I very frequently find myself wishing I could simply save an entire TOPCAT session, all the tables, at once into a new "table" package that I could restore at a later time. the instances that most often elicit this desire include backing up my work, or wishing I had because the power at my work went out wiping out my TOPCAT session, creating a package of tables from a project to share, and portability between desktop/laptop/home/work. such a feature has been prominent in our CfA python users group and astropython.org discussions, which is just to say that prob any work "environ" could lead to such a feature request. I suppose metadata/session packaging might be an issue. I personally would think that a multi-extension FITS would upset the fewest number of users. Of course xml+fits ref might alleviate metadata translation issues. I don't know. anyway, just 1 user's comment/suggestion. i hope i haven't overlooked some basic feature yet also wonder if there are other's who have similar experiences. best, - gus muench From m.b.taylor@bristol.XXX Wed Apr 7 16:45:43 2010 Date: Wed, 7 Apr 2010 16:45:43 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving Hi Gus, On Wed, 7 Apr 2010, august.fly@gmail.XXX wrote: > hi, > > > it may be entirely a product of my particular use case(s) but I very frequently > find myself wishing I could simply save an entire TOPCAT session, all the > tables, at once into a new "table" package that I could restore at a later > time. This is a request that has come up several times, but so far I've not been persuaded to do it. I can see the usefulness, but it would be quite a lot of work to do it properly - there is quite a lot of state in a TOPCAT session, and it's not clear what's the best way to save some of it - e.g. should table data itself be saved, or references to external tables (which is what TOPCAT does internally in many cases). > I suppose metadata/session packaging might be an issue. I personally would > think that a multi-extension FITS would upset the fewest number of users. Of > course xml+fits ref might alleviate metadata translation issues. I don't know. Just saving table data would be straightforward enough (of course you can do it one table at a time right now...). It would certainly be feasible to add an option to save either all currently loaded tables, or a selection of currently loaded tables, into a single multi-extension FITS file or multi-table VOTable file. This would store table data and metadata, but not for instance row subsets, activation actions, open plots, state of crossmatch windows etc etc. How far would that go to addressing your wishes? Comments from other users welcome too. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Apr 7 16:56:05 2010 Date: Wed, 7 Apr 2010 16:56:00 +0100 (BST) From: Mark Taylor To: Frank Primini Cc: topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving On Wed, 7 Apr 2010, Frank Primini wrote: > Hi, > > Rather than saving state, would it be possible to simply write the stilts equivalents to the various actions that have been performed in a session, say in a log file? Then one could easily reproduce the session at will. > > Frank Primini Frank, that's also an excellent idea! I hope to be able to provide something along those lines in the forseeable future. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bianca@lambrate.XXX Wed Apr 7 17:57:15 2010 Date: Wed, 7 Apr 2010 17:56:55 +0200 From: Bianca Garilli To: topcat-user@mlist2.XXX Cc: Mark Taylor , topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving Hi I'm a very frequent user of topcat, as well as other people in my institute, and we are all big fans of it But yes, we (and I talk also for my 4-5 colleagues, probably not subscribers of this list) would very much appreciate to have a way to "save" and "restore" the session. For example, now I have 2 topcat session open, in each of them ~5 tables, and each table has from 5 to 8 subsets In 20 minutes I have to go home and I will carry the laptop with me. The only idea of saving all tables, one by one, on disk, then reimporting them all, and (worst of everything) repeating all this work of defining subsets makes me cry. Not to mention my favourite settings for plots And this happens quite often, as topcat for me has become a fundamental tool for examining data and finding relations Thus I'm strongly in favour of a save/restore utility for the future. No idea how it could be done. Nor if it can be done so extensively as I would like (subsets, plotting preferences, etc etc etc). But any, even small, improvemnt on the current state would be greatly appreciated Bianca On Wednesday 07 April 2010 17:45, Mark Taylor wrote: > Hi Gus, > > On Wed, 7 Apr 2010, august.fly@gmail.XXX wrote: > > hi, > > > > > > it may be entirely a product of my particular use case(s) but I very > > frequently find myself wishing I could simply save an entire TOPCAT > > session, all the tables, at once into a new "table" package that I could > > restore at a later time. > > This is a request that has come up several times, but so far I've not > been persuaded to do it. I can see the usefulness, but it would be > quite a lot of work to do it properly - there is quite a lot of > state in a TOPCAT session, and it's not clear what's the best way > to save some of it - e.g. should table data itself be saved, or > references to external tables (which is what TOPCAT does internally > in many cases). > > > I suppose metadata/session packaging might be an issue. I personally > > would think that a multi-extension FITS would upset the fewest number of > > users. Of course xml+fits ref might alleviate metadata translation > > issues. I don't know. > > Just saving table data would be straightforward enough (of course you > can do it one table at a time right now...). It would certainly be > feasible to add an option to save either all currently loaded tables, > or a selection of currently loaded tables, into a single multi-extension > FITS file or multi-table VOTable file. This would store table data and > metadata, but not for instance row subsets, activation actions, open > plots, state of crossmatch windows etc etc. How far would that go to > addressing your wishes? > > Comments from other users welcome too. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ -- ************** NOT IN MY NAME ******************************** -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 ------------------------------------------------------------------------- From august.fly@gmail.XXX Wed Apr 7 17:59:13 2010 Date: Wed, 7 Apr 2010 11:58:59 -0400 From: "August (Gus) Muench" To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving I do hope others weigh in... >> I suppose metadata/session packaging might be an issue.  I personally would >> think that a multi-extension FITS would upset the fewest number of users.  Of >> course xml+fits ref might alleviate metadata translation issues. I don't know. > > Just saving table data would be straightforward enough (of course you > can do it one table at a time right now...).  It would certainly be > feasible to add an option to save either all currently loaded tables, > or a selection of currently loaded tables, into a single multi-extension > FITS file or multi-table VOTable file.  This would store table data and > metadata, but not for instance row subsets, activation actions, open > plots, state of crossmatch windows etc etc.  How far would that go to > addressing your wishes? > I think that would go along, long way. of all the other session variables, row subsets are very useful, so I frequently convert them to boolean columns when a table approaches a high degree of revision. - gus -- August (Gus) Muench august.fly@gmail.XXX @augustmuench From jtlz2@astro.XXX Wed Apr 7 17:03:56 2010 Date: Wed, 7 Apr 2010 12:03:28 -0400 From: Jonathan Zwart To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving Hi Mark, Gus and others, On 7 April 2010 11:45, Mark Taylor wrote: > Hi Gus, > > On Wed, 7 Apr 2010, august.fly@gmail.XXX wrote: > > > hi, > > it may be entirely a product of my particular use case(s) but I very > frequently > > find myself wishing I could simply save an entire TOPCAT session, all the > > tables, at once into a new "table" package that I could restore at a > later > > time. > So I asked Mark for exactly this just a few days ago. It's definitely top of my wish list. If I'm playing with a data set for a few days on and off, there is a huge overhead associated with getting back to where I was, and so I have to try and leave TOPCAT running throughout. But see below. This is a request that has come up several times, but so far I've not > been persuaded to do it. I can see the usefulness, but it would be > quite a lot of work to do it properly - there is quite a lot of > state in a TOPCAT session, and it's not clear what's the best way > to save some of it - e.g. should table data itself be saved, or > references to external tables (which is what TOPCAT does internally > in many cases). > > > I suppose metadata/session packaging might be an issue. I personally > would > > think that a multi-extension FITS would upset the fewest number of users. > Of > > course xml+fits ref might alleviate metadata translation issues. I don't > know. > > Just saving table data would be straightforward enough (of course you > can do it one table at a time right now...). It would certainly be > feasible to add an option to save either all currently loaded tables, > or a selection of currently loaded tables, into a single multi-extension > FITS file or multi-table VOTable file. I agree this would be a great start! And I love Frank's STILTS session-logging idea. > This would store table data and > metadata, but not for instance row subsets, activation actions, open > plots, state of crossmatch windows etc etc. How far would that go to > addressing your wishes? > The next most useful thing after saving tables would definitely be to have the ability to store row subsets. But these can be tested for each row, no? So what about, when exporting and when at least one subset has been defined, adding extra columns, one for each subset with this information? Of course you'd need to be able to read it back in and set up the subsets. Jon From ivan.zolotukhin@gmail.XXX Wed Apr 7 20:42:03 2010 Date: Wed, 7 Apr 2010 22:41:47 +0400 From: Ivan Zolotukhin To: Jonathan Zwart Cc: Mark Taylor , topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving Hi all, Glad to see this issue escalating, thanks to public mailing list! First time I asked Mark about that in Dec 2008 (just found a mail), next time during Strasbourg Interop in May 2009 asking to provide at least STILTS script session log similar to existing Aladin script history functionality, all in vain :) Hope that now the time has come. Owing to bright use cases mentioned in this thread we now can see that this ability to save a session state (at least somehow) is probably the most anticipated TOPCAT feature. I totally support any move in this direction that Mark considers appropriate, because it would greatly simplify the way people (and myself) work with TOPCAT every day. -- Regards, Ivan On Wed, Apr 7, 2010 at 8:03 PM, Jonathan Zwart wrote: > Hi Mark, Gus and others, > > On 7 April 2010 11:45, Mark Taylor wrote: >> >> Hi Gus, >> >> On Wed, 7 Apr 2010, august.fly@gmail.XXX wrote: >> >> > hi, >> > it may be entirely a product of my particular use case(s) but I very >> > frequently >> > find myself wishing I could simply save an entire TOPCAT session, all >> > the >> > tables, at once into a new "table" package that I could restore at a >> > later >> > time. > > So I asked Mark for exactly this just a few days ago. It's definitely top of > my wish list. If I'm playing with a data set for a few days on and off, > there is a huge overhead associated with getting back to where I was, and so > I have to try and leave TOPCAT running throughout. But see below. > >> This is a request that has come up several times, but so far I've not >> been persuaded to do it.  I can see the usefulness, but it would be >> quite a lot of work to do it properly - there is quite a lot of >> state in a TOPCAT session, and it's not clear what's the best way >> to save some of it - e.g. should table data itself be saved, or >> references to external tables (which is what TOPCAT does internally >> in many cases). >> >> > I suppose metadata/session packaging might be an issue.  I personally >> > would >> > think that a multi-extension FITS would upset the fewest number of >> > users.  Of >> > course xml+fits ref might alleviate metadata translation issues. I don't >> > know. >> >> Just saving table data would be straightforward enough (of course you >> can do it one table at a time right now...).  It would certainly be >> feasible to add an option to save either all currently loaded tables, >> or a selection of currently loaded tables, into a single multi-extension >> FITS file or multi-table VOTable file. > > I agree this would be a great start! > > And I love Frank's STILTS session-logging idea. > >> >>  This would store table data and >> metadata, but not for instance row subsets, activation actions, open >> plots, state of crossmatch windows etc etc.  How far would that go to >> addressing your wishes? > > The next most useful thing after saving tables would definitely be to have > the ability to store row subsets. But these can be tested for each row, no? > So what about, when exporting and when at least one subset has been defined, > adding extra columns, one for each subset with this information? Of course > you'd need to be able to read it back in and set up the subsets. > > Jon From cgp@star.XXX Thu Apr 8 10:14:24 2010 Date: Thu, 08 Apr 2010 09:14:11 +0100 From: Clive Page To: topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving On 07/04/2010 19:41, Ivan Zolotukhin wrote: > Glad to see this issue escalating, thanks to public mailing list! I hope that Mark does not discontinue the public mailing list, now that he sees how easily a feature request turns into a lobby group, and then into a juggernaut. If so I suggest he deploys the classical response: "There is no demand for this feature, and by the way you are the tenth person this morning to whom I have had to say this." Regards -- Clive Page From m.b.taylor@bristol.XXX Thu Apr 8 09:30:44 2010 Date: Thu, 8 Apr 2010 09:30:44 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving On Wed, 7 Apr 2010, Ivan Zolotukhin wrote: > Hi all, > > Glad to see this issue escalating, thanks to public mailing list! > First time I asked Mark about that in Dec 2008 (just found a mail), > next time during Strasbourg Interop in May 2009 asking to provide at > least STILTS script session log similar to existing Aladin script > history functionality, all in vain :) Hope that now the time has come. not in vain! I keep a mental (and sometimes physical) note of all such user requests, and the more often I see the same thing cropping up, the more likely I am to do something about it. It is this kind of user-driven design that has to a large extent made TOPCAT what it is today, and I'm very grateful for all the user feedback I get. I'm happy to see this sort of discussion happening on the mailing list, it's partly why I wanted to set it up. For this particular issue, I can now see how to implement much of the functionality that people want with a modest amount of effort (this approach has only become possible since infrastructure changes - multiple table load - introduced near the end of last year), so I will make it a priority, hopefully the next release. In the first instance I expect the new functionality to be: - save all or a selection of loaded tables to a single file - all row subsets will get serialized as boolean columns in their tables When the single file is loaded back in, all the tables will be present with their defined subsets, but columns and subsets which were defined in terms of algebraic expressions may return as fixed data rather than algebraically defined. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From mattia.vaccari@gmail.XXX Thu Apr 29 11:30:25 2010 Date: Thu, 29 Apr 2010 11:29:35 +0200 From: Mattia Vaccari To: topcat-user@bristol.XXX Subject: Sky Map Projections I understand topcat doesn't provide built-in equal-area (Hammer-Aitoff and/or Mollweide etc) sky map projection capabilities. *If* that's right, what's the handiest (interoperable?) tool one can send catalogs to for such projections? Cheers - Mattia -- Mattia Vaccari ------ Post Doctoral Research Assistant Department of Astronomy --------- University of Padova Vicolo Osservatorio 3 --- I-35122 --- Padova --- Italy mattia@mattiavaccari.XXX ---- http://mattiavaccari.net +39 348 2541209 (Mobile) ---- +39 049 8278251 (Office) +39 049 8278212 (Fax) - +44 (0)77 60421086 (UK Mobile) From m.b.taylor@bristol.XXX Thu Apr 29 10:40:24 2010 Date: Thu, 29 Apr 2010 10:40:23 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Sky Map Projections On Thu, 29 Apr 2010, Mattia Vaccari wrote: > I understand topcat doesn't provide built-in equal-area (Hammer-Aitoff and/or > Mollweide etc) sky map projection capabilities. *If* that's right, what's the > handiest (interoperable?) tool one can send catalogs to for such projections? That is right, sky coordinate systems are not well supported in TOPCAT graphics. I hope to overhaul the graphics in the next few months to do something about this. In the mean time, depending on what you want to do, I should think Aladin is a good choice? Others may have different suggestions. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From thomas.boch@astro.XXX Thu Apr 29 11:44:42 2010 Date: Thu, 29 Apr 2010 11:44:31 +0200 From: Thomas Boch To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Sky Map Projections Mark Taylor wrote: > On Thu, 29 Apr 2010, Mattia Vaccari wrote: > > >> I understand topcat doesn't provide built-in equal-area (Hammer-Aitoff and/or >> Mollweide etc) sky map projection capabilities. *If* that's right, what's the >> handiest (interoperable?) tool one can send catalogs to for such projections? >> > > That is right, sky coordinate systems are not well supported in > TOPCAT graphics. I hope to overhaul the graphics in the next few months > to do something about this. > > In the mean time, depending on what you want to do, I should think > Aladin is a good choice? Others may have different suggestions. > Aladin does indeed support various sky projections (AITOFF, TAN, SIN, etc) for both images and catalogues. Mattia : if you could give a bit more details about what you'd like to do, I could assess whether Aladin is or not the right tool for your needs. Thomas > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From mattia.vaccari@gmail.XXX Thu Apr 29 12:03:18 2010 Date: Thu, 29 Apr 2010 12:02:59 +0200 From: Mattia Vaccari To: Thomas Boch Cc: Mark Taylor , topcat-user@bristol.XXX Subject: Re: Sky Map Projections i mostly wanted to have an handy tool to display large-area catalogs and maps in a number of ccordinate systems, and indeed, Aladin seems to be doing it out of the box. cheers - mattia -- Mattia Vaccari ------ Post Doctoral Research Assistant Department of Astronomy --------- University of Padova Vicolo Osservatorio 3 --- I-35122 --- Padova --- Italy mattia@mattiavaccari.XXX ---- http://mattiavaccari.net +39 348 2541209 (Mobile) ---- +39 049 8278251 (Office) +39 049 8278212 (Fax) - +44 (0)77 60421086 (UK Mobile) From m.b.taylor@bristol.XXX Mon May 24 13:53:07 2010 Date: Mon, 24 May 2010 13:53:05 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Java versions Dear TOPCAT/STILTS/STIL users, in the near future, I am planning to change the build of TOPCAT, STILTS and STIL to Java 1.5. Up till now, it's been built using Java 1.4. The effect of this is that it will no longer be possible to use the applications with an installed java version of 1.4, and it will no longer be possible to build other software based on the libraries with a JDK of 1.4. A corollary of this is it will not be possible to build software which uses [new versions of] STIL/TS as library support to run on 1.4 JVMs. I don't expect this to cause any problems, since Java 1.4 is now very old (Sun considered J2SE 1.4.2 to have reached its End Of Service Life in October 2008). I believe that nearly all java installations in use now are 1.5 or 1.6 (sometimes known as 5.0 and 6.0). However, if anybody has concerns about this or would like me to reconsider the policy, please get in touch. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ipf@iac.XXX Wed Jun 9 16:36:27 2010 Date: Wed, 09 Jun 2010 15:35:15 +0100 From: Ismael Perez Fournon To: Mark Taylor Cc: topcat-user@bristol.XXX Ismael Perez Fournon Subject: STILTS public server? Hi Mark, I am learning to use the "server" option of stilts to execute STILTS commands on a server. I have tested it on my local machine and it works fine, but if I want to use an external server, is one available somewhere so that I do not need to start one? Something like: http://somewhere_stilts_public_server/stilts/task/plot2d?in=http://surveys.roe.ac.uk/wsa/tmp/tmp_sql/results9_5_54_46_6702.fits&xdata=ra&ydata=dec Thanks, Ismael --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci?n de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Tue Jun 15 12:26:09 2010 Date: Tue, 15 Jun 2010 12:26:08 +0100 (BST) From: Mark Taylor To: Ismael Perez Fournon Cc: topcat-user@bristol.XXX Subject: Re: STILTS public server? On Wed, 9 Jun 2010, Ismael Perez Fournon wrote: > Hi Mark, > > I am learning to use the "server" option of stilts to execute STILTS > commands on a server. > > I have tested it on my local machine and it works fine, but if I want to > use an external server, is one available somewhere so that I do not need > to start one? > > Something like: > > http://somewhere_stilts_public_server/stilts/task/plot2d?in=http://surveys.roe.ac.uk/wsa/tmp/tmp_sql/results9_5_54_46_6702.fits&xdata=ra&ydata=dec Ismael, sorry for the delay in replying, I was away. I don't run a public STILTS server, and I'm not aware of other people who run one for general purpose use. The main use case I envisaged for this functionality was so that data providers could run a server local to their data, so that external users could run STILTS commands in such a way that the results of the command, rather than the (potentially large) table data, got transferred over the network - some projects are doing this. Of course people are free to use the server functionality for anything they find useful, but I haven't been expecting a general purpose STILTS server to be set up. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From anthonysmith80@gmail.XXX Wed Jun 23 17:10:21 2010 Date: Wed, 23 Jun 2010 16:10:03 +0100 From: Anthony Smith To: topcat-user@bristol.XXX Subject: STILTS tpipe cmd: strings within strings Hi all, I'm trying to manipulate strings in a stilts tpipe command, and I'm having problems. Here's the simple version, more detail below stilts tpipe cmd="addcol Test '"Hello There "'" in=$1 out=$2 --> Not extant file or valid URL: There stilts tpipe cmd="addcol Test 'radiansToHms(ra*DEGREE).replace(":","")'" in=$1 out=$2 --> Bad expression "radiansToHms(ra*DEGREE).replace(:,): Encountered unexpected character ','. Any ideas? Thanks, Anthony -- astronomy.sussex.ac.uk/~anthonys www.anthonysmith.me.uk =========== MORE DETAIL Here's the idea: stilts calc '"0HERMES S250 "+radiansToHms((15.-0.05/15/3600)*DEGREE, 1).replace(":","")+radiansToDms((10.-0.5/3600)*DEGREE).replace(":","")' --> 0HERMES S250 010000.0+100000 Now try the same in a tpipe cmd: stilts tpipe cmd="addcol HermesName '"0HERMES S250 "+radiansToHms((15.-0.05/15/3600)*DEGREE,1).replace(":","") +radiansToDms((10.-0.5/3600)*DEGREE).replace(":","")'" in=$1 out=$2 --> Not extant file or valid URL: S250 Simplify to show another problem: stilts tpipe cmd="addcol HermesName 'radiansToHms((15.-0.05/15/3600)*DEGREE,1).replace(":","") +radiansToDms((10.-0.5/3600)*DEGREE).replace(":","")'" in=$1 out=$2 --> Bad expression "radiansToHms((15.-0.05/15/3600)*DEGREE,1).replace(:,) +radiansToDms((10.-0.5/3600)*DEGREE).replace(:,): Encountered unexpected character ','. From m.b.taylor@bristol.XXX Wed Jun 23 16:50:59 2010 Date: Wed, 23 Jun 2010 16:50:59 +0100 (BST) From: Mark Taylor To: Anthony Smith Cc: topcat-user@bristol.XXX Subject: Re: STILTS tpipe cmd: strings within strings On Wed, 23 Jun 2010, Anthony Smith wrote: > Hi all, > > I'm trying to manipulate strings in a stilts tpipe command, and I'm having > problems. Antony, I'm sorry to say that specifying STILTS commands containing literal strings (i.e. things containing quote characters) from the shell is pretty horrible. The problem is that there are three separate quoting syntaxes (java, shell and STILTS) which end up fighting each other. In most cases it is possible to achieve what you want to do by sufficent use/abuse of various escaping mechanisms; from the C-shell, I can make the examples you quote work as follows: > stilts tpipe cmd="addcol Test '"Hello There "'" in=$1 out=$2 cmd='addcol Test \"Hello\ there\"' > stilts tpipe cmd="addcol Test 'radiansToHms(ra*DEGREE).replace(":","")'" in=$1 > out=$2 cmd='addcol Test radiansToHms(ra*DEGREE).replace(\":\",\"\")' However, this is really not fun, and probably there are some things which are simply not possible in any given shell. STILTS users trying to do this have my apologies. If you're trying to do much of this stuff, the recommended approach is to use JyStilts (http://www.starlink.ac.uk/stilts/sun256/jystilts.html). Since python/jython has its own well-defined quoting mechanisms STILTS can co-opt these rather than having to reinvent the same thing badly, and it all works fine. The above example would go something like: import stilts t = stilts.tread(file1) t = t.cmd_addcol('Test', '"Hello There"') t.write(file2) Using JyStilts can be good advice for readability and in some cases efficiency in any case if you're doing long complicated manipulations. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From anthonysmith80@gmail.XXX Wed Jun 23 18:16:02 2010 Date: Wed, 23 Jun 2010 17:15:46 +0100 From: Anthony Smith To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: STILTS tpipe cmd: strings within strings Hi Mark, On 23 Jun 2010, at 16:50, Mark Taylor wrote: > If you're trying to do much of this stuff, the recommended approach is > to use JyStilts Thanks - JyStilts does the job fine (I tried using it before my first email, but was using backslashes in the wrong places). Cheers! Anthony -- astronomy.sussex.ac.uk/~anthonys www.anthonysmith.me.uk 07751 143941 From thomas.boch@astro.XXX Mon Jun 28 16:38:03 2010 Date: Mon, 28 Jun 2010 16:36:16 +0200 From: Thomas Boch To: topcat-user@mlist2.XXX Subject: Problem with datatype 'long' while reading IPAC table Hi, I have retrieved a table from the IPAC web site, and STILTS has troubles reading it because it does not recognize the 'long' datatype : > stilts tpipe ifmt=IPAC in=SAGELMCcatalogIRAC.tbl ofmt=csv out=SAGELMCcatalogIRAC.csv Can't open SAGELMCcatalogIRAC.tbl as IPAC (Unknown IPAC data type long) (Can't open SAGELMCcatalogIRAC.tbl as IPAC (Unknown IPAC data type long) (Unknown IPAC data type long)) I have attached a small sample of the file, including the header. Cheers, Thomas -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch [ Part 2: "Attached Text" ] | globalSourceID | sourceCatalog | epoch | designation | tmass_cntr | tmass_designation | ra | dec | dra | ddec | closeFlag | closeFullFlag | magJ | dmagJ | magH | dmagH | magK | dmagK | mag3_6 | dmag3_6 | mag4_5 | dmag4_5 | mag5_8 | dmag5_8 | mag8_0 | dmag8_0 | fluxJ | dfluxJ | fluxH | dfluxH | fluxK | dfluxK | flux3_6 | dflux3_6 | flux4_5 | dflux4_5 | flux5_8 | dflux5_8 | flux8_0 | dflux8_0 | rms_f3_6 | rms_f4_5 | rms_f5_8 | rms_f8_0 | sky3_6 | sky4_5 | sky5_8 | sky8_0 | SNJ | SNH | SNK | SN3_6 | SN4_5 | SN5_8 | SN8_0 | srcDensity3_6 | srcDensity4_5 | srcDensity5_8 | srcDensity8_0 | m3_6 | m4_5 | m5_8 | m8_0 | n3_6 | n4_5 | n5_8 | n8_0 | sqfJ | sqfH | sqfK | sqf3_6 | sqf4_5 | sqf5_8 | sqf8_0 | mf3_6 | mf4_5 | mf5_8 | mf8_0 | versionNo | versionDate | cx | cy | cz | htmID | | int | char | char | char | int | char | double | double | double | double | int | int | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | real | int | int | int | int | int | int | int | int | int | int | int | int | int | int | int | int | int | int | int | real | char | double | double | double | long | | | | | | | | deg | deg | arc seconds | arc seconds | | | mag | mag | mag | mag | mag | mag | mag | mag | mag | mag | mag | mag | mag | mag | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | mJy | MJy/sr | MJy/sr | MJy/sr | MJy/sr | | | | | | | | #/sq arcmin | #/sq arcmin | #/sq arcmin | #/sq arcmin | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 | | | | | | | | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | 99.999 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -999.9 | -9.99 | -9.99 | -9.99 | -9.99 | -9.99 | -9.99 | -9.99 | -9.9 | -9.9 | -9.9 | -9.9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | -9 | | | | | | | 1529671267 iracc SMP SSTISAGEMC J041826.63-693904.4 119920193 04182666-6939043 64.610980 -69.651227 0.3 0.3 0 0 11.361 0.025 11.062 0.024 10.930 0.023 10.905 0.044 99.999 99.999 99.999 99.999 99.999 99.999 45.510 1.048 38.500 0.851 28.310 0.600 12.200 0.495 -999.9 -999.9 -999.9 -999.9 -999.9 -999.9 0.000 -999.9 -999.9 -999.9 0.042 -999.9 1.069 -999.9 43.43 45.24 47.21 24.65 -9.99 -9.99 -9.99 26.4 -9.9 2.4 -9.9 2 0 1 0 2 0 1 0 29360128 29360128 29360128 16896 -9 1024 -9 195 -9 192 -9 3.10 Aug 17 2009 0.14909508830022126000 0.31414888521599066000 -0.93759326606066203000 9328366705324 1529671268 iracc SMP SSTISAGEMC J041833.52-693803.4 52622101 04183348-6938034 64.639679 -69.634299 0.3 0.3 0 0 16.979 0.081 16.456 0.127 15.894 0.112 15.764 0.075 99.999 99.999 99.999 99.999 99.999 99.999 0.258 0.019 0.268 0.031 0.293 0.030 0.139 0.010 -999.9 -999.9 -999.9 -999.9 -999.9 -999.9 0.000 -999.9 -999.9 -999.9 0.013 -999.9 0.803 -999.9 13.42 8.53 9.67 14.48 -9.99 -9.99 -9.99 0.0 -9.9 0.0 -9.9 1 0 1 0 1 0 1 0 29368320 25174016 25174016 16384 -9 8397824 -9 12352 -9 12288 -9 3.10 Aug 17 2009 0.14905635498392862000 0.31447382894625646000 -0.93749048739002454000 9328364396263 1529671269 iracc SMP SSTISAGEMC J041836.36-693946.6 52639174 04183633-6939458 64.651504 -69.662955 0.3 0.3 0 1 17.415 0.115 16.737 0.153 16.507 0.224 16.352 0.060 99.999 99.999 99.999 99.999 99.999 99.999 0.172 0.018 0.207 0.029 0.166 0.034 0.081 0.004 -999.9 -999.9 -999.9 -999.9 -999.9 -999.9 0.000 -999.9 -999.9 -999.9 0.006 -999.9 -999.9 -999.9 9.47 7.09 4.84 17.98 -9.99 -9.99 -9.99 0.0 -9.9 -9.9 -9.9 1 0 1 0 1 0 1 0 25174016 25174016 16785408 0 -9 -9 -9 12352 -9 -9 -9 3.10 Aug 17 2009 0.14879069304040900000 0.31408081134065700000 -0.93766442484086454000 9328366609175 1529671270 iracc SMP SSTISAGEMC J041836.79-693601.2 52639012 04183680-6936011 64.653332 -69.600355 0.3 0.3 0 0 15.655 0.032 15.274 0.053 15.317 0.079 15.157 0.052 99.999 99.999 99.999 99.999 99.999 99.999 0.872 0.026 0.796 0.039 0.498 0.036 0.243 0.012 -999.9 -999.9 -999.9 -999.9 -999.9 -999.9 0.000 -999.9 -999.9 -999.9 0.002 -999.9 0.831 -999.9 33.61 20.60 13.66 20.95 -9.99 -9.99 -9.99 0.0 -9.9 0.0 -9.9 1 0 1 0 1 0 1 0 29368320 29368320 29368320 8192 -9 8396800 -9 12352 -9 12288 -9 3.10 Aug 17 2009 0.14921915290989164000 0.31501121895418769000 -0.93728414919801724000 9328360110087 1529671271 iracc SMP SSTISAGEMC J041837.61-693512.1 52638975 04183761-6935123 64.656722 -69.586702 0.3 0.3 0 2 16.321 0.048 15.584 0.065 15.447 0.103 15.584 0.077 99.999 99.999 99.999 99.999 99.999 99.999 0.472 0.021 0.598 0.036 0.442 0.042 0.164 0.012 -999.9 -999.9 -999.9 -999.9 -999.9 -999.9 0.000 -999.9 -999.9 -999.9 0.008 -999.9 0.859 -999.9 22.39 16.81 10.50 14.14 -9.99 -9.99 -9.99 0.0 -9.9 0.0 -9.9 1 0 1 0 1 0 1 0 29368320 29368320 29368320 16384 -9 8397312 -9 12352 -9 12288 -9 3.10 Aug 17 2009 0.14929611109431100000 0.31522188865003473000 -0.93720106280777347000 9328360642001 1529671272 iracc SMP SSTISAGEMC J041838.90-693709.1 52639065 04183910-6937100 64.662098 -69.619213 0.3 0.3 0 0 17.614 0.130 16.916 0.189 16.705 0.265 16.776 0.120 99.999 99.999 99.999 99.999 99.999 99.999 0.144 0.017 0.175 0.031 0.139 0.034 0.055 0.006 -999.9 -999.9 -999.9 -999.9 -999.9 -999.9 0.000 -999.9 -999.9 -999.9 0.018 -999.9 -999.9 -999.9 8.38 5.74 4.10 9.03 -9.99 -9.99 -9.99 0.0 -9.9 -9.9 -9.9 1 0 0 0 1 0 1 0 25174016 20979712 16785408 512 -9 -9 -9 12352 -9 -9 -9 3.10 Aug 17 2009 0.14903892670554176000 0.31475521288106370000 -0.93739882349545133000 9328359937459 From m.b.taylor@bristol.XXX Mon Jun 28 16:48:07 2010 Date: Mon, 28 Jun 2010 16:48:06 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Problem with datatype 'long' while reading IPAC table On Mon, 28 Jun 2010, Thomas Boch wrote: > Hi, > > I have retrieved a table from the IPAC web site, and STILTS has troubles > reading it because it does not recognize the 'long' datatype : > > > stilts tpipe ifmt=IPAC in=SAGELMCcatalogIRAC.tbl ofmt=csv > out=SAGELMCcatalogIRAC.csv > Can't open SAGELMCcatalogIRAC.tbl as IPAC (Unknown IPAC data type long) > (Can't open SAGELMCcatalogIRAC.tbl as IPAC (Unknown IPAC data type long) > (Unknown IPAC data type long)) > > I have attached a small sample of the file, including the header. > Cheers, Thomas, thanks for this report. Type "long" is not mentioned in the IPAC table format documentation (http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html), so the fault is not with the client. I'll report it to IPAC for a clarification so that either the data can get fixed or the file format description can be amended. Having said that, last time I reported an error along these lines to the IRSA helpdesk it took three reports and almost a year before the issue was resolved, so since it's pretty obvious what the intention is, I won't wait for a fix at their end before I do something about it. I've made the necessary small adjustment; you can find pre-releases of TOPCAT/STILTS which will cope with this data at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Jul 12 15:01:25 2010 Date: Mon, 12 Jul 2010 15:01:25 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Crossmatching issues in crowded fields To any crossmatchers out there, I've had one or two queries recently from people who think that TOPCAT/STILTS might be making errors in its crossmatching. While this isn't inconceivable, I certainly hope it's not the case, and investigation of these reports have shown that it's misunderstanding of the match options or the output, rather than software errors, that has been at issue. A common case is "Best" matching in crowded fields where it may look like not all the matches are being found - what's actually happening is that Best Match mode intentionally excludes some matches, and these may not be the ones you're expecting. I've written an entry in the FAQ about this: see http://www.starlink.ac.uk/topcat/faq.html#xmatchIssues Please follow up here if anybody has comments or queries about this, or, of course, if you think that you may have found a crossmatching error. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Wed Jul 14 11:58:22 2010 Date: Wed, 14 Jul 2010 10:58:07 +0100 From: Seb Oliver To: topcat-user@bristol.XXX Subject: Re: activation / browser problem Hi Mark, That tip has helped narrow down the problem. It is clearly something specific to the URLs I am using. http://www.google.com works in both their test tool AND topcat running on my machine. However the URLs I am trying to use are like this... http://nedwww.ipac.caltech.edu/cgi-bin/nph-objsearch?in_csys=Equatorial&in_equinox=J2000.0&lon=%2070.765219d&lat=%20-54.780657d&radius=1.0&search_type=Near+Position+Search&out_csys=Equatorial&out_equinox=J2000.0&obj_sort=Distance+to+search+center&of=pre_text&zv_breaker=30000.0&list_limit=5&img_stamp=YES&z_constraint=Unconstrained&z_value1=&z_value2=&z_unit=z&ot_include=ANY&nmp_op=ANY Which work fine when cut and pasted into a browser (or in your basic browser) but not with system browser or their testing tool. I also had problems with the same URLs when working with an undergraduate using linux. Maybe you could check if you can get that URL to work with your installation of topcat (or their testing tool) and then we can see whether this is Mac / my local environment problem or a bug in their application. Seb. On 14 Jul 2010, at 09:59, Mark Taylor wrote: > On Tue, 13 Jul 2010, Seb Oliver wrote: > >> Hi Mark, >> >> I wonder if you can help. I've often used the activation action in >> topcat. >> Pointing to URLS which I put in tables. I've just tried it now and >> it seems >> to be broken. >> >> It works with the basic browser. But with the "system browser" it >> seems to >> just send the URL to the std out. firefox and netscape are not >> recognised as >> commands. >> >> I realise this is probably a system problem at my end, but wondered >> if you >> have any ideas. >> >> I down-loaded the latest version of TOPCAT and rebooted by machine >> but nothing >> helps. I've also tried with and without http_proxy environment >> variable set. >> >> I am using a mac os x 10.5.8 > > Hi Seb, > > the system browser launching is done using some third party code > (browserlaunch2.sourceforge.net), and I'm not sure exactly what it > gets up to. I don't have a Mac to play with, so I can't look > at the details easily. The browser launcher library has a diagnostic > tool (http://browserlaunch2.sourceforge.net/docs.shtml#testapp) > you could try running this (download > http://sourceforge.net/projects/browserlaunch2/files/browserlauncher2/1.3/BrowserLauncher2-all-1_3.jar/download > and do java -jar BrowserLauncher2-all-1_3.jar) and see if it says > anything revealing. > > An alternative is to use the "Execute custom code" activation action. > If I fill that in with > exec("firefox", URL) > (where "URL" is the name of the column that has a URL in it) > it works for me; I'm not sure how you start a browser from the command > line on OSX, but you can probably get something similar to work. > > Other possible hacks might be to put a script on your path which > is called firefox but which calls safari or something ... > > In Java 1.6 there is supposed to be a more reliable way to start a > system browser, so I'll put it on the to-do list, but I'm probably > not going to get to that right away. > > Mark > > PS the topcat-user mailing list > (http://www.star.bris.ac.uk/~mbt/topcat-lists.html) > is now the preferred channel for queries, requests, bug reports > etc, though mails to me are OK if you prefer it for some reason. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, > UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/ > ~mbt/ From m.b.taylor@bristol.XXX Wed Jul 14 11:16:51 2010 Date: Wed, 14 Jul 2010 11:16:51 +0100 (BST) From: Mark Taylor To: Seb Oliver Cc: topcat-user@bristol.XXX Subject: Re: activation / browser problem On Wed, 14 Jul 2010, Seb Oliver wrote: > Hi Mark, > > That tip has helped narrow down the problem. It is clearly something specific > to the URLs I am using. http://www.google.com works in both their test tool > AND topcat running on my machine. However the URLs I am trying to use are > like this... > > http://nedwww.ipac.caltech.edu/cgi-bin/nph-objsearch?in_csys=Equatorial&in_equinox=J2000.0&lon=%2070.765219d&lat=%20-54.780657d&radius=1.0&search_type=Near+Position+Search&out_csys=Equatorial&out_equinox=J2000.0&obj_sort=Distance+to+search+center&of=pre_text&zv_breaker=30000.0&list_limit=5&img_stamp=YES&z_constraint=Unconstrained&z_value1=&z_value2=&z_unit=z&ot_include=ANY&nmp_op=ANY > > Which work fine when cut and pasted into a browser (or in your basic browser) > but not with system browser or their testing tool. > > I also had problems with the same URLs when working with an undergraduate > using linux. > > Maybe you could check if you can get that URL to work with your installation > of topcat (or their testing tool) and then we can see whether this is Mac / my > local environment problem or a bug in their application. Yes, that URL seems to work in the system browser from TOPCAT and from the test tool for me. Are there any possibly enlightening log messages in the test tool when it fails for you? -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Wed Jul 14 13:17:42 2010 Date: Wed, 14 Jul 2010 12:17:37 +0100 From: Seb Oliver To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: activation / browser problem More progress. Anthony Smith spotted the %20 in the URL I circulated which actually comes from a space in the fits file column. If we remove the spaces it works. So I guess we could put this down to a badly formed URL. Seb. On 14 Jul 2010, at 11:16, Mark Taylor wrote: > On Wed, 14 Jul 2010, Seb Oliver wrote: > >> Hi Mark, >> >> That tip has helped narrow down the problem. It is clearly >> something specific >> to the URLs I am using. http://www.google.com works in both their >> test tool >> AND topcat running on my machine. However the URLs I am trying to >> use are >> like this... >> >> http://nedwww.ipac.caltech.edu/cgi-bin/nph-objsearch?in_csys=Equatorial&in_equinox=J2000.0&lon=%2070.765219d&lat=%20-54.780657d&radius=1.0&search_type=Near+Position+Search&out_csys=Equatorial&out_equinox=J2000.0&obj_sort=Distance+to+search+center&of=pre_text&zv_breaker=30000.0&list_limit=5&img_stamp=YES&z_constraint=Unconstrained&z_value1=&z_value2=&z_unit=z&ot_include=ANY&nmp_op=ANY >> >> Which work fine when cut and pasted into a browser (or in your >> basic browser) >> but not with system browser or their testing tool. >> >> I also had problems with the same URLs when working with an >> undergraduate >> using linux. >> >> Maybe you could check if you can get that URL to work with your >> installation >> of topcat (or their testing tool) and then we can see whether this >> is Mac / my >> local environment problem or a bug in their application. > > Yes, that URL seems to work in the system browser from TOPCAT and from > the test tool for me. Are there any possibly enlightening log > messages > in the test tool when it fails for you? > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, > UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/ > ~mbt/ From m.b.taylor@bristol.XXX Wed Jul 14 13:29:15 2010 Date: Wed, 14 Jul 2010 13:29:15 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: activation / browser problem On Wed, 14 Jul 2010, Seb Oliver wrote: > More progress. Anthony Smith spotted the %20 in the URL I circulated which > actually comes from a space in the fits file column. If we remove the spaces > it works. So I guess we could put this down to a badly formed URL. By my reading of RFC1738 it should be OK to encode a space as %20, though sec 17.13.4 of the HTML 4.01 spec may suggest otherwise ... So, I wouldn't like to say if it's a badly formed URL or a bug somewhere, but if removing the "%20" (or replacing it with "+", which is probably how spaces should be encoded here) does the trick, then good. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From anthonysmith80@gmail.XXX Wed Jul 14 15:40:38 2010 Date: Wed, 14 Jul 2010 14:40:27 +0100 From: Anthony Smith To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: activation / browser problem Hi Mark, It was actually a genuine space in the URL column, not a "%20". (The "%20" appeared when the URL was manually pasted into the browser.) The TOPCAT activation action works fine if the " " are replaced by "%20". I suppose TOPCAT could do its own " " --> "%20" conversion when opening URLs, but I'm not too worried... Cheers, Anthony On 14 Jul 2010, at 13:29, Mark Taylor wrote: > On Wed, 14 Jul 2010, Seb Oliver wrote: > >> More progress. Anthony Smith spotted the %20 in the URL I >> circulated which >> actually comes from a space in the fits file column. If we remove >> the spaces >> it works. So I guess we could put this down to a badly formed URL. > > By my reading of RFC1738 it should be OK to encode a space as %20, > though sec 17.13.4 of the HTML 4.01 spec may suggest otherwise ... > So, I wouldn't like to say if it's a badly formed URL or a bug > somewhere, but if removing the "%20" (or replacing it with "+", > which is probably how spaces should be encoded here) does the > trick, then good. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, > UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/ > ~mbt/ > From m.b.taylor@bristol.XXX Mon Aug 9 11:33:12 2010 Date: Mon, 9 Aug 2010 11:33:12 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Table IO/ Session Saving Gus, Bianca, Jon, Ivan and others who have requested session save, I'm happy to report that this feature is now present in TOPCAT v3.6 (just released). The save window now gives you three options: - Current Table - as before, saves the current 'apparent table' (no subsets, only currently visible rows and columns included) - Multiple Tables - saves many apparent tables into a single container file (generally multi-extension FITS or multi-TABLE VOTable) - Session - saves (nearly) all the state of each loaded table, into a single container file. This preserves row subsets, current subset and sort order, label, hidden columns and some other things. A saved session file can be loaded in just like a normal multi-table file. Note however that a couple of things the session save does not currently do are saving non-table information like plots and save activation actions. Also, subsets are effectively serialized as boolean columns (though this is hidden from the user), so that subsets defined by expressions, when saved and restored, are no longer defined by expressions. It's possible some of these things may be improved in a future release. Thanks for your input that's encouraged me to introduce this feature. Any further comments very welcome on this list. Mark On Wed, 7 Apr 2010, august.fly@gmail.XXX wrote: > hi, > > > it may be entirely a product of my particular use case(s) but I very frequently > find myself wishing I could simply save an entire TOPCAT session, all the > tables, at once into a new "table" package that I could restore at a later > time. > > > the instances that most often elicit this desire include backing up my work, or > wishing I had because the power at my work went out wiping out my TOPCAT > session, creating a package of tables from a project to share, and portability > between desktop/laptop/home/work. > > > such a feature has been prominent in our CfA python users group and > astropython.org discussions, which is just to say that prob any work "environ" > could lead to such a feature request. > > > I suppose metadata/session packaging might be an issue. I personally would > think that a multi-extension FITS would upset the fewest number of users. Of > course xml+fits ref might alleviate metadata translation issues. I don't know. > > > anyway, just 1 user's comment/suggestion. i hope i haven't overlooked some > basic feature yet also wonder if there are other's who have similar > experiences. > > > best, > > > - gus muench > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Aug 9 11:37:08 2010 Date: Mon, 9 Aug 2010 11:37:08 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Join many tables in a big one On Wed, 31 Mar 2010, Mark Taylor wrote: > On Wed, 31 Mar 2010, Jorge wrote: > > > Hi Mark. > > > > I've got about 50 separate tables and I need to join all of them in a big one. > > I thought two different solutions to do that but, Idon´t know how to make > > them. > > > > 1: Save the 50 separate tables in an only file which, when I open it, Topcat > > opens the 50 separate tables at once. > > > > 2: In the case of concatenate the tables, I need to keep all the information > > possible. This is so difficult whith the topcat task. Is possible to join the > > tables in a big one with out losing columns of information? > > Jorge, > > it is possible to prepare a single file which contains many different > tables, which will all be loaded into TOPCAT when the file is loaded. > There are two possibilities: > > 1. a VOTable with one TABLE element for each table you want > 2. a FITS file with one BINTABLE (or TABLE) HDU for each table you want > > However, you can't currently prepare either of these formats using > TOPCAT or STILTS :-(. > I am considering adding that possibility in a future release. Now it is! See the new commands tmulti and tmultin in STILTS v2.2. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Wed Aug 11 14:02:04 2010 Date: Wed, 11 Aug 2010 13:01:56 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Changing internal constant value in TOPCAT (SDSS_BASE_URL) Hi, I'm using the SDSS cutouts activation action in TOPCAT however I've realized that the program it is using the DR4 instead of the DR7. I've seen that this is connected with the constant SDSS_BASE_URL that in my version it is set to "http://casjobs.sdss.org/ImgCutoutDR4/getjpeg.aspx". Is it possible to change the value of this constant without having to update the program (I have no privileges to do it in my computer at work). BTW, congratulation to the developers for the program. I've found it really useful! Thanks! Jesús Varela From m.b.taylor@bristol.XXX Wed Aug 11 13:19:26 2010 Date: Wed, 11 Aug 2010 13:19:26 +0100 (BST) From: Mark Taylor To: jmvarela@iac.XXX Cc: topcat-user@sympa.XXX Subject: Re: Changing internal constant value in TOPCAT (SDSS_BASE_URL) Jesús, On Wed, 11 Aug 2010, jmvarela@iac.XXX wrote: > I'm using the SDSS cutouts activation action in TOPCAT however I've realized > that the program it is using the DR4 instead of the DR7. I've seen that this is > connected with the constant SDSS_BASE_URL that in my version it is set to > "http://casjobs.sdss.org/ImgCutoutDR4/getjpeg.aspx". > > Is it possible to change the value of this constant without having to update > the program (I have no privileges to do it in my computer at work). It's not possible to change it as such, but you can do what the sdssCutout function is doing by hand, and substitute in a URL of your choice. If you use the "Execute Custom Code" option for Activation Action, you can use an expression like displayImage("SDSS", "http://casjobs.sdss.org/ImgCutoutDR7/getjpeg.aspx?ra="+ra+"&dec="+dec) assuming that your RA,Dec columns are named "ra" and "dec" respectively. Other parameters (specified like &name=value in the URL) for that SDSS service include "height", "width" and "scale", so you can set these explicitly if you want, or just get whatever the defaults are (I forget where this service is documented, but probably you can find it on the SDSS web site). However, upgrading to a more recent version of TOPCAT is probably a good idea (to get other new/improved features as well as the DR4->DR7 update). I'm not sure exactly how much privilege you don't have on the machine in question, but it's very easy to install a copy in your own file space - just download the topcat-full.jar file from the web page (and optionally the topcat script too, if you're using *nix) and that's it. > BTW, congratulation to the developers for the program. I've found it really > useful! Good! thanks for the feedback. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Wed Aug 11 15:06:57 2010 Date: Wed, 11 Aug 2010 14:06:50 +0100 From: Jesús Varela To: topcat-user@sympa.XXX Subject: TOPCAT 3.6 doesn't get Java flags in command line Hi Mark, I'm just installed the last version of TOPCAT (3.6) and I'm running it with JDK 1.6.0_21 in a X84_64 Linux system with the command: /usr/pkg/jdk/x86_64/jdk1.6.0_21/bin/java -jar ~/Software/TOPCAT/topcat And it works :-) However I want to set the temporary directory as I did with the previous version of TOPCAT adding the "-Djava.io.tmpdir=/scratch/tmp/" flag as follows: /usr/pkg/jdk/x86_64/jdk1.6.0_21/bin/java -jar ~/Software/TOPCAT/topcat -Djava.io.tmpdir=/scratch/tmp/ However it returns me the usage message: Usage: topcat [-help] [-version] [-stilts ] [-jsamp ] [-verbose] [-demo] [-memory] [-disk] [-hub] [-exthub] [-samp] [-plastic] [-noserv] [[-f ] table ...] But if I do: /usr/pkg/jdk/x86_64/jdk1.6.0_21/bin/java -jar ~/Software/TOPCAT/topcat -help It returns: Usage: topcat [[-f ] ...] General flags: -help print this message and exit -version print component versions etc and exit -verbose increase verbosity of reports to console -demo start with demo data -memory use memory storage for tables -disk use disk backing store for large tables -samp use SAMP for tool interoperability -plastic use PLASTIC for tool interoperability -hub run internal SAMP/PLASTIC hub -exthub run external SAMP/PLASTIC hub -noserv don't run any services (PLASTIC or SAMP) -stilts run STILTS not TOPCAT -jsamp run JSAMP not TOPCAT Useful Java flags: -classpath jar1:jar2.. specify additional classes -XmxnnnM use nnn megabytes of memory -Dname=value set system property Auto-detected formats: fits-plus, colfits-plus, colfits-basic, fits, votable All known formats: fits-plus, colfits-plus, colfits-basic, fits, votable, ascii, csv, tst, ipac, wdc Useful system properties (-Dname=value - lists are colon-separated): java.io.tmpdir temporary filespace directory jdbc.drivers JDBC driver classes jel.classes custom algebraic function classes jel.classes.activation custom action function classes star.connectors custom remote filestore classes startable.load.dialogs custom load dialogue classes startable.readers custom table input handlers startable.writers custom table output handlers startable.storage storage policy mark.workaround work around mark/reset bug myspace.cache MySpace performance workaround (see topcat -jsamp -help for more) Which it is the usual "help" message that I got in a previous version. I've tried using also the flag "-XmxnnnM" and I have the same problem. I have no idea of the reason of this behaviour. Any clue? Thanks! Jesús Varela From m.b.taylor@bristol.XXX Wed Aug 11 14:21:49 2010 Date: Wed, 11 Aug 2010 14:21:49 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: TOPCAT 3.6 doesn't get Java flags in command line On Wed, 11 Aug 2010, Jesús Varela wrote: > Hi Mark, > > I'm just installed the last version of TOPCAT (3.6) and I'm running it with > JDK 1.6.0_21 in a X84_64 Linux system with the command: > > /usr/pkg/jdk/x86_64/jdk1.6.0_21/bin/java -jar ~/Software/TOPCAT/topcat > > And it works :-) > > However I want to set the temporary directory as I did with the previous > version of TOPCAT adding the "-Djava.io.tmpdir=/scratch/tmp/" flag as follows: > > /usr/pkg/jdk/x86_64/jdk1.6.0_21/bin/java -jar ~/Software/TOPCAT/topcat > -Djava.io.tmpdir=/scratch/tmp/ Flag order is a bit tricky - the "-D" and "-X" flags have to be supplied to the java runtime itself, and not to the TOPCAT application. The upshot of this is that if you're running with "java -jar" you need to put the -D (and -X) flags before the -jar flag: java -Djava.io.tmpdir=/scratch/tmp/ -jar topcat-full.jar The topcat shell script (http://www.star.bristol.ac.uk/~mbt/topcat/topcat) shuffles some of the flags around to try to make life easier, so if you install that (which just consists of putting it in the same directory as the topcat-full.jar or topcat-lite.jar file) you can put the flags in almost any order and it will work OK. > I've tried using also the flag "-XmxnnnM" and I have the same problem. Same solution. Note however that memory allocation is a bit smarter in recent versions, so you may find you don't need -XmxnnnM any more (though you might not). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Wed Aug 11 16:30:19 2010 Date: Wed, 11 Aug 2010 15:29:57 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: TOPCAT 3.6 doesn't get Java flags in command line >> >> I'm just installed the last version of TOPCAT (3.6) and I'm >> running it with >> JDK 1.6.0_21 in a X84_64 Linux system with the command: >> >> /usr/pkg/jdk/x86_64/jdk1.6.0_21/bin/java -jar ~/Software/TOPCAT/topcat >> >> And it works :-) >> >> However I want to set the temporary directory as I did with the previous >> version of TOPCAT adding the "-Djava.io.tmpdir=/scratch/tmp/" flag >> as follows: >> >> /usr/pkg/jdk/x86_64/jdk1.6.0_21/bin/java -jar ~/Software/TOPCAT/topcat >> -Djava.io.tmpdir=/scratch/tmp/ > > Flag order is a bit tricky - the "-D" and "-X" flags have to be supplied > to the java runtime itself, and not to the TOPCAT application. > The upshot of this is that if you're running with "java -jar" you > need to put the -D (and -X) flags before the -jar flag: > > java -Djava.io.tmpdir=/scratch/tmp/ -jar topcat-full.jar > > The topcat shell script (http://www.star.bristol.ac.uk/~mbt/topcat/topcat) > shuffles some of the flags around to try to make life easier, so if > you install that (which just consists of putting it in the same directory > as the topcat-full.jar or topcat-lite.jar file) you can put the flags > in almost any order and it will work OK. > >> I've tried using also the flag "-XmxnnnM" and I have the same problem. > > Same solution. Note however that memory allocation is a bit smarter > in recent versions, so you may find you don't need -XmxnnnM any more > (though you might not). > > Mark > Thanks again Mark for the quick solution! :) I should read more carefully the installation instructions and I would have seen the script that you said. Then, I had to mess up with the "java" binary because the default installation is "gij" but I've found a Sun Version somewhere in the system and after tweaking a bit in the script now it works perfectly. Thanks! Jesús -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From jmvarela@iac.XXX Thu Aug 12 14:03:36 2010 Date: Thu, 12 Aug 2010 13:03:32 +0100 From: Jesús Varela To: topcat-user@sympa.XXX Subject: TOPCAT doesn't found installed SoG viewer Hi Mark, still trying to improve my TOPCAT experience :) I was trying to display the images using the "SoG" viewer but TOPCAT insists in using the internal viewer when using the "Display Cutout Image" Activation Action. I guess the problem can be connected with the way "sog" works in our system. The command is not available by default and one has to load first the "starlink" environment (just using "starlink" command). Then the command "sog" is active (in fact, it is an alias to a command called "runSOG") and the viewer can be launched. However, TOPCAT continues to use always the internal viewer. I've made several tests including opening a new terminal, launch starlink, launch sog and then TOPCAT. No way, it looks like TOPCAT doesn't find the SoG executable. Any idea? Thanks in advance! Jesús From m.b.taylor@bristol.XXX Thu Aug 12 13:38:33 2010 Date: Thu, 12 Aug 2010 13:38:33 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: TOPCAT doesn't found installed SoG viewer Jesús, On Thu, 12 Aug 2010, Jesús Varela wrote: > I was trying to display the images using the "SoG" viewer but TOPCAT insists > in using the internal viewer when using the "Display Cutout Image" Activation > Action. > > I guess the problem can be connected with the way "sog" works in our system. > The command is not available by default and one has to load first the > "starlink" environment (just using "starlink" command). Then the command "sog" > is active (in fact, it is an alias to a command called "runSOG") and the viewer > can be launched. However, TOPCAT continues to use always the internal viewer. > > I've made several tests including opening a new terminal, launch starlink, > launch sog and then TOPCAT. No way, it looks like TOPCAT doesn't find the SoG > executable. The issue here is (most likely) that SoG needs a version of java with JAI (the Java Advanced Imaging extension) installed. If you run it using your normal java (probably /usr/bin/java or something), it's likely that JAI is absent. The java which is distributed with the Starlink system does have JAI installed. So, to get this working, you can either install JAI in your system default java (see http://java.sun.com/products/java-media/jai/downloads/) or, probably easier, use the starlink version of java to run TOPCAT. Something like /star/java/bin/java -jar topcat-full.jar should do the trick. You don't need to initialise the starlink environment for this to work. Note: supplying the "-version" flag, or looking at the Help|About TOPCAT menu item, will tell you whether SoG is available or absent. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Thu Aug 12 16:16:10 2010 Date: Thu, 12 Aug 2010 15:15:53 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: TOPCAT doesn't found installed SoG viewer Hi Mark, > Jesús, > > On Thu, 12 Aug 2010, Jesús Varela wrote: > >> I was trying to display the images using the "SoG" viewer but >> TOPCAT insists >> in using the internal viewer when using the "Display Cutout Image" >> Activation >> Action. >> >> I guess the problem can be connected with the way "sog" works in our system. >> The command is not available by default and one has to load first the >> "starlink" environment (just using "starlink" command). Then the >> command "sog" >> is active (in fact, it is an alias to a command called "runSOG") >> and the viewer >> can be launched. However, TOPCAT continues to use always the >> internal viewer. >> >> I've made several tests including opening a new terminal, launch starlink, >> launch sog and then TOPCAT. No way, it looks like TOPCAT doesn't >> find the SoG >> executable. > > The issue here is (most likely) that SoG needs a version of java with JAI > (the Java Advanced Imaging extension) installed. If you run it using your > normal java (probably /usr/bin/java or something), it's likely that JAI is > absent. The java which is distributed with the Starlink system does have > JAI installed. > > So, to get this working, you can either install JAI in your system default > java (see http://java.sun.com/products/java-media/jai/downloads/) or, > probably easier, use the starlink version of java to run TOPCAT. > Something like > > /star/java/bin/java -jar topcat-full.jar > > should do the trick. You don't need to initialise the starlink > environment for this to work. Note: supplying the "-version" flag, > or looking at the Help|About TOPCAT menu item, will tell you > whether SoG is available or absent. > We are getting closer... I found a "starjava" command in the "starlink" directory and doing: /usr/pkg/starlink/Hawaiki/starjava/bin/starjava -Djava.io.tmpdir="/scratch/tmp/" -Xmx3000M -jar ~/Software/TOPCAT/topcat-full.jar TOPCAT is launched and SoG is available. Either it is said in the "About TOPCAT" and when selecting a galaxy SoG is automatically launched. However,... it produces the following error when selecting a row to Display Cutout Image: sog(http://casjobs.sdss.org/ImgCutoutDR7/getjpeg.aspx?ra=220.85998743001852&dec=40.135940900487995&height=100&width=100) java.lang.RuntimeException: java.net.SocketException: Connection reset at jsky.util.FileUtil.makeURLStream(FileUtil.java:69) at jsky.util.gui.ProgressBarFilterInputStream.(ProgressBarFilterInputStream.java:86) at jsky.util.gui.StatusPanel.getLoggedInputStream(StatusPanel.java:189) at jsky.util.gui.ProgressPanel.getLoggedInputStream(ProgressPanel.java:378) at jsky.image.gui.DivaMainImageDisplay$14.construct(DivaMainImageDisplay.java:920) at jsky.util.SwingWorker$2.run(SwingWorker.java:131) at java.lang.Thread.run(Thread.java:595) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:313) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:681) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:626) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:983) at java.net.URL.openStream(URL.java:1007) at jsky.util.FileUtil.makeURLStream(FileUtil.java:63) ... 6 more ...and no image is loaded. Any clue? Thanks for the support again! Jesús -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Thu Aug 12 15:26:41 2010 Date: Thu, 12 Aug 2010 15:26:41 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: TOPCAT doesn't found installed SoG viewer On Thu, 12 Aug 2010, jmvarela@iac.XXX wrote: > TOPCAT is launched and SoG is available. Either it is said in the "About > TOPCAT" and when selecting a galaxy SoG is automatically launched. However,... > it produces the following error when selecting a row to Display Cutout Image: > > sog(http://casjobs.sdss.org/ImgCutoutDR7/getjpeg.aspx?ra=220.85998743001852&dec=40.135940900487995&height=100&width=100) > java.lang.RuntimeException: java.net.SocketException: Connection reset As far as I can see, this is nothing to do with TOPCAT; that SDSS cutout service is just broken at the moment. I get the same result, but the other image cutout services are OK. I think you'll see the same error for the version you had working before (SDSS DR4 cutout server seems to be suffering the same thing). You'll just have to wait until SDSS fixes itself I think. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Thu Aug 12 16:36:16 2010 Date: Thu, 12 Aug 2010 15:36:04 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: TOPCAT doesn't found installed SoG viewer UPDATE of previous message... the problem with the Cutout Images only happens with SDSS. There is no problem to view images from SuperCosmos and 2MASS. I don't know if it is connected with the problem but when using SuperCosmos or 2MASS after selecting a row the message that appears in the terminal is on the kind: http://irsa.ipac.caltech.edu/cgi-bin/Oasis/2MASSImg /nph-2massimg?objstr=14%3a36%3a22.54350%20%2b39%3a30%3a47.4499&size=100&band=k http://www-wfau.roe.ac.uk/~sss/cgi-bin/sss_topcat_pix.cgi?ra=14%3a40%3a39.52216&dec=%2b37%3a39%3a42.0104&mime-type=image%2fx-gfits&x=1.11&y=1.11&waveband=blue While for SDSS is slightly different: sog(http://casjobs.sdss.org/ImgCutoutDR7/getjpeg.aspx?ra=220.85998743001852&dec=40.135940900487995&height=100&width=100) Just in case it can be useful. Jesús > Hi Mark, > >> Jesús, >> >> On Thu, 12 Aug 2010, Jesús Varela wrote: >> >>> I was trying to display the images using the "SoG" viewer but >>> TOPCAT insists >>> in using the internal viewer when using the "Display Cutout Image" >>> Activation >>> Action. >>> >>> I guess the problem can be connected with the way "sog" works in >>> our system. >>> The command is not available by default and one has to load first the >>> "starlink" environment (just using "starlink" command). Then the >>> command "sog" >>> is active (in fact, it is an alias to a command called "runSOG") >>> and the viewer >>> can be launched. However, TOPCAT continues to use always the >>> internal viewer. >>> >>> I've made several tests including opening a new terminal, launch starlink, >>> launch sog and then TOPCAT. No way, it looks like TOPCAT doesn't >>> find the SoG >>> executable. >> >> The issue here is (most likely) that SoG needs a version of java with JAI >> (the Java Advanced Imaging extension) installed. If you run it using your >> normal java (probably /usr/bin/java or something), it's likely that JAI is >> absent. The java which is distributed with the Starlink system does have >> JAI installed. >> >> So, to get this working, you can either install JAI in your system default >> java (see http://java.sun.com/products/java-media/jai/downloads/) or, >> probably easier, use the starlink version of java to run TOPCAT. >> Something like >> >> /star/java/bin/java -jar topcat-full.jar >> >> should do the trick. You don't need to initialise the starlink >> environment for this to work. Note: supplying the "-version" flag, >> or looking at the Help|About TOPCAT menu item, will tell you >> whether SoG is available or absent. >> > > We are getting closer... I found a "starjava" command in the "starlink" > directory and doing: > > /usr/pkg/starlink/Hawaiki/starjava/bin/starjava > -Djava.io.tmpdir="/scratch/tmp/" -Xmx3000M -jar > ~/Software/TOPCAT/topcat-full.jar > > TOPCAT is launched and SoG is available. Either it is said in the > "About TOPCAT" and when selecting a galaxy SoG is automatically > launched. However,... it produces the following error when selecting a > row to Display Cutout Image: > > sog(http://casjobs.sdss.org/ImgCutoutDR7/getjpeg.aspx?ra=220.85998743001852&dec=40.135940900487995&height=100&width=100) > java.lang.RuntimeException: java.net.SocketException: Connection reset > at jsky.util.FileUtil.makeURLStream(FileUtil.java:69) > at > jsky.util.gui.ProgressBarFilterInputStream.(ProgressBarFilterInputStream.java:86) > at > jsky.util.gui.StatusPanel.getLoggedInputStream(StatusPanel.java:189) > at > jsky.util.gui.ProgressPanel.getLoggedInputStream(ProgressPanel.java:378) > at > jsky.image.gui.DivaMainImageDisplay$14.construct(DivaMainImageDisplay.java:920) > at jsky.util.SwingWorker$2.run(SwingWorker.java:131) > at java.lang.Thread.run(Thread.java:595) > Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:168) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) > at java.io.BufferedInputStream.read(BufferedInputStream.java:313) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:681) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:626) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:983) > at java.net.URL.openStream(URL.java:1007) > at jsky.util.FileUtil.makeURLStream(FileUtil.java:63) > ... 6 more > > > ...and no image is loaded. > > Any clue? > --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From jmvarela@iac.XXX Thu Aug 12 16:55:22 2010 Date: Thu, 12 Aug 2010 15:55:04 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: TOPCAT doesn't found installed SoG viewer Oh! We cross the messages :) Quoting Mark Taylor : > On Thu, 12 Aug 2010, jmvarela@iac.XXX wrote: > >> TOPCAT is launched and SoG is available. Either it is said in the "About >> TOPCAT" and when selecting a galaxy SoG is automatically launched. >> However,... >> it produces the following error when selecting a row to Display >> Cutout Image: >> >> sog(http://casjobs.sdss.org/ImgCutoutDR7/getjpeg.aspx?ra=220.85998743001852&dec=40.135940900487995&height=100&width=100) >> java.lang.RuntimeException: java.net.SocketException: Connection reset > > As far as I can see, this is nothing to do with TOPCAT; that SDSS > cutout service is just broken at the moment. > I get the same result, but the other image cutout services are OK. > I think you'll see the same error for the version you had working before > (SDSS DR4 cutout server seems to be suffering the same thing). > You'll just have to wait until SDSS fixes itself I think. > Oh! You are right. The SDSS server has stopped working right today... what a messy coincidence... sorry for the annoyance. Cheers! Jesús --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From jmvarela@iac.XXX Fri Aug 13 14:43:34 2010 Date: Fri, 13 Aug 2010 13:43:26 +0100 From: Jesús Varela To: topcat-user@sympa.XXX Subject: Controlling TOPCAT from terminal command line Hi Mark, I was wondering if there is any way to control TOPCAT from the command line. I would find very useful at least to have the possibility of loading a table in an open TOPCAT from the command line, but if other TOPCAT commands would be accessible from the command line it would be great. Maybe there is some possibility using the interoperability capabilities of TOPCAT through SAMP. Thanks! Jesús From m.b.taylor@bristol.XXX Fri Aug 13 16:09:15 2010 Date: Fri, 13 Aug 2010 16:09:14 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Controlling TOPCAT from terminal command line On Fri, 13 Aug 2010, Jesús Varela wrote: > Hi Mark, > > I was wondering if there is any way to control TOPCAT from the command line. > I would find very useful at least to have the possibility of loading a table in > an open TOPCAT from the command line, but if other TOPCAT commands would be > accessible from the command line it would be great. Maybe there is some > possibility using the interoperability capabilities of TOPCAT through SAMP. Jesús, you can send a table to a running TOPCAT using STILTS. If you've got a table t.fits, you can just do stilts tpipe in=t.fits omode=topcat For non-auto-detected formats (not FITS or VOTable) you need to give the table format as well: stilts tpipe in=t.csv ifmt=csv omode=topcat It's best to have a SAMP hub running when doing this, but it's not essential - if no hub is running, STILTS will start up a short-lived hub to manage the communications. In case you don't know, STILTS is TOPCAT's command-line counterpart (http://www.starlink.ac.uk/stilts/); you can copy the http://www.starlink.ac.uk/stilts/stilts script into the same directory as the topcat jar file, or you can use "topcat -stilts" instead of "stilts" in the invocations above. omode=samp is another alternative, which lets you load tables into other SAMP/table-aware applications as well. For further information, see: http://www.starlink.ac.uk/stilts/sun256/mode-topcat.html http://www.starlink.ac.uk/stilts/sun256/mode-samp.html You can do some other control of TOPCAT using other SAMP MTypes that it supports, but at present there's no way of performing more detailed control (such as plotting). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Fri Aug 13 17:17:07 2010 Date: Fri, 13 Aug 2010 16:16:52 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Controlling TOPCAT from terminal command line Hi Mark, Quoting Mark Taylor : > On Fri, 13 Aug 2010, Jesús Varela wrote: > >> Hi Mark, >> >> I was wondering if there is any way to control TOPCAT from the >> command line. >> I would find very useful at least to have the possibility of >> loading a table in >> an open TOPCAT from the command line, but if other TOPCAT commands would be >> accessible from the command line it would be great. Maybe there is some >> possibility using the interoperability capabilities of TOPCAT through SAMP. > > Jesús, > > you can send a table to a running TOPCAT using STILTS. If you've got > a table t.fits, you can just do > > stilts tpipe in=t.fits omode=topcat > > For non-auto-detected formats (not FITS or VOTable) you need to give the > table format as well: > > stilts tpipe in=t.csv ifmt=csv omode=topcat > > It's best to have a SAMP hub running when doing this, but it's not > essential - if no hub is running, STILTS will start up a short-lived > hub to manage the communications. In case you don't know, STILTS is > TOPCAT's command-line counterpart (http://www.starlink.ac.uk/stilts/); > you can copy the http://www.starlink.ac.uk/stilts/stilts script into > the same directory as the topcat jar file, or you can use > "topcat -stilts" instead of "stilts" in the invocations above. > omode=samp is another alternative, which lets you load tables into > other SAMP/table-aware applications as well. For further information, > see: > > http://www.starlink.ac.uk/stilts/sun256/mode-topcat.html > http://www.starlink.ac.uk/stilts/sun256/mode-samp.html > > You can do some other control of TOPCAT using other SAMP MTypes that > it supports, but at present there's no way of performing more detailed > control (such as plotting). > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ Thanks Mark! That's the kind of thing that I was looking for. I've tried and it works perfectly. BTW I also took a closer look to STILTS and found many useful commands that for sure I will use from now on. Thanks for all! Jesús -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From bianca@lambrate.XXX Mon Aug 23 08:52:43 2010 Date: Fri, 20 Aug 2010 08:41:41 +0200 From: Bianca Garilli To: topcat-user@mlist2.XXX Subject: Re: Table IO/ Session Saving Hi Mark, yes I saw the new version announcement. It's great and I'll cenrtainly install it as soon as I'm back from holydays and have the new computer up and running (which usually takes some time, for whatever reason :-( ) Thanks again for the work on topcat, a really great tool Bianca On Monday 09 August 2010 12:33, Mark Taylor wrote: > Gus, Bianca, Jon, Ivan and others who have requested session save, > > I'm happy to report that this feature is now present in TOPCAT v3.6 > (just released). The save window now gives you three options: > > - Current Table > - as before, saves the current 'apparent table' (no subsets, > only currently visible rows and columns included) > - Multiple Tables > - saves many apparent tables into a single container file > (generally multi-extension FITS or multi-TABLE VOTable) > - Session > - saves (nearly) all the state of each loaded table, into a > single container file. This preserves row subsets, current > subset and sort order, label, hidden columns and some > other things. > > A saved session file can be loaded in just like a normal multi-table > file. Note however that a couple of things the session save does > not currently do are saving non-table information like plots and > save activation actions. Also, subsets are effectively serialized > as boolean columns (though this is hidden from the user), so that > subsets defined by expressions, when saved and restored, are no > longer defined by expressions. It's possible some of these things > may be improved in a future release. > > Thanks for your input that's encouraged me to introduce this feature. > Any further comments very welcome on this list. > > Mark > > On Wed, 7 Apr 2010, august.fly@gmail.XXX wrote: > > hi, > > > > > > it may be entirely a product of my particular use case(s) but I very > > frequently find myself wishing I could simply save an entire TOPCAT > > session, all the tables, at once into a new "table" package that I could > > restore at a later time. > > > > > > the instances that most often elicit this desire include backing up my > > work, or wishing I had because the power at my work went out wiping out > > my TOPCAT session, creating a package of tables from a project to share, > > and portability between desktop/laptop/home/work. > > > > > > such a feature has been prominent in our CfA python users group and > > astropython.org discussions, which is just to say that prob any work > > "environ" could lead to such a feature request. > > > > > > I suppose metadata/session packaging might be an issue. I personally > > would think that a multi-extension FITS would upset the fewest number of > > users. Of course xml+fits ref might alleviate metadata translation > > issues. I don't know. > > > > > > anyway, just 1 user's comment/suggestion. i hope i haven't overlooked > > some basic feature yet also wonder if there are other's who have similar > > experiences. > > > > > > best, > > > > > > - gus muench > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ -- ******************************************************** Adopt an Italian Astronomer http://adoptitaastronom.altervista.org/index.html Support Italian Research and Education http://no-brain-no-gain.net/ ********************************************************* -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 ------------------------------------------------------------------------- From thomas.boch@astro.XXX Thu Aug 26 09:31:19 2010 Date: Thu, 26 Aug 2010 09:31:00 +0200 From: Thomas Boch To: topcat-user@mlist2.XXX Subject: Removing duplicating entries in a table Hi Mark and TOPCAT enthusiasts, I would like to eliminate duplicate entries in a table (ie rows having the exact same values for all fields). Is there a smart way of doing this in TOPCAT ? Cheers, Thomas -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From m.b.taylor@bristol.XXX Thu Aug 26 09:44:39 2010 Date: Thu, 26 Aug 2010 09:44:38 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Removing duplicating entries in a table On Thu, 26 Aug 2010, Thomas Boch wrote: > Hi Mark and TOPCAT enthusiasts, > > I would like to eliminate duplicate entries in a table (ie rows having the > exact same values for all fields). Is there a smart way of doing this in > TOPCAT ? Hi Thomas, there is not. STILTS can probably help though. There is a STILTS filter 'uniq' (http://www.starlink.ac.uk/stilts/sun256/uniq.html), named after the unix command of the same name. Used without arguments, it will eliminate any duplicated rows which appear adjacent to each other. So, if you can sort your table so that any identical rows appear next to each other, you can use it to eliminate duplicate rows. The 'sort' filter (http://www.starlink.ac.uk/stilts/sun256/sort.html) can do this, but as it stands the only way to ensure that all fields are sorted on is to name all the columns explicitly. So, if your table t.fits has three columns ID, RA, DEC, then the following would do it: stilts tpipe in=t.fits cmd='sort "ID RA DEC"' cmd='uniq' or cmd='sort "$1 $2 $3"' cmd='uniq' If you happen to know that a sort on ID would produce the same order, then you could just do cmd='sort ID' cmd='uniq' and if you just want to eliminate rows with the same ID regardless of other columns, you can do cmd='sort ID' cmd='uniq ID' If you (or anyone else) would like to be able to do variations on this, e.g. sort on all columns without having to name them, or eliminate identical rows without modifying the table order, let me know and I'll think about generalising the functionality. Regarding having this sort of thing in TOPCAT - it feels to me like the sort of thing which is sufficiently "programmatic" that it wouldn't fit there very well (a GUI interface with all the required options would be unweildy). If anybody would like to persuade me otherwise, they are welcome to have a go ... Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Thu Aug 26 10:48:43 2010 Date: Thu, 26 Aug 2010 09:48:27 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Removing duplicating entries in a table Hi Thomas, you can try with "Internal Match" -> Algorithm = Exact Value ; Action = Eliminate All But First of Each Group/Eliminate All Grouped Rows (depending whether you want to keep one row or no row of the duplicate ones) Hope this helps, Jesús Quoting Thomas Boch : > Hi Mark and TOPCAT enthusiasts, > > I would like to eliminate duplicate entries in a table (ie rows having > the exact same values for all fields). Is there a smart way of doing > this in TOPCAT ? > Cheers, > > Thomas > > -- > Thomas Boch > CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 > 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 > F-67000 Strasbourg Email : thomas.boch@astro.XXX > France http://cdsweb.u-strasbg.fr/~boch -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Thu Aug 26 09:56:12 2010 Date: Thu, 26 Aug 2010 09:56:11 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Removing duplicating entries in a table Jesús (and, off-list, Evanthia), thank you! You're quite right, I didn't think of that. I'll leave answering queries to the experts in future :-). That beats the method I suggested in that it doesn't reorder the rows, but it will only work on the value of a single column (or at least a single calculated value). I think I could fix that by providing a function which hashes all the values in a row, if required. Mark On Thu, 26 Aug 2010, jmvarela@iac.XXX wrote: > Hi Thomas, > > you can try with "Internal Match" -> Algorithm = Exact Value ; Action = > Eliminate All But First of Each Group/Eliminate All Grouped Rows (depending > whether you want to keep one row or no row of the duplicate ones) > > Hope this helps, > > Jesús > > Quoting Thomas Boch : > > > Hi Mark and TOPCAT enthusiasts, > > > > I would like to eliminate duplicate entries in a table (ie rows having > > the exact same values for all fields). Is there a smart way of doing > > this in TOPCAT ? > > Cheers, > > > > Thomas > > > > -- > > Thomas Boch > > CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 > > 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 > > F-67000 Strasbourg Email : thomas.boch@astro.XXX > > France http://cdsweb.u-strasbg.fr/~boch > > > > -- > > Dr. Jesús Varela López > Instituto de Astrofísica de Canarias (IAC) > E-38205 La Laguna, Tenerife, Spain > Phone: +34 922 605 278 > Fax: +34 922 605 210 > Website: http://www.iac.es > > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de > Datos, acceda a http://www.iac.es/disclaimer.php > WARNING: For more information on privacy and fulfilment of the Law concerning > the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Thu Aug 26 11:06:23 2010 Date: Thu, 26 Aug 2010 10:05:44 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Removing duplicating entries in a table > > thank you! You're quite right, I didn't think of that. I'll leave > answering queries to the experts in future :-). > :D Thanks to you for making everything available :D > That beats the method I suggested in that it doesn't reorder the rows, > but it will only work on the value of a single column (or at least > a single calculated value). I think I could fix that by providing > a function which hashes all the values in a row, if required. > Maybe it would be nice to have directly a command to remove duplicates rows (matching one, several or all the fields). This could be added to the options of the Internal Matching. Jesús > Mark > > On Thu, 26 Aug 2010, jmvarela@iac.XXX wrote: > >> Hi Thomas, >> >> you can try with "Internal Match" -> Algorithm = Exact Value ; Action = >> Eliminate All But First of Each Group/Eliminate All Grouped Rows (depending >> whether you want to keep one row or no row of the duplicate ones) >> >> Hope this helps, >> >> Jesús >> >> Quoting Thomas Boch : >> >> > Hi Mark and TOPCAT enthusiasts, >> > >> > I would like to eliminate duplicate entries in a table (ie rows having >> > the exact same values for all fields). Is there a smart way of doing >> > this in TOPCAT ? >> > Cheers, >> > >> > Thomas >> > >> > -- >> > Thomas Boch >> > CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 >> > 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 >> > F-67000 Strasbourg Email : thomas.boch@astro.XXX >> > France http://cdsweb.u-strasbg.fr/~boch >> >> >> >> -- >> >> Dr. Jesús Varela López >> Instituto de Astrofísica de Canarias (IAC) >> E-38205 La Laguna, Tenerife, Spain >> Phone: +34 922 605 278 >> Fax: +34 922 605 210 >> Website: http://www.iac.es >> >> --------------------------------------------------------------------------------------------- >> ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de >> Datos, acceda a http://www.iac.es/disclaimer.php >> WARNING: For more information on privacy and fulfilment of the Law >> concerning >> the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en >> >> >> > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From price@astro.XXX Thu Sep 2 10:30:07 2010 Date: Wed, 01 Sep 2010 13:44:52 -0400 From: Paul Price To: topcat-user@bristol.XXX Subject: Plots suitable for publication? Hi Mark et al. I've enjoyed using TOPCAT for exploring data, but when it comes to making a figure for a paper or talk, I've been forced to use a different tool in order to get more control over the line widths, etc., so that they will be readable. Of course, this usually involves programming and manually fiddling with SM/gnuplot to get what I can very quickly produce in TOPCAT with a few clicks. It would be great to have either a "publication mode" for plots (e.g., line width=3, font size=30) or a configuration dialog for controlling the plot appearances. Thanks, Paul. From m.b.taylor@bristol.XXX Thu Sep 2 09:43:47 2010 Date: Thu, 2 Sep 2010 09:43:46 +0100 (BST) From: Mark Taylor To: Paul Price Cc: topcat-user@bristol.XXX Subject: Re: Plots suitable for publication? On Wed, 1 Sep 2010, Paul Price wrote: > Hi Mark et al. > > I've enjoyed using TOPCAT for exploring data, but when it comes to making > a figure for a paper or talk, I've been forced to use a different tool in > order to get more control over the line widths, etc., so that they will be > readable. Of course, this usually involves programming and manually > fiddling with SM/gnuplot to get what I can very quickly produce in TOPCAT > with a few clicks. It would be great to have either a "publication mode" > for plots (e.g., line width=3, font size=30) or a configuration dialog for > controlling the plot appearances. Paul, thanks for this suggestion - you're not the first to make it, but all feedback is welcome, the more requests I get for a given feature the more likely I am to implement it. In fact I had hoped that an overhaul of TOPCAT's graphics, including this, was going to reach the top of my to-do list over the last few months, but with one thing and another it seems that it hasn't. In the mean time, you can get some control over line width/font size simply by changing the size of the plot window on screen before you export the graphic; since the text/line width stay the same, a large window effectively means (relatively) thin lines and small font, so resizing the plot window smaller might help a bit. But I realise this is a rather blunt instrument. Depending on funding, I hope this will be improved in the forseeable future, but it's probably fair to say it's unlikely to become as sophisticated as a dedicated programmatic plotting tool along the lines of Gnuplot. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Thu Sep 2 11:59:18 2010 Date: Thu, 02 Sep 2010 10:58:03 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Plots suitable for publication? Hi Mark, firs of all, I add my vote to this request to make it go a little bit higher in your to-do list ;-) However, I'm thinking that maybe having an option "export as table" that just creates a table (fits or ascii) with the plotted values would make easier to use an alternative tool redo the plot, until the more sophisticate plotting tool is available in TOPCAT. Greetings, Jesús Quoting Mark Taylor : > On Wed, 1 Sep 2010, Paul Price wrote: > >> Hi Mark et al. >> >> I've enjoyed using TOPCAT for exploring data, but when it comes to making >> a figure for a paper or talk, I've been forced to use a different tool in >> order to get more control over the line widths, etc., so that they will be >> readable. Of course, this usually involves programming and manually >> fiddling with SM/gnuplot to get what I can very quickly produce in TOPCAT >> with a few clicks. It would be great to have either a "publication mode" >> for plots (e.g., line width=3, font size=30) or a configuration dialog for >> controlling the plot appearances. > > Paul, > > thanks for this suggestion - you're not the first to make it, but > all feedback is welcome, the more requests I get for a given feature > the more likely I am to implement it. In fact I had hoped that an > overhaul of TOPCAT's graphics, including this, was going to reach > the top of my to-do list over the last few months, but with one > thing and another it seems that it hasn't. > > In the mean time, you can get some control over line width/font size > simply by changing the size of the plot window on screen before you > export the graphic; since the text/line width stay the same, > a large window effectively means (relatively) thin lines and small > font, so resizing the plot window smaller might help a bit. > But I realise this is a rather blunt instrument. > > Depending on funding, I hope this will be improved in the forseeable > future, but it's probably fair to say it's unlikely to become as > sophisticated as a dedicated programmatic plotting tool along the > lines of Gnuplot. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Thu Sep 2 11:05:21 2010 Date: Thu, 2 Sep 2010 11:05:21 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Plots suitable for publication? Jesús In general, I agree with the idea that (as things stand) it makes sense to use TOPCAT interactively to work out what it is that you want to plot, and then export the data in such a way that it can be plotted by some other tool for more polished presentation purposes. However, I'm not sure what kind of export you've got in mind here. Obviously, you can already output tables from TOPCAT in all kinds of different forms. What are you suggesting that is not already available? Mark On Thu, 2 Sep 2010, jmvarela@iac.XXX wrote: > Hi Mark, > > firs of all, I add my vote to this request to make it go a little bit higher > in your to-do list ;-) > > However, I'm thinking that maybe having an option "export as table" that just > creates a table (fits or ascii) with the plotted values would make easier to > use an alternative tool redo the plot, until the more sophisticate plotting > tool is available in TOPCAT. > > Greetings, > > Jesús > > > Quoting Mark Taylor : > > > On Wed, 1 Sep 2010, Paul Price wrote: > > > > > Hi Mark et al. > > > > > > I've enjoyed using TOPCAT for exploring data, but when it comes to making > > > a figure for a paper or talk, I've been forced to use a different tool in > > > order to get more control over the line widths, etc., so that they will be > > > readable. Of course, this usually involves programming and manually > > > fiddling with SM/gnuplot to get what I can very quickly produce in TOPCAT > > > with a few clicks. It would be great to have either a "publication mode" > > > for plots (e.g., line width=3, font size=30) or a configuration dialog for > > > controlling the plot appearances. > > > > Paul, > > > > thanks for this suggestion - you're not the first to make it, but > > all feedback is welcome, the more requests I get for a given feature > > the more likely I am to implement it. In fact I had hoped that an > > overhaul of TOPCAT's graphics, including this, was going to reach > > the top of my to-do list over the last few months, but with one > > thing and another it seems that it hasn't. > > > > In the mean time, you can get some control over line width/font size > > simply by changing the size of the plot window on screen before you > > export the graphic; since the text/line width stay the same, > > a large window effectively means (relatively) thin lines and small > > font, so resizing the plot window smaller might help a bit. > > But I realise this is a rather blunt instrument. > > > > Depending on funding, I hope this will be improved in the forseeable > > future, but it's probably fair to say it's unlikely to become as > > sophisticated as a dedicated programmatic plotting tool along the > > lines of Gnuplot. > > > > Mark > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > > > > > > -- > > Dr. Jesús Varela López > Instituto de Astrofísica de Canarias (IAC) > E-38205 La Laguna, Tenerife, Spain > Phone: +34 922 605 278 > Fax: +34 922 605 210 > Website: http://www.iac.es > > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de > Datos, acceda a http://www.iac.es/disclaimer.php > WARNING: For more information on privacy and fulfilment of the Law concerning > the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ipf@iac.XXX Thu Sep 2 12:23:40 2010 Date: Thu, 02 Sep 2010 11:22:28 +0100 From: ipf@iac.XXX To: Mark Taylor Cc: topcat-user@bristol.XXX ipf@iac.XXX Subject: URLs in tables saved from Topcat in HTML format Hi Mark, a question about URLs in tables in HTML format. I have topcat tables with active links to URLs. When I save them in HTML format, the URLs are saved as in the original topcat table but not in the HTML format that makes the link work in a browser. Is there a way in topcat or with other programmes to write the appropriate HTML format so that the links work? Thanks, Ismael --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From jmvarela@iac.XXX Thu Sep 2 12:33:59 2010 Date: Thu, 02 Sep 2010 11:33:40 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Plots suitable for publication? Hi Mark, > Jesús > > In general, I agree with the idea that (as things stand) it makes sense > to use TOPCAT interactively to work out what it is that you want to plot, > and then export the data in such a way that it can be plotted by some > other tool for more polished presentation purposes. > > However, I'm not sure what kind of export you've got in mind here. > Obviously, you can already output tables from TOPCAT in all kinds of > different forms. What are you suggesting that is not already > available? My idea is to export the actual (X,Y), (x,y,z), (bin,hist),... values from the plot. Usually a plot is a selection of rows and selection of columns and maybe different samples that in some way are produced on the fly while producing the plot. So maybe to reproduce the selection and export the table in the usual way in TOPCAT can be tedious. So my idea would be something like being producing a plot and them just make "import as table" that would produce a list of (x,y) values or (bin,n) values so I can take other tool read the exported table and let's say do "read x y ; plot x y". For example, this would avoid the need to hidden the columns just to have the ones to be plotted. Also if several subsamples are plotted a third column could be add in the exported table to identify easily the points corresponding to each group. Right now I don't see a fast way to save different subsamples apart of selecting each one and saving them in different files. So my idea is just to take the information needed to reproduce the plot without having to deal with the whole table. I admit that this sounds easy for scatter plots and maybe it can become a nightmare for other kind of plots. I'm not sure if I was able to explained myself. Hope so :) Jesús > > Mark > > On Thu, 2 Sep 2010, jmvarela@iac.XXX wrote: > >> Hi Mark, >> >> firs of all, I add my vote to this request to make it go a little >> bit higher >> in your to-do list ;-) >> >> However, I'm thinking that maybe having an option "export as >> table" that just >> creates a table (fits or ascii) with the plotted values would make easier to >> use an alternative tool redo the plot, until the more sophisticate plotting >> tool is available in TOPCAT. >> >> Greetings, >> >> Jesús >> >> >> Quoting Mark Taylor : >> >> > On Wed, 1 Sep 2010, Paul Price wrote: >> > >> > > Hi Mark et al. >> > > >> > > I've enjoyed using TOPCAT for exploring data, but when it comes >> to making >> > > a figure for a paper or talk, I've been forced to use a >> different tool in >> > > order to get more control over the line widths, etc., so that >> they will be >> > > readable. Of course, this usually involves programming and manually >> > > fiddling with SM/gnuplot to get what I can very quickly produce >> in TOPCAT >> > > with a few clicks. It would be great to have either a >> "publication mode" >> > > for plots (e.g., line width=3, font size=30) or a configuration >> dialog for >> > > controlling the plot appearances. >> > >> > Paul, >> > >> > thanks for this suggestion - you're not the first to make it, but >> > all feedback is welcome, the more requests I get for a given feature >> > the more likely I am to implement it. In fact I had hoped that an >> > overhaul of TOPCAT's graphics, including this, was going to reach >> > the top of my to-do list over the last few months, but with one >> > thing and another it seems that it hasn't. >> > >> > In the mean time, you can get some control over line width/font size >> > simply by changing the size of the plot window on screen before you >> > export the graphic; since the text/line width stay the same, >> > a large window effectively means (relatively) thin lines and small >> > font, so resizing the plot window smaller might help a bit. >> > But I realise this is a rather blunt instrument. >> > >> > Depending on funding, I hope this will be improved in the forseeable >> > future, but it's probably fair to say it's unlikely to become as >> > sophisticated as a dedicated programmatic plotting tool along the >> > lines of Gnuplot. >> > >> > Mark >> > >> > -- >> > Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> > >> > >> >> >> >> -- >> >> Dr. Jesús Varela López >> Instituto de Astrofísica de Canarias (IAC) >> E-38205 La Laguna, Tenerife, Spain >> Phone: +34 922 605 278 >> Fax: +34 922 605 210 >> Website: http://www.iac.es >> >> --------------------------------------------------------------------------------------------- >> ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de >> Datos, acceda a http://www.iac.es/disclaimer.php >> WARNING: For more information on privacy and fulfilment of the Law >> concerning >> the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en >> >> >> > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Thu Sep 2 12:24:26 2010 Date: Thu, 2 Sep 2010 12:24:26 +0100 (BST) From: Mark Taylor To: ipf@iac.XXX Cc: topcat-user@bristol.XXX Subject: Re: URLs in tables saved from Topcat in HTML format Ismael, On Thu, 2 Sep 2010, ipf@iac.XXX wrote: > Hi Mark, > > a question about URLs in tables in HTML format. > > I have topcat tables with active links to URLs. When I save them > in HTML format, the URLs are saved as in the original topcat table but > not in the HTML format that makes the link work in a browser. > > Is there a way in topcat or with other programmes to write the > appropriate HTML format so that the links work? That's a good idea. I've made an adjustment to the HTML output handler so that cell contents which look like URLs get output as HTML links instead of plain text. This will be in the next public release; in the mean time, you can get a pre-release copy with this function from ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Thanks for the suggestion. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jbrinchmann@gmail.XXX Thu Sep 2 15:00:13 2010 Date: Thu, 2 Sep 2010 15:00:01 +0200 From: Jarle Brinchmann To: topcat-user@sympa.XXX Subject: Re: Plots suitable for publication? To add my 2c, I am a bit along the lines of Jesús here. Personally I never use topcat, nor any other interactive program (except to touch up perhaps) to produce publication plots for the simple reason that I want my plots to be easy to recreate years hence and I find writing a plot script with comments is much the better way for posterity. However I often use topcat for exploring data and sometimes to select data for subsequent analysis in IDL/PDL/R and creation data selections for subsequent creation of plots is one of those tasks. One thing I then have missed is a button in the plot window, particularly scatter plots, but other windows could also be relevant, where I can quickly save the data using my current display as a constraint. Ie. say I zoomed in on useful data in a big dataset and I select two subsamples interactively based on some (often irregular) criterion. It would then be great to be able to have a one-click solution to save the data I have displayed as a snapshot with a boolean or flag column indicating the samples selected. That would certainly ease the selection/output loop quite a bit. It is of course still possible to do this but it is slightly more time-consuming. Having some control over the plot in details would also be nice but that is something that ought to be implemented in a thorough fashion and that is probably a significant amount of work - speaking entirely from personal preference I am also happy to have tools for specific tasks - I don't much like a tool that does everything, but I know I am in the minority :) Cheers, Jarle. On Thu, Sep 2, 2010 at 12:33 PM, wrote: > Hi Mark, > >> Jesús >> >> In general, I agree with the idea that (as things stand) it makes sense >> to use TOPCAT interactively to work out what it is that you want to plot, >> and then export the data in such a way that it can be plotted by some >> other tool for more polished presentation purposes. >> >> However, I'm not sure what kind of export you've got in mind here. >> Obviously, you can already output tables from TOPCAT in all kinds of >> different forms.  What are you suggesting that is not already >> available? > > My idea is to export the actual (X,Y), (x,y,z), (bin,hist),... values from > the plot. Usually a plot is a selection of rows and selection of columns and > maybe different samples that in some way are produced on the fly while > producing the plot. So maybe to reproduce the selection and export the table > in the usual way in TOPCAT can be tedious. So my idea would be something > like being producing a plot and them just make "import as table" that would > produce a list of (x,y) values or (bin,n) values so I can take other tool > read the exported table and let's say do "read x y ; plot x y". > >  For example, this would avoid the need to hidden the columns just to have > the ones to be plotted. Also if several subsamples are plotted a third > column could be add in the exported table to identify easily the points > corresponding to each group. Right now I don't see a fast way to save > different subsamples apart of selecting each one and saving them in > different files. > >  So my idea is just to take the information needed to reproduce the plot > without having to deal with the whole table. > >  I admit that this sounds easy for scatter plots and maybe it can become a > nightmare for other kind of plots. > >  I'm not sure if I was able to explained myself. Hope so :) > >  Jesús > >> >> Mark >> >> On Thu, 2 Sep 2010, jmvarela@iac.XXX wrote: >> >>> Hi Mark, >>> >>>  firs of all, I add my vote to this request to make it go a little  bit >>> higher >>> in your to-do list ;-) >>> >>>  However, I'm thinking that maybe having an option "export as  table" >>> that just >>> creates a table (fits or ascii) with the plotted values would make easier >>> to >>> use an alternative tool redo the plot, until the more sophisticate >>> plotting >>> tool is available in TOPCAT. >>> >>>  Greetings, >>> >>> Jesús >>> >>> >>> Quoting Mark Taylor : >>> >>> > On Wed, 1 Sep 2010, Paul Price wrote: >>> > >>> > > Hi Mark et al. >>> > > >>> > > I've enjoyed using TOPCAT for exploring data, but when it comes  to >>> > > making >>> > > a figure for a paper or talk, I've been forced to use a  different >>> > > tool in >>> > > order to get more control over the line widths, etc., so that  they >>> > > will be >>> > > readable.  Of course, this usually involves programming and manually >>> > > fiddling with SM/gnuplot to get what I can very quickly produce  in >>> > > TOPCAT >>> > > with a few clicks.  It would be great to have either a  "publication >>> > > mode" >>> > > for plots (e.g., line width=3, font size=30) or a configuration >>> > >  dialog for >>> > > controlling the plot appearances. >>> > >>> > Paul, >>> > >>> > thanks for this suggestion - you're not the first to make it, but >>> > all feedback is welcome, the more requests I get for a given feature >>> > the more likely I am to implement it.  In fact I had hoped that an >>> > overhaul of TOPCAT's graphics, including this, was going to reach >>> > the top of my to-do list over the last few months, but with one >>> > thing and another it seems that it hasn't. >>> > >>> > In the mean time, you can get some control over line width/font size >>> > simply by changing the size of the plot window on screen before you >>> > export the graphic; since the text/line width stay the same, >>> > a large window effectively means (relatively) thin lines and small >>> > font, so resizing the plot window smaller might help a bit. >>> > But I realise this is a rather blunt instrument. >>> > >>> > Depending on funding, I hope this will be improved in the forseeable >>> > future, but it's probably fair to say it's unlikely to become as >>> > sophisticated as a dedicated programmatic plotting tool along the >>> > lines of Gnuplot. >>> > >>> > Mark >>> > >>> > -- >>> > Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK >>> > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >>> > >>> > >>> >>> >>> >>> -- >>> >>> Dr. Jesús Varela López >>> Instituto de Astrofísica de Canarias (IAC) >>> E-38205 La Laguna, Tenerife, Spain >>> Phone:   +34 922 605 278 >>> Fax:     +34 922 605 210 >>> Website: http://www.iac.es >>> >>> >>> --------------------------------------------------------------------------------------------- >>> ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion >>> de >>> Datos, acceda a http://www.iac.es/disclaimer.php >>> WARNING: For more information on privacy and fulfilment of the Law >>>  concerning >>> the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en >>> >>> >>> >> >> -- >> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > > -- > > Dr. Jesús Varela López > Instituto de Astrofísica de Canarias (IAC) > E-38205 La Laguna, Tenerife, Spain > Phone:   +34 922 605 278 > Fax:     +34 922 605 210 > Website: http://www.iac.es > > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de > Datos, acceda a http://www.iac.es/disclaimer.php > WARNING: For more information on privacy and fulfilment of the Law > concerning the Protection of Data, consult > http://www.iac.es/disclaimer.php?lang=en > > > > From jtlz2@astro.XXX Thu Sep 2 15:54:33 2010 Date: Thu, 2 Sep 2010 10:53:47 -0400 From: Jonathan Zwart To: topcat-user@sympa.XXX Subject: Re: Plots suitable for publication? Hi Mark - another vote from me for a publication-quality mode (it's always disappointing after using TOPCAT so powerfully to have to move to gnuplot etc to finish off) and a quick-dump-the-very-plot-you-see-to-a-table button. Massive thanks, as always, for this wonderful software. Jon On 2 September 2010 09:00, Jarle Brinchmann wrote: > To add my 2c, I am a bit along the lines of Jesús here. > > Personally I never use topcat, nor any other interactive program > (except to touch up perhaps) to produce publication plots for the > simple reason that I want my plots to be easy to recreate years hence > and I find writing a plot script with comments is much the better way > for posterity. However I often use topcat for exploring data and > sometimes to select data for subsequent analysis in IDL/PDL/R and > creation data selections for subsequent creation of plots is one of > those tasks. > > One thing I then have missed is a button in the plot window, > particularly scatter plots, but other windows could also be relevant, > where I can quickly save the data using my current display as a > constraint. Ie. say I zoomed in on useful data in a big dataset and I > select two subsamples interactively based on some (often irregular) > criterion. It would then be great to be able to have a one-click > solution to save the data I have displayed as a snapshot with a > boolean or flag column indicating the samples selected. That would > certainly ease the selection/output loop quite a bit. It is of course > still possible to do this but it is slightly more time-consuming. > > Having some control over the plot in details would also be nice but > that is something that ought to be implemented in a thorough fashion > and that is probably a significant amount of work - speaking entirely > from personal preference I am also happy to have tools for specific > tasks - I don't much like a tool that does everything, but I know I am > in the minority :) > > Cheers, > Jarle. > > > > On Thu, Sep 2, 2010 at 12:33 PM, wrote: > > Hi Mark, > > > >> Jesús > >> > >> In general, I agree with the idea that (as things stand) it makes sense > >> to use TOPCAT interactively to work out what it is that you want to > plot, > >> and then export the data in such a way that it can be plotted by some > >> other tool for more polished presentation purposes. > >> > >> However, I'm not sure what kind of export you've got in mind here. > >> Obviously, you can already output tables from TOPCAT in all kinds of > >> different forms. What are you suggesting that is not already > >> available? > > > > My idea is to export the actual (X,Y), (x,y,z), (bin,hist),... values > from > > the plot. Usually a plot is a selection of rows and selection of columns > and > > maybe different samples that in some way are produced on the fly while > > producing the plot. So maybe to reproduce the selection and export the > table > > in the usual way in TOPCAT can be tedious. So my idea would be something > > like being producing a plot and them just make "import as table" that > would > > produce a list of (x,y) values or (bin,n) values so I can take other tool > > read the exported table and let's say do "read x y ; plot x y". > > > > For example, this would avoid the need to hidden the columns just to > have > > the ones to be plotted. Also if several subsamples are plotted a third > > column could be add in the exported table to identify easily the points > > corresponding to each group. Right now I don't see a fast way to save > > different subsamples apart of selecting each one and saving them in > > different files. > > > > So my idea is just to take the information needed to reproduce the plot > > without having to deal with the whole table. > > > > I admit that this sounds easy for scatter plots and maybe it can become > a > > nightmare for other kind of plots. > > > > I'm not sure if I was able to explained myself. Hope so :) > > > > Jesús > > > >> > >> Mark > >> > >> On Thu, 2 Sep 2010, jmvarela@iac.XXX wrote: > >> > >>> Hi Mark, > >>> > >>> firs of all, I add my vote to this request to make it go a little bit > >>> higher > >>> in your to-do list ;-) > >>> > >>> However, I'm thinking that maybe having an option "export as table" > >>> that just > >>> creates a table (fits or ascii) with the plotted values would make > easier > >>> to > >>> use an alternative tool redo the plot, until the more sophisticate > >>> plotting > >>> tool is available in TOPCAT. > >>> > >>> Greetings, > >>> > >>> Jesús > >>> > >>> > >>> Quoting Mark Taylor : > >>> > >>> > On Wed, 1 Sep 2010, Paul Price wrote: > >>> > > >>> > > Hi Mark et al. > >>> > > > >>> > > I've enjoyed using TOPCAT for exploring data, but when it comes to > >>> > > making > >>> > > a figure for a paper or talk, I've been forced to use a different > >>> > > tool in > >>> > > order to get more control over the line widths, etc., so that they > >>> > > will be > >>> > > readable. Of course, this usually involves programming and > manually > >>> > > fiddling with SM/gnuplot to get what I can very quickly produce in > >>> > > TOPCAT > >>> > > with a few clicks. It would be great to have either a > "publication > >>> > > mode" > >>> > > for plots (e.g., line width=3, font size=30) or a configuration > >>> > > dialog for > >>> > > controlling the plot appearances. > >>> > > >>> > Paul, > >>> > > >>> > thanks for this suggestion - you're not the first to make it, but > >>> > all feedback is welcome, the more requests I get for a given feature > >>> > the more likely I am to implement it. In fact I had hoped that an > >>> > overhaul of TOPCAT's graphics, including this, was going to reach > >>> > the top of my to-do list over the last few months, but with one > >>> > thing and another it seems that it hasn't. > >>> > > >>> > In the mean time, you can get some control over line width/font size > >>> > simply by changing the size of the plot window on screen before you > >>> > export the graphic; since the text/line width stay the same, > >>> > a large window effectively means (relatively) thin lines and small > >>> > font, so resizing the plot window smaller might help a bit. > >>> > But I realise this is a rather blunt instrument. > >>> > > >>> > Depending on funding, I hope this will be improved in the forseeable > >>> > future, but it's probably fair to say it's unlikely to become as > >>> > sophisticated as a dedicated programmatic plotting tool along the > >>> > lines of Gnuplot. > >>> > > >>> > Mark > >>> > > >>> > -- > >>> > Mark Taylor Astronomical Programmer Physics, Bristol University, > UK > >>> > m.b.taylor@bris.XXX +44-117-928-8776 > http://www.star.bris.ac.uk/~mbt/ > >>> > > >>> > > >>> > >>> > >>> > >>> -- > >>> > >>> Dr. Jesús Varela López > >>> Instituto de Astrofísica de Canarias (IAC) > >>> E-38205 La Laguna, Tenerife, Spain > >>> Phone: +34 922 605 278 > >>> Fax: +34 922 605 210 > >>> Website: http://www.iac.es > >>> > >>> > >>> > --------------------------------------------------------------------------------------------- > >>> ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion > >>> de > >>> Datos, acceda a http://www.iac.es/disclaimer.php > >>> WARNING: For more information on privacy and fulfilment of the Law > >>> concerning > >>> the Protection of Data, consult > http://www.iac.es/disclaimer.php?lang=en > >>> > >>> > >>> > >> > >> -- > >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK > >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > > > > > > -- > > > > Dr. Jesús Varela López > > Instituto de Astrofísica de Canarias (IAC) > > E-38205 La Laguna, Tenerife, Spain > > Phone: +34 922 605 278 > > Fax: +34 922 605 210 > > Website: http://www.iac.es > > > > > --------------------------------------------------------------------------------------------- > > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion > de > > Datos, acceda a http://www.iac.es/disclaimer.php > > WARNING: For more information on privacy and fulfilment of the Law > > concerning the Protection of Data, consult > > http://www.iac.es/disclaimer.php?lang=en > > > > > > > > > > > From ipf@iac.XXX Thu Sep 2 16:59:07 2010 Date: Thu, 02 Sep 2010 15:58:58 +0100 From: Ismael Perez Fournon To: Mark Taylor Cc: topcat-user@bristol.XXX Ismael Perez Fournon Subject: Re: URLs in tables saved from Topcat in HTML format many thanks, Mark. please let us know when a similar improvement to the HTML output handler is available in stilts. Ismael Mark Taylor wrote: > Ismael, > > On Thu, 2 Sep 2010, ipf@iac.XXX wrote: > > >> Hi Mark, >> >> a question about URLs in tables in HTML format. >> >> I have topcat tables with active links to URLs. When I save them >> in HTML format, the URLs are saved as in the original topcat table but >> not in the HTML format that makes the link work in a browser. >> >> Is there a way in topcat or with other programmes to write the >> appropriate HTML format so that the links work? >> > > That's a good idea. I've made an adjustment to the HTML output handler > so that cell contents which look like URLs get output as HTML links > instead of plain text. This will be in the next public release; > in the mean time, you can get a pre-release copy with this function > from > > ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ > > Thanks for the suggestion. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci?n de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Thu Sep 2 16:09:13 2010 Date: Thu, 2 Sep 2010 16:09:11 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: URLs in tables saved from Topcat in HTML format On Thu, 2 Sep 2010, Ismael Perez Fournon wrote: > many thanks, Mark. > > please let us know when a similar improvement to the HTML output handler > is available in stilts. Done - find it at: ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ In fact since the TOPCAT jar file contains all the STILTS classes, if you have TOPCAT then you have STILTS. There's a little-known convenience feature that lets you exploit this, TOPCAT's "-stilts" flag, e.g.: topcat -stilts tpipe in=x.fits out=x.html or java -jar topcat-full.jar -stilts tpipe in=x.fits out=x.html Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ipf@iac.XXX Thu Sep 2 17:27:44 2010 Date: Thu, 02 Sep 2010 16:26:37 +0100 From: Ismael Perez Fournon To: Mark Taylor Cc: topcat-user@bristol.XXX Ismael Perez Fournon Subject: Re: URLs in tables saved from Topcat in HTML format great! many thanks again, also for the trick to use stilts from topcat. Ismael Mark Taylor wrote: > On Thu, 2 Sep 2010, Ismael Perez Fournon wrote: > > >> many thanks, Mark. >> >> please let us know when a similar improvement to the HTML output handler >> is available in stilts. >> > > Done - find it at: > > ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ > > In fact since the TOPCAT jar file contains all the STILTS classes, > if you have TOPCAT then you have STILTS. There's a little-known > convenience feature that lets you exploit this, TOPCAT's "-stilts" > flag, e.g.: > > topcat -stilts tpipe in=x.fits out=x.html > > or > > java -jar topcat-full.jar -stilts tpipe in=x.fits out=x.html > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci?n de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From jmvarela@iac.XXX Thu Sep 2 17:46:39 2010 Date: Thu, 02 Sep 2010 16:46:19 +0100 From: jmvarela@iac.XXX To: topcat-user@bristol.XXX Subject: Small shell script to easily load a catalogue from command-line in an open TOPCAT Hello everybody, connected with another thread about the possibility of loading a catalogue in TOPCAT directly from the command-line, I've written a small script that you can use to do it. Here it is: ----------------------------------------------------------- #!/usr/bin/env tcsh # # Script to load a table in an already open TOPCAT window if ( $1 == "" ) then echo "Usage: $0 Catalogue_FITS" exit(1) endif stilts tpipe in=$1 omode=topcat # END -------------------------------------------------------------- You can just copied in a file (e.g. CatToTOPCAT), put it in your path (e.g. ~/bin) and make it executable (chmod u+x CatToTOPCAT) and then to load a catalogue from the command-line to an already open TOPCAT you just have to do. > CatToTOPCAT catalogue.fits Sorry for those that prefer "bash". I'm sure you will be able to convert it to your favorite shell ;-) Cheers, Jesús -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Fri Sep 3 09:31:09 2010 Date: Fri, 3 Sep 2010 09:31:08 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Plots suitable for publication? Thanks all for your thoughts on this. Personally I'm with Jarle about wanting to use scriptable tools for publication plots, though I know this isn't how everybody likes to work. That's one reason why I introduced the plotting tools into STILTS; these do allow you to write a script that generates plots, and they also have a bit more configurability (e.g. label fonts and sizes) than TOPCAT. However, it's a long way from the flexibility of a dedicated plotting system like Gnuplot or IDL. It's also a bit unwieldy (see e.g. the documentation for the scatter plot command, plot2d: http://www.starlink.ac.uk/stilts/sun256/plot2d.html). One thing I would like to do is to introduce a button in TOPCAT that would tell you the STILTS command which generates the plot you're looking at. Combined with a bit more configurability in STILTS (things like line width, fairly easy to add) this would get some of the way to allowing you to generate more configurable plots without too much difficulty. Another possibility would be a button which spits out, say, a self-contained Gnuplot file containing all the data and enough commands to generate a plot more or less like the visible one. I hadn't thought of the idea suggested by Jesús and Jarle of outputting table data directly from a plot window. It is possible to do this sort of thing as it stands (New Subset From Visible button, maybe combine that subset with existing ones to select only some points from each, add the relevant subsets as columns, save the table), but you're right that in practice it is a bit fiddly, especially if you're doing it multiple times. I will give consideration to this, though you probably won't see it in the near future. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Fri Sep 3 13:28:55 2010 Date: Fri, 03 Sep 2010 12:27:34 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Plots suitable for publication? Hi Mark > One thing I would like to do is to introduce a button in TOPCAT > that would tell you the STILTS command which generates the plot > you're looking at. Combined with a bit more configurability > in STILTS (things like line width, fairly easy to add) this would > get some of the way to allowing you to generate more configurable > plots without too much difficulty. > This could be also a good idea... and it would do easier to get used to and learn STITLS ;-) > Another possibility would be a button which spits out, say, a > self-contained Gnuplot file containing all the data and enough > commands to generate a plot more or less like the visible one. > Also a possibility. > I hadn't thought of the idea suggested by Jesús and Jarle of > outputting table data directly from a plot window. It is possible > to do this sort of thing as it stands (New Subset From Visible > button, maybe combine that subset with existing ones to select > only some points from each, add the relevant subsets as columns, > save the table), but you're right that in practice it is a bit > fiddly, especially if you're doing it multiple times. > I will give consideration to this, though you probably won't see > it in the near future. > I think we will continue using TOPCAT anyway ;-) so, of course, take your time and follow your priorities... ... and thanks for all your work!! Cheers! Jesús -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From jtlz2@astro.XXX Fri Sep 3 15:20:33 2010 Date: Fri, 3 Sep 2010 10:20:06 -0400 From: Jonathan Zwart To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Plots suitable for publication? Hi Mark, On 3 September 2010 04:31, Mark Taylor wrote: > Thanks all for your thoughts on this. > > Personally I'm with Jarle about wanting to use scriptable tools for > publication plots, though I know this isn't how everybody likes to work. > That's one reason why I introduced the plotting tools into STILTS; > these do allow you to write a script that generates plots, and they > also have a bit more configurability (e.g. label fonts and sizes) > than TOPCAT. However, it's a long way from the flexibility of a > dedicated plotting system like Gnuplot or IDL. It's also a bit > unwieldy (see e.g. the documentation for the scatter plot command, > plot2d: http://www.starlink.ac.uk/stilts/sun256/plot2d.html). > > One thing I would like to do is to introduce a button in TOPCAT > that would tell you the STILTS command which generates the plot > you're looking at. Combined with a bit more configurability > in STILTS (things like line width, fairly easy to add) this would > get some of the way to allowing you to generate more configurable > plots without too much difficulty. > This is a wonderful idea! Jon > Another possibility would be a button which spits out, say, a > self-contained Gnuplot file containing all the data and enough > commands to generate a plot more or less like the visible one. > > I hadn't thought of the idea suggested by Jesús and Jarle of > outputting table data directly from a plot window. It is possible > to do this sort of thing as it stands (New Subset From Visible > button, maybe combine that subset with existing ones to select > only some points from each, add the relevant subsets as columns, > save the table), but you're right that in practice it is a bit > fiddly, especially if you're doing it multiple times. > I will give consideration to this, though you probably won't see > it in the near future. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From jtlz2@astro.XXX Wed Sep 8 21:45:10 2010 Date: Wed, 8 Sep 2010 16:44:44 -0400 From: Jonathan Zwart To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: ATpy and TOPCAT Hi Mark, There was an interesting proto-discussion about interfacing of the (pythonic) ATpy with TOPCAT, via SAMP --- www.astrobetter.com/atpy-and-asciitable Jon From jmvarela@iac.XXX Fri Sep 10 19:54:24 2010 Date: Fri, 10 Sep 2010 18:52:33 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Reloading a table without creating a new one Hi Mark and TOPCATers! I am wondering if there is any way (or if it could be added) to reload a table (whose content has change in the disk) without creating a new one. Kind of Emacs' "Revert Buffer". I use TOPCAT quite often to check tables that are output of another program. So while I'm testing the program the table has always the same name and only the content changes. So I would find quite comfortable to have an option to reload a table updating the content from the disk without having to create a new table in TOPCAT and delete the previous one with the obsolete content. Just an idea :) Thanks in advance! Jesús -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Mon Sep 13 10:57:35 2010 Date: Mon, 13 Sep 2010 10:57:35 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Reloading a table without creating a new one On Fri, 10 Sep 2010, jmvarela@iac.XXX wrote: > Hi Mark and TOPCATers! > > I am wondering if there is any way (or if it could be added) to reload a > table (whose content has change in the disk) without creating a new one. Kind > of Emacs' "Revert Buffer". > > I use TOPCAT quite often to check tables that are output of another program. > So while I'm testing the program the table has always the same name and only > the content changes. So I would find quite comfortable to have an option to > reload a table updating the content from the disk without having to create a > new table in TOPCAT and delete the previous one with the obsolete content. > > Just an idea :) > > Thanks in advance! > > Jesús Chaps, I'll think about this. I think Jesús's idea as stated, that is just to provide a one-click way of discarding an old table and loading a new one, from the same source as the old one, in its place should be reasonably straightforward (though there may be complications, I'd have to think a bit more). However Jon's and Hauke's requests for all the additional business of subsets, added columns, parameters, plots(?) etc defined for the initial table to be retained in the replacement would be harder - partly because there's no guarantee that the reloaded version will have the same structure as the original, so for instance a subset defined based on a column might find that its column has unexpectedly disappeared when the table is reloaded. Jesús, which of these variants did you have in mind? Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Sep 13 11:00:38 2010 Date: Mon, 13 Sep 2010 11:00:38 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Reloading a table without creating a new one On Sun, 12 Sep 2010, Jonathan Zwart wrote: > Forwarding Hauke's message below. > > Hauke I'd like to second your idea for generating multiple sets at once - I > suggested exactly this to Mark a few days ago. > > Jon > > > On 12 September 2010 19:08, Hauke Worpel (Sci) wrote: > > > Speaking of defining subsets, is it possible to implement an extension of > > the Boolean column feature that creates a separate subset for each distinct > > value in the column? What I mean is, say a column contains a number from 1 > > to 10- what I'd like to see is an automatic way to create a subset for all > > the 1's, all the 2's, etc, all in one go. Is this possible? > > > > Hauke this is on the to-do list. I'll try to prioritise it. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Mon Sep 13 15:29:09 2010 Date: Mon, 13 Sep 2010 14:26:59 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Reloading a table without creating a new one Hi everyone! Quoting Mark Taylor : > On Fri, 10 Sep 2010, jmvarela@iac.XXX wrote: > >> Hi Mark and TOPCATers! >> >> I am wondering if there is any way (or if it could be added) to reload a >> table (whose content has change in the disk) without creating a new >> one. Kind >> of Emacs' "Revert Buffer". >> >> I use TOPCAT quite often to check tables that are output of >> another program. >> So while I'm testing the program the table has always the same name and only >> the content changes. So I would find quite comfortable to have an option to >> reload a table updating the content from the disk without having to create a >> new table in TOPCAT and delete the previous one with the obsolete content. >> >> Just an idea :) >> >> Thanks in advance! >> >> Jesús > > Chaps, > > I'll think about this. I think Jesús's idea as stated, that is just > to provide a one-click way of discarding an old table and loading a new > one, from the same source as the old one, in its place should be > reasonably straightforward (though there may be complications, I'd have > to think a bit more). However Jon's and Hauke's requests for all > the additional business of subsets, added columns, parameters, plots(?) > etc defined for the initial table to be retained in the replacement > would be harder - partly because there's no guarantee that the reloaded > version will have the same structure as the original, so for instance > a subset defined based on a column might find that its column has > unexpectedly disappeared when the table is reloaded. > > Jesús, which of these variants did you have in mind? > At the time of writing I was thinking in the easy one, i.e. just reload the table, because what I was doing was to check the FITS output of a program. Then after reading the additional comments I thought it would be cool also to have these but I think this also would mean a lot of work for you, Mark. So right now I would be happy with a "reload table from disk" option maybe with a warning "All your changes will be lost!" :D Cheers! Jesús -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From sebastien.derriere@astro.XXX Mon Sep 13 18:24:55 2010 Date: Mon, 13 Sep 2010 18:28:16 +0200 From: Sebastien Derriere To: topcat-user@mlist2.XXX Subject: Re: Reloading a table without creating a new one On 09/13/2010 12:00 PM, Mark Taylor wrote: > On Sun, 12 Sep 2010, Jonathan Zwart wrote: > >> Forwarding Hauke's message below. >> >> Hauke I'd like to second your idea for generating multiple sets at once - I >> suggested exactly this to Mark a few days ago. >> >> Jon >> >> >> On 12 September 2010 19:08, Hauke Worpel (Sci)wrote: >> >>> Speaking of defining subsets, is it possible to implement an extension of >>> the Boolean column feature that creates a separate subset for each distinct >>> value in the column? What I mean is, say a column contains a number from 1 >>> to 10- what I'd like to see is an automatic way to create a subset for all >>> the 1's, all the 2's, etc, all in one go. Is this possible? >>> >>> Hauke > > this is on the to-do list. I'll try to prioritise it. Hello all, I remember the make-subsets-from-distinct-values is a suggestion I made a while ago, so I'm happy to see others push for it too ;o) Sebastien -- ((( Sebastien Derriere sebastien.derriere@astro.XXX (. .) Observatoire de Strasbourg Phone +33 (0) 368 852 444 (( v )) 11, rue de l'universite Telefax +33 (0) 368 852 417 ---m-m--- F-67000 Strasbourg France From m.b.taylor@bristol.XXX Mon Sep 13 17:28:21 2010 Date: Mon, 13 Sep 2010 17:28:21 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Reloading a table without creating a new one On Mon, 13 Sep 2010, Sebastien Derriere wrote: > On 09/13/2010 12:00 PM, Mark Taylor wrote: > > On Sun, 12 Sep 2010, Jonathan Zwart wrote: > > > > > Forwarding Hauke's message below. > > > > > > Hauke I'd like to second your idea for generating multiple sets at once - > > > I > > > suggested exactly this to Mark a few days ago. > > > > > > Jon > > > > > > > > > On 12 September 2010 19:08, Hauke Worpel > > > (Sci)wrote: > > > > > > > Speaking of defining subsets, is it possible to implement an extension > > > > of > > > > the Boolean column feature that creates a separate subset for each > > > > distinct > > > > value in the column? What I mean is, say a column contains a number from > > > > 1 > > > > to 10- what I'd like to see is an automatic way to create a subset for > > > > all > > > > the 1's, all the 2's, etc, all in one go. Is this possible? > > > > > > > > Hauke > > > > this is on the to-do list. I'll try to prioritise it. > > Hello all, > > I remember the make-subsets-from-distinct-values is a suggestion I made a > while ago, so I'm happy to see others push for it too ;o) 20 March 2008 in fact. So yes, it really does deserve to get to the top of that list some time soon... -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From JJ.Kavelaars@nrc-cnrc.XXX Mon Sep 20 10:42:54 2010 Date: Sat, 18 Sep 2010 19:11:02 -0700 From: "Kavelaars, JJ" To: topcat-user@sympa.XXX Subject: Accessing URLs protected via basic authentication from STILTS Hello, Is there a way to pass authentication credentials to stilts so that a protected URL can be used as input? I'm thinking of something like.... Stilts plot2d in=http://user:passwd@place.XXX JJ From m.b.taylor@bristol.XXX Mon Sep 20 10:09:30 2010 Date: Mon, 20 Sep 2010 10:09:30 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Accessing URLs protected via basic authentication from STILTS On Sat, 18 Sep 2010, Kavelaars, JJ wrote: > > Hello, > > Is there a way to pass authentication credentials to stilts so that a protected URL can be used as input? I'm thinking of something like.... > > Stilts plot2d in=http://user:passwd@place.XXX JJ, I think that formulation would work, *if* the server in question accepts simple user/password authentication along those lines, since if STILTS gets a URL as an input table value it just passes it straight on to java's URL resolution machinery, which in turn passes the request to the server in question in the standard way. If it's something more complicated (certificate-based?) then you'd have to work harder. I must admit, I'm far from expert on security issues, but if you post more details on what kind of authentication is protecting this service, then I or (preferably) someone better informed than me might be able to help. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Tue Sep 21 10:39:03 2010 Date: Tue, 21 Sep 2010 10:39:02 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Accessing URLs protected via basic authentication from STILTS On Mon, 20 Sep 2010, JJ Kavelaars wrote: > > On 2010-09-20, at 2:09 AM, Mark Taylor wrote: > > > On Sat, 18 Sep 2010, Kavelaars, JJ wrote: > > > >> > >> Hello, > >> > >> Is there a way to pass authentication credentials to stilts so that a protected URL can be used as input? I'm thinking of something like.... > >> > >> Stilts plot2d in=http://user:passwd@place.XXX > > > > JJ, > > > > I think that formulation would work, *if* the server in question > > accepts simple user/password authentication along those lines, > > since if STILTS gets a URL as an input table value it just passes > > it straight on to java's URL resolution machinery, which in turn > > passes the request to the server in question in the standard way. > > > I've attempted to use the above syntax for access to files protected via basic authentication and this does not work. Java responds with: > > Server returned HTTP response code: 401 for URL: http://user:passwd@place.XXX > > however > > curl `http://user:passwd@place.XXX` | stilts plot2d in=- ifmt=votable > > works.... so the access is correct but passing that directly into stilts (which then hands off to Java) doesn't work so well. OK - I thought that java might handle this itself, but it seems not. > there you need to do something like: > > BASE64Encoder enc = new sun.misc.BASE64Encoder(); > String userpassword = username + ":" + password; > String encodedAuthorization = enc.encode( userpassword.getBytes() ); > connection.setRequestProperty("Authorization", "Basic "+ > encodedAuthorization); > > > where you grab (via whatever method) username and password from someplace.... > > how to implement this is a developer / usage question. Since you've done the hard bit by supplying the code above, it's the least I can do to put it into the STIL codebase. Can you try the pre-release version at ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ and let me know if that seems to do the trick? Thanks for your input, Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From wheresmytab@gmail.XXX Wed Sep 22 00:10:32 2010 Date: Tue, 21 Sep 2010 23:10:24 +0100 From: wheresmytab@gmail.XXX To: topcat-user@sympa.XXX Subject: add row Short question. Is it possible to add a row to a table by hand? Thanks, ~Danny From jj.kavelaars@nrc.XXX Wed Sep 22 00:14:04 2010 Date: Tue, 21 Sep 2010 15:13:39 -0700 From: JJ Kavelaars To: topcat-user@sympa.XXX Subject: Re: add row If you edit the file by hand... yes. But if you want to add inside topcat. no. I've asked Mark about this before, seams you should be able to do this but, AFAIK, this is currently not a feature.... JJ On 2010-09-21, at 3:10 PM, wrote: > Short question. > Is it possible to add a row to a table by hand? > Thanks, > ~Danny > From jacobsda@sas.XXX Wed Sep 22 10:46:37 2010 Date: Tue, 21 Sep 2010 22:19:01 +0000 From: Danny Jacobs To: JJ Kavelaars Cc: topcat-user@sympa.XXX Subject: Re: add row Thanks JJ, you confirm my observation. I assume its on a long list of feature requests somewhere. ~Danny On Tue, Sep 21, 2010 at 10:13 PM, JJ Kavelaars wrote: > If you edit the file by hand... yes.  But if you want to add inside topcat.  no.  I've asked Mark about this before, seams you should be able to do this but, AFAIK, this is currently not a feature.... > > JJ > On 2010-09-21, at 3:10 PM, wrote: > >> Short question. >> Is it possible to add a row to a table by hand? >> Thanks, >> ~Danny >> > > > From S.Oliver@sussex.XXX Wed Sep 22 10:49:32 2010 Date: Wed, 22 Sep 2010 09:49:23 +0100 From: Seb Oliver To: jacobsda@sas.XXX Cc: JJ Kavelaars , topcat-user@sympa.XXX Subject: Re: add row presumably one could script this up in stilts quite easily by using head to get a file with one row and then merging with the orginal. On 21 Sep 2010, at 23:19, Danny Jacobs wrote: > Thanks JJ, you confirm my observation. I assume its on a long list of > feature requests somewhere. > ~Danny > > > On Tue, Sep 21, 2010 at 10:13 PM, JJ Kavelaars > wrote: >> If you edit the file by hand... yes. But if you want to add inside >> topcat. no. I've asked Mark about this before, seams you should >> be able to do this but, AFAIK, this is currently not a feature.... >> >> JJ >> On 2010-09-21, at 3:10 PM, > > wrote: >> >>> Short question. >>> Is it possible to add a row to a table by hand? >>> Thanks, >>> ~Danny >>> >> >> >> > From m.b.taylor@bristol.XXX Wed Sep 22 09:54:21 2010 Date: Wed, 22 Sep 2010 09:54:21 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: add row Rather near the bottom of said list I'm afraid. The assumption that tables keep the same number of rows is built in at quite a low level to the design, so this feature is not going to appear in the near future. Although for certain usage scenarios it's the sort of thing that's required, I believe that most TOPCAT use is for manipulating tables where the underlying data is basically fixed, so adding rows is a bit of a niche requirement. If you really needed to do it you could concatenate a table with a subset of itself ... but as JJ suggests, it's probably better to edit the table file in ascii format or something like that. Mark On Tue, 21 Sep 2010, Danny Jacobs wrote: > Thanks JJ, you confirm my observation. I assume its on a long list of > feature requests somewhere. > ~Danny > > > On Tue, Sep 21, 2010 at 10:13 PM, JJ Kavelaars wrote: > > If you edit the file by hand... yes.  But if you want to add inside topcat.  no.  I've asked Mark about this before, seams you should be able to do this but, AFAIK, this is currently not a feature.... > > > > JJ > > On 2010-09-21, at 3:10 PM, wrote: > > > >> Short question. > >> Is it possible to add a row to a table by hand? > >> Thanks, > >> ~Danny > >> > > > > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Sep 22 09:57:40 2010 Date: Wed, 22 Sep 2010 09:57:39 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: add row Good thinking Seb, that's another option. For a more convenient version of the same thing, I could provide a filter in STILTS to add a number of empty rows at the end of a table. If people are keen let me know and I'll do that. Mark On Wed, 22 Sep 2010, Seb Oliver wrote: > presumably one could script this up in stilts quite easily by using head to > get a file with one row and then merging with the orginal. > > > On 21 Sep 2010, at 23:19, Danny Jacobs wrote: > > > Thanks JJ, you confirm my observation. I assume its on a long list of > > feature requests somewhere. > > ~Danny > > > > > > On Tue, Sep 21, 2010 at 10:13 PM, JJ Kavelaars > > wrote: > > > If you edit the file by hand... yes. But if you want to add inside > > > topcat. no. I've asked Mark about this before, seams you should be able > > > to do this but, AFAIK, this is currently not a feature.... > > > > > > JJ > > > On 2010-09-21, at 3:10 PM, > > > wrote: > > > > > > > Short question. > > > > Is it possible to add a row to a table by hand? > > > > Thanks, > > > > ~Danny > > > > > > > > > > > > > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Wed Sep 22 13:12:17 2010 Date: Wed, 22 Sep 2010 12:08:48 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: add row Hi everybody, here comes my workaround in case you want to edit or add several lines keeping the column definition within TOPCAT. 1- Duplicate the original table 2- In the new copy of the table, edit by hand as many lines as you need. 3- Always in the copied table, define a subsample selecting the edited lines and select this new sample ("New Subset" -> "Add and Set Current subset") 4- Concatenate the original table with the new table in which only the edited lines are set (Check "Row Subset") A little bit messy but I think does the work :) Cheers, Jesús Mark Taylor ha escrito: > Good thinking Seb, that's another option. > > For a more convenient version of the same thing, I could provide a > filter in STILTS to add a number of empty rows at the end of a table. > If people are keen let me know and I'll do that. > > Mark > > On Wed, 22 Sep 2010, Seb Oliver wrote: > >> presumably one could script this up in stilts quite easily by using head to >> get a file with one row and then merging with the orginal. >> >> >> On 21 Sep 2010, at 23:19, Danny Jacobs wrote: >> >> > Thanks JJ, you confirm my observation. I assume its on a long list of >> > feature requests somewhere. >> > ~Danny >> > >> > >> > On Tue, Sep 21, 2010 at 10:13 PM, JJ Kavelaars >> > wrote: >> > > If you edit the file by hand... yes. But if you want to add inside >> > > topcat. no. I've asked Mark about this before, seams you >> should be able >> > > to do this but, AFAIK, this is currently not a feature.... >> > > >> > > JJ >> > > On 2010-09-21, at 3:10 PM, >> >> > > wrote: >> > > >> > > > Short question. >> > > > Is it possible to add a row to a table by hand? >> > > > Thanks, >> > > > ~Danny >> > > > >> > > >> > > >> > > >> > >> >> > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From jacobsda@sas.XXX Wed Sep 22 17:16:26 2010 Date: Wed, 22 Sep 2010 15:09:32 +0000 From: Danny Jacobs To: jmvarela@iac.XXX Cc: topcat-user@sympa.XXX Subject: Re: add row Excellent suggestions everyone. I'm using Jesús' suggestion, which uses the new copy feature, with fine results. I will be learning stilts though for future batch work. What a responsive group! Thanks everyone, ~Danny On Wed, Sep 22, 2010 at 11:08 AM, wrote: > > Hi everybody, > >  here comes my workaround in case you want to edit or add several lines > keeping the column definition within TOPCAT. > > 1- Duplicate the original table > 2- In the new copy of the table, edit by hand as many lines as you need. > 3- Always in the copied table, define a subsample selecting the edited lines > and select this new sample ("New Subset" -> "Add and Set Current subset") > 4- Concatenate the original table with the new table in which only the > edited lines are set (Check "Row Subset") > > A little bit messy but I think does the work :) > > Cheers, > > Jesús > > Mark Taylor ha escrito: > >> Good thinking Seb, that's another option. >> >> For a more convenient version of the same thing, I could provide a >> filter in STILTS to add a number of empty rows at the end of a table. >> If people are keen let me know and I'll do that. >> >> Mark >> >> On Wed, 22 Sep 2010, Seb Oliver wrote: >> >>> presumably one could script this up in stilts quite easily by using head >>> to >>> get a file with one row and then merging with the orginal. >>> >>> >>> On 21 Sep 2010, at 23:19, Danny Jacobs wrote: >>> >>> > Thanks JJ, you confirm my observation.  I assume its on a long list of >>> > feature requests somewhere. >>> > ~Danny >>> > >>> > >>> > On Tue, Sep 21, 2010 at 10:13 PM, JJ Kavelaars >>> > wrote: >>> > > If you edit the file by hand... yes.  But if you want to add inside >>> > > topcat.  no.  I've asked Mark about this before, seams you  should be >>> > > able >>> > > to do this but, AFAIK, this is currently not a feature.... >>> > > >>> > > JJ >>> > > On 2010-09-21, at 3:10 PM, >>> > >   >>> > > wrote: >>> > > >>> > > > Short question. >>> > > > Is it possible to add a row to a table by hand? >>> > > > Thanks, >>> > > > ~Danny >>> > > > >>> > > >>> > > >>> > > >>> > >>> >>> >> >> -- >> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> >> > > > > -- > > Dr. Jesús Varela López > Instituto de Astrofísica de Canarias (IAC) > E-38205 La Laguna, Tenerife, Spain > Phone:   +34 922 605 278 > Fax:     +34 922 605 210 > Website: http://www.iac.es > > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de > Datos, acceda a http://www.iac.es/disclaimer.php > WARNING: For more information on privacy and fulfilment of the Law > concerning the Protection of Data, consult > http://www.iac.es/disclaimer.php?lang=en > > > > From jmvarela@iac.XXX Fri Sep 24 19:04:11 2010 Date: Fri, 24 Sep 2010 18:04:00 +0100 From: jmvarela@iac.XXX To: topcat-user@bristol.XXX Subject: Matching table with duplications and request for matching entire lines Hi Mark, I've just discovered a weird behaviour when matching tables in which there are element with the same matching columns. The situation is as follows. I'm matching two tables in which one is a subsample of the other (so complete lines are the same in both tables). For example, Small_Table: ---------more objects ----------- a1 b1 c1 ----- RA DEC ----- y1 z1 a2 b2 c2 ----- RA DEC ----- y2 z2 ---------more objects ----------- Large_Table: -----much more objects ---------- a1 b1 c1 ----- RA DEC ----- y1 z1 a2 b2 c2 ----- RA DEC ----- y2 z2 -----much more objects ----------- So there are lines in which (RA,DEC) are the same but the rest of values in other columns are different. Since I'm used to, I've just used (RA,DEC) for the matching. And I got: Matching Small_Table - Large_Table: ----------------------matched objects ----------------------------- a1 b1 c1 ----- RA DEC ----- y1 z1 a1 b1 c1 ----- RA DEC ----- y1 z1 ----------------------matched objects ----------------------------- Therefore, losing the 2nd line in both tables. It took me a while to discover this issue. So my request it is if it would be possible to add an option for matching entire lines in the list of "Matching Algorithms". This could be connected with your proposal in another thread of providing a function that hasses an entire line. Thanks in advance! Jesús -- Dr. Jesús Varela López Instituto de Astrofísica de Canarias (IAC) E-38205 La Laguna, Tenerife, Spain Phone: +34 922 605 278 Fax: +34 922 605 210 Website: http://www.iac.es --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Mon Sep 27 09:56:51 2010 Date: Mon, 27 Sep 2010 09:56:50 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Matching table with duplications and request for matching entire lines > Jesús On Fri, 24 Sep 2010, jmvarela@iac.XXX wrote: > Hi Mark, > > I've just discovered a weird behaviour when matching tables in which there > are element with the same matching columns. > > The situation is as follows. I'm matching two tables in which one is a > subsample of the other (so complete lines are the same in both tables). For > example, this seems like a somewhat strange thing to do; no doubt you have your reasons, but I can't see it being a very popular requirement. Why do you want to match tables with (some) identical rows? Could whatever you're trying to do be achieved by using an internal match (with Eliminate All Grouped Rows or Eliminate All But First Of Each Group option)? > Small_Table: > > ---------more objects ----------- > a1 b1 c1 ----- RA DEC ----- y1 z1 > a2 b2 c2 ----- RA DEC ----- y2 z2 > ---------more objects ----------- > > Large_Table: > > -----much more objects ---------- > a1 b1 c1 ----- RA DEC ----- y1 z1 > a2 b2 c2 ----- RA DEC ----- y2 z2 > -----much more objects ----------- > > > So there are lines in which (RA,DEC) are the same but the rest of values in > other columns are different. > > Since I'm used to, I've just used (RA,DEC) for the matching. And I got: > > Matching Small_Table - Large_Table: > > ----------------------matched objects ----------------------------- > a1 b1 c1 ----- RA DEC ----- y1 z1 a1 b1 c1 ----- RA DEC ----- y1 z1 > ----------------------matched objects ----------------------------- > > Therefore, losing the 2nd line in both tables. You might get something more like what you want by selecting "All Matches" rather than "Best Match Only", thugh I'm not sure of exactly what you're after. > It took me a while to discover this issue. > > So my request it is if it would be possible to add an option for matching > entire lines in the list of "Matching Algorithms". This could be connected > with your proposal in another thread of providing a function that hasses an > entire line. Adding a whole-row hash would be relatively easy, I'll put it on the list; it wouldn't provide a guarantee of getting the matching right because of the possibility of hash collisions, but it could probably be made to have a pretty good chance of getting it right. A rigorous full-row match would be rather harder to fit into the existing framework. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Mon Sep 27 12:14:56 2010 Date: Mon, 27 Sep 2010 11:14:13 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Matching table with duplications and request for matching entire lines Hi Mark, Quoting Mark Taylor : >> Jesús > On Fri, 24 Sep 2010, jmvarela@iac.XXX wrote: > >> Hi Mark, >> >> I've just discovered a weird behaviour when matching tables in which there >> are element with the same matching columns. >> >> The situation is as follows. I'm matching two tables in which one is a >> subsample of the other (so complete lines are the same in both tables). For >> example, > > this seems like a somewhat strange thing to do; no doubt you have your > reasons, but I can't see it being a very popular requirement. Why do > you want to match tables with (some) identical rows? Could whatever > you're trying to do be achieved by using an internal match > (with Eliminate All Grouped Rows or Eliminate All But First Of Each Group > option)? > My situation was the following. I had an old table and then I produced a new larger one that should contain the old one and I wanted to check whether actually the first small one was completely inside the second larger one. That's the reason why matches should be better done row by row. The other reason is that the catalogues contain galaxies in different structures and the table contains data of the galaxies and of the structures in which they are, but one galaxy can be simultaneously in two structures so the data of the galaxy is repeated (and therefore the coordinates) but the data of the structures change. >> Small_Table: >> >> ---------more objects ----------- >> a1 b1 c1 ----- RA DEC ----- y1 z1 >> a2 b2 c2 ----- RA DEC ----- y2 z2 >> ---------more objects ----------- >> >> Large_Table: >> >> -----much more objects ---------- >> a1 b1 c1 ----- RA DEC ----- y1 z1 >> a2 b2 c2 ----- RA DEC ----- y2 z2 >> -----much more objects ----------- >> >> >> So there are lines in which (RA,DEC) are the same but the rest of values in >> other columns are different. >> >> Since I'm used to, I've just used (RA,DEC) for the matching. And I got: >> >> Matching Small_Table - Large_Table: >> >> ----------------------matched objects ----------------------------- >> a1 b1 c1 ----- RA DEC ----- y1 z1 a1 b1 c1 ----- RA DEC ----- y1 z1 >> ----------------------matched objects ----------------------------- >> >> Therefore, losing the 2nd line in both tables. > > You might get something more like what you want by selecting > "All Matches" rather than "Best Match Only", thuogh I'm not sure of > exactly what you're after. > I could try although at the end I found a workaround generating a new column combining different columns to produce a unique value for each row and using this value to do the matching. This remained me that once you proposed to add a function to produce a unique value per row. >> It took me a while to discover this issue. >> >> So my request it is if it would be possible to add an option for matching >> entire lines in the list of "Matching Algorithms". This could be connected >> with your proposal in another thread of providing a function that hasses an >> entire line. > > Adding a whole-row hash would be relatively easy, I'll put it on the list; > it wouldn't provide a guarantee of getting the matching right because > of the possibility of hash collisions, but it could probably be made > to have a pretty good chance of getting it right. A rigorous full-row > match would be rather harder to fit into the existing framework. > I think this would be ok. I already thought that maybe this is not very popular thing to do but I thought it would be interesting to share it :) Cheers! Jesús --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From ipf@iac.XXX Wed Sep 29 19:25:51 2010 Date: Wed, 29 Sep 2010 18:25:35 +0100 From: Ismael Perez Fournon To: Mark Taylor Cc: topcat-user@bristol.XXX "ipf >> Ismael Perez Fournon" Subject: HTML output in HTML sortable table? Mark, something to add to the topcat wish list, that the output tables in HTML format are sortable by the column contents. Is this easy to implement? Thanks, Ismael Mark Taylor wrote: > Ismael, > > On Thu, 2 Sep 2010, ipf@iac.XXX wrote: > > >> Hi Mark, >> >> a question about URLs in tables in HTML format. >> >> I have topcat tables with active links to URLs. When I save them >> in HTML format, the URLs are saved as in the original topcat table but >> not in the HTML format that makes the link work in a browser. >> >> Is there a way in topcat or with other programmes to write the >> appropriate HTML format so that the links work? >> > > That's a good idea. I've made an adjustment to the HTML output handler > so that cell contents which look like URLs get output as HTML links > instead of plain text. This will be in the next public release; > in the mean time, you can get a pre-release copy with this function > from > > ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ > > Thanks for the suggestion. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci?n de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Thu Sep 30 09:42:16 2010 Date: Thu, 30 Sep 2010 09:42:16 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: HTML output in HTML sortable table? On Wed, 29 Sep 2010, Ismael Perez Fournon wrote: > Mark, > > something to add to the topcat wish list, that the output tables in HTML > format > are sortable by the column contents. Is this easy to implement? Ismael, thanks for the suggestion, but this is probably a bit outside the scope of TOPCAT. The HTML output is fairly quick and dirty; if you want to reconfigure the presentation significantly I think it would be more appropriate to use some other software. The most obvious (though not necessarily easiest) way to do this would be to take VOTable output and write an XSLT stylesheet to turn it into whatever kind of all-singing all-dancing HTML result you want (I guess some javascript would be involved to effect row sorting? I'm no expert on clever HTML). I'm sure similar things have been done before - possibly by voview (http://code.google.com/p/voview/)? Alternatively you might get away with just taking the HTML output and stuffing an appropriate bit of code near the top. Having said that, if somebody who actually knows about this stuff can persuade me it's a small amount of effort to add it into the STIL HTML output handler, I might be persuaded. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From eglez@ast.XXX Thu Sep 30 11:14:14 2010 Date: Thu, 30 Sep 2010 10:14:04 +0100 From: Eduardo Gonzalez To: Mark Taylor Cc: Eduardo Gonzalez , topcat-user@bristol.XXX Subject: Re: HTML output in HTML sortable table? Hi Mark, There are several jQuery plugins to do this, e.g. http://tablesorter.com/docs/. The only changes necessary are to add some line to the HEAD tag and and ID to the table. I will send you an example a bit later. But not sure if you want to add that to TOPCAT. Maybe the easiest is to add some example to the documentation so that users can do it (so that they can also selected the theme and several parameters). Cheers, Eduardo On 30 Sep 2010, at 09:42, Mark Taylor wrote: > On Wed, 29 Sep 2010, Ismael Perez Fournon wrote: > >> Mark, >> >> something to add to the topcat wish list, that the output tables in HTML >> format >> are sortable by the column contents. Is this easy to implement? > > Ismael, > > thanks for the suggestion, but this is probably a bit outside the scope > of TOPCAT. The HTML output is fairly quick and dirty; if you want to > reconfigure the presentation significantly I think it would be more > appropriate to use some other software. The most obvious (though not > necessarily easiest) way to do this would be to take VOTable output > and write an XSLT stylesheet to turn it into whatever kind of > all-singing all-dancing HTML result you want (I guess some javascript > would be involved to effect row sorting? I'm no expert on clever HTML). > I'm sure similar things have been done before - possibly by voview > (http://code.google.com/p/voview/)? > > Alternatively you might get away with just taking the HTML output > and stuffing an appropriate bit of code near the top. > > Having said that, if somebody who actually knows about this stuff > can persuade me it's a small amount of effort to add it into the > STIL HTML output handler, I might be persuaded. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ Eduardo Gonzalez-Solares Institute of Astronomy, Cambridge Astronomy Survey Unit Madingley Rd, CB3 0HA, UK ONLINE INFO: web ∴ blog ∴ skype There are two ways to write error-free programs. Only the third one works. From eglez@ast.XXX Thu Sep 30 11:23:25 2010 Date: Thu, 30 Sep 2010 10:23:17 +0100 From: Eduardo Gonzalez To: Mark Taylor Cc: Eduardo Gonzalez , topcat-user@bristol.XXX Subject: Re: HTML output in HTML sortable table? Mark, Find below my modification to make the tables sortable. Cheers, Eduardo

...etc... On 30 Sep 2010, at 09:42, Mark Taylor wrote: > On Wed, 29 Sep 2010, Ismael Perez Fournon wrote: > >> Mark, >> >> something to add to the topcat wish list, that the output tables in HTML >> format >> are sortable by the column contents. Is this easy to implement? > > Ismael, > > thanks for the suggestion, but this is probably a bit outside the scope > of TOPCAT. The HTML output is fairly quick and dirty; if you want to > reconfigure the presentation significantly I think it would be more > appropriate to use some other software. The most obvious (though not > necessarily easiest) way to do this would be to take VOTable output > and write an XSLT stylesheet to turn it into whatever kind of > all-singing all-dancing HTML result you want (I guess some javascript > would be involved to effect row sorting? I'm no expert on clever HTML). > I'm sure similar things have been done before - possibly by voview > (http://code.google.com/p/voview/)? > > Alternatively you might get away with just taking the HTML output > and stuffing an appropriate bit of code near the top. > > Having said that, if somebody who actually knows about this stuff > can persuade me it's a small amount of effort to add it into the > STIL HTML output handler, I might be persuaded. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ Eduardo Gonzalez eglez@ast.XXX From m.b.taylor@bristol.XXX Thu Sep 30 14:55:35 2010 Date: Thu, 30 Sep 2010 14:55:35 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: HTML output in HTML sortable table? Eduardo, thanks for that and the sample code. In line with your comments and my earlier ones I don't think I'll add it to the TOPCAT codebase, but I may put a reference in the FAQ or documentation somewhere. In the mean time, this should provide the functionality that Ismael was after. Mark On Thu, 30 Sep 2010, Eduardo Gonzalez wrote: > Hi Mark, > > There are several jQuery plugins to do this, e.g. http://tablesorter.com/docs/. The only changes necessary are to add some line to the HEAD tag and and ID to the table. I will send you an example a bit later. > > But not sure if you want to add that to TOPCAT. Maybe the easiest is to add some example to the documentation so that users can do it (so that they can also selected the theme and several parameters). > > Cheers, > > Eduardo > > On 30 Sep 2010, at 09:42, Mark Taylor wrote: > > > On Wed, 29 Sep 2010, Ismael Perez Fournon wrote: > > > >> Mark, > >> > >> something to add to the topcat wish list, that the output tables in HTML > >> format > >> are sortable by the column contents. Is this easy to implement? > > > > Ismael, > > > > thanks for the suggestion, but this is probably a bit outside the scope > > of TOPCAT. The HTML output is fairly quick and dirty; if you want to > > reconfigure the presentation significantly I think it would be more > > appropriate to use some other software. The most obvious (though not > > necessarily easiest) way to do this would be to take VOTable output > > and write an XSLT stylesheet to turn it into whatever kind of > > all-singing all-dancing HTML result you want (I guess some javascript > > would be involved to effect row sorting? I'm no expert on clever HTML). > > I'm sure similar things have been done before - possibly by voview > > (http://code.google.com/p/voview/)? > > > > Alternatively you might get away with just taking the HTML output > > and stuffing an appropriate bit of code near the top. > > > > Having said that, if somebody who actually knows about this stuff > > can persuade me it's a small amount of effort to add it into the > > STIL HTML output handler, I might be persuaded. > > > > Mark > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > Eduardo Gonzalez-Solares > Institute of Astronomy, Cambridge Astronomy Survey Unit > Madingley Rd, CB3 0HA, UK > ONLINE INFO: web ∴ blog ∴ skype > There are two ways to write error-free programs. Only the third one works. > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From fran.jimenez-esteban@cab.XXX Thu Sep 30 22:34:24 2010 Date: Thu, 30 Sep 2010 18:30:17 +0100 From: fran.jimenez-esteban@cab.XXX To: topcat-user@sympa.XXX Subject: different result with STILTS than with TopCat Dear Mark, I am obtaining different result when doing (I think) the same thing with Topcat and Stilt. I am downloading data from Tycho-2. --- Topcat: VO-Visier Catalog Service Cone selection: RA 0.00000 deg Deg 80.0000 deg Radius: 10.0000 deg Maximum Row Count: unlimited Catalog Selection: Surveys Tycho-2 It returns a table with data in a 10deg radius circle around (0.0,80.0) Perfect. -- Stilts: stilts coneskymatch ifmt=ascii in=coord.txt ra=RA dec=DEC sr=10.0000 out=0.00000_80.0000.xml serviceurl=http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/259 find=all Coord.txt fiel: # RA DEC 0.00000 85.0000 Its returns only part of the data, only part of the circle !?!? Regards, Fran. From JJ.Kavelaars@nrc-cnrc.XXX Fri Oct 1 00:04:05 2010 Date: Thu, 30 Sep 2010 15:03:17 -0700 From: "Kavelaars, JJ" To: "" Cc: "topcat-user@sympa.XXX" Subject: Re: different result with STILTS than with TopCat Hi Fran, do you mean to have DEG=80 in the Topcat search and but 85 in the STITLS version? JJ On 2010-09-30, at 10:30 AM, wrote: > Dear Mark, > I am obtaining different result when doing (I think) the same thing with Topcat > and Stilt. I am downloading data from Tycho-2. > > --- Topcat: > VO-Visier Catalog Service > Cone selection: RA 0.00000 deg > Deg 80.0000 deg > Radius: 10.0000 deg > Maximum Row Count: unlimited > Catalog Selection: Surveys Tycho-2 > > It returns a table with data in a 10deg radius circle around (0.0,80.0) > Perfect. > > -- Stilts: > stilts coneskymatch ifmt=ascii in=coord.txt ra=RA dec=DEC sr=10.0000 > out=0.00000_80.0000.xml > serviceurl=http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/259 find=all > > Coord.txt fiel: > # RA DEC > 0.00000 85.0000 > > Its returns only part of the data, only part of the circle !?!? > > > Regards, > Fran. > From m.b.taylor@bristol.XXX Fri Oct 1 10:17:32 2010 Date: Fri, 1 Oct 2010 10:17:32 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Cc: "" Subject: Re: different result with STILTS than with TopCat Fran, As JJ points out, your positions are different. I'll assume that you mean RA=80 in both cases. You noted that your TOPCAT VizieR search used a Maximum Row Count of unlimited. This gives you 19228 points, a nice circle if you plot it. If you use the default limit (50000), then you get only 12062 points, a crescent shape. So, the limit is important. I don't know why a 50k limit cuts off around 12000, I'll leave that for any VizieR experts listening to comment on if they wish. The STILTS command doesn't have the row limit option (since no such limit is defined by the cone search protocol it's based on), and so uses VizieR's default limit, which is not unlimited (I think it corresponds to the 50000 value). So, you get only 12061 points here (not certain why it's 1 less than in TOPCAT, some rounding error probably). You can adjust the limit when using the cone search URL by reading the VizieR option usage at http://vizier.u-strasbg.fr/doc/asu-summary.htx. Or, if you supply the "-verbose" option to either TOPCAT or STILTS, they will print logging messages that tell you exactly what queries they've made, and you can crib from there. Either way, you'll find that appending "&-out.max=unlimited" to your coneskymatch serviceurl parameter will do the trick. Having said that, coneskymatch is a more complicated tool than you need to do a single cone search. You can juse use curl/wget or a web browser with the service URL and "&RA=0&DEC=80&SR=10" to retrieve the VOTable. Alternatively, if you want, you can supply the URL to any STILTS command as a table input, so for instance: stilts tpipe in='http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/259&-out.max=unlimited&RA=0&DEC=80&SR=10' out=0-80.fits will retrieve the VOTable from the VizieR service and output it as a FITS file. Mark On Thu, 30 Sep 2010, Kavelaars, JJ wrote: > > Hi Fran, > > do you mean to have DEG=80 in the Topcat search and but 85 in the STITLS version? > > JJ > > On 2010-09-30, at 10:30 AM, wrote: > > > Dear Mark, > > I am obtaining different result when doing (I think) the same thing with Topcat > > and Stilt. I am downloading data from Tycho-2. > > > > --- Topcat: > > VO-Visier Catalog Service > > Cone selection: RA 0.00000 deg > > Deg 80.0000 deg > > Radius: 10.0000 deg > > Maximum Row Count: unlimited > > Catalog Selection: Surveys Tycho-2 > > > > It returns a table with data in a 10deg radius circle around (0.0,80.0) > > Perfect. > > > > -- Stilts: > > stilts coneskymatch ifmt=ascii in=coord.txt ra=RA dec=DEC sr=10.0000 > > out=0.00000_80.0000.xml > > serviceurl=http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/259 find=all > > > > Coord.txt fiel: > > # RA DEC > > 0.00000 85.0000 > > > > Its returns only part of the data, only part of the circle !?!? > > > > > > Regards, > > Fran. > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From sebastien.derriere@astro.XXX Fri Oct 1 15:47:08 2010 Date: Fri, 01 Oct 2010 15:50:47 +0200 From: Sebastien Derriere To: Mark Taylor Cc: topcat-user@bristol.XXX "" , Francois Ochsenbein , "Question@Simbad (CDS Helpdesk)" Subject: Re: different result with STILTS than with TopCat Hello all, Maybe I can help with the VizieR issue : there is indeed a VizieR bug when you have a proper mixing of : 1) querying certain catalogues (does not affect very large ones) 2) a search area containing a celestial pole (as is the case for your query, if the cone stops at +89.999 declination, no problem) 3) combining a large search radius with a limited number of tuples in the output What happens is that the row limit is processed too early in the selection and this causes for example to *test* a max of 50k rows instead of *selecting* a max of 50k rows. Hence the strange results you have noticed. We'll try to fix it shortly. If you want to avoid this problem, just use a max rows value larger than the catalogue size, and it will be ok. Also, be careful that catalogue I/259 contains several tables, so make sure to precise I/259/tyc2 if you want just the main table without the supplements. Sebastien. On 10/01/2010 11:17 AM, Mark Taylor wrote: > Fran, > > As JJ points out, your positions are different. I'll assume that > you mean RA=80 in both cases. > > You noted that your TOPCAT VizieR search used a Maximum Row Count > of unlimited. This gives you 19228 points, a nice circle if you > plot it. If you use the default limit (50000), then you get only > 12062 points, a crescent shape. So, the limit is important. > I don't know why a 50k limit cuts off around 12000, I'll leave that > for any VizieR experts listening to comment on if they wish. > > The STILTS command doesn't have the row limit option (since no such > limit is defined by the cone search protocol it's based on), and so uses > VizieR's default limit, which is not unlimited (I think it corresponds > to the 50000 value). So, you get only 12061 points here (not certain > why it's 1 less than in TOPCAT, some rounding error probably). > > You can adjust the limit when using the cone search URL by reading the > VizieR option usage at http://vizier.u-strasbg.fr/doc/asu-summary.htx. > Or, if you supply the "-verbose" option to either TOPCAT or STILTS, > they will print logging messages that tell you exactly what queries > they've made, and you can crib from there. Either way, you'll find > that appending "&-out.max=unlimited" to your coneskymatch serviceurl > parameter will do the trick. > > Having said that, coneskymatch is a more complicated tool than you > need to do a single cone search. You can juse use curl/wget or a > web browser with the service URL and "&RA=0&DEC=80&SR=10" to retrieve > the VOTable. Alternatively, if you want, you can supply the URL to > any STILTS command as a table input, so for instance: > > stilts tpipe in='http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/259&-out.max=unlimited&RA=0&DEC=80&SR=10' > out=0-80.fits > > will retrieve the VOTable from the VizieR service and output it > as a FITS file. > > Mark > > On Thu, 30 Sep 2010, Kavelaars, JJ wrote: > >> >> Hi Fran, >> >> do you mean to have DEG=80 in the Topcat search and but 85 in the STITLS version? >> >> JJ >> >> On 2010-09-30, at 10:30 AM, wrote: >> >>> Dear Mark, >>> I am obtaining different result when doing (I think) the same thing with Topcat >>> and Stilt. I am downloading data from Tycho-2. >>> >>> --- Topcat: >>> VO-Visier Catalog Service >>> Cone selection: RA 0.00000 deg >>> Deg 80.0000 deg >>> Radius: 10.0000 deg >>> Maximum Row Count: unlimited >>> Catalog Selection: Surveys Tycho-2 >>> >>> It returns a table with data in a 10deg radius circle around (0.0,80.0) >>> Perfect. >>> >>> -- Stilts: >>> stilts coneskymatch ifmt=ascii in=coord.txt ra=RA dec=DEC sr=10.0000 >>> out=0.00000_80.0000.xml >>> serviceurl=http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/259 find=all >>> >>> Coord.txt fiel: >>> # RA DEC >>> 0.00000 85.0000 >>> >>> Its returns only part of the data, only part of the circle !?!? >>> >>> >>> Regards, >>> Fran. >>> >> >> >> > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > -- ((( Sebastien Derriere sebastien.derriere@astro.XXX (. .) Observatoire de Strasbourg Phone +33 (0) 368 852 444 (( v )) 11, rue de l'universite Telefax +33 (0) 368 852 417 ---m-m--- F-67000 Strasbourg France From sebastien.derriere@astro.XXX Fri Oct 1 16:58:03 2010 Date: Fri, 01 Oct 2010 17:01:36 +0200 From: Sebastien Derriere To: topcat-user@bristol.XXX Subject: SQL queries from TopCat Hello Mark, I think there is still a bug with the way SQL direct access can be done through TopCat with the latest version. I am making tests on a Sybase installation and driver jconn3.jar, launching TopCat with all the proper arguments : java -classpath pathto/topcat-full.jar:pathto/jconn3.jar -Djdbc.drivers=com.sybase.jdbc3.jdbc.SybDriver uk.ac.starlink.topcat.Driver With TopCat 3.5-1, I can get it to work by typing in the Load new table > Location the full jdbc string which looks like : jdbc:sybase:Tds:hostname:port/dbname?user=XXX&password=XXX#select * from mytable where... However, with version 3.5-2, the dedicated "Open JDBC table" window does not work with the following error : Error making connection jdbc:sybase:Tds://hostname:port/dbname As you can see this is probably due to the // before hostname. Now with TopCat version 3.6, I can't get it to work at all ! Load new table > Location fails with this error : java.io.FileNotFoundException: Not extant file or valid URL: jdbc:sybase:Tds:hostname:port/dbname?user=XXX&password=XXX at uk.ac.starlink.util.DataSource.makeDataSource(DataSource.java:613) at uk.ac.starlink.table.gui.TableLoadChooser$8.attemptLoads(TableLoadChooser.java:587) at uk.ac.starlink.table.gui.LoadWorker$1.run(LoadWorker.java:82) Caused by: java.net.MalformedURLException: unknown protocol: jdbc at java.net.URL.(URL.java:574) at java.net.URL.(URL.java:464) at java.net.URL.(URL.java:413) at uk.ac.starlink.util.DataSource.makeDataSource(DataSource.java:609) ... 2 more The dedicated "Open JDBC table" window still does not work with the following error : java.io.IOException: Error making connection jdbc:sybase:Tds://hostname:port/dbname - known JDBC drivers: sun.jdbc.odbc.JdbcOdbcDriver com.sybase.jdbc3.jdbc.SybDriver at uk.ac.starlink.table.jdbc.JDBCHandler.makeStarTable(JDBCHandler.java:122) at uk.ac.starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java:485) at uk.ac.starlink.table.gui.SQLReadDialog$1.attemptLoads(SQLReadDialog.java:67) at uk.ac.starlink.table.gui.LoadWorker$1.run(LoadWorker.java:82) Caused by: java.sql.SQLException: JZ003: Le format de l'URL est incorrect at com.sybase.jdbc3.jdbc.ErrorMessage.raiseError(Unknown Source) at com.sybase.jdbc3.tds.Tds.login(Unknown Source) at com.sybase.jdbc3.jdbc.SybConnection.a(Unknown Source) at com.sybase.jdbc3.jdbc.SybConnection.a(Unknown Source) at com.sybase.jdbc3.jdbc.SybConnection.(Unknown Source) at com.sybase.jdbc3.jdbc.SybConnection.(Unknown Source) at com.sybase.jdbc3.jdbc.SybDriver.connect(Unknown Source) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:185) at uk.ac.starlink.table.jdbc.JDBCHandler.getConnection(JDBCHandler.java:167) at uk.ac.starlink.table.jdbc.JDBCHandler.access$000(JDBCHandler.java:14) at uk.ac.starlink.table.jdbc.JDBCHandler$1.getConnection(JDBCHandler.java:73) at uk.ac.starlink.table.jdbc.JDBCStarTable.(JDBCStarTable.java:74) at uk.ac.starlink.table.jdbc.JDBCHandler.makeStarTable(JDBCHandler.java:91) ... 3 more I remember when I first discussed this with you a while ago you said it was quite difficult to have proper URL parsing, and I guess this is what is at stake here. Do you think you could build the jdbc string without the // that is causing trouble, without breaking other more used expressions in the "Location" dialog ? Sebastien -- ((( Sebastien Derriere sebastien.derriere@astro.XXX (. .) Observatoire de Strasbourg Phone +33 (0) 368 852 444 (( v )) 11, rue de l'universite Telefax +33 (0) 368 852 417 ---m-m--- F-67000 Strasbourg France From m.b.taylor@bristol.XXX Fri Oct 1 18:35:03 2010 Date: Fri, 1 Oct 2010 18:35:02 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: SQL queries from TopCat Hi Sebastien, thanks for these very useful reports. There are two items: 1. jdbc: locations don't work at all since v3.6 in the Location field. Oops! You're right, this was an error I introduced when allowing it to load multiple tables at once. Sorry about that. Now fixed. 2. The SQL read/write dialogue boxes make it impossible to enter jdbc URLs appropriate for some (likely, most) JDBC drivers. Quite right again. I can't remember what I said to you last time, but it's clear to me now that those dialogue boxes have the wrong form. I need to rework them so that you can enter a more freeform URL base: I think that just collapsing the "Protocol", "Host" and "Database name" fields into a single "base URL" field will do the trick. (Another issue is that the SQL Query field is not big enough, but that's a different matter). As it happens I've recently reworked all the load dialogues, so this is a good time to fix this. I'll get on it. I'll let you know when there is a version with these fixes in. Mark On Fri, 1 Oct 2010, Sebastien Derriere wrote: > > Hello Mark, > > I think there is still a bug with the way SQL direct access can > be done through TopCat with the latest version. > > I am making tests on a Sybase installation and driver jconn3.jar, > launching TopCat with all the proper arguments : > java -classpath pathto/topcat-full.jar:pathto/jconn3.jar > -Djdbc.drivers=com.sybase.jdbc3.jdbc.SybDriver uk.ac.starlink.topcat.Driver > > > With TopCat 3.5-1, I can get it to work by typing in the > Load new table > Location the full jdbc string which looks like : > > jdbc:sybase:Tds:hostname:port/dbname?user=XXX&password=XXX#select * from > mytable where... > > However, with version 3.5-2, the dedicated "Open JDBC table" window > does not work with the following error : > Error making connection jdbc:sybase:Tds://hostname:port/dbname > > As you can see this is probably due to the // before hostname. > > > Now with TopCat version 3.6, I can't get it to work at all ! > Load new table > Location fails with this error : > > java.io.FileNotFoundException: Not extant file or valid URL: > jdbc:sybase:Tds:hostname:port/dbname?user=XXX&password=XXX > at uk.ac.starlink.util.DataSource.makeDataSource(DataSource.java:613) > at > uk.ac.starlink.table.gui.TableLoadChooser$8.attemptLoads(TableLoadChooser.java:587) > at uk.ac.starlink.table.gui.LoadWorker$1.run(LoadWorker.java:82) > Caused by: java.net.MalformedURLException: unknown protocol: jdbc > at java.net.URL.(URL.java:574) > at java.net.URL.(URL.java:464) > at java.net.URL.(URL.java:413) > at uk.ac.starlink.util.DataSource.makeDataSource(DataSource.java:609) > ... 2 more > > The dedicated "Open JDBC table" window still does not work with the following > error : > > java.io.IOException: Error making connection > jdbc:sybase:Tds://hostname:port/dbname - known JDBC drivers: > sun.jdbc.odbc.JdbcOdbcDriver > com.sybase.jdbc3.jdbc.SybDriver > > at > uk.ac.starlink.table.jdbc.JDBCHandler.makeStarTable(JDBCHandler.java:122) > at > uk.ac.starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java:485) > at > uk.ac.starlink.table.gui.SQLReadDialog$1.attemptLoads(SQLReadDialog.java:67) > at uk.ac.starlink.table.gui.LoadWorker$1.run(LoadWorker.java:82) > Caused by: java.sql.SQLException: JZ003: Le format de l'URL est incorrect > at com.sybase.jdbc3.jdbc.ErrorMessage.raiseError(Unknown Source) > at com.sybase.jdbc3.tds.Tds.login(Unknown Source) > at com.sybase.jdbc3.jdbc.SybConnection.a(Unknown Source) > at com.sybase.jdbc3.jdbc.SybConnection.a(Unknown Source) > at com.sybase.jdbc3.jdbc.SybConnection.(Unknown Source) > at com.sybase.jdbc3.jdbc.SybConnection.(Unknown Source) > at com.sybase.jdbc3.jdbc.SybDriver.connect(Unknown Source) > at java.sql.DriverManager.getConnection(DriverManager.java:582) > at java.sql.DriverManager.getConnection(DriverManager.java:185) > at > uk.ac.starlink.table.jdbc.JDBCHandler.getConnection(JDBCHandler.java:167) > at > uk.ac.starlink.table.jdbc.JDBCHandler.access$000(JDBCHandler.java:14) > at > uk.ac.starlink.table.jdbc.JDBCHandler$1.getConnection(JDBCHandler.java:73) > at > uk.ac.starlink.table.jdbc.JDBCStarTable.(JDBCStarTable.java:74) > at > uk.ac.starlink.table.jdbc.JDBCHandler.makeStarTable(JDBCHandler.java:91) > ... 3 more > > > I remember when I first discussed this with you a while ago you said > it was quite difficult to have proper URL parsing, and I guess this is > what is at stake here. > Do you think you could build the jdbc string without the // that > is causing trouble, without breaking other more used expressions in > the "Location" dialog ? > > Sebastien > > > -- > > ((( Sebastien Derriere sebastien.derriere@astro.XXX > (. .) Observatoire de Strasbourg Phone +33 (0) 368 852 444 > (( v )) 11, rue de l'universite Telefax +33 (0) 368 852 417 > ---m-m--- F-67000 Strasbourg France -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Francois.Ochsenbein@unistra.XXX Sun Oct 3 11:23:34 2010 Date: Sat, 02 Oct 2010 08:26:14 +0200 From: Francois Ochsenbein To: topcat-user@bristol.XXX Subject: Re: different result with STILTS than with TopCat Hi Mark et al., Sébastien showed to me the problem of the incomplete selection on the Tycho-2 when the pole is included; his diagnostic is right, it is in fact a limitation on the number of tuples to read in the database. The origin of the problem is located and will be fixed, but it could take a few weeks because we're just in the process of testing a new vizier interface... Still about the Tycho2 it would likely be better to merge the main part and the appendix into a single table (with an additional column telling the origin of the row: main, apx1 or 2). Any objection to that ? I just registered to the top-cat-user list -- not only I'm a use of tapcat (so easy to get statistical views and plots), also topcat is now among the 5top agents using vizier ;-) Cheers, François -- ====================================================================== Francois Ochsenbein --- Observatoire Astronomique de Strasbourg francois.ochsenbein@astro.XXX +33 (0)3 68 85 24 29 ====================================================================== From m.b.taylor@bristol.XXX Mon Oct 4 09:25:52 2010 Date: Mon, 4 Oct 2010 09:25:52 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: different result with STILTS than with TopCat Francois, On Sat, 2 Oct 2010, Francois Ochsenbein wrote: > Still about the Tycho2 it would likely be better to merge the > main part and the appendix into a single table (with an additional > column telling the origin of the row: main, apx1 or 2). > Any objection to that ? that sounds like good sense to me, I think it should make it easier to use that catalogue. > I just registered to the top-cat-user list -- not only I'm a > use of tapcat (so easy to get statistical views and plots), > also topcat is now among the 5top agents using vizier ;-) interesting, thanks for passing that on. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From fran.jimenez-esteban@cab.XXX Tue Oct 5 10:24:35 2010 Date: Tue, 05 Oct 2010 09:35:22 +0200 From: Fran Jimenez-Esteban To: Sebastien Derriere Cc: Mark Taylor , topcat-user@bristol.XXX Francois Ochsenbein , "Question@Simbad (CDS Helpdesk)" Subject: Re: different result with STILTS than with TopCat Thank you Sebastian and Mark for your instructions and advices. Regards, Francisco. Sebastien Derriere wrote: > > Hello all, > > Maybe I can help with the VizieR issue : there is indeed a VizieR bug > when you have a proper mixing of : > 1) querying certain catalogues (does not affect very large ones) > 2) a search area containing a celestial pole (as is the case for > your query, if the cone stops at +89.999 declination, no problem) > 3) combining a large search radius with a limited number of tuples > in the output > > What happens is that the row limit is processed too early in the > selection and this causes for example to *test* a max of 50k rows > instead of *selecting* a max of 50k rows. Hence the strange results > you have noticed. > > We'll try to fix it shortly. If you want to avoid this problem, > just use a max rows value larger than the catalogue size, and it > will be ok. > > Also, be careful that catalogue I/259 contains several tables, so > make sure to precise I/259/tyc2 if you want just the main table > without the supplements. > > Sebastien. > > > On 10/01/2010 11:17 AM, Mark Taylor wrote: >> Fran, >> >> As JJ points out, your positions are different. I'll assume that >> you mean RA=80 in both cases. >> >> You noted that your TOPCAT VizieR search used a Maximum Row Count >> of unlimited. This gives you 19228 points, a nice circle if you >> plot it. If you use the default limit (50000), then you get only >> 12062 points, a crescent shape. So, the limit is important. >> I don't know why a 50k limit cuts off around 12000, I'll leave that >> for any VizieR experts listening to comment on if they wish. >> >> The STILTS command doesn't have the row limit option (since no such >> limit is defined by the cone search protocol it's based on), and so uses >> VizieR's default limit, which is not unlimited (I think it corresponds >> to the 50000 value). So, you get only 12061 points here (not certain >> why it's 1 less than in TOPCAT, some rounding error probably). >> >> You can adjust the limit when using the cone search URL by reading the >> VizieR option usage at http://vizier.u-strasbg.fr/doc/asu-summary.htx. >> Or, if you supply the "-verbose" option to either TOPCAT or STILTS, >> they will print logging messages that tell you exactly what queries >> they've made, and you can crib from there. Either way, you'll find >> that appending "&-out.max=unlimited" to your coneskymatch serviceurl >> parameter will do the trick. >> >> Having said that, coneskymatch is a more complicated tool than you >> need to do a single cone search. You can juse use curl/wget or a >> web browser with the service URL and "&RA=0&DEC=80&SR=10" to retrieve >> the VOTable. Alternatively, if you want, you can supply the URL to >> any STILTS command as a table input, so for instance: >> >> stilts tpipe >> in='http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/259&-out.max=unlimited&RA=0&DEC=80&SR=10' >> >> out=0-80.fits >> >> will retrieve the VOTable from the VizieR service and output it >> as a FITS file. >> >> Mark >> >> On Thu, 30 Sep 2010, Kavelaars, JJ wrote: >> >>> >>> Hi Fran, >>> >>> do you mean to have DEG=80 in the Topcat search and but 85 in the >>> STITLS version? >>> >>> JJ >>> >>> On 2010-09-30, at 10:30 AM, >>> wrote: >>> >>>> Dear Mark, >>>> I am obtaining different result when doing (I think) the same thing >>>> with Topcat >>>> and Stilt. I am downloading data from Tycho-2. >>>> >>>> --- Topcat: >>>> VO-Visier Catalog Service >>>> Cone selection: RA 0.00000 deg >>>> Deg 80.0000 deg >>>> Radius: 10.0000 deg >>>> Maximum Row Count: unlimited >>>> Catalog Selection: Surveys Tycho-2 >>>> >>>> It returns a table with data in a 10deg radius circle around (0.0,80.0) >>>> Perfect. >>>> >>>> -- Stilts: >>>> stilts coneskymatch ifmt=ascii in=coord.txt ra=RA dec=DEC sr=10.0000 >>>> out=0.00000_80.0000.xml >>>> serviceurl=http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/259 >>>> find=all >>>> >>>> Coord.txt fiel: >>>> # RA DEC >>>> 0.00000 85.0000 >>>> >>>> Its returns only part of the data, only part of the circle !?!? >>>> >>>> >>>> Regards, >>>> Fran. >>>> >>> >>> >>> >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> > > -- +++++++++++++++++++++++++++++++++ Francisco Jiménez-Esteban Spanish Virtual Observatory Centro de Astrobiología (INTA-CSIC) LAEFF Campus. POB 78 28691 Villanueva de la Cañada Madrid, Spain Tel: +34 - 91 813 11 93 Fax: +34 - 91 813 11 60 +++++++++++++++++++++++++++++++++ From jmangeri@gmail.XXX Thu Oct 7 10:25:57 2010 Date: Wed, 6 Oct 2010 21:43:52 -0700 From: John Mangeri To: topcat-user@bristol.XXX Subject: TOPCAT Question Hi, After increasing useable memory to 512M, it seems that loading large 400,000+ tables will not work for some reason. Is there a way to get around this through a specialized command line script? Thanks -John M. From m.b.taylor@bristol.XXX Thu Oct 7 09:34:35 2010 Date: Thu, 7 Oct 2010 09:34:35 +0100 (BST) From: Mark Taylor To: John Mangeri Cc: topcat-user@bristol.XXX Subject: Re: TOPCAT Question On Wed, 6 Oct 2010, John Mangeri wrote: > Hi, > > After increasing useable memory to 512M, it seems that loading large > 400,000+ tables will not work for some reason. Is there a way to get around > this through a specialized command line script? John, can you run topcat with the -verbose flag ("topcat -verbose", or "java -jar topcat-full.jar -verbose") and tell me what you see on the console when attempting this load, as well as what error you get that tells you the load has failed. Also can you tell me the output from running with the -version flag. It might help for me to see the table as well; if your network connection is up to it, you can upload it by anonymous ftp to andromeda.star.bris.ac.uk in /incoming. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From mara.salvato@ipp.XXX Thu Oct 7 17:58:30 2010 Date: Thu, 07 Oct 2010 18:00:39 +0200 From: Mara Salvato To: topcat-user@mlist2.XXX Subject: density map Hi Mark et al, is there a way to 1) invert the grayscale in the density map plot 2) save the result of the matrix in a file ? Thanks, Mara From m.b.taylor@bristol.XXX Thu Oct 7 17:14:14 2010 Date: Thu, 7 Oct 2010 17:14:14 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: density map On Thu, 7 Oct 2010, Mara Salvato wrote: > Hi Mark et al, > > is there a way to > 1) invert the grayscale in the density map plot I thought there was, but it seems not. However you could achieve roughly this visual effect by adding a Weight axis with the value -1. > 2) save the result of the matrix in a file ? You can export it as a 2-d FITS image (look in the Export menu or the equivalent toolbar button). Alternatively, you can export it directly using SAMP to some other tool which is better at handling 2-d array (=image) data. There are lots of these (Aladin, GAIA, ds9) which is why I haven't put a lot of effort into the density map viewing capabilities. To do that you need to have the SAMP Hub running (Interop|SAMP Status|Connect in the main Control Window - see http://www.starlink.ac.uk/topcat/sun253/interop.html for more background on SAMP), and of course a suitable image viewing tool running, then use one of the options from the Interop menu in the Density Plot window. Mark > > > Thanks, > Mara > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Fri Oct 15 00:16:57 2010 Date: Thu, 14 Oct 2010 23:16:43 +0100 From: Seb Oliver To: topcat-user@bristol.XXX Subject: seek out of range error Anyone come across an error like this? > stilts tpipe head=1 /export/latour/sams/4times3.fits Seek out of range: -1170910336 3Gb file with 4.8e6 rows. Fits file was generated with IDL, though I don't normally have a problem with that. Tried it with -disk option. Using stilts 2.1-2+ Maybe I should be using -memory options, but I can't find any doc on those. Seb. From m.b.taylor@bristol.XXX Fri Oct 15 11:01:50 2010 Date: Fri, 15 Oct 2010 11:01:49 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: seek out of range error On Thu, 14 Oct 2010, Seb Oliver wrote: > Anyone come across an error like this? > > > stilts tpipe head=1 /export/latour/sams/4times3.fits > Seek out of range: -1170910336 > > 3Gb file with 4.8e6 rows. > > Fits file was generated with IDL, though I don't normally have a problem with > that. > > Tried it with -disk option. > > Using stilts 2.1-2+ > > Maybe I should be using -memory options, but I can't find any doc on those. Seb, not sure - can you try it with -debug -verbose options and tell me what it says? You shouldn't need any other funny options to make it work. Also are you using a 64-bit JVM and OS? You probably need to with a file that size. Even if not, it ought to give you a better error message, so I should maybe do something about it. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Fri Oct 15 12:04:48 2010 Date: Fri, 15 Oct 2010 11:04:42 +0100 From: Seb Oliver To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: seek out of range error [11:03 sjo@emperor:~][41]% stilts -debug -verbose tpipe head=1 /export/ latour/sams/4times3.fits INFO: Table not FITS-plus - Doesn't look like a FITS-plus file (uk.ac.starlink.table.StarTableFactory.makeStarTable) INFO: Table not colfits-plus - Doesn't look like a colfits-plus file (uk.ac.starlink.table.StarTableFactory.makeStarTable) INFO: Table not colfits-basic - Doesn't have exactly one row (uk.ac.starlink.table.StarTableFactory.makeStarTable) INFO: /export/latour/sams/4times3.fits too long for monolithic map (uk.ac.starlink.fits.FitsConstants.getInputStreamStart) INFO: /export/latour/sams/4times3.fits - mapping in blocks (uk.ac.starlink.fits.FitsConstants.getInputStreamStart) INFO: FITS file mapped as 12 blocks of 268435456 bytes (uk.ac.starlink.fits.MultiMappedFile.) java.lang.IllegalArgumentException: Seek out of range: -1170910336 at uk.ac.starlink.fits.MultiMappedFile.seek(MultiMappedFile.java:79) at uk.ac.starlink.fits.FitsTableBuilder.isEof(FitsTableBuilder.java: 461) at uk .ac .starlink.fits.FitsTableBuilder.attemptReadTable(FitsTableBuilder.java: 401) at uk .ac .starlink.fits.FitsTableBuilder.attemptReadTable(FitsTableBuilder.java: 362) at uk .ac.starlink.fits.FitsTableBuilder.makeStarTable(FitsTableBuilder.java: 126) at uk .ac .starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java: 359) at uk .ac .starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java: 535) at uk .ac .starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java: 676) at uk .ac .starlink .ttools .task .AbstractInputTableParameter .makeTable(AbstractInputTableParameter.java:121) at uk .ac .starlink .ttools.task.InputTableParameter.tableValue(InputTableParameter.java:36) at uk .ac.starlink.ttools.task.ConsumerTask.createProducer(ConsumerTask.java: 180) at uk .ac .starlink .ttools .task.SingleMapperTask.createInputProducer(SingleMapperTask.java:74) at uk.ac.starlink.ttools.task.TablePipe.createProducer(TablePipe.java: 34) at uk .ac .starlink.ttools.task.ConsumerTask.createExecutable(ConsumerTask.java: 99) at uk.ac.starlink.ttools.task.LineInvoker.invoke(LineInvoker.java:269) at uk.ac.starlink.ttools.Stilts.main(Stilts.java:38) On 15 Oct 2010, at 11:01, Mark Taylor wrote: >>> stilts tpipe head=1 /export/latour/sams/4times3.fits From S.Oliver@sussex.XXX Fri Oct 15 12:28:02 2010 Date: Fri, 15 Oct 2010 11:27:56 +0100 From: Seb Oliver To: Seb Oliver Cc: Mark Taylor , topcat-user@bristol.XXX Subject: Re: seek out of range error I've also created smaller versions of the file which read into topcat fine and have some of the same warning messages.. NFO: Table not FITS-plus - Doesn't look like a FITS-plus file (uk.ac.starlink.table.StarTableFactory.makeStarTable) INFO: Table not colfits-plus - Doesn't look like a colfits-plus file (uk.ac.starlink.table.StarTableFactory.makeStarTable) INFO: Table not colfits-basic - Doesn't have exactly one row (uk.ac.starlink.table.StarTableFactory.makeStarTable) INFO: tpipe in=sams10.fits ifmt=(auto) istream=false cmd=null omode=meta (uk.ac.starlink.ttools.task.LineInvoker.logParameterValues) Name: sams10.fits Columns: 155 Rows: 20650 On 15 Oct 2010, at 11:04, Seb Oliver wrote: > [11:03 sjo@emperor:~][41]% stilts -debug -verbose tpipe head=1 / > export/latour/sams/4times3.fits > INFO: Table not FITS-plus - Doesn't look like a FITS-plus file > (uk.ac.starlink.table.StarTableFactory.makeStarTable) > INFO: Table not colfits-plus - Doesn't look like a colfits-plus file > (uk.ac.starlink.table.StarTableFactory.makeStarTable) > INFO: Table not colfits-basic - Doesn't have exactly one row > (uk.ac.starlink.table.StarTableFactory.makeStarTable) > INFO: /export/latour/sams/4times3.fits too long for monolithic map > (uk.ac.starlink.fits.FitsConstants.getInputStreamStart) > INFO: /export/latour/sams/4times3.fits - mapping in blocks > (uk.ac.starlink.fits.FitsConstants.getInputStreamStart) > INFO: FITS file mapped as 12 blocks of 268435456 bytes > (uk.ac.starlink.fits.MultiMappedFile.) > java.lang.IllegalArgumentException: Seek out of range: -1170910336 > at uk.ac.starlink.fits.MultiMappedFile.seek(MultiMappedFile.java:79) > at uk.ac.starlink.fits.FitsTableBuilder.isEof(FitsTableBuilder.java: > 461) > at > uk > .ac > .starlink > .fits.FitsTableBuilder.attemptReadTable(FitsTableBuilder.java:401) > at > uk > .ac > .starlink > .fits.FitsTableBuilder.attemptReadTable(FitsTableBuilder.java:362) > at > uk > .ac > .starlink.fits.FitsTableBuilder.makeStarTable(FitsTableBuilder.java: > 126) > at > uk > .ac > .starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java: > 359) > at > uk > .ac > .starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java: > 535) > at > uk > .ac > .starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java: > 676) > at > uk > .ac > .starlink > .ttools > .task > .AbstractInputTableParameter > .makeTable(AbstractInputTableParameter.java:121) > at > uk > .ac > .starlink > .ttools.task.InputTableParameter.tableValue(InputTableParameter.java: > 36) > at > uk > .ac > .starlink.ttools.task.ConsumerTask.createProducer(ConsumerTask.java: > 180) > at > uk > .ac > .starlink > .ttools > .task.SingleMapperTask.createInputProducer(SingleMapperTask.java:74) > at > uk.ac.starlink.ttools.task.TablePipe.createProducer(TablePipe.java:34) > at > uk > .ac > .starlink > .ttools.task.ConsumerTask.createExecutable(ConsumerTask.java:99) > at uk.ac.starlink.ttools.task.LineInvoker.invoke(LineInvoker.java: > 269) > at uk.ac.starlink.ttools.Stilts.main(Stilts.java:38) > > On 15 Oct 2010, at 11:01, Mark Taylor wrote: > >>>> stilts tpipe head=1 /export/latour/sams/4times3.fits > > From m.b.taylor@bristol.XXX Fri Oct 15 11:33:47 2010 Date: Fri, 15 Oct 2010 11:33:47 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: seek out of range error On Thu, 14 Oct 2010, Seb Oliver wrote: > Anyone come across an error like this? > > > stilts tpipe head=1 /export/latour/sams/4times3.fits > Seek out of range: -1170910336 > > 3Gb file with 4.8e6 rows. > > Fits file was generated with IDL, though I don't normally have a problem with > that. > > Tried it with -disk option. > > Using stilts 2.1-2+ v2.2 might behave better come to think of it - can you try that? -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Sun Oct 17 10:06:43 2010 Date: Sun, 17 Oct 2010 09:06:37 +0100 From: Seb Oliver To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: seek out of range error Hi Mark, This version has same problem STILTS version 2.2 STIL version 2.9-3x Starjava revision: 9468 JVM: Java HotSpot(TM) 64-Bit Server VM version 14.3-b01-101 I've checked, recreating the files on my mac (still running OSX 10.5 not 10.6, so I'm not sure how 64-bit it is). The problem occurs with a file of 3776329 rows 155 columns 2.26Gb but not with one of 1849877 rows 1.11 Gb. Seb. On 15 Oct 2010, at 11:33, Mark Taylor wrote: > On Thu, 14 Oct 2010, Seb Oliver wrote: > >> Anyone come across an error like this? >> >>> stilts tpipe head=1 /export/latour/sams/4times3.fits >> Seek out of range: -1170910336 >> >> 3Gb file with 4.8e6 rows. >> >> Fits file was generated with IDL, though I don't normally have a >> problem with >> that. >> >> Tried it with -disk option. >> >> Using stilts 2.1-2+ > > v2.2 might behave better come to think of it - can you try that? > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, > UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/ > ~mbt/ > From m.b.taylor@bristol.XXX Sun Oct 17 18:31:17 2010 Date: Sun, 17 Oct 2010 18:31:16 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: seek out of range error Seb, looks like you are in 64-bit land, and those INFO messages you see are harmless. I can't reproduce the error (likely dependent on the details of the file), but I *think* I can see and fix what was going wrong, an overflow bug due to sloppy coding of integer arithmetic. Can you try the version at ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/stilts.jar and see if that sorts it out? Mark On Sun, 17 Oct 2010, Seb Oliver wrote: > Hi Mark, > > This version has same problem > > STILTS version 2.2 > STIL version 2.9-3x > Starjava revision: 9468 > JVM: Java HotSpot(TM) 64-Bit Server VM version 14.3-b01-101 > > I've checked, recreating the files on my mac (still running OSX 10.5 not 10.6, > so I'm not sure how 64-bit it is). The problem occurs with a file of 3776329 > rows 155 columns 2.26Gb but not with one of 1849877 rows 1.11 Gb. > > Seb. > > > On 15 Oct 2010, at 11:33, Mark Taylor wrote: > > > On Thu, 14 Oct 2010, Seb Oliver wrote: > > > > > Anyone come across an error like this? > > > > > > > stilts tpipe head=1 /export/latour/sams/4times3.fits > > > Seek out of range: -1170910336 > > > > > > 3Gb file with 4.8e6 rows. > > > > > > Fits file was generated with IDL, though I don't normally have a problem > > > with > > > that. > > > > > > Tried it with -disk option. > > > > > > Using stilts 2.1-2+ > > > > v2.2 might behave better come to think of it - can you try that? > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From sylvain.lafrasse@obs.XXX Thu Oct 21 13:04:04 2010 Date: Thu, 21 Oct 2010 14:03:46 +0200 From: Sylvain LAFRASSE To: topcat-user@bristol.XXX Subject: SAMP VoTable Loading Bug ? Dear Mark, I am in the process of adding SAMP VoTable export to JMMC SearchCal, using jsamp. I use TOPCAT and Aladin to test/debug my implementation, and Aladin handles my export gracefully, but TOPCAT doesn't manage to process it. I use a File.createTempFile() to write my VoTable to disk, and then send its URL or URI through SAMP, to get this stack trace : java.io.FileNotFoundException: No such file /var/folders/qa/qa-VvbaF2RWBU 8ZONSy 19o/-Tmp-/SearchCal7671109803842499423scvot at uk.ac.starlink.util.FileDataSource.(FileDataSource.java:33) at uk.ac.starlink.util.FileDataSource.(FileDataSource.java:55) at uk.ac.starlink.topcat.interop.TopcatSampControl.createTable(TopcatSampControl.java:456) at uk.ac.starlink.topcat.interop.TopcatSampControl.access$500(TopcatSampControl.java:48) at uk.ac.starlink.topcat.interop.TopcatSampControl$TableLoadHandler.processCall(TopcatSampControl.java:554) at org.astrogrid.samp.client.AbstractMessageHandler.receiveNotification(AbstractMessageHandler.java:99) at org.astrogrid.samp.client.HubConnector$ConnectorCallableClient.receiveNotification(HubConnector.java:819) at org.astrogrid.samp.gui.MessageTrackerHubConnector$MessageTrackerCallableClient.receiveNotification(MessageTrackerHubConnector.java:685) at org.astrogrid.samp.xmlrpc.ClientXmlRpcHandler$ClientActorImpl.receiveNotification(ClientXmlRpcHandler.java:61) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.astrogrid.samp.xmlrpc.ActorHandler.handleCall(ActorHandler.java:96) at org.astrogrid.samp.xmlrpc.internal.InternalServer.handleCall(InternalServer.java:207) at org.astrogrid.samp.xmlrpc.internal.InternalServer.getXmlRpcResult(InternalServer.java:193) at org.astrogrid.samp.xmlrpc.internal.InternalServer.getXmlRpcResponse(InternalServer.java:99) at org.astrogrid.samp.xmlrpc.internal.InternalServer$1.serveRequest(InternalServer.java:56) at org.astrogrid.samp.httpd.HttpServer.serve(HttpServer.java:153) at org.astrogrid.samp.httpd.HttpServer.serveRequest(HttpServer.java:267) at org.astrogrid.samp.httpd.HttpServer.access$200(HttpServer.java:41) at org.astrogrid.samp.httpd.HttpServer$1$1.run(HttpServer.java:187) The original URL was /var/folders/qa/qa-VvbaF2RWBU++8ZONSy+++19o/-Tmp-/SearchCal4299931296293182547scvot Something wrong with '+' ??? Hope all this will help ! Best regards, ------------------------------------------------------------------------------------------------------------------------ Sylvain LAFRASSE Tel : +33 (0)4 76 63 55 30 Fax : +33 (0)4 76 44 88 21 Laboratoire d'Astrophysique de Grenoble (LAOG) 414, rue de la Piscine Domaine Universitaire de Saint-Martin-d'Hères BP 53 France 38041 Grenoble cedex 9 ------------------------------------------------------------------------------------------------------------------------ From m.b.taylor@bristol.XXX Fri Oct 22 12:31:55 2010 Date: Fri, 22 Oct 2010 12:31:55 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: seek out of range error All, After a bit of to-ing and fro-ing, I finally nailed this. There was an integer arithmetic bug which could sometimes cause failures with funny error messages about negative seek values when looking at large (>2Gb) FITS files. Fixed pre-release versions for STILTS and TOPCAT can be found at ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ If you use the topcat versions here, beware that the load dialogue interface has changed significantly, and the documentation has not caught up with it. Mark On Sun, 17 Oct 2010, Mark Taylor wrote: > Seb, > > looks like you are in 64-bit land, and those INFO messages you see > are harmless. I can't reproduce the error (likely dependent on the > details of the file), but I *think* I can see and fix what was going > wrong, an overflow bug due to sloppy coding of integer arithmetic. > > Can you try the version at > > ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/stilts.jar > > and see if that sorts it out? > > Mark > > On Sun, 17 Oct 2010, Seb Oliver wrote: > > > Hi Mark, > > > > This version has same problem > > > > STILTS version 2.2 > > STIL version 2.9-3x > > Starjava revision: 9468 > > JVM: Java HotSpot(TM) 64-Bit Server VM version 14.3-b01-101 > > > > I've checked, recreating the files on my mac (still running OSX 10.5 not 10.6, > > so I'm not sure how 64-bit it is). The problem occurs with a file of 3776329 > > rows 155 columns 2.26Gb but not with one of 1849877 rows 1.11 Gb. > > > > Seb. > > > > > > On 15 Oct 2010, at 11:33, Mark Taylor wrote: > > > > > On Thu, 14 Oct 2010, Seb Oliver wrote: > > > > > > > Anyone come across an error like this? > > > > > > > > > stilts tpipe head=1 /export/latour/sams/4times3.fits > > > > Seek out of range: -1170910336 > > > > > > > > 3Gb file with 4.8e6 rows. > > > > > > > > Fits file was generated with IDL, though I don't normally have a problem > > > > with > > > > that. > > > > > > > > Tried it with -disk option. > > > > > > > > Using stilts 2.1-2+ > > > > > > v2.2 might behave better come to think of it - can you try that? > > > > > > -- > > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Sat Oct 23 18:01:55 2010 Date: Sat, 23 Oct 2010 18:01:54 +0100 (BST) From: Mark Taylor To: Sylvain LAFRASSE Cc: topcat-user@bristol.XXX Subject: Re: SAMP VoTable Loading Bug ? Sylvain, thanks for reporting this, and sorry it took a while to respond, I've been away at meetings. This is indeed a bug to do with "+" characters and the way TOPCAT decodes URLs for incoming SAMP tables. The fix will be in the next release, but meanwhile you can find a pre-release version at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Let me know if that works OK. Mark On Thu, 21 Oct 2010, Sylvain LAFRASSE wrote: > Dear Mark, > > I am in the process of adding SAMP VoTable export to JMMC SearchCal, using jsamp. > I use TOPCAT and Aladin to test/debug my implementation, and Aladin handles my export gracefully, but TOPCAT doesn't manage to process it. > > I use a File.createTempFile() to write my VoTable to disk, and then send its URL or URI through SAMP, to get this stack trace : > java.io.FileNotFoundException: No such file /var/folders/qa/qa-VvbaF2RWBU 8ZONSy 19o/-Tmp-/SearchCal7671109803842499423scvot > at uk.ac.starlink.util.FileDataSource.(FileDataSource.java:33) > at uk.ac.starlink.util.FileDataSource.(FileDataSource.java:55) > at uk.ac.starlink.topcat.interop.TopcatSampControl.createTable(TopcatSampControl.java:456) > at uk.ac.starlink.topcat.interop.TopcatSampControl.access$500(TopcatSampControl.java:48) > at uk.ac.starlink.topcat.interop.TopcatSampControl$TableLoadHandler.processCall(TopcatSampControl.java:554) > at org.astrogrid.samp.client.AbstractMessageHandler.receiveNotification(AbstractMessageHandler.java:99) > at org.astrogrid.samp.client.HubConnector$ConnectorCallableClient.receiveNotification(HubConnector.java:819) > at org.astrogrid.samp.gui.MessageTrackerHubConnector$MessageTrackerCallableClient.receiveNotification(MessageTrackerHubConnector.java:685) > at org.astrogrid.samp.xmlrpc.ClientXmlRpcHandler$ClientActorImpl.receiveNotification(ClientXmlRpcHandler.java:61) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.astrogrid.samp.xmlrpc.ActorHandler.handleCall(ActorHandler.java:96) > at org.astrogrid.samp.xmlrpc.internal.InternalServer.handleCall(InternalServer.java:207) > at org.astrogrid.samp.xmlrpc.internal.InternalServer.getXmlRpcResult(InternalServer.java:193) > at org.astrogrid.samp.xmlrpc.internal.InternalServer.getXmlRpcResponse(InternalServer.java:99) > at org.astrogrid.samp.xmlrpc.internal.InternalServer$1.serveRequest(InternalServer.java:56) > at org.astrogrid.samp.httpd.HttpServer.serve(HttpServer.java:153) > at org.astrogrid.samp.httpd.HttpServer.serveRequest(HttpServer.java:267) > at org.astrogrid.samp.httpd.HttpServer.access$200(HttpServer.java:41) > at org.astrogrid.samp.httpd.HttpServer$1$1.run(HttpServer.java:187) > > The original URL was /var/folders/qa/qa-VvbaF2RWBU++8ZONSy+++19o/-Tmp-/SearchCal4299931296293182547scvot > Something wrong with '+' ??? > > Hope all this will help ! > Best regards, > > > ------------------------------------------------------------------------------------------------------------------------ > Sylvain LAFRASSE > > Tel : +33 (0)4 76 63 55 30 Fax : +33 (0)4 76 44 88 21 > Laboratoire d'Astrophysique de Grenoble (LAOG) 414, rue de la Piscine > Domaine Universitaire de Saint-Martin-d'Hères BP 53 > France 38041 Grenoble cedex 9 > ------------------------------------------------------------------------------------------------------------------------ -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From thomas.boch@astro.XXX Mon Oct 25 09:26:19 2010 Date: Mon, 25 Oct 2010 09:26:07 +0200 From: Thomas Boch To: topcat-user@bristol.XXX Subject: Questions related to searching services in the registry Hi Mark, Trying to search for PPMX (keywords : 'ppmx') cone search services in Topcat triggered the following questions : - why do the results from the different registries differ ? Using the default Astrogrid one, I get 2 services back whereas using the NVO or the Euro-VO one did not bring any result back. In the same time, querying one of the latter registries through their Web interfaces(eg http://registry.euro-vo.org/search.jsp ) does bring back results. - do you have an idea why the CDS cone search service ivo://CDS.VizieR/I/312 does not show up in the result list ? According to http://registry.euro-vo.org/result.jsp?searchMethod=GetResource&identifier=ivo://CDS.VizieR/I/312 , it exists and is active. Cheers, Thomas -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From m.b.taylor@bristol.XXX Mon Oct 25 09:56:13 2010 Date: Mon, 25 Oct 2010 09:56:12 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Questions related to searching services in the registry On Mon, 25 Oct 2010, Thomas Boch wrote: > Hi Mark, > > Trying to search for PPMX (keywords : 'ppmx') cone search services in Topcat > triggered the following questions : > > - why do the results from the different registries differ ? Using the default > Astrogrid one, I get 2 services back whereas using the NVO or the Euro-VO one > did not bring any result back. In the same time, querying one of the latter > registries through their Web interfaces(eg > http://registry.euro-vo.org/search.jsp ) does bring back results. > > - do you have an idea why the CDS cone search service ivo://CDS.VizieR/I/312 > does not show up in the result list ? According to > http://registry.euro-vo.org/result.jsp?searchMethod=GetResource&identifier=ivo://CDS.VizieR/I/312 > , it exists and is active. Hi Thomas, the three registry services all fail to find your service for completely different reasons. It's hard not to see this as an illustration of the state of IVOA standard registry services to date. Searches against the Euro-VO registry, unlike the AstroGrid or NVO registries, are case-sensitive. Case-sensitivity is unfortunately unspecified by the RI 1.0 search interface standard. If you search for 'PPMX' you'll get a result. I highlighted this in my talk at the last IVOA Interop meeting (http://www.ivoa.net/internal/IVOA/InterOpMay2010Reg/reg.pdf); I seem to recall one of the Euro-VO people saying that they would change the implementation to be case-insensitive, which would be a more sensible choice, but I don't remember who said this, and it doesn't appear to have been changed. The NVO RI 1.0 search interface has for the last few months not been working at all, as far as I can see, though it used to. Any search you make against it from TOPCAT returns zero results, and I'm pretty sure the problem is at the service end. I reported this to NVO/VAO in August and a couple of times since; apparently it is on their to-do list, but they have other higher priorities (presumably VAO rather than IVOA). The AstroGrid registry is, as usual, working and standard-compliant, but for some reason this resource is not registered in it (though ivo://cds.vizier/I/311 and ivo://cds.vizier/I/313 are there). I'll look into that. One further thing: the registry search user interface in TOPCAT will be improved in the next version, you'll be able to select which fields you want to query, which will make it easier to get the results you want. A preview is available at the usual place (ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/). This wouldn't solve any of your problems above though. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From cstc@swansea.XXX Tue Oct 26 23:10:27 2010 Date: Tue, 26 Oct 2010 22:10:22 +0100 From: Tim Crawford To: topcat-user@sympa.XXX Subject: TOPCAT: Request for additional HOW TOs, tutorials or youtube videos Dear Topcat people, Does anyone know of additional resources, example data and other links to tutorials for TOPCAT? Any help would be much appreciated. Regards, Tim Crawford From jtlz2@astro.XXX Wed Oct 27 00:19:49 2010 Date: Tue, 26 Oct 2010 19:19:18 -0400 From: Jonathan Zwart To: Tim Crawford Cc: topcat-user@sympa.XXX Subject: Re: TOPCAT: Request for additional HOW TOs, tutorials or youtube videos www.youtube.com/watch?v=XMFxVpP_F1o (via www.astrobetter.com/topcat-leader-of-the-catalogue-manipulation-gang) Jon On 26 October 2010 17:10, Tim Crawford wrote: > Dear Topcat people, > > Does anyone know of additional resources, example data and other links to > tutorials for TOPCAT? > > Any help would be much appreciated. > > Regards, > > Tim Crawford > > From murphysj@mso.XXX Wed Oct 27 01:24:07 2010 Date: Wed, 27 Oct 2010 10:23:49 +1100 From: Simon Murphy To: Tim Crawford Cc: topcat-user@mlist2.XXX Subject: Re: TOPCAT: Request for additional HOW TOs, tutorials or youtube videos I have a small tutorial with example data available at: http://www.mso.anu.edu.au/~murphysj/topcat/ All the best, Simon On 27/10/2010, at 8:10 AM, Tim Crawford wrote: > Dear Topcat people, > > Does anyone know of additional resources, example data and other > links to tutorials for TOPCAT? > > Any help would be much appreciated. > > Regards, > > Tim Crawford From sylvain.lafrasse@obs.XXX Tue Nov 2 13:16:07 2010 Date: Tue, 2 Nov 2010 14:15:57 +0100 From: Sylvain LAFRASSE To: Mark Taylor , topcat-user@bristol.XXX Cc: jmmc-tech-group@ujf-grenoble.XXX Subject: Re: SAMP VoTable Loading Bug ? Dear Mark, On 23 oct. 2010, at 19:01, Mark Taylor wrote: > Sylvain, > > thanks for reporting this, and sorry it took a while to respond, I've > been away at meetings. I was myself away last week, so no problem ! > This is indeed a bug to do with "+" characters and the way TOPCAT > decodes URLs for incoming SAMP tables. The fix will be in the next > release, but meanwhile you can find a pre-release version at > > ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ > > Let me know if that works OK. It now works fine ! '+' were the culprits... Thank you again for your great support. Cheers, Sylvain > Mark > > On Thu, 21 Oct 2010, Sylvain LAFRASSE wrote: > >> Dear Mark, >> >> I am in the process of adding SAMP VoTable export to JMMC SearchCal, using jsamp. >> I use TOPCAT and Aladin to test/debug my implementation, and Aladin handles my export gracefully, but TOPCAT doesn't manage to process it. >> >> I use a File.createTempFile() to write my VoTable to disk, and then send its URL or URI through SAMP, to get this stack trace : >> java.io.FileNotFoundException: No such file /var/folders/qa/qa-VvbaF2RWBU 8ZONSy 19o/-Tmp-/SearchCal7671109803842499423scvot >> at uk.ac.starlink.util.FileDataSource.(FileDataSource.java:33) >> at uk.ac.starlink.util.FileDataSource.(FileDataSource.java:55) >> at uk.ac.starlink.topcat.interop.TopcatSampControl.createTable(TopcatSampControl.java:456) >> at uk.ac.starlink.topcat.interop.TopcatSampControl.access$500(TopcatSampControl.java:48) >> at uk.ac.starlink.topcat.interop.TopcatSampControl$TableLoadHandler.processCall(TopcatSampControl.java:554) >> at org.astrogrid.samp.client.AbstractMessageHandler.receiveNotification(AbstractMessageHandler.java:99) >> at org.astrogrid.samp.client.HubConnector$ConnectorCallableClient.receiveNotification(HubConnector.java:819) >> at org.astrogrid.samp.gui.MessageTrackerHubConnector$MessageTrackerCallableClient.receiveNotification(MessageTrackerHubConnector.java:685) >> at org.astrogrid.samp.xmlrpc.ClientXmlRpcHandler$ClientActorImpl.receiveNotification(ClientXmlRpcHandler.java:61) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.astrogrid.samp.xmlrpc.ActorHandler.handleCall(ActorHandler.java:96) >> at org.astrogrid.samp.xmlrpc.internal.InternalServer.handleCall(InternalServer.java:207) >> at org.astrogrid.samp.xmlrpc.internal.InternalServer.getXmlRpcResult(InternalServer.java:193) >> at org.astrogrid.samp.xmlrpc.internal.InternalServer.getXmlRpcResponse(InternalServer.java:99) >> at org.astrogrid.samp.xmlrpc.internal.InternalServer$1.serveRequest(InternalServer.java:56) >> at org.astrogrid.samp.httpd.HttpServer.serve(HttpServer.java:153) >> at org.astrogrid.samp.httpd.HttpServer.serveRequest(HttpServer.java:267) >> at org.astrogrid.samp.httpd.HttpServer.access$200(HttpServer.java:41) >> at org.astrogrid.samp.httpd.HttpServer$1$1.run(HttpServer.java:187) >> >> The original URL was /var/folders/qa/qa-VvbaF2RWBU++8ZONSy+++19o/-Tmp-/SearchCal4299931296293182547scvot >> Something wrong with '+' ??? >> >> Hope all this will help ! >> Best regards, >> >> >> ------------------------------------------------------------------------------------------------------------------------ >> Sylvain LAFRASSE >> >> Tel : +33 (0)4 76 63 55 30 Fax : +33 (0)4 76 44 88 21 >> Laboratoire d'Astrophysique de Grenoble (LAOG) 414, rue de la Piscine >> Domaine Universitaire de Saint-Martin-d'Hères BP 53 >> France 38041 Grenoble cedex 9 >> ------------------------------------------------------------------------------------------------------------------------ > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jacobsda@sas.XXX Tue Nov 9 23:37:50 2010 Date: Tue, 9 Nov 2010 12:29:51 -0700 From: Danny Jacobs To: topcat-user@sympa.XXX Subject: axis labels on log scale Hi Mark et al., In making plots in xy mode with I am noticing something funny about the labels on logarithmic mode axes. Specifically they seem to be off by a factor of 10 or perhaps missing a "x10" notation. I enclose an example set of plots showing the problem on the x axis. The linear version shows the true range of the x axis values, while the log shows the "off-by-x-10" issue. Thanks for any insight you (or anyone else on this list) might have. Cheers, ~Danny -- -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Danny Jacobs Dept. of Physics and Astronomy University of Pennsylvania 209 South 33rd Street Philadelphia, PA 19104-6396 ------------------------------------------------- Office:        Rittenhouse Lab - 1n2a Phone:        (215) 280-7357 Fax:        (215) 898-2010 Homepage:     http://dannycjacobs.googlepages.com [ Part 2, Image/PNG (Name: "topcat_log_axis_label_example_lin.png") 13 ] [ KB. ] [ Unable to print this part. ] [ Part 3, Image/PNG (Name: "topcat_log_axis_label_example.png") 13 KB. ] [ Unable to print this part. ] From JJ.Kavelaars@nrc-cnrc.XXX Tue Nov 9 23:47:28 2010 Date: Tue, 9 Nov 2010 15:48:30 -0800 From: "Kavelaars, JJ" To: "jacobsda@sas.XXX" Cc: "topcat-user@sympa.XXX" Subject: Re: axis labels on log scale The plots don't put the log scaling range in the sub-intervals. in your plot the 1e1 major tic mark is missing and this results in the difficult to understand log scale access. I've encountered similar issues with the tick-mark feature but not investigated possible solutions. JJ On 2010-11-09, at 3:37 PM, "Danny Jacobs" wrote: > Hi Mark et al., > In making plots in xy mode with I am noticing something funny about > the labels on logarithmic mode axes. Specifically they seem to be off > by a factor of 10 or perhaps missing a "x10" notation. I enclose an > example set of plots showing the problem on the x axis. The linear > version shows the true range of the x axis values, while the log shows > the "off-by-x-10" issue. > Thanks for any insight you (or anyone else on this list) might have. > > Cheers, > ~Danny > > -- > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Danny Jacobs > > Dept. of Physics and Astronomy > University of Pennsylvania > 209 South 33rd Street > Philadelphia, PA 19104-6396 > ------------------------------------------------- > Office: Rittenhouse Lab - 1n2a > Phone: (215) 280-7357 > Fax: (215) 898-2010 > Homepage: http://dannycjacobs.googlepages.com > > From m.b.taylor@bristol.XXX Wed Nov 10 08:04:42 2010 Date: Wed, 10 Nov 2010 08:04:42 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Cc: "jacobsda@sas.XXX" Subject: Re: axis labels on log scale Thanks JJ - as you say, the labelling is not wrong, though I admit it's not particularly good. Long time users will know that I've been promising to improve axis, especially log axis, labelling for ages. Maybe next year. Mark On Tue, 9 Nov 2010, Kavelaars, JJ wrote: > The plots don't put the log scaling range in the sub-intervals. in your plot the 1e1 major tic mark is missing and this results in the difficult to understand log scale access. I've encountered similar issues with the tick-mark feature but not investigated possible solutions. > > JJ > > On 2010-11-09, at 3:37 PM, "Danny Jacobs" wrote: > > > Hi Mark et al., > > In making plots in xy mode with I am noticing something funny about > > the labels on logarithmic mode axes. Specifically they seem to be off > > by a factor of 10 or perhaps missing a "x10" notation. I enclose an > > example set of plots showing the problem on the x axis. The linear > > version shows the true range of the x axis values, while the log shows > > the "off-by-x-10" issue. > > Thanks for any insight you (or anyone else on this list) might have. > > > > Cheers, > > ~Danny > > > > -- > > -- > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Danny Jacobs > > > > Dept. of Physics and Astronomy > > University of Pennsylvania > > 209 South 33rd Street > > Philadelphia, PA 19104-6396 > > ------------------------------------------------- > > Office: Rittenhouse Lab - 1n2a > > Phone: (215) 280-7357 > > Fax: (215) 898-2010 > > Homepage: http://dannycjacobs.googlepages.com > > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From dasmus@eso.XXX Fri Nov 19 09:30:24 2010 Date: Thu, 18 Nov 2010 17:41:26 -0300 From: Daniel Asmus To: topcat-user@bristol.XXX Subject: Question: Does topcat save the Expression? Hi! I am a new topcat user (coming from openoffice), so excuse me for 2 simple questions. But first I would like to mention that I am amazed by this program, I love it! Many thanks for doing this very useful peace of software for the community! 1. I generate a new column with some kind of expression using other column for the computation. Is there any way/format to save the table so that next time I open it the expression will still be in place? So that changing a value in one of the other columns still leads to an update in the column using the expression (as it works in Excel/openoffice)? As far as I can see topcat is only saving the values in the column and moves the expression to the description. If not I (and some colleagues) would consider it very useful, if it could be implemented somehow. 2. Isn't there any easy way to just add one row to your table? Shouldn't this be possible with just one click by the user? Maybe as an explanation: I work with tables of ~20-1000 rows and ~40 columns, which contain various samples of galaxies and their properties. I am used to do quite some editing inside the table directly, like updating individual values or adding objects from new publications. Finally, I have 2 related suggestions for the personal wish list: 1. I would appreciate a lot an "undo"-button (at least in the table view). 2. I would love to be able to open tables in "read-only" mode or be able to "lock" columns from direct editing. This is due to the fact that I am working with a lot of windows in parallel and I am always afraid of accidentally changing a value in the table. Many thanks for any help! Best regards, Daniel -- ---------------------------- Daniel ASMUS ESO Office Santiago Alonso de Cordova 3107 Vitacura Casilla 19001 Santiago 19 Chile Tel.: (+56 2) 463 3065 Skype: daniel.asmus From m.b.taylor@bristol.XXX Fri Nov 19 11:05:22 2010 Date: Fri, 19 Nov 2010 11:05:22 +0000 (GMT) From: Mark Taylor To: Daniel Asmus Cc: topcat-user@bristol.XXX Subject: Re: Question: Does topcat save the Expression? Daniel, On Thu, 18 Nov 2010, Daniel Asmus wrote: > Hi! > > I am a new topcat user (coming from openoffice), so excuse me for 2 simple > questions. > But first I would like to mention that I am amazed by this program, I love it! > Many thanks for doing this very useful peace of software for the community! glad you like it. I should probably note, as I think you're finding out, that while TOPCAT resembles a spreasheet application in some ways, it's really a different beast, and so it won't have all the same capabilities that you would expect from Excel or its OpenOffice equivalent. > 1. I generate a new column with some kind of expression using other column for > the computation. Is there any way/format to save the table so that next time > I open it the expression will still be in place? So that changing a value in > one of the other columns still leads to an update in the column using the > expression (as it works in Excel/openoffice)? As far as I can see topcat is > only saving the values in the column and moves the expression to the > description. Yes, when the table is saved, only the values are written. The text of the expression is saved in the metadata for the column, but this is just for reference (for humans looking at the metadata), it's not re-read by the tool on load. > If not I (and some colleagues) would consider it very useful, if it could be > implemented somehow. Noted. It is on the to-do list somewhere, but not likely to get done in the near future. > 2. Isn't there any easy way to just add one row to your table? Shouldn't this > be possible with just one click by the user? Maybe as an explanation: I work > with tables of ~20-1000 rows and ~40 columns, which contain various samples > of galaxies and their properties. I am used to do quite some editing inside > the table directly, like updating individual values or adding objects from new > publications. It's not. Because of the internal design (loaded tables are assumed throughout the code to have a fixed number of rows), it would not be easy to add this feature. Although there are one or two people using TOPCAT in the way that you are, to edit or author tables by hand, it seems to be rather a minority interest for the user base, so again I'm afraid this is not something I plan to add any time soon. > Finally, I have 2 related suggestions for the personal wish list: > > 1. I would appreciate a lot an "undo"-button (at least in the table view). That's a popular request, though not so easy to do. Maybe one day. > 2. I would love to be able to open tables in "read-only" mode or be able to > "lock" columns from direct editing. This is due to the fact that I am working > with a lot of windows in parallel and I am always afraid of accidentally > changing a value in the table. Per-column lock is possible, I'll think about it. As far as read-only mode goes - just don't save/overwrite the table! TOPCAT never alters the data of a table on disk, changes are only made if you explicitly perform a save. It's possible that you could use STILTS, the command-line counterpart to TOPCAT to do some of these things, but as a command-line tool, it's considerably more effort to use. Thanks for the feedback. Apologies that the answer to most of your questions is no, but it is useful to hear how people are using the application. Best wishes Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From cgp@star.XXX Fri Nov 19 11:22:13 2010 Date: Fri, 19 Nov 2010 11:21:56 +0000 From: Clive Page To: topcat-user@sympa.XXX Subject: Re: Question: Does topcat save the Expression? On 19/11/2010 11:05, Mark Taylor wrote: > I should probably note, as I think you're finding out, that while TOPCAT > resembles a spreasheet application in some ways, it's really a different > beast, and so it won't have all the same capabilities that you would > expect from Excel or its OpenOffice equivalent. Perhaps it's obvious, but probably the best thing to do is to use a genuine spreadsheet like Excel (or one of the free equivalents) when creating a table or appending rows to one, then save the results in CSV format (comma-separated value). Excel works well enough reading and writing CSV files (thought it keeps on saying something like "do you really want to save your data in this format which loses many of the wonderful Excel features", and you have to keep on saying yes). Since CSV isn't any kind of standard, different packages have various options for saving data in it, e.g. whether the first line is a list of column names or not, and whether strings including commas are automatically enclosed in quotation marks. But mostly it works well enough. When you need the features that only TOPCAT provides, it will, of course, import CSV files. And can even export in that format, if you need it with or without the header line. While it would be nicer if a single package did everything one wanted, I can see that there's a fundamental design difference, which makes that hard to achieve. Vive la difference. Regards -- Clive Page From fran@cab.XXX Fri Nov 19 15:53:38 2010 Date: Fri, 19 Nov 2010 16:50:50 +0100 From: fran@cab.XXX To: topcat-user@bristol.XXX Cc: Fran Jimenez Subject: access to SIMBAD Hi, I would like to do a multiple cone search in simbad using topcat or stilts. Is there any way to do that? Thank you for the help. Cheers, Fran. -- +++++++++++++++++++++++++++++++++ Francisco Jiménez-Esteban Spanish Virtual Observatory Centro de Astrobiología (INTA-CSIC) LAEFF Campus. POB 78 28691 Villanueva de la Cañada Madrid, Spain Tel: +34 - 91 813 11 93 Fax: +34 - 91 813 11 60 +++++++++++++++++++++++++++++++++ From m.b.taylor@bristol.XXX Fri Nov 19 16:08:52 2010 Date: Fri, 19 Nov 2010 16:08:51 +0000 (GMT) From: Mark Taylor To: Fran Jimenez Cc: topcat-user@bristol.XXX Subject: Re: access to SIMBAD On Fri, 19 Nov 2010, fran@cab.XXX wrote: > Hi, > I would like to do a multiple cone search in simbad using topcat or stilts. Is > there any way to do that? Yes. The cone search service URL for Simbad is: http://simbad.u-strasbg.fr/simbad/sim-cone? In principle you can find this out using the registry search in the multi-cone search window. However, for reasons outside of TOPCAT's control, this requires some non-obvious expertise (for discussion see http://sympa.bris.ac.uk/sympa/arc/topcat-user/2010-10/msg00022.html). You can get the correct service record using the multi-cone window (http://www.starlink.ac.uk/topcat/sun253/ConeMultiWindow.html) if you select the Euro-VO registry (http://registry.euro-vo.org/services/RegistrySearch) in the Registry selector at the top, and then search for "Simbad" (capital first letter required). Alternatively, you can forget the registry search and just cut'n'paste the service URL I've cited above into the "Cone Search URL" box half way down TOPCAT's multi-cone window. Then in TOPCAT, just follow the normal Multi-Cone procedure. STILTS equivalent would look something like: stilts coneskymatch in=in.vot ra=ra_col dec=dec_col sr=0.0003 parallel=5 servicetype=cone serviceurl='http://simbad.u-strasbg.fr/simbad/sim-cone?' (details at http://www.starlink.ac.uk/stilts/sun256/coneskymatch.html). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jacobsda@sas.XXX Sun Nov 28 11:40:10 2010 Date: Fri, 26 Nov 2010 22:15:18 -0700 From: Danny Jacobs To: topcat-user@sympa.XXX Subject: legends Hi Mark et. al. Is there a way to get the legend to lie inside the plot axis, instead of outside? My apologies if it's another feature request, I just wanted to be sure I wasn't missing something. I still occasionally stumble upon features I wasn't expecting! Thanks, ~Danny -- -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Danny Jacobs Dept. of Physics and Astronomy University of Pennsylvania 209 South 33rd Street Philadelphia, PA 19104-6396 ------------------------------------------------- Office:        Rittenhouse Lab - 1n2a Phone:        (215) 280-7357 Fax:        (215) 898-2010 Homepage:     http://dannycjacobs.googlepages.com From m.b.taylor@bristol.XXX Sun Nov 28 18:59:26 2010 Date: Sun, 28 Nov 2010 18:59:26 +0000 (GMT) From: Mark Taylor To: Danny Jacobs Cc: topcat-user@sympa.XXX Subject: Re: legends On Fri, 26 Nov 2010, Danny Jacobs wrote: > Hi Mark et. al. > Is there a way to get the legend to lie inside the plot axis, instead > of outside? My apologies if it's another feature request, I just > wanted to be sure I wasn't missing something. I still occasionally > stumble upon features I wasn't expecting! I'm afraid you're not missing anything - inside/outside axes is one of the long list of detailed configurability options that TOPCAT doesn't give you for plots. Maybe one day, though I must confess I have been saying that about plot configurability for a long time now. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Thu Dec 23 16:28:08 2010 Date: Thu, 23 Dec 2010 16:28:08 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: TOPCAT, STILTS, STIL releases Hi, for those subscribed to this list but not to topcat-announce, I've just posted a release announcement there. You can see it at http://sympa.bris.ac.uk/sympa/arc/topcat-announce/2010-12/msg00000.html Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Christoph.Deil@mpi-hd.XXX Fri Dec 24 00:23:38 2010 Date: Fri, 24 Dec 2010 00:50:50 +0100 From: Christoph Deil To: topcat-user@mlist2.XXX Subject: Christmas wish: PyStilts Hi Mark, I was wondering if you have any plans to implement a CPython wrapper for STILTS? This thread just started on the scipy-user mailing list, calling Java from Python seems possible, although I have no idea how much work it would be for STILTS: http://mail.scipy.org/pipermail/scipy-user/2010-December/027974.html Merry Christmas, Christoph From bilicki@camk.XXX Tue Dec 28 19:14:25 2010 Date: Tue, 28 Dec 2010 19:14:16 +0000 From: bilicki@camk.XXX To: topcat-user@sympa.XXX Subject: Problem with loading SDSS from VizieR When I load SDSS DR7 data via VO -> Vizier Catalogue Service, I get only 137 rows. This also happens if I limit the number of rows to be loaded. Anybody knows what's the issue? For 2MASS XSC for example there is no problem, all 1.6 M rows are loaded. Maciek From thomas.boch@astro.XXX Tue Dec 28 19:41:56 2010 Date: Tue, 28 Dec 2010 20:41:27 +0100 From: Thomas Boch To: bilicki@camk.XXX Cc: topcat-user@mlist2.XXX Subject: Re: Problem with loading SDSS from VizieR Hi Maciek, When querying SDSS from VizieR, which target and radius did you request ? Cheers, Thomas On 28/12/10 20:14, bilicki@camk.XXX wrote: > When I load SDSS DR7 data via VO -> Vizier Catalogue Service, I get only 137 > rows. This also happens if I limit the number of rows to be loaded. Anybody > knows what's the issue? For 2MASS XSC for example there is no problem, all 1.6 > M rows are loaded. > > Maciek > -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From bilicki@camk.XXX Tue Dec 28 19:50:09 2010 Date: Tue, 28 Dec 2010 20:49:49 +0100 From: Maciej Bilicki To: topcat-user@sympa.XXX Subject: Re: Problem with loading SDSS from VizieR Hi Thomas, I didn't choose any target nor radius. In the menu VO -> VizieR Catalogue Service I choose "All Rows" and take a Maximum Row Count e.g. 10000 (or whatever). For Output Columns I take "all". In Catalogue Selection I use the "Surveys" tab and click SDSS-DR7, which contains > 357 M rows. There are no more options here. I click OK... and obtain only 137 rows. Maciek On 28.12.2010 20:41, From Thomas Boch: > Hi Maciek, > > When querying SDSS from VizieR, which target and radius did you request ? > Cheers, > > Thomas > > On 28/12/10 20:14, bilicki@camk.XXX wrote: >> When I load SDSS DR7 data via VO -> Vizier Catalogue Service, I get >> only 137 >> rows. This also happens if I limit the number of rows to be loaded. >> Anybody >> knows what's the issue? For 2MASS XSC for example there is no problem, >> all 1.6 >> M rows are loaded. >> >> Maciek >> > > From thomas.boch@astro.XXX Tue Dec 28 19:57:06 2010 Date: Tue, 28 Dec 2010 20:56:54 +0100 From: Thomas Boch To: Maciej Bilicki Cc: topcat-user@mlist2.XXX Subject: Re: Problem with loading SDSS from VizieR Hi, For "large" catalogues as SDSS, VizieR puts a default target (0 +0 I think) and a default radius, even if the user didn't provide one. I CC this message to François Ochsenbein, in charge of the VizieR service, who might give details about the best way to query SDSS in order to get all the sources. Thomas On 28/12/10 20:49, Maciej Bilicki wrote: > Hi Thomas, > > I didn't choose any target nor radius. In the menu VO -> VizieR > Catalogue Service I choose "All Rows" and take a Maximum Row Count > e.g. 10000 (or whatever). For Output Columns I take "all". In > Catalogue Selection I use the "Surveys" tab and click SDSS-DR7, which > contains > 357 M rows. There are no more options here. I click OK... > and obtain only 137 rows. > > Maciek > > On 28.12.2010 20:41, From Thomas Boch: >> Hi Maciek, >> >> When querying SDSS from VizieR, which target and radius did you >> request ? >> Cheers, >> >> Thomas >> >> On 28/12/10 20:14, bilicki@camk.XXX wrote: >>> When I load SDSS DR7 data via VO -> Vizier Catalogue Service, I get >>> only 137 >>> rows. This also happens if I limit the number of rows to be loaded. >>> Anybody >>> knows what's the issue? For 2MASS XSC for example there is no problem, >>> all 1.6 >>> M rows are loaded. >>> >>> Maciek >>> >> >> > > -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From francois@cdsarc.XXX Thu Dec 30 12:39:14 2010 Date: Wed, 29 Dec 2010 11:01:58 +0100 From: "Francois Ochsenbein (ext.52429)" To: Maciej Bilicki Cc: topcat-user@mlist2.XXX Subject: Re: Fwd: Re: Problem with loading SDSS from VizieR Hi, Thomas is right -- catalogs with hundreds of millions of rows can't be downloaded from the vizier server without any further constraint, a default center / radius are then assumed (in the case of SDSS-DR7, the assumed center is effectively 0+0, as Thomas guessed). We'll check what can be done in the very beginning of next year... Season's greetings in the meantime ! Francois Ochsenbein CDS, Strasbourg > >Hi, > >For "large" catalogues as SDSS, VizieR puts a default target (0 +0 I >think) and a default radius, even if the user didn't provide one. >I CC this message to François Ochsenbein, in charge of the VizieR >service, who might give details about the best way to query SDSS in >order to get all the sources. > >Thomas > >On 28/12/10 20:49, Maciej Bilicki wrote: >> Hi Thomas, >> >> I didn't choose any target nor radius. In the menu VO -> VizieR >> Catalogue Service I choose "All Rows" and take a Maximum Row Count >> e.g. 10000 (or whatever). For Output Columns I take "all". In >> Catalogue Selection I use the "Surveys" tab and click SDSS-DR7, which >> contains> 357 M rows. There are no more options here. I click OK... >> and obtain only 137 rows. >> >> Maciek >> >> On 28.12.2010 20:41, From Thomas Boch: >>> Hi Maciek, >>> >>> When querying SDSS from VizieR, which target and radius did you >>> request ? >>> Cheers, >>> >>> Thomas >>> >>> On 28/12/10 20:14, bilicki@camk.XXX wrote: >>>> When I load SDSS DR7 data via VO -> Vizier Catalogue Service, I get >>>> only 137 >>>> rows. This also happens if I limit the number of rows to be loaded. >>>> Anybody >>>> knows what's the issue? For 2MASS XSC for example there is no problem, >>>> all 1.6 >>>> M rows are loaded. >>>> >>>> Maciek >>>> >>> >>> >> >> > > >-- >Thomas Boch > CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 > 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 > F-67000 Strasbourg Email : thomas.boch@astro.XXX > France http://cdsweb.u-strasbg.fr/~boch > > ======================================================================= Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 Email: francois@astro.XXX (France) Fax: +33-(0)368 85 24 17 ======================================================================= From m.b.taylor@bristol.XXX Thu Dec 30 13:45:40 2010 Date: Thu, 30 Dec 2010 13:45:40 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Problem with loading SDSS from VizieR Another point: 357M rows is rather a lot to download. At a rough calculation the file would be around 50Gb and the download would take a few days. Moreover, TOPCAT certainly wouldn't be able to cope with a table this size. 10 million rows is a rough maximum size for a table to load into TOPCAT - that figure can go up and down a bit depending on your platform and what you need to do with it, but 357M is definitely too big (though you could possibly do useful things with a table that size using STILTS). Mark On Tue, 28 Dec 2010, Maciej Bilicki wrote: > Hi Thomas, > > I didn't choose any target nor radius. In the menu VO -> VizieR Catalogue > Service I choose "All Rows" and take a Maximum Row Count e.g. 10000 (or > whatever). For Output Columns I take "all". In Catalogue Selection I use the > "Surveys" tab and click SDSS-DR7, which contains > 357 M rows. There are no > more options here. I click OK... and obtain only 137 rows. > > Maciek > > On 28.12.2010 20:41, From Thomas Boch: > > Hi Maciek, > > > > When querying SDSS from VizieR, which target and radius did you request ? > > Cheers, > > > > Thomas > > > > On 28/12/10 20:14, bilicki@camk.XXX wrote: > > > When I load SDSS DR7 data via VO -> Vizier Catalogue Service, I get > > > only 137 > > > rows. This also happens if I limit the number of rows to be loaded. > > > Anybody > > > knows what's the issue? For 2MASS XSC for example there is no problem, > > > all 1.6 > > > M rows are loaded. > > > > > > Maciek > > > > > > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Tue Jan 4 12:44:08 2011 Date: Tue, 4 Jan 2011 12:44:08 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Christmas wish: PyStilts On Fri, 24 Dec 2010, Christoph Deil wrote: > Hi Mark, > > I was wondering if you have any plans to implement a CPython wrapper for STILTS? > > This thread just started on the scipy-user mailing list, calling Java from Python seems possible, although I have no idea how much work it would be for STILTS: > http://mail.scipy.org/pipermail/scipy-user/2010-December/027974.html Christoph, this is probably feasible, but I don't have plans to do it. I am aware that it would useful to considerably more people than the existing Jython-based version, but the fact that it would have to mix C and Java would make the build and distribution a lot more complicated than it currently is (no longer a single platform- independent jar file), not to mention the additional coding effort. If it turned out that there was massive demand I might just be persuaded ... but I'm not expecting to. Happy new year! Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Tue Jan 4 13:22:35 2011 Date: Tue, 4 Jan 2011 13:22:35 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: SQL queries from TopCat Sebastien, following up this matter, the first one (jdbc: URLs) is fixed in the recent release v3.7, but the second one (form of the SQL load dialogue) is not. It's on the to-do list for some release in the future. Thanks again for these reports. Mark On Fri, 1 Oct 2010, Mark Taylor wrote: > Hi Sebastien, > > thanks for these very useful reports. There are two items: > > 1. jdbc: locations don't work at all since v3.6 in the Location field. > > Oops! You're right, this was an error I introduced when allowing > it to load multiple tables at once. Sorry about that. Now fixed. > 2. The SQL read/write dialogue boxes make it impossible to enter > jdbc URLs appropriate for some (likely, most) JDBC drivers. > > Quite right again. I can't remember what I said to you last time, > but it's clear to me now that those dialogue boxes have the wrong > form. I need to rework them so that you can enter a more > freeform URL base: I think that just collapsing the "Protocol", > "Host" and "Database name" fields into a single "base URL" field > will do the trick. (Another issue is that the SQL Query field > is not big enough, but that's a different matter). > > As it happens I've recently reworked all the load dialogues, so > this is a good time to fix this. I'll get on it. > > I'll let you know when there is a version with these fixes in. > > Mark > > > On Fri, 1 Oct 2010, Sebastien Derriere wrote: > > > > > Hello Mark, > > > > I think there is still a bug with the way SQL direct access can > > be done through TopCat with the latest version. > > > > I am making tests on a Sybase installation and driver jconn3.jar, > > launching TopCat with all the proper arguments : > > java -classpath pathto/topcat-full.jar:pathto/jconn3.jar > > -Djdbc.drivers=com.sybase.jdbc3.jdbc.SybDriver uk.ac.starlink.topcat.Driver > > > > > > With TopCat 3.5-1, I can get it to work by typing in the > > Load new table > Location the full jdbc string which looks like : > > > > jdbc:sybase:Tds:hostname:port/dbname?user=XXX&password=XXX#select * from > > mytable where... > > > > However, with version 3.5-2, the dedicated "Open JDBC table" window > > does not work with the following error : > > Error making connection jdbc:sybase:Tds://hostname:port/dbname > > > > As you can see this is probably due to the // before hostname. > > > > > > Now with TopCat version 3.6, I can't get it to work at all ! > > Load new table > Location fails with this error : > > > > java.io.FileNotFoundException: Not extant file or valid URL: > > jdbc:sybase:Tds:hostname:port/dbname?user=XXX&password=XXX > > at uk.ac.starlink.util.DataSource.makeDataSource(DataSource.java:613) > > at > > uk.ac.starlink.table.gui.TableLoadChooser$8.attemptLoads(TableLoadChooser.java:587) > > at uk.ac.starlink.table.gui.LoadWorker$1.run(LoadWorker.java:82) > > Caused by: java.net.MalformedURLException: unknown protocol: jdbc > > at java.net.URL.(URL.java:574) > > at java.net.URL.(URL.java:464) > > at java.net.URL.(URL.java:413) > > at uk.ac.starlink.util.DataSource.makeDataSource(DataSource.java:609) > > ... 2 more > > > > The dedicated "Open JDBC table" window still does not work with the following > > error : > > > > java.io.IOException: Error making connection > > jdbc:sybase:Tds://hostname:port/dbname - known JDBC drivers: > > sun.jdbc.odbc.JdbcOdbcDriver > > com.sybase.jdbc3.jdbc.SybDriver > > > > at > > uk.ac.starlink.table.jdbc.JDBCHandler.makeStarTable(JDBCHandler.java:122) > > at > > uk.ac.starlink.table.StarTableFactory.makeStarTable(StarTableFactory.java:485) > > at > > uk.ac.starlink.table.gui.SQLReadDialog$1.attemptLoads(SQLReadDialog.java:67) > > at uk.ac.starlink.table.gui.LoadWorker$1.run(LoadWorker.java:82) > > Caused by: java.sql.SQLException: JZ003: Le format de l'URL est incorrect > > at com.sybase.jdbc3.jdbc.ErrorMessage.raiseError(Unknown Source) > > at com.sybase.jdbc3.tds.Tds.login(Unknown Source) > > at com.sybase.jdbc3.jdbc.SybConnection.a(Unknown Source) > > at com.sybase.jdbc3.jdbc.SybConnection.a(Unknown Source) > > at com.sybase.jdbc3.jdbc.SybConnection.(Unknown Source) > > at com.sybase.jdbc3.jdbc.SybConnection.(Unknown Source) > > at com.sybase.jdbc3.jdbc.SybDriver.connect(Unknown Source) > > at java.sql.DriverManager.getConnection(DriverManager.java:582) > > at java.sql.DriverManager.getConnection(DriverManager.java:185) > > at > > uk.ac.starlink.table.jdbc.JDBCHandler.getConnection(JDBCHandler.java:167) > > at > > uk.ac.starlink.table.jdbc.JDBCHandler.access$000(JDBCHandler.java:14) > > at > > uk.ac.starlink.table.jdbc.JDBCHandler$1.getConnection(JDBCHandler.java:73) > > at > > uk.ac.starlink.table.jdbc.JDBCStarTable.(JDBCStarTable.java:74) > > at > > uk.ac.starlink.table.jdbc.JDBCHandler.makeStarTable(JDBCHandler.java:91) > > ... 3 more > > > > > > I remember when I first discussed this with you a while ago you said > > it was quite difficult to have proper URL parsing, and I guess this is > > what is at stake here. > > Do you think you could build the jdbc string without the // that > > is causing trouble, without breaking other more used expressions in > > the "Location" dialog ? > > > > Sebastien > > > > > > -- > > > > ((( Sebastien Derriere sebastien.derriere@astro.XXX > > (. .) Observatoire de Strasbourg Phone +33 (0) 368 852 444 > > (( v )) 11, rue de l'universite Telefax +33 (0) 368 852 417 > > ---m-m--- F-67000 Strasbourg France > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Tue Jan 25 15:23:47 2011 Date: Tue, 25 Jan 2011 16:23:37 +0100 From: Ivan Zolotukhin To: topcat-user@bristol.XXX Subject: draw surveys footprints on a sky plot Hi Mark, Just out of curiosity, how difficult it would be to draw footprints (spherical polygons, defined either by using IVOA recommendation or any other mean) of some survey (e.g. SDSS) on a sky plot in TOPCAT? Can you suggest any workaround? The motivation for this feature is quite simple: a visual test of some set of sources to present in wide area surveys (instead of going to their SIAP or ConeSearch services one by one and test it directly; for many of coming surveys they are not available at all, though footprints are already defined). Seems that this simple thing is not achievable with any of the VO tools, though it would encourage data re-use in the field in my mind, which is an important goal of the VO. -- With best regards, Ivan From ivan.zolotukhin@gmail.XXX Tue Jan 25 16:05:44 2011 Date: Tue, 25 Jan 2011 17:05:36 +0100 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: draw surveys footprints on a sky plot Hi again, On Tue, Jan 25, 2011 at 4:39 PM, Mark Taylor wrote: > On Tue, 25 Jan 2011, Ivan Zolotukhin wrote: > >> Hi Mark, >> >> Just out of curiosity, how difficult it would be to draw footprints >> (spherical polygons, defined either by using IVOA recommendation or >> any other mean) of some survey (e.g. SDSS) on a sky plot in TOPCAT? >> Can you suggest any workaround? >> >> The motivation for this feature is quite simple: a visual test of some >> set of sources to present in wide area surveys (instead of going to >> their SIAP or ConeSearch services one by one and test it directly; for >> many of coming surveys they are not available at all, though >> footprints are already defined). Seems that this simple thing is not >> achievable with any of the VO tools, though it would encourage data >> re-use in the field in my mind, which is an important goal of the VO. > > Hi Ivan, > > I don't expect to introduce this as a capability of the existing > plotting machinery in TOPCAT.  When[*] the long-threatened Great > Graphics Overhaul comes about I will give serious consideration > to some kind of footprint functionality that would serve this kind > of use case, which I can see would be useful. Pity... > > As regards a workaround, you could use some spatially > random or random-like subsample of points from the survey in question > and plot those points, which would give you a good idea of the spatial > limits of the survey.  This is not elegant and depending on what > data you have to hand may not be easy to do either, though I do > occasionally use it as a quick hack, e.g. by plotting the SDSS DR5 QSO > catalog (http://www.sdss.org/dr7/products/value_added/qsocat_dr5.html) > to look at the footprint of SDSS DR5. Yeah, I do the same thing by plotting Witham et al. catalog to get an idea of IPHAS footprint, but it's silly. Some surveys (e.g. VISTA stuff) haven't published anything yet, but their footprints are well known. Moreover, it's a mess to check catalogs one by one. So it's a very limited workaround idea, unfortunately. > As regards other VO tools - doesn't Aladin have quite a bit of > footprint functionality built in?  If Thomas or another Aladin > person is listening perhaps they could comment. I'm in contact with Thomas privately. Initially, Aladin is able to plot small sized FOVs only, and there's no option to use large scale footprints in all-sky mode. But seems that Thomas has got some solution. I'll post it in this thread if it works. > > [*] now officially a When rather than an If since as of this week I >    have 2.5 years funding in the bag Wow! That's a really good news! Now I'm not even disappointed with the lack of positive answer on my question :) Congratulations! > -- > Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From Thomas.A.McGlynn@nasa.XXX Tue Jan 25 16:09:02 2011 Date: Tue, 25 Jan 2011 11:08:50 -0500 From: Tom McGlynn To: Mark Taylor Cc: Ivan Zolotukhin , "topcat-user@bristol.XXX" Subject: Re: draw surveys footprints on a sky plot Mark Taylor wrote: ... > As regards other VO tools - doesn't Aladin have quite a bit of > footprint functionality built in? If Thomas or another Aladin > person is listening perhaps they could comment. > I think both Aladin and SkyView (in its standalone incarnation) can do this to the extent that the footprint is a set of polygons in the sky. I think in Aladin the underlying framework was originally intended for displaying instrumental fields of view but I don't think it's limited to that. In SkyView we've used its drawing facility for both instrumental overlays and constellation boundaries. If you know can define the boundaries, SkyView can draw them. For SkyView the required format is just an ASCII list of points (in either coordinates or some projection) between which you draw the boundary lines. E.g., the following draws the boundaries for the constellation Lyra (see below). Tom McGlynn Example of SkyView overlay file (draws boundary of Lyra with text LYR in the center of the constellation) # Constellation boundary file for LYR # Boundaries in B1875 coordinates. project Coo B1875 scale 1d thick 2 283. 25.5 283. 26.0 275.5 26.0 275.5 30.0 272.625 30.0 272.625 47.5 273.5 47.5 286.25 47.5 287.5 47.5 287.5 43.5 291 43.5 291 36.5 290.375 36.5 290.375 30.0 288.875 30.0 288.875 27.5 288.875 25.5 283. 25.5 text -76.0418661368841 34.8527443823087 0 LYR From jmvarela@iac.XXX Tue Jan 25 16:15:52 2011 Date: Tue, 25 Jan 2011 16:14:36 +0000 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: draw surveys footprints on a sky plot > > [*] now officially a When rather than an If since as of this week I > have 2.5 years funding in the bag > Congrats, Mark! That's great news for you... and for us, TOPCAT users! :) Jesús -- **************************************************************** * Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * * * * Instituto de Astrofísica de Canarias * * --------------- www.iac.es ------------- * * * * C/ Vía Láctea s/n Tel:+34 922 605278 * * E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * **************************************************************** --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From S.Oliver@sussex.XXX Tue Jan 25 17:51:31 2011 Date: Tue, 25 Jan 2011 17:51:22 +0000 From: Seb Oliver To: Tom McGlynn Cc: Mark Taylor , Ivan Zolotukhin , "topcat-user@bristol.XXX" Subject: Re: draw surveys footprints on a sky plot what would be really good would be if polygons could be easily used to define catalogue subsets. Seb. On 25 Jan 2011, at 16:08, Tom McGlynn wrote: > Mark Taylor wrote: > ... > >> As regards other VO tools - doesn't Aladin have quite a bit of >> footprint functionality built in? If Thomas or another Aladin >> person is listening perhaps they could comment. >> > > I think both Aladin and SkyView (in its standalone incarnation) can > do this to the extent that the footprint is a set of polygons in the > sky. I think in Aladin the underlying framework was originally > intended for displaying instrumental fields of view but I don't > think it's limited to that. In SkyView we've used its drawing > facility for both instrumental overlays and constellation > boundaries. If you know can define the boundaries, SkyView can draw > them. > > For SkyView the required format is just an ASCII list of points (in > either coordinates or some projection) between which you draw the > boundary lines. E.g., the following draws the boundaries for the > constellation Lyra (see below). > > Tom McGlynn > > > Example of SkyView overlay file (draws boundary > of Lyra with text LYR in the center of the constellation) > > # Constellation boundary file for LYR > # Boundaries in B1875 coordinates. > project Coo B1875 > scale 1d > thick 2 > > 283. 25.5 > 283. 26.0 > 275.5 26.0 > 275.5 30.0 > 272.625 30.0 > 272.625 47.5 > 273.5 47.5 > 286.25 47.5 > 287.5 47.5 > 287.5 43.5 > 291 43.5 > 291 36.5 > 290.375 36.5 > 290.375 30.0 > 288.875 30.0 > 288.875 27.5 > 288.875 25.5 > 283. 25.5 > > text -76.0418661368841 34.8527443823087 0 LYR > From thomas.boch@astro.XXX Wed Jan 26 14:20:22 2011 Date: Wed, 26 Jan 2011 15:19:59 +0100 From: Thomas Boch To: Ivan Zolotukhin Cc: topcat-user@bristol.XXX Subject: Re: draw surveys footprints on a sky plot Hi Ivan, My cheap and dirty solution is the following : - starting from the Healpix FITS density map of a catalogue, I generated a VOTable which one row for each Healpix cell having at least one source. The position of this row is the center of the Healpix cell. - when displayed in Aladin on an all-sky image (see for instance http://cdsweb.u-strasbg.fr/~boch/cats_footprints/snapshot1.png or http://cdsweb.u-strasbg.fr/~boch/cats_footprints/snapshot2.png ) , it gives an interesting view of the catalogue coverage. I have generated such VOTables for SDSS7, C2D, GLIMPSE and DENIS : http://cdsweb.u-strasbg.fr/~boch/cats_footprints/ I could easily generate this for every VizieR catalogue with positions. Hope this helps. Now, I'm looking for a smart way to display this coverage using the Aladin footprint facility (primarily created to describe instrument FoV), possibly based on STC-S. Cheers, Thomas Ivan Zolotukhin wrote: > Hi Mark, > > Just out of curiosity, how difficult it would be to draw footprints > (spherical polygons, defined either by using IVOA recommendation or > any other mean) of some survey (e.g. SDSS) on a sky plot in TOPCAT? > Can you suggest any workaround? > > The motivation for this feature is quite simple: a visual test of some > set of sources to present in wide area surveys (instead of going to > their SIAP or ConeSearch services one by one and test it directly; for > many of coming surveys they are not available at all, though > footprints are already defined). Seems that this simple thing is not > achievable with any of the VO tools, though it would encourage data > re-use in the field in my mind, which is an important goal of the VO. > > -- > With best regards, > Ivan > > -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From ivan.zolotukhin@gmail.XXX Wed Jan 26 16:28:23 2011 Date: Wed, 26 Jan 2011 17:26:14 +0100 From: Ivan Zolotukhin To: Thomas Boch Cc: topcat-user@bristol.XXX Subject: Re: draw surveys footprints on a sky plot Hi Thomas, Thanks for this idea. I'll probably try to generate similar VOTables for surveys I need (e.g. IPHAS or UKIDSS which are not in Vizier), based on rough footprint polygons specifications available from literature. In the meantime, don't forget that some users would want to specify polygons manually, without having any set of sources that can help determine survey boundaries. What regards to STC-S formatted footprints, I think most of the interested parties strongly support this idea. Seems that http://www.voservices.net/footprint/ has a footprint builder which suits me quite well and the service can export user generated footprints in STC format. I also know Chandra guys export their footprints in STC format which would be quite nice to have that in Aladin (I would become a power user of that). So if only Aladin can display them, that would probably solve many problems at once. -- With best regards, Ivan On Wed, Jan 26, 2011 at 3:19 PM, Thomas Boch wrote: > Hi Ivan, > > My cheap and dirty solution is the following : > - starting from the Healpix FITS density map of a catalogue, I generated a > VOTable which one row for each Healpix cell having at least one source. The > position of this row is the center of the Healpix cell. > - when displayed in Aladin on an all-sky image (see for instance > http://cdsweb.u-strasbg.fr/~boch/cats_footprints/snapshot1.png or > http://cdsweb.u-strasbg.fr/~boch/cats_footprints/snapshot2.png ) , it gives > an interesting view of the catalogue coverage. > > I have generated such VOTables for SDSS7, C2D, GLIMPSE and DENIS : > http://cdsweb.u-strasbg.fr/~boch/cats_footprints/ > > I could easily generate this for every VizieR catalogue with positions. > > Hope this helps. > > Now, I'm looking for a smart way to display this coverage using the Aladin > footprint facility (primarily created to describe instrument FoV), possibly > based on STC-S. > > Cheers, > > Thomas > > Ivan Zolotukhin wrote: > > Hi Mark, > > Just out of curiosity, how difficult it would be to draw footprints > (spherical polygons, defined either by using IVOA recommendation or > any other mean) of some survey (e.g. SDSS) on a sky plot in TOPCAT? > Can you suggest any workaround? > > The motivation for this feature is quite simple: a visual test of some > set of sources to present in wide area surveys (instead of going to > their SIAP or ConeSearch services one by one and test it directly; for > many of coming surveys they are not available at all, though > footprints are already defined). Seems that this simple thing is not > achievable with any of the VO tools, though it would encourage data > re-use in the field in my mind, which is an important goal of the VO. > > -- > With best regards, > Ivan > > > > -- > Thomas Boch > CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 > 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 > F-67000 Strasbourg Email : thomas.boch@astro.XXX > France http://cdsweb.u-strasbg.fr/~boch > From cgp@star.XXX Fri Feb 18 09:28:01 2011 Date: Fri, 18 Feb 2011 09:27:48 +0000 From: Clive Page To: topcat-user@sympa.XXX Subject: Galactic coordinate conversions I've just found that I need to find galactic long and lat for a table which has only RA and DEC. I half expected to find that TOPCAT had suitable functions there already, but it doesn't (so I'll have to roll my own, no big deal). I wonder, however, if anyone else would find these useful, if so maybe we could persuade Mark to provide them in a future release? There would probably have to be two functions, I guess e.g. GalLat(RA,DEC) and GalLong(RA,DEC) which is not terribly elegant or efficient, but would be useful all the same. Regards -- Clive Page Dept of Physics & Astronomy, University of Leicester, Leicester, LE1 7RH, U.K. From JJ.Kavelaars@nrc-cnrc.XXX Fri Mar 4 01:52:35 2011 Date: Thu, 3 Mar 2011 17:51:59 -0800 From: "Kavelaars, JJ" To: "topcat-user@bristol.XXX" Subject: STILTS plot2d Hello, Is there a way of controlling the style of the line that option lineN=DotToDot provides? I'd like to make the line a bit thicker and have the line be 'dashed' instead of solid... cheers, JJ From pierre.didelon@cea.XXX Fri Mar 4 09:37:16 2011 Date: Fri, 04 Mar 2011 08:31:42 +0100 From: Pierre Didelon To: "Kavelaars, JJ" Cc: "topcat-user@bristol.XXX" Subject: Re: STILTS plot2d Kavelaars, JJ wrote: >Hello, > >Is there a way of controlling the style of the line that option lineN=DotToDot provides? I'd like to make the line a bit thicker and have the line be 'dashed' instead of solid... > it is in the same window where you select dot to dot (plot style editor), above you have the thickness and dash style choice > >cheers, >JJ > > > > -- Pierre ------------------------------------------------------------------- DIDELON :@: pdidelon_at_cea.fr Phone : 33 (0)1 69 08 58 89 CEA SACLAY - Service d'Astrophysique 91191 Gif-Sur-Yvette Cedex ------------------------------------------------------------------- Aidez les enfants Tibétains : http://www.a-e-t.org/actions/ d'autres : http://www.sosesf.org/ ou trouvez une autre ONG -------------------------------------------------------------------- Seule compte la démarche. Car c'est elle qui dure et non le but qui n'est qu'illusion du voyageur... Saint Exupery - Citadelle -------------------------------------------------------------------- From m.b.taylor@bristol.XXX Fri Mar 4 10:38:16 2011 Date: Fri, 4 Mar 2011 10:38:08 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Cc: "Kavelaars, JJ" Subject: Re: STILTS plot2d On Fri, 4 Mar 2011, Pierre Didelon wrote: > Kavelaars, JJ wrote: > > > Hello, > > > > Is there a way of controlling the style of the line that option > > lineN=DotToDot provides? I'd like to make the line a bit thicker and have > > the line be 'dashed' instead of solid... > > > it is in the same window where you select dot to dot (plot style editor), > above you have the thickness and dash style choice Works for TOPCAT, but not for STILTS. It seems that plot2d has been missing these parameters. Better download a bleeding-edge version: ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/stilts.jar The plot2d version here has new parameters linewidthNS and dashNS. For help stilts plot2d help=linewidthNS stilts plot2d help=dashNS Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From H.L.Rose@sms.XXX Fri Mar 4 14:04:57 2011 Date: Fri, 04 Mar 2011 13:50:49 +0000 From: HL Rose To: topcat-user@bristol.XXX Subject: Topcat Match Tables query Hello! I am trying to match two tables together in TOPCAT, assigning new columns to each row. I understand that you can do this easily if the rows are in order, or by matching a common column. However, I am struggling, given that I want them to be matched based on three seperate identifier columns. For instance, both of my tables have an ID1, ID2 and ID3 column, and only matching the tables based on one of these would not be sufficient as ID1, ID2 and ID3 can have repeated values. Is there are way of matching two tables with 3 matched value columns? Thank you Hannah Rose -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From S.Oliver@sussex.XXX Fri Mar 4 14:21:46 2011 Date: Fri, 4 Mar 2011 14:21:22 +0000 From: Seb Oliver To: HL Rose Cc: topcat-user@bristol.XXX Subject: Re: Topcat Match Tables query why don't you create a new column which is ID1*xxxxxx+ID2*xxx+ID3 Seb. On 4 Mar 2011, at 13:50, HL Rose wrote: > Hello! > > I am trying to match two tables together in TOPCAT, assigning new > columns to each row. I understand that you can do this easily if the > rows are in order, or by matching a common column. However, I am > struggling, given that I want them to be matched based on three > seperate identifier columns. For instance, both of my tables have an > ID1, ID2 and ID3 column, and only matching the tables based on one > of these would not be sufficient as ID1, ID2 and ID3 can have > repeated values. > > Is there are way of matching two tables with 3 matched value columns? > > Thank you > > Hannah Rose > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > From aaronsmashing@gmail.XXX Fri Mar 4 14:29:32 2011 Date: Fri, 4 Mar 2011 09:59:21 -0430 From: Aaron Machado To: HL Rose Cc: topcat-user@bristol.XXX Subject: Re: Topcat Match Tables query *"Is there are way of matching two tables with 3 matched value columns?"* Maybe I'm wrong but I think you can just go to the match window, then click in "Algorithm" menu and select 3-d Cartesian. Then Select your two tables, and asing X, Y and Z column your concerning three columns. Let me know if it work or if I misunderstood your question. 2011/3/4 HL Rose > Hello! > > I am trying to match two tables together in TOPCAT, assigning new columns > to each row. I understand that you can do this easily if the rows are in > order, or by matching a common column. However, I am struggling, given that > I want them to be matched based on three seperate identifier columns. For > instance, both of my tables have an ID1, ID2 and ID3 column, and only > matching the tables based on one of these would not be sufficient as ID1, > ID2 and ID3 can have repeated values. > > Is there are way of matching two tables with 3 matched value columns? > > Thank you > > Hannah Rose > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > From jtlz2@astro.XXX Fri Mar 4 14:39:00 2011 Date: Fri, 4 Mar 2011 09:38:22 -0500 From: Jonathan Zwart To: Aaron Machado Cc: HL Rose , topcat-user@bristol.XXX Subject: Re: Topcat Match Tables query On 4 March 2011 09:29, Aaron Machado wrote: > *"Is there are way of matching two tables with 3 matched value columns?"* > > Maybe I'm wrong but I think you can just go to the match window, then click > in "Algorithm" menu and select 3-d Cartesian. Then Select your two tables, > and asing X, Y and Z column your concerning three columns. > > Let me know if it work or if I misunderstood your question. > That's how I usually do it, but I like the new-column idea. Jon From barclay.astro@gmail.XXX Fri Mar 4 15:05:44 2011 Date: Fri, 4 Mar 2011 14:57:14 +0000 From: Tom Barclay To: HL Rose Cc: topcat-user@bristol.XXX Subject: Re: Topcat Match Tables query On 4 March 2011 14:38, Jonathan Zwart wrote: > On 4 March 2011 09:29, Aaron Machado wrote: > >> *"Is there are way of matching two tables with 3 matched value columns?"* >> >> Maybe I'm wrong but I think you can just go to the match window, then >> click in "Algorithm" menu and select 3-d Cartesian. Then Select your two >> tables, and asing X, Y and Z column your concerning three columns. >> >> Let me know if it work or if I misunderstood your question. >> > > That's how I usually do it, but I like the new-column idea. > I find the 3-d Cartesian is fine when the tables are small but the tables are large (in my case, over a million rows), this method is, understandably, very slow and memory intensive. In this case I usually write a quick python script to match multiple rows of id's. However, I think Seb's method is quite neat. A similar method I sometimes use is to create a column with values toString(ID1)+"-"+toString(ID2)+"-"+toString(ID3) and match these using exact value. Tom From jmvarela@iac.XXX Fri Mar 4 14:59:13 2011 Date: Fri, 04 Mar 2011 14:58:17 +0000 From: jmvarela@iac.XXX To: Jonathan Zwart Cc: Aaron Machado , HL Rose , topcat-user@bristol.XXX Subject: Re: Topcat Match Tables query Hi Hanna, if you want exact matching for your IDs, you can try the following: 1- Create a new column in both tables with the expression: concat(toString(ID1),toString(ID2),toString(ID3)) 2- Match the tables using the option "Exact value". Another alternative would be to create the new column as an array with: array(ID1,ID2,ID3) But I'm not sure how the matching would work. Cheers, Jesús Jonathan Zwart ha escrito: > On 4 March 2011 09:29, Aaron Machado wrote: > >> *"Is there are way of matching two tables with 3 matched value columns?"* >> >> Maybe I'm wrong but I think you can just go to the match window, then click >> in "Algorithm" menu and select 3-d Cartesian. Then Select your two tables, >> and asing X, Y and Z column your concerning three columns. >> >> Let me know if it work or if I misunderstood your question. >> > > That's how I usually do it, but I like the new-column idea. > > Jon > -- **************************************************************** * Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * * * * Instituto de Astrofísica de Canarias * * --------------- www.iac.es ------------- * * * * C/ Vía Láctea s/n Tel:+34 922 605278 * * E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * **************************************************************** --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From dabrusco@gmail.XXX Tue Mar 8 22:31:55 2011 Date: Tue, 8 Mar 2011 17:31:11 -0500 From: Raffaele D'Abrusco To: topcat-user@bristol.XXX Subject: Galactic extinction calculator in Topcat? Hi Mark and Topcat users, I know that there are tons of webservices out there for the evaluation of galactic extinction, but I think it would be very handy to get it from inside Topcat, the very place where all my precious tables are retrieved and manipulated... Is there any chance that such feature will be introduced in a future release of the software? Thanks, Raffaele From m.b.taylor@bristol.XXX Wed Mar 9 10:51:07 2011 Date: Wed, 9 Mar 2011 10:51:07 +0000 (GMT) From: Mark Taylor To: Raffaele D'Abrusco Cc: topcat-user@bristol.XXX Subject: Re: Galactic extinction calculator in Topcat? On Tue, 8 Mar 2011, Raffaele D'Abrusco wrote: > Hi Mark and Topcat users, > > I know that there are tons of webservices out there for the evaluation > of galactic extinction, but I think it would be very handy to get it from > inside Topcat, the very place where all my precious tables are > retrieved and manipulated... > Is there any chance that such feature will be introduced in a future > release of the software? Raffaele, Thanks for this suggestion. I don't know much about galactic extinction, but I've had a quick Google. Although it doesn't look impossible to add functionality along these lines, I don't think I'm likely to do it. In general I'm happy to add functions which people will find useful if there is a formula or algorithm for the calculation which is uncontroversial, not too complicated, and with a relatively small number of input parameters. As far as I can tell, as well as being fairly complicated to calculate, in order to provide galactic extinctions I'd need to carry around very large data files inside TOPCAT (200Mb, from the look of http://astro.berkeley.edu/~marc/dust/data/data.html), and the inconvenience of burdening the software with that many extra bytes wouldn't be worth the benefit. I'm also a bit wary of adding black box type functions which rely on data and assumptions rather than ones that I can document the exact expressions for. It would be feasible to write a plugin to do, e.g., Schlegel type extinction calculations that could be used with TOPCAT, but... the amount of work would be considerable (converting the IDL code to Java), so again it doesn't look too likely. Another option would be to have TOPCAT talk to an external web service which does the calculations. This would require some additional infrastructure in the program, but that might be useful for use of other similar type calculation services, so it might be worth it. However, I don't see any web services out there for galactic extinction calculation that are capable of being (easily) driven by software rather than by humans typing numbers into web forms, so it wouldn't be easy. So, probably not. But I'm happy to hear other people's perspectives, especially if it sounds like I may have misunderstood what the requirement is here. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Wed Mar 9 11:49:27 2011 Date: Wed, 9 Mar 2011 12:49:02 +0100 (CET) From: Maciej Bilicki To: Mark Taylor Cc: Raffaele D'Abrusco , topcat-user@bristol.XXX Subject: Re: Galactic extinction calculator in Topcat? Dear All, The paper by Schlegel, Finkbeiner and Davis, which is now the main reference paper to calculate the Galactic extinction, is accompanied by easy-to-use routines in C, Fortran and IDL, under the link that Mark gave. All one needs is to export the coordinates of the objects and apply the maps for them with some smartly chosen parameters. This can easily be recombined with the table in TOPCAT, I guess... Maciek On Wed, 9 Mar 2011, Mark Taylor wrote: > On Tue, 8 Mar 2011, Raffaele D'Abrusco wrote: > >> Hi Mark and Topcat users, >> >> I know that there are tons of webservices out there for the evaluation >> of galactic extinction, but I think it would be very handy to get it from >> inside Topcat, the very place where all my precious tables are >> retrieved and manipulated... >> Is there any chance that such feature will be introduced in a future >> release of the software? > > Raffaele, > > Thanks for this suggestion. I don't know much about galactic > extinction, but I've had a quick Google. Although it doesn't look > impossible to add functionality along these lines, I don't think I'm > likely to do it. > > In general I'm happy to add functions which people will find useful > if there is a formula or algorithm for the calculation which is > uncontroversial, not too complicated, and with a relatively small > number of input parameters. > > As far as I can tell, as well as being fairly complicated to > calculate, in order to provide galactic extinctions I'd need > to carry around very large data files inside TOPCAT (200Mb, from > the look of http://astro.berkeley.edu/~marc/dust/data/data.html), > and the inconvenience of burdening the software with that many > extra bytes wouldn't be worth the benefit. I'm also a bit wary > of adding black box type functions which rely on data and assumptions > rather than ones that I can document the exact expressions for. > > It would be feasible to write a plugin to do, e.g., Schlegel type > extinction calculations that could be used with TOPCAT, but... > the amount of work would be considerable (converting the IDL code > to Java), so again it doesn't look too likely. > > Another option would be to have TOPCAT talk to an external web service > which does the calculations. This would require some additional > infrastructure in the program, but that might be useful for use of > other similar type calculation services, so it might be worth it. > However, I don't see any web services out there for galactic > extinction calculation that are capable of being (easily) driven > by software rather than by humans typing numbers into web forms, > so it wouldn't be easy. > > So, probably not. But I'm happy to hear other people's perspectives, > especially if it sounds like I may have misunderstood what the > requirement is here. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > From S.Oliver@sussex.XXX Wed Mar 9 13:56:37 2011 Date: Wed, 9 Mar 2011 13:56:27 +0000 From: Seb Oliver To: Maciej Bilicki Cc: Mark Taylor , Raffaele D'Abrusco , topcat-user@bristol.XXX Subject: Re: Galactic extinction calculator in Topcat? I'd have thought this is comfortably outside the scope of TOPCAT - unless there is a server. Also even a SFD calculation would be controversial. Seb, On 9 Mar 2011, at 11:49, Maciej Bilicki wrote: > Dear All, > > The paper by Schlegel, Finkbeiner and Davis, which is now the main > reference paper to calculate the Galactic extinction, is accompanied > by easy-to-use routines in C, Fortran and IDL, under the link that > Mark gave. All one needs is to export the coordinates of the objects > and apply the maps for them with some smartly chosen parameters. > This can easily be recombined with the table in TOPCAT, I guess... > > Maciek > > On Wed, 9 Mar 2011, Mark Taylor wrote: > >> On Tue, 8 Mar 2011, Raffaele D'Abrusco wrote: >> >>> Hi Mark and Topcat users, >>> >>> I know that there are tons of webservices out there for the >>> evaluation >>> of galactic extinction, but I think it would be very handy to get >>> it from >>> inside Topcat, the very place where all my precious tables are >>> retrieved and manipulated... >>> Is there any chance that such feature will be introduced in a future >>> release of the software? >> >> Raffaele, >> >> Thanks for this suggestion. I don't know much about galactic >> extinction, but I've had a quick Google. Although it doesn't look >> impossible to add functionality along these lines, I don't think I'm >> likely to do it. >> >> In general I'm happy to add functions which people will find useful >> if there is a formula or algorithm for the calculation which is >> uncontroversial, not too complicated, and with a relatively small >> number of input parameters. >> >> As far as I can tell, as well as being fairly complicated to >> calculate, in order to provide galactic extinctions I'd need >> to carry around very large data files inside TOPCAT (200Mb, from >> the look of http://astro.berkeley.edu/~marc/dust/data/data.html), >> and the inconvenience of burdening the software with that many >> extra bytes wouldn't be worth the benefit. I'm also a bit wary >> of adding black box type functions which rely on data and assumptions >> rather than ones that I can document the exact expressions for. >> >> It would be feasible to write a plugin to do, e.g., Schlegel type >> extinction calculations that could be used with TOPCAT, but... >> the amount of work would be considerable (converting the IDL code >> to Java), so again it doesn't look too likely. >> >> Another option would be to have TOPCAT talk to an external web >> service >> which does the calculations. This would require some additional >> infrastructure in the program, but that might be useful for use of >> other similar type calculation services, so it might be worth it. >> However, I don't see any web services out there for galactic >> extinction calculation that are capable of being (easily) driven >> by software rather than by humans typing numbers into web forms, >> so it wouldn't be easy. >> >> So, probably not. But I'm happy to hear other people's perspectives, >> especially if it sounds like I may have misunderstood what the >> requirement is here. >> >> Mark >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol >> University, UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> >> > From JJ.Kavelaars@nrc-cnrc.XXX Wed Mar 9 18:47:34 2011 Date: Wed, 9 Mar 2011 10:47:19 -0800 From: "Kavelaars, JJ" To: "topcat-user@bristol.XXX" Subject: Re: Galactic extinction calculator in Topcat? would using the FORTRAN extinction code inside FORTRESS work? http://projectfortress.sun.com/Projects/Community/ JJ On 2011-03-09, at 3:49 AM, "Maciej Bilicki" wrote: > Dear All, > > The paper by Schlegel, Finkbeiner and Davis, which is now the main > reference paper to calculate the Galactic extinction, is accompanied by > easy-to-use routines in C, Fortran and IDL, under the link that Mark gave. > All one needs is to export the coordinates of the objects and apply the > maps for them with some smartly chosen parameters. This can easily be > recombined with the table in TOPCAT, I guess... > > Maciek > > On Wed, 9 Mar 2011, Mark Taylor wrote: > >> On Tue, 8 Mar 2011, Raffaele D'Abrusco wrote: >> >>> Hi Mark and Topcat users, >>> >>> I know that there are tons of webservices out there for the evaluation >>> of galactic extinction, but I think it would be very handy to get it from >>> inside Topcat, the very place where all my precious tables are >>> retrieved and manipulated... >>> Is there any chance that such feature will be introduced in a future >>> release of the software? >> >> Raffaele, >> >> Thanks for this suggestion. I don't know much about galactic >> extinction, but I've had a quick Google. Although it doesn't look >> impossible to add functionality along these lines, I don't think I'm >> likely to do it. >> >> In general I'm happy to add functions which people will find useful >> if there is a formula or algorithm for the calculation which is >> uncontroversial, not too complicated, and with a relatively small >> number of input parameters. >> >> As far as I can tell, as well as being fairly complicated to >> calculate, in order to provide galactic extinctions I'd need >> to carry around very large data files inside TOPCAT (200Mb, from >> the look of http://astro.berkeley.edu/~marc/dust/data/data.html), >> and the inconvenience of burdening the software with that many >> extra bytes wouldn't be worth the benefit. I'm also a bit wary >> of adding black box type functions which rely on data and assumptions >> rather than ones that I can document the exact expressions for. >> >> It would be feasible to write a plugin to do, e.g., Schlegel type >> extinction calculations that could be used with TOPCAT, but... >> the amount of work would be considerable (converting the IDL code >> to Java), so again it doesn't look too likely. >> >> Another option would be to have TOPCAT talk to an external web service >> which does the calculations. This would require some additional >> infrastructure in the program, but that might be useful for use of >> other similar type calculation services, so it might be worth it. >> However, I don't see any web services out there for galactic >> extinction calculation that are capable of being (easily) driven >> by software rather than by humans typing numbers into web forms, >> so it wouldn't be easy. >> >> So, probably not. But I'm happy to hear other people's perspectives, >> especially if it sounds like I may have misunderstood what the >> requirement is here. >> >> Mark >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> >> > From thomas.boch@astro.XXX Wed Mar 9 18:54:03 2011 Date: Wed, 09 Mar 2011 19:53:20 +0100 From: Thomas Boch To: Maciej Bilicki Cc: Mark Taylor , Raffaele D'Abrusco , topcat-user@bristol.XXX Subject: Re: Galactic extinction calculator in Topcat? Hi, IMHO, it would make more sense if someone sets up a REST service allowing this kind of calculation, and possibly following the IVOA UWS pattern. This would allow Topcat (and other VO-aware applications) to easily access it. Cheers, Thomas On 09/03/11 12:49, Maciej Bilicki wrote: > Dear All, > > The paper by Schlegel, Finkbeiner and Davis, which is now the main > reference paper to calculate the Galactic extinction, is accompanied > by easy-to-use routines in C, Fortran and IDL, under the link that > Mark gave. All one needs is to export the coordinates of the objects > and apply the maps for them with some smartly chosen parameters. This > can easily be recombined with the table in TOPCAT, I guess... > > Maciek > > On Wed, 9 Mar 2011, Mark Taylor wrote: > >> On Tue, 8 Mar 2011, Raffaele D'Abrusco wrote: >> >>> Hi Mark and Topcat users, >>> >>> I know that there are tons of webservices out there for the evaluation >>> of galactic extinction, but I think it would be very handy to get it >>> from >>> inside Topcat, the very place where all my precious tables are >>> retrieved and manipulated... >>> Is there any chance that such feature will be introduced in a future >>> release of the software? >> >> Raffaele, >> >> Thanks for this suggestion. I don't know much about galactic >> extinction, but I've had a quick Google. Although it doesn't look >> impossible to add functionality along these lines, I don't think I'm >> likely to do it. >> >> In general I'm happy to add functions which people will find useful >> if there is a formula or algorithm for the calculation which is >> uncontroversial, not too complicated, and with a relatively small >> number of input parameters. >> >> As far as I can tell, as well as being fairly complicated to >> calculate, in order to provide galactic extinctions I'd need >> to carry around very large data files inside TOPCAT (200Mb, from >> the look of http://astro.berkeley.edu/~marc/dust/data/data.html), >> and the inconvenience of burdening the software with that many >> extra bytes wouldn't be worth the benefit. I'm also a bit wary >> of adding black box type functions which rely on data and assumptions >> rather than ones that I can document the exact expressions for. >> >> It would be feasible to write a plugin to do, e.g., Schlegel type >> extinction calculations that could be used with TOPCAT, but... >> the amount of work would be considerable (converting the IDL code >> to Java), so again it doesn't look too likely. >> >> Another option would be to have TOPCAT talk to an external web service >> which does the calculations. This would require some additional >> infrastructure in the program, but that might be useful for use of >> other similar type calculation services, so it might be worth it. >> However, I don't see any web services out there for galactic >> extinction calculation that are capable of being (easily) driven >> by software rather than by humans typing numbers into web forms, >> so it wouldn't be easy. >> >> So, probably not. But I'm happy to hear other people's perspectives, >> especially if it sounds like I may have misunderstood what the >> requirement is here. >> >> Mark >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> >> > -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From m.b.taylor@bristol.XXX Thu Mar 10 09:55:52 2011 Date: Thu, 10 Mar 2011 09:55:52 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Galactic extinction calculator in Topcat? Thomas, I think you're right; although the amount of data involved (a few hundred Mb) is not so great that the calculation must be done server-side, it would be useful to have this functionality available as a service so that the burden of implementation for what is probably a fairly niche function does not have to be replicated. However, I think UWS might be overkill since, as I understand it, this is basically a lookup, not very computationally intensive, so asynchronous operation would not be necessary. In fact, given that the main input parameters are sky position, it's almost a Cone Search. The semantics are a bit different (search radius is irrelevant, and only one result is required for each query) but existing cone search clients (e.g. the multi-cone capabilities in TOPCAT or STILTS) could be used to access a service exposed in such a way. The same could be said of anything which is basically an all-sky pixel map where you might want to know the results a pixel at a time (does the CMB fall into this category? Anything else?) Any volunteers to set up such a service? Mark On Wed, 9 Mar 2011, Thomas Boch wrote: > Hi, > > IMHO, it would make more sense if someone sets up a REST service allowing this > kind of calculation, and possibly following the IVOA UWS pattern. This would > allow Topcat (and other VO-aware applications) to easily access it. > Cheers, > > Thomas > > On 09/03/11 12:49, Maciej Bilicki wrote: > > Dear All, > > > > The paper by Schlegel, Finkbeiner and Davis, which is now the main reference > > paper to calculate the Galactic extinction, is accompanied by easy-to-use > > routines in C, Fortran and IDL, under the link that Mark gave. All one needs > > is to export the coordinates of the objects and apply the maps for them with > > some smartly chosen parameters. This can easily be recombined with the table > > in TOPCAT, I guess... > > > > Maciek > > > > On Wed, 9 Mar 2011, Mark Taylor wrote: > > > > > On Tue, 8 Mar 2011, Raffaele D'Abrusco wrote: > > > > > > > Hi Mark and Topcat users, > > > > > > > > I know that there are tons of webservices out there for the evaluation > > > > of galactic extinction, but I think it would be very handy to get it > > > > from > > > > inside Topcat, the very place where all my precious tables are > > > > retrieved and manipulated... > > > > Is there any chance that such feature will be introduced in a future > > > > release of the software? > > > > > > Raffaele, > > > > > > Thanks for this suggestion. I don't know much about galactic > > > extinction, but I've had a quick Google. Although it doesn't look > > > impossible to add functionality along these lines, I don't think I'm > > > likely to do it. > > > > > > In general I'm happy to add functions which people will find useful > > > if there is a formula or algorithm for the calculation which is > > > uncontroversial, not too complicated, and with a relatively small > > > number of input parameters. > > > > > > As far as I can tell, as well as being fairly complicated to > > > calculate, in order to provide galactic extinctions I'd need > > > to carry around very large data files inside TOPCAT (200Mb, from > > > the look of http://astro.berkeley.edu/~marc/dust/data/data.html), > > > and the inconvenience of burdening the software with that many > > > extra bytes wouldn't be worth the benefit. I'm also a bit wary > > > of adding black box type functions which rely on data and assumptions > > > rather than ones that I can document the exact expressions for. > > > > > > It would be feasible to write a plugin to do, e.g., Schlegel type > > > extinction calculations that could be used with TOPCAT, but... > > > the amount of work would be considerable (converting the IDL code > > > to Java), so again it doesn't look too likely. > > > > > > Another option would be to have TOPCAT talk to an external web service > > > which does the calculations. This would require some additional > > > infrastructure in the program, but that might be useful for use of > > > other similar type calculation services, so it might be worth it. > > > However, I don't see any web services out there for galactic > > > extinction calculation that are capable of being (easily) driven > > > by software rather than by humans typing numbers into web forms, > > > so it wouldn't be easy. > > > > > > So, probably not. But I'm happy to hear other people's perspectives, > > > especially if it sounds like I may have misunderstood what the > > > requirement is here. > > > > > > Mark > > > > > > -- > > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > > > > > > > > > -- > Thomas Boch > CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 > 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 > F-67000 Strasbourg Email : thomas.boch@astro.XXX > France http://cdsweb.u-strasbg.fr/~boch > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From eglez@ast.XXX Fri Mar 11 14:55:40 2011 Date: Fri, 11 Mar 2011 14:54:34 +0000 From: Eduardo Gonzalez Solares To: topcat-user@bristol.XXX Subject: Using FITS keywords from HDU Hi Mark, Is there currently a way of reading FITS keywords from the headers and use them in calculations, or to create new columns? If not, is this something you would consider implementing? The rationale is that for cases in which there is a unique number (like magnitude zero point, aperture correction, exposure time, ....) valid for all elements of a table, instead of having multiple columns with just all rows the same number it is preferable to store that information on the header of the FITS table saving space and making the FITS table tidier and simpler. It also makes table updating much easier, e..g if the zero point changes, then I only need to change a value in the header, not several thousand values in a column. What do you think? Cheers, Eduardo Eduardo Gonzalez-Solares Institute of Astronomy, Cambridge Astronomy Survey Unit Madingley Rd, CB3 0HA, UK ONLINE INFO: web ∴ blog ∴ skype There are two ways to write error-free programs. Only the third one works. From S.Oliver@sussex.XXX Fri Mar 11 17:06:52 2011 Date: Fri, 11 Mar 2011 17:06:26 +0000 From: Seb Oliver To: Eduardo Gonzalez Solares Cc: topcat-user@bristol.XXX Subject: Re: Using FITS keywords from HDU that would be good. Topcat does understand parameters which have the same function. On 11 Mar 2011, at 14:54, Eduardo Gonzalez Solares wrote: > Hi Mark, > > Is there currently a way of reading FITS keywords from the headers > and use them in calculations, or to create new columns? If not, is > this something you would consider implementing? > > The rationale is that for cases in which there is a unique number > (like magnitude zero point, aperture correction, exposure > time, ....) valid for all elements of a table, instead of having > multiple columns with just all rows the same number it is preferable > to store that information on the header of the FITS table saving > space and making the FITS table tidier and simpler. It also makes > table updating much easier, e..g if the zero point changes, then I > only need to change a value in the header, not several thousand > values in a column. > > What do you think? > > Cheers, > > Eduardo > > Eduardo Gonzalez-Solares > Institute of Astronomy, Cambridge Astronomy Survey Unit > Madingley Rd, CB3 0HA, UK > ONLINE INFO: web ∴ blog ∴ skype > There are two ways to write error-free programs. Only the third one > works. > From jtlz2@astro.XXX Fri Mar 11 17:58:59 2011 Date: Fri, 11 Mar 2011 12:58:08 -0500 From: Jonathan Zwart To: topcat-user@bristol.XXX Eduardo Gonzalez Solares Subject: Re: Using FITS keywords from HDU You can check by clicking on the 'Display table metadata' button, right..? In the help, A3.2 implies that FITS header values should be present, and you can then refer to them in formulae etc. Jon On 11 March 2011 12:06, Seb Oliver wrote: > that would be good. Topcat does understand parameters which have the same > function. > > On 11 Mar 2011, at 14:54, Eduardo Gonzalez Solares wrote: > > Hi Mark, > > Is there currently a way of reading FITS keywords from the headers and use > them in calculations, or to create new columns? If not, is this something > you would consider implementing? > > The rationale is that for cases in which there is a unique number (like > magnitude zero point, aperture correction, exposure time, ....) valid for > all elements of a table, instead of having multiple columns with just all > rows the same number it is preferable to store that information on the > header of the FITS table saving space and making the FITS table tidier and > simpler. It also makes table updating much easier, e..g if the zero point > changes, then I only need to change a value in the header, not several > thousand values in a column. > > What do you think? > > Cheers, > > Eduardo > > [image: Eduardo] *Eduardo Gonzalez-Solares* > Institute of Astronomy, Cambridge Astronomy Survey Unit > Madingley Rd, CB3 0HA, UK > ONLINE INFO: web ∴ blog∴ > skype > There are two ways to write error-free programs. Only the third one > works. > > > From jmvarela@iac.XXX Fri Mar 11 18:34:16 2011 Date: Fri, 11 Mar 2011 18:34:00 +0000 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Using FITS keywords from HDU Hi, as Jon said, it is possible to refer to table parameters (I guess also FITS parameters) using the expression: param$param_name For example, if the zero point is in a keyword called "ZP" you can refer to it in formulae using: param$ZP This is explained in section "7.3 Referencing Table Parameters" of the help. There is a drawback however, and it is that names cannot have spaces. In this case, maybe it is needed to create a new keyword with a suitable name. BTW, Mark, would it be possible to allow to reference to columns with spaces in their names? Cheers, Jesús Jonathan Zwart ha escrito: > You can check by clicking on the 'Display table metadata' button, right..? > In the help, A3.2 implies that FITS header values should be present, and you > can then refer to them in formulae etc. > > Jon > > > On 11 March 2011 12:06, Seb Oliver wrote: > >> that would be good. Topcat does understand parameters which have the same >> function. >> >> On 11 Mar 2011, at 14:54, Eduardo Gonzalez Solares wrote: >> >> Hi Mark, >> >> Is there currently a way of reading FITS keywords from the headers and use >> them in calculations, or to create new columns? If not, is this something >> you would consider implementing? >> >> The rationale is that for cases in which there is a unique number (like >> magnitude zero point, aperture correction, exposure time, ....) valid for >> all elements of a table, instead of having multiple columns with just all >> rows the same number it is preferable to store that information on the >> header of the FITS table saving space and making the FITS table tidier and >> simpler. It also makes table updating much easier, e..g if the zero point >> changes, then I only need to change a value in the header, not several >> thousand values in a column. >> >> What do you think? >> >> Cheers, >> >> Eduardo >> >> [image: Eduardo] *Eduardo Gonzalez-Solares* >> Institute of Astronomy, Cambridge Astronomy Survey Unit >> Madingley Rd, CB3 0HA, UK >> ONLINE INFO: web ∴ >> blog∴ >> skype >> There are two ways to write error-free programs. Only the third one >> works. >> >> >> > -- **************************************************************** * Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * * * * Instituto de Astrofísica de Canarias * * --------------- www.iac.es ------------- * * * * C/ Vía Láctea s/n Tel:+34 922 605278 * * E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * **************************************************************** --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Mon Mar 14 09:49:41 2011 Date: Mon, 14 Mar 2011 09:49:41 +0000 (GMT) From: Mark Taylor To: Jonathan Zwart Cc: topcat-user@bristol.XXX Eduardo Gonzalez Solares Subject: Re: Using FITS keywords from HDU Eduardo, Exactly as Jon says. So for instance if a FITS header card 'REF_FREQ' exists in a FITS table extension, you can write something like stilts tpipe in=BL146_1.fits cmd='addcol ref_lambda 3e8/param$ref_freq' and similarly in expressions within TOPCAT. Mark On Fri, 11 Mar 2011, Jonathan Zwart wrote: > You can check by clicking on the 'Display table metadata' button, right..? > In the help, A3.2 implies that FITS header values should be present, and you > can then refer to them in formulae etc. > > Jon > > > On 11 March 2011 12:06, Seb Oliver wrote: > > > that would be good. Topcat does understand parameters which have the same > > function. > > > > On 11 Mar 2011, at 14:54, Eduardo Gonzalez Solares wrote: > > > > Hi Mark, > > > > Is there currently a way of reading FITS keywords from the headers and use > > them in calculations, or to create new columns? If not, is this something > > you would consider implementing? > > > > The rationale is that for cases in which there is a unique number (like > > magnitude zero point, aperture correction, exposure time, ....) valid for > > all elements of a table, instead of having multiple columns with just all > > rows the same number it is preferable to store that information on the > > header of the FITS table saving space and making the FITS table tidier and > > simpler. It also makes table updating much easier, e..g if the zero point > > changes, then I only need to change a value in the header, not several > > thousand values in a column. > > > > What do you think? > > > > Cheers, > > > > Eduardo -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Mar 14 10:04:32 2011 Date: Mon, 14 Mar 2011 10:04:32 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Using FITS keywords from HDU On Fri, 11 Mar 2011, jmvarela@iac.XXX wrote: > For example, if the zero point is in a keyword called "ZP" you can refer to it > in formulae using: > > param$ZP > > This is explained in section "7.3 Referencing Table Parameters" of the help. > > There is a drawback however, and it is that names cannot have spaces. In this > case, maybe it is needed to create a new keyword with a suitable name. This restriction doesn't apply to FITS headers, since they can't have spaces in their names (though they can contain minus signs, which would make them unsuitable for the TOPCAT/STILTS syntax). > BTW, Mark, would it be possible to allow to reference to columns with spaces > in their names? For column names, the most straightforward thing is to use the $ID form rather than the column name, so the third colum can be referred to as "$3". Using STILTS there is another handy way to get round it: use the fixcolnames filter. stilts tpipe in=t1.fits out=t2.fits cmd=fixcolnames will ensure that all the column and parameter names in the output file have suitable syntax for use in TOPCAT/STILTS expressions (mostly it just replaces unhelpful characters with underscores). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Mon Mar 14 14:40:50 2011 Date: Mon, 14 Mar 2011 15:40:28 +0000 From: Seb Oliver To: topcat-user@bristol.XXX Subject: weird bug Dear All, I appear to have found a weird bug. If I create some subsets of a table and then look at the statistics of the subsets and then change the expressions for the subsets then go back and look at the statistics it seems to remember the previous expression so that not only does it get the Row statistics wrong but it also resets the value of the "Size" in the Row Subset window See in these screen grabs. Originally ID_8 is not of ID _6 and has Size 2955. In the first I've changed ID _8 so that it is the same as ID _3 with size = 5108 [ Part 2: "Attached Text" ] [ The following attachment was DELETED when this message was saved: ] [ A Image/PNG (Name="Picture 27.png") segment of about 22,914 bytes. ] [ Part 3: "Attached Text" ] Then I go to the Row Statistics window and there are 2955 nGoods (same as it was originally not 5108 as it should now be) [ Part 4: "Attached Text" ] [ The following attachment was DELETED when this message was saved: ] [ A Image/PNG (Name="Picture 29.png") segment of about 58,285 bytes. ] [ Part 5: "Attached Text" ] And low and behold back in the other window [ Part 6: "Attached Text" ] [ The following attachment was DELETED when this message was saved: ] [ A Image/PNG (Name="Picture 28.png") segment of about 22,332 bytes. ] [ Part 7: "Attached Text" ] Anyone else had a similar problem? Seb. From m.b.taylor@bristol.XXX Mon Mar 14 16:24:22 2011 Date: Mon, 14 Mar 2011 16:24:22 +0000 (GMT) From: Mark Taylor To: Seb Oliver Cc: topcat-user@bristol.XXX Subject: Re: weird bug On Mon, 14 Mar 2011, Seb Oliver wrote: > Dear All, > > I appear to have found a weird bug. If I create some subsets of a table and > then look at the statistics of the subsets and then change the expressions for > the subsets then go back and look at the statistics it seems to remember the > previous expression so that not only does it get the Row statistics wrong but > it also resets the value of the "Size" in the Row Subset window > > > > See in these screen grabs. Originally ID_8 is not of ID _6 and has Size 2955. > In the first I've changed ID _8 so that it is the same as ID _3 with size = > 5108 Hmm, I see what you mean. I will try to do something about it. In the mean time you can fix the behaviour by hand by hitting the "Recalculate" button in the Statistics window if you think that it has got, or might have got, confused in this way (if that doesn't work, let me know). Thanks for the report, Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From molinaro@oats.XXX Tue Mar 22 14:55:30 2011 Date: Tue, 22 Mar 2011 15:53:58 +0100 From: Marco Molinaro To: astro-aladin , topcat-user Subject: SAMP broadcasting error Hi all, I take advantage of the newly created Aladin list merged to the TOPCAT users' one. Searching for data from the SIA service: http://palantir10.oats.inaf.it:8080/ivoa/resources/siap/58? m33 2 deg in TOPCAT I get some results. The same happens for the similar query done from Aladin (server selector URL directly) http://palantir10.oats.inaf.it:8080/ivoa/resources/siap/58?POS=23.4261,30.66&SIZE=2.0 However when I try to send the table from TOPCAT to Aladin I get an error, criptic as a java.lang.NullPointerException can be (see details below). Do you have any suggestion where to search for the problem? I think it's on my side (I prepared that service) but I don't know how to check the table for errors. I already validate the SIA service through the euro-vo.org validator, and it looks like it's compliant. Thank you all in advance cheers Marco details --------- alert: java.lang.NullPointerException SAMP status response text: could not load VOTable java out: TOPCAT: WARNING: table.load.votable: error sending to Aladin (org.astrogrid.samp.client.LogResultHandler.result) WARNING: Could not load VOTable ! (org.astrogrid.samp.client.LogResultHandler.result) Aladin: java.lang.NullPointerException at cds.aladin.TreeView.createNewBranch(TreeView.java:337) at cds.aladin.TreeView.updateTree(TreeView.java:165) at cds.aladin.Server.updateMetaData(Server.java:230) at cds.aladin.Server.updateMetaData(Server.java:237) at cds.aladin.ServerFile.creatLocalPlane(ServerFile.java:318) at cds.aladin.Calque.newPlan(Calque.java:3018) at cds.aladin.SAMPManager.loadVOTFromURL(SAMPManager.java:1341) at cds.aladin.SAMPManager.execute(SAMPManager.java:436) at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:746) at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:626) at java.lang.Thread.run(Unknown Source) !!! java.lang.NullPointerException I'm using: TOPCAT Version 3.6 STIL Version 2.9-3x Starjava revision: 9468 JVM: Java HotSpot(TM) Client VM version 19.0-b09 SoG: absent Aladin: version 7.005 From buehler@stanford.XXX Fri Mar 25 09:37:36 2011 Date: Thu, 24 Mar 2011 16:50:56 -0700 From: Rolf Buehler To: topcat-user@bristol.XXX Subject: histogram erros Dear topcat coders, thanks for the great program. I have one suggestion: it would be great if the histogram viewer had the option to plot poissonian error bars (as mostly counts will be displayed). Cheers, Rolf -- Rolf Buehler +01-650-9265545 www.rolfbuehler.net From m.b.taylor@bristol.XXX Fri Mar 25 09:42:50 2011 Date: Fri, 25 Mar 2011 09:42:50 +0000 (GMT) From: Mark Taylor To: Rolf Buehler Cc: topcat-user@bristol.XXX Subject: Re: histogram erros On Thu, 24 Mar 2011, Rolf Buehler wrote: > Dear topcat coders, > thanks for the great program. I have one suggestion: > it would be great if the histogram viewer had the option > to plot poissonian error bars (as mostly counts will be > displayed). Cheers, Rolf, I'm not quite sure what you mean. Do you mean vertical error bounds on each histogram bar of size sqrt(c) if the bar is c counts high? Or have you got something else in mind. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From JJ.Kavelaars@nrc-cnrc.XXX Fri Mar 25 16:12:58 2011 Date: Fri, 25 Mar 2011 09:12:21 -0700 From: "Kavelaars, JJ" To: "topcat-user@bristol.XXX" Subject: Re: histogram erros Poisson error bars would be non-symetric. P(N=x) = (exp(-l) l^x)/ x! where lambda is value in the bin. Then, you need to choose what P values you want, 95% confidence intervals are typical, and determine the 'x' that provides P=0.05 and 0.95. Those are your errorbars. JJ On 2011-03-25, at 2:42 AM, Mark Taylor wrote: > On Thu, 24 Mar 2011, Rolf Buehler wrote: > >> Dear topcat coders, >> thanks for the great program. I have one suggestion: >> it would be great if the histogram viewer had the option >> to plot poissonian error bars (as mostly counts will be >> displayed). Cheers, > > Rolf, > > I'm not quite sure what you mean. Do you mean vertical error bounds > on each histogram bar of size sqrt(c) if the bar is c counts high? > Or have you got something else in mind. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From ttosu423@gmail.XXX Mon Mar 28 16:38:57 2011 Date: Mon, 28 Mar 2011 14:43:27 +0100 From: PhanTom To: topcat-user@bristol.XXX Subject: Topcat problems Hi, Experienced a row of troubles with TOPCAT v3.5-2 1. Loading a file causes an error. Was working on this file for a few weeks, and now seems to be lost to the bug: java.io.IOException: FITS read error at uk.ac.starlink.fits.ColFitsTableBuilder.makeStarTable(ColFitsTableBuilder.java:72) at uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:429) at uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:597) at uk.ac.starlink.table.gui.FilestoreTableLoadDialog$2.getTables(FilestoreTableLoadDialog.java:97) at uk.ac.starlink.table.gui.MultiTableLoadDialog$MultiLoader.run(MultiTableLoadDialog.java:159) Caused by: nom.tam.fits.TruncatedFileException at uk.ac.starlink.fits.FitsConstants.readHeader(FitsConstants.java:312) at uk.ac.starlink.fits.ColFitsTableBuilder.makeStarTable(ColFitsTableBuilder.java:69) ... 4 more 2. Editing a column expression/description fails to update the values in the table itself. So if one notices a mistake in a formula, you have to go back and redefine (make new columns) all the columns that were made after the mis-entry. 3. No ability to plot multiple columns on one graph - very useful feature missing. 4. Clicking "Load file using tree-view browser" immediatelly crashes the program, including all open and yet unsaved tables. 5. Randomly freezing up. From jtlz2@astro.XXX Mon Mar 28 16:45:51 2011 Date: Mon, 28 Mar 2011 10:45:16 -0400 From: Jonathan Zwart To: PhanTom Cc: topcat-user@bristol.XXX Subject: Re: Topcat problems Hi PhanTom, On 28 March 2011 09:43, PhanTom wrote: > 2. Editing a column expression/description fails to update the values in > the table itself. So if one notices a mistake in a formula, you have to go > back and redefine (make new columns) all the columns that were made after > the mis-entry. This works for me under MacOS - I change the formula and all the entries are updated as soon I hit return, as expected. Jon From thomas.boch@astro.XXX Mon Mar 28 17:38:27 2011 Date: Mon, 28 Mar 2011 17:03:06 +0200 From: Thomas Boch To: PhanTom Cc: topcat-user@bristol.XXX Subject: Re: Topcat problems Hi, Which version of Java are you running ? (you can find out by typing 'java -version' in a terminal) I used to have some problems with some Java applications when running them with OpenJDK. Cheers, Thomas On 28/03/11 15:43, PhanTom wrote: > Hi, > Experienced a row of troubles with TOPCAT v3.5-2 > 1. Loading a file causes an error. Was working on this file for a few > weeks, and now seems to be lost to the bug: > java.io.IOException: FITS read error > at > uk.ac.starlink.fits.ColFitsTableBuilder.makeStarTable(ColFitsTableBuilder.java:72) > at > uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:429) > at > uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:597) > at > uk.ac.starlink.table.gui.FilestoreTableLoadDialog$2.getTables(FilestoreTableLoadDialog.java:97) > at > uk.ac.starlink.table.gui.MultiTableLoadDialog$MultiLoader.run(MultiTableLoadDialog.java:159) > Caused by: nom.tam.fits.TruncatedFileException > at uk.ac.starlink.fits.FitsConstants.readHeader(FitsConstants.java:312) > at > uk.ac.starlink.fits.ColFitsTableBuilder.makeStarTable(ColFitsTableBuilder.java:69) > ... 4 more > 2. Editing a column expression/description fails to update the values > in the table itself. So if one notices a mistake in a formula, you > have to go back and redefine (make new columns) all the columns that > were made after the mis-entry. > 3. No ability to plot multiple columns on one graph - very useful > feature missing. > 4. Clicking "Load file using tree-view browser" immediatelly crashes > the program, including all open and yet unsaved tables. > 5. Randomly freezing up. -- Thomas Boch CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From m.b.taylor@bristol.XXX Tue Mar 29 09:09:18 2011 Date: Tue, 29 Mar 2011 09:09:18 +0100 (BST) From: Mark Taylor To: PhanTom Cc: topcat-user@bristol.XXX Subject: Re: Topcat problems Dear Mr Tom, On Mon, 28 Mar 2011, PhanTom wrote: > Hi, > > Experienced a row of troubles with TOPCAT v3.5-2 > > 1. Loading a file causes an error. Was working on this file for a few weeks, > and now seems to be lost to the bug: > java.io.IOException: FITS read error > at > uk.ac.starlink.fits.ColFitsTableBuilder.makeStarTable(ColFitsTableBuilder.java:72) > at > uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:429) > at > uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:597) > at > uk.ac.starlink.table.gui.FilestoreTableLoadDialog$2.getTables(FilestoreTableLoadDialog.java:97) > at > uk.ac.starlink.table.gui.MultiTableLoadDialog$MultiLoader.run(MultiTableLoadDialog.java:159) > Caused by: nom.tam.fits.TruncatedFileException I'm assuming this was a FITS file. This error looks like the file has been truncated - a good check for this is to see if the file length is a multiple of 2880, if not then it can't be a legal FITS file, it's missing some bytes. Most likely cause I should think is that when you were saving it the file system ran out of space and couldn't write it all. I think this would have generated an error at save time. Or possibly the write was interrupted in some other way - I haven't seen TOPCAT write incomplete FITS files spontaneously before. If there is important information in the file, I may be able to help read what's there by performing some repair on the FITS file. Any data that should have been in the part of the file that's missing is of course gone. > 2. Editing a column expression/description fails to update the values in the > table itself. So if one notices a mistake in a formula, you have to go back > and redefine (make new columns) all the columns that were made after the > mis-entry. As Jon Zwart says, editing the contents of the Expression field ought to update the columns. However if you've saved the file and reloaded it, the expression text is only saved for documentation purposes in the Description column; the saved file contains numeric data in the columns not the rules for how to calculate them. Editing the content of the Description field won't change any values. > 3. No ability to plot multiple columns on one graph - very useful feature > missing. You can overplot multiple data sets from the same or different tables on a single graph. Use the Add Dataset button (see http://www.starlink.ac.uk/topcat/sun253/PointSelector.html). > 4. Clicking "Load file using tree-view browser" immediatelly crashes the > program, including all open and yet unsaved tables. Sorry if you had that problem. It is usually related to unhappy interaction between TOPCAT and older versions of Starlink software installed on the same machine. I believe it is fixed in the current version (3.7), but I have thought that about some earlier versions and it's come back again. > 5. Randomly freezing up. Difficult to comment without more detail. Possibly to do with running out of memory, may need to start it with more, though it might be something else. As Thomas says, if you're using OpenJDK it's advised to move to the Sun version. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Tue Mar 29 15:16:20 2011 Date: Tue, 29 Mar 2011 15:16:09 +0200 From: Ivan Zolotukhin To: topcat-user Subject: auto table format recognition Hi Mark, All this time I was suffering from TOPCAT's inability to properly recognize straightforward data formats when opening files and finally decided to ask you about this problem. Can you please explain the logic behind 'format: (auto)' default option in the open file dialogue? It seems to be somewhat strange because when one opens file.csv it fails with the error message uk.ac.starlink.table.TableFormatException: Can't make StarTables from "/path/to/file.csv" (tried FITS-plus, colfits-plus, colfits-basic, FITS, VOTable) at uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:466) at uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:597) at uk.ac.starlink.table.gui.SystemTableLoadDialog$1.run(SystemTableLoadDialog.java:130) and user has to explicitly set 'format: CSV' setting to open it. I think the same thing happens with some other formats, not only CSV like in this example. Wouldn't it be better to figure out the format from the file extension or try brute force parsing with all available parsers? I think it's cheap to try/catch many formats and save a couple of clicks, making user a bit more happy about clever software. -- Regards, Ivan From ivan.zolotukhin@gmail.XXX Tue Mar 29 16:08:41 2011 Date: Tue, 29 Mar 2011 16:08:26 +0200 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user Subject: Re: auto table format recognition Mark, >> All this time I was suffering from TOPCAT's inability to properly >> recognize straightforward data formats when opening files and finally >> decided to ask you about this problem. >> >> Can you please explain the logic behind 'format: (auto)' default >> option in the open file dialogue? It seems to be somewhat strange >> because when one opens file.csv it fails with the error message >> >> uk.ac.starlink.table.TableFormatException: Can't make StarTables from >> "/path/to/file.csv" (tried FITS-plus, colfits-plus, colfits-basic, >> FITS, VOTable) >>       at uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:466) >>       at uk.ac.starlink.table.StarTableFactory.makeStarTables(StarTableFactory.java:597) >>       at uk.ac.starlink.table.gui.SystemTableLoadDialog$1.run(SystemTableLoadDialog.java:130) >> >> and user has to explicitly set 'format: CSV' setting to open it. I >> think the same thing happens with some other formats, not only CSV >> like in this example. Wouldn't it be better to figure out the format >> from the file extension or try brute force parsing with all available >> parsers? I think it's cheap to try/catch many formats and save a >> couple of clicks, making user a bit more happy about clever software. > > Since I find nothing more annonying than a machine that thinks it's > smarter than me, I try to avoid clever software at all costs :-). Don't leave (auto) mode then and machine will do only what you explicitly ask. Aren't you annoyed when you know that machine can do the job for you but misses this functionality? When user leaves (auto) he/she expects a machine to be smart, doesn't he? If you're not agree with these thoughts, consider renaming (auto) to FITS/VOTable (or better wording) to remove the confusion because it is what actually happens. > The logic for this behaviour is that determining ASCII-like file > formats by examining file contents is essentially an artificial > intelligence job.  What's this: > >    I am a file, >    with some, data in. > > a plain ASCII table?  a CSV file?  neither?  both?   Brute force > might get it wrong, and worse, might need to suck in a lot of > bytes before it has worked out whether it was a CSV file or ASCII > file or core dump or what.  Trial and error probably is cheap > (though potentially error prone) if the files are short.  If the > files are very long (always an eventuality I have in mind with > TOPCAT/STILTS), it could be very expensive. I'm asking about 20/80 functionality. 80% of CSV files are named .csv. 80% of ASCII files are named .ascii or .txt or .tab (does not matter, if there are couple of more variants). 80% of real life *text* tables are less than 1MB which you can detect at load time. From the first rows of a file you can understand what text format it is. If there's an error after, it would also happen when user set the format explicitly, but you still saved those "choose the format" clicks. So from my experience, there are no problems here to make 80% users happy for 20% of complete solution coding time. The existing architecture design can make it more difficult though. But still, (auto) for many users means "guess it for me among all available formats", not just among those that are easily detectable. At least miriads of other programs that open files work like that by default. > > FITS and VOTable (and variants) are reasonably easy to identify from > their magic numbers and (auto) mode can pick these up. > For any other currently supported file type, which basically means > variants of ASCII, (auto) doesn't work and you are required to specify > the type explicitly. > > I don't take notice of file names (extensions) on load for two reasons: > >   1.  file names can lie >   2.  the loading framework is for the most part based on streams, >       so a filename may not be available to the code > > Mark > > -- > Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From m.b.taylor@bristol.XXX Tue Mar 29 16:48:06 2011 Date: Tue, 29 Mar 2011 16:48:04 +0100 (BST) From: Mark Taylor To: PhanTom Cc: topcat-user@bristol.XXX Subject: Re: Topcat problems On Tue, 29 Mar 2011, PhanTom wrote: > Hi Mark, > > Thanks for the tips. What Thomas said, doesn't quite apply to me: I can't > change the platform we're using, as it's set up by the university :) Also > unsure how to change the amount of memory it starts with, it seems limited > to 64M. In order to change the amount of memory, you'll have to start it from the command line (DOS window/Unix shell), and write something like java -Xmx512M topcat-full.jar (either look for the topcat-full.jar file wherever the thing is installed at the moment, or download it from http://www.starlink.ac.uk/topcat/topcat-full.jar). It should be possible to download a Java version from Oracle (Java SE JRE) and install it in your own disk space if necessary on a unix-like OS. For MS Windows I don't know, but it's likely that you've got a sensible version if you're on Windows in any case. > I have attached the corrupted file, if you can do something with it, > it would be great. How fascinating. The file wasn't truncated by saving to a full disk as I suspected, the corruption had to do with a pound sign (£) in the description of column 241. This must be a nasty multi-byte character handling bug in TOPCAT. Thanks for pointing this out - I will look into why it has happened and try to fix it ASAP. In the mean time, try to avoid strange characters in column descriptions. I'll send you the fixed file in a separate off-list mail. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From price@astro.XXX Tue Mar 29 17:59:25 2011 Date: Tue, 29 Mar 2011 11:56:11 -0400 From: Paul Price To: Mark Taylor Cc: PhanTom , topcat-user@bristol.XXX Subject: Re: Topcat problems On Tue, 2011-03-29 at 16:48 +0100, Mark Taylor wrote: > How fascinating. The file wasn't truncated by saving to a full disk > as I suspected, the corruption had to do with a pound sign (£) in the > description of column 241. This must be a nasty multi-byte character > handling bug in TOPCAT. From the FITS definition, v3.0, section 4.2.1: "A character string value shall be composed only of the set of restricted ASCII text characters, decimal 32 through 126". The British pound symbol is not in ASCII. P. [ Part 2, "This is a digitally signed message part" ] [ Application/PGP-SIGNATURE (Name: "signature.asc") 205 bytes. ] [ Unable to print this part. ] From cgp@star.XXX Tue Mar 29 19:17:03 2011 Date: Tue, 29 Mar 2011 18:16:53 +0100 From: Clive Page To: topcat-user@sympa.XXX Subject: Re: auto table format recognition > The logic for this behaviour is that determining ASCII-like file > formats by examining file contents is essentially an artificial > intelligence job. What's this: Agreed, I've grappled with this myself in the past, it's not trivial. > I don't take notice of file names (extensions) on load for two reasons: > > 1. file names can lie > 2. the loading framework is for the most part based on streams, > so a filename may not be available to the code True, but if the extension is wrong then it is the user's own fault, surely. I accept that the filename might not be easily accessible to the lower layers of software, but would like to support the original suggestion that, if it isn't really difficult, then the default (auto) option ought to be slightly cleverer, e.g. to accept that files with extensions of .csv or .CSV are indeed in comma-separated-value format. I know it's only a few clicks to set an explicit input format, but that tends to annoy when it happens repeatedly. TOPCAT is so easy to use in most other ways, it's a pity that reading non-FITS input files is not quite as smooth as one feels it could be. Regards -- Clive Page From jtlz2@astro.XXX Tue Mar 29 19:42:26 2011 Date: Tue, 29 Mar 2011 13:41:55 -0400 From: Jonathan Zwart To: Clive Page Cc: topcat-user@sympa.XXX Subject: Re: auto table format recognition > I accept that the filename might not be easily accessible to the lower > layers of software, but would like to support the original suggestion that, > if it isn't really difficult, then the default (auto) option ought to be > slightly cleverer, e.g. to accept that files with extensions of .csv or .CSV > are indeed in comma-separated-value format. I know it's only a few clicks > to set an explicit input format, but that tends to annoy when it happens > repeatedly. TOPCAT is so easy to use in most other ways, it's a pity that > reading non-FITS input files is not quite as smooth as one feels it could > be. > I've asked for this before, so I feel I ought to second or third it now :-) Indeed don't most OSs operate along these lines? I can make MacOS's preview choke on a PDF just by changing its extension to PS - the file is the same but the OS is assuming that the new extension is correct - nothing more sophisticated. Jon From S.Oliver@sussex.XXX Wed Mar 30 00:16:43 2011 Date: Tue, 29 Mar 2011 23:16:31 +0100 From: Seb Oliver To: Jonathan Zwart Cc: Clive Page , topcat-user@sympa.XXX Subject: Re: auto table format recognition I know nothing about this so forgive my intrusion...but don't we (as users) train our OS and browsers to recognise particular file types through their extensions and whatever (is this mime-types or is that just mail browsers). Would it be possible for topcat to tap into this kind of thing so the user was in charge of what auto did? Seb. On 29 Mar 2011, at 18:41, Jonathan Zwart wrote: > > I accept that the filename might not be easily accessible to the > lower layers of software, but would like to support the original > suggestion that, if it isn't really difficult, then the default > (auto) option ought to be slightly cleverer, e.g. to accept that > files with extensions of .csv or .CSV are indeed in comma-separated- > value format. I know it's only a few clicks to set an explicit > input format, but that tends to annoy when it happens repeatedly. > TOPCAT is so easy to use in most other ways, it's a pity that > reading non-FITS input files is not quite as smooth as one feels it > could be. > > I've asked for this before, so I feel I ought to second or third it > now :-) > > Indeed don't most OSs operate along these lines? I can make MacOS's > preview choke on a PDF just by changing its extension to PS - the > file is the same but the OS is assuming that the new extension is > correct - nothing more sophisticated. > > Jon From dabrusco@gmail.XXX Wed Mar 30 21:40:36 2011 Date: Wed, 30 Mar 2011 15:40:06 -0400 From: Raffaele D'Abrusco To: topcat-user@bristol.XXX Subject: Reordering list of columns from the "Table Columns" window? Hi Mark, I don't remember if this point has already been raised before in the Topcat mailing list (or somewhere else), but I would find very handy to be able to re-order the columns of a table from the "Table Columns" window instead of from the "Table Browser" window. This would be particularly useful when dealing with tables containing many columns, in which case the user is required to drag one column at a time from one side to the other of the visible portion of the table in the "Table Browser" window, then scrolling the table, dragging the column again, etc. Would implementing a similar mechanism in the "Table Columns" window require a great effort? By adding, for example, an additional column with "handles" (or something similar) to move the columns vertically in the window? Thanks! Raffaele From m.b.taylor@bristol.XXX Thu Mar 31 08:56:50 2011 Date: Thu, 31 Mar 2011 08:56:50 +0100 (BST) From: Mark Taylor To: Raffaele D'Abrusco Cc: topcat-user@bristol.XXX Subject: Re: Reordering list of columns from the "Table Columns" window? Raffeale, thanks for this suggestion. I agree that as it stands it's a pain to reorder columns when there are a lot of them. This one has been lurking on the to-do list for quite a while - I'll see if I can get to it sooner rather than later. Mark On Wed, 30 Mar 2011, Raffaele D'Abrusco wrote: > Hi Mark, > I don't remember if this point has already been raised before in the > Topcat mailing list > (or somewhere else), but I would find very handy to be able to > re-order the columns of > a table from the "Table Columns" window instead of from the "Table > Browser" window. > This would be particularly useful when dealing with tables containing > many columns, in which > case the user is required to drag one column at a time from one side > to the other of the > visible portion of the table in the "Table Browser" window, then > scrolling the table, dragging the > column again, etc. > > Would implementing a similar mechanism in the "Table Columns" window > require a great effort? > By adding, for example, an additional column with "handles" (or > something similar) > to move the columns vertically in the window? > > Thanks! > > Raffaele -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From mara.salvato@ipp.XXX Fri Apr 15 09:29:13 2011 Date: Fri, 15 Apr 2011 09:30:10 +0200 From: Mara Salvato To: topcat-user@mlist2.XXX Subject: acknowledgments Hello, is there a preferred form for acknowledging TOPCAT and STILTS (beside citing the introductory papers?) Thanks, Mara From m.b.taylor@bristol.XXX Fri Apr 15 10:29:07 2011 Date: Fri, 15 Apr 2011 10:29:07 +0100 (BST) From: Mark Taylor To: Mara Salvato Cc: topcat-user@bristol.XXX Subject: Re: acknowledgments On Fri, 15 Apr 2011, Mara Salvato wrote: > Hello, > > is there a preferred form for acknowledging TOPCAT and STILTS (beside citing > the introductory papers?) > > Thanks, > Mara Mara, thanks for asking. There is no recommended form of words, but it is appreciated if you can cite the relevant paper(s) and if you like URL(s) noted here: http://www.star.bristol.ac.uk/~mbt/topcat/faq.html#acknowledgement Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From elle@oal.XXX Tue Apr 26 13:19:56 2011 Date: Mon, 18 Apr 2011 15:01:54 +0100 (WEST) From: Elvira Leonardo To: topcat-user@bristol.XXX Subject: How to save a file in ascii format Good evening, I am a new user of TOPCAT and I have a .fits file which I would like to save as Ascii. How can I do that? Thanks in advance From m.b.taylor@bristol.XXX Tue Apr 26 12:29:27 2011 Date: Tue, 26 Apr 2011 12:29:27 +0100 (BST) From: Mark Taylor To: Elvira Leonardo Cc: topcat-user@bristol.XXX Subject: Re: How to save a file in ascii format On Mon, 18 Apr 2011, Elvira Leonardo wrote: > Good evening, > I am a new user of TOPCAT and I have a .fits file which I would like to save > as Ascii. How can I do that? > Thanks in advance Elvira, first load the FITS file in using the load window: - File|Load Table menu item, or equivalent toolbar button - Use the Filestore Browser button to locate the file, select it by clicking, and click OK then save it using the save window: - Select the table to save in the list on the left of the Control Window (if you've only just loaded it, it will be selected already) - File|Save Table(s)/Session menu item, or equivalent toolbar button - Use the Filestore Browser to indicate where you want to save it - Select "ascii" from the drop-down list in the Output Format selector - Click OK to save There are variants on this procedure, but the basic idea is the same. Mark PS The long delay before this message appeared on the list is because you are not subscribed to it. If you are subscribed to the list, your messages will appear immediately. If you are not subscribed, they appear only when the moderator (me) explicitly approves them. Normally I do that straight away, but I was on holiday last week. So, it's not compulsory to subscribe to the list in order to post to it, but you'll get faster service if you do. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ianc@ucla.XXX Mon May 9 10:30:48 2011 Date: Sun, 08 May 2011 19:37:02 -0700 From: Ian Crossfield Reply-To: ianc@astro.XXX To: topcat-user@bristol.XXX Subject: Saving synthetic column expressions Hello, First-time question from a new TOPCAT user: after creating a synthetic column using an algebraic expression that is dependent on other columns, is there any way to save the current table so that the algebraic expression is saved (instead of saving the calculated values)? Documentation says only that "the new column and its contents will be written to the new output table," but none of the formats I have tried (various VOT, FITS, etc.) seem to implement the functionality I want: they save the calculated value, not the expression that computes it. This means that anytime I close TOPCAT and reopen my data, updating any of my low-level values requires me to re-enter the (long) algebraic expression that calculates the higher-level values. Any spreadsheet program can do this, but for a host of reasons I'd much rather be able to do it in TOPCAT. How can I do so? Thanks, -- Ian Crossfield UCLA Astronomy KH 3-145J http://www.astro.ucla.edu/~ianc/ From jmvarela@iac.XXX Mon May 9 12:37:58 2011 Date: Mon, 09 May 2011 11:37:38 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Saving synthetic column expressions Hi Ian, I'm not sure about the possibility of storing the expressions through TOPCAT. However, you can use STILTS (~ the command line of TOPCAT) as a workaround. For example, to add a new column (newcol1) in table "Table1.fits" using expression "expression1" and loading it into TOPCAT you can use the command: stilts tpipe cmd='addcol newcol1 expression1' in="Tabl1.fits" omode=topcat You can keep this command (or several commands) in a text file (AddNewColumn) and do: source AddNewColumn (this works using tcsh as shell, not sure with bash) each time you want to load the table with the new column. You can put as many "cmd='...'" as you need, so you can add many columns. However, I think that this doesn't keep the expression to be modified within TOPCAT :( Of course, you need to have stilts installed since, I think, it is not install with TOPCAT. Hope this helps, Jesús Ian Crossfield ha escrito: > Hello, > > First-time question from a new TOPCAT user: after creating a synthetic > column using an algebraic expression that is dependent on other > columns, is there any way to save the current table so that the > algebraic expression is saved (instead of saving the calculated > values)? Documentation says only that "the new column and its contents > will be written to the new output table," but none of the formats I > have tried (various VOT, FITS, etc.) seem to implement the > functionality I want: they save the calculated value, not the > expression that computes it. > > This means that anytime I close TOPCAT and reopen my data, updating any > of my low-level values requires me to re-enter the (long) algebraic > expression that calculates the higher-level values. Any spreadsheet > program can do this, but for a host of reasons I'd much rather be able > to do it in TOPCAT. How can I do so? > > Thanks, > > -- > Ian Crossfield > UCLA Astronomy > KH 3-145J > http://www.astro.ucla.edu/~ianc/ -- **************************************************************** * Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * * * * Instituto de Astrofísica de Canarias * * --------------- www.iac.es ------------- * * * * C/ Vía Láctea s/n Tel:+34 922 605278 * * E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * **************************************************************** From m.b.taylor@bristol.XXX Mon May 9 20:17:03 2011 Date: Mon, 9 May 2011 20:17:02 +0100 (BST) From: Mark Taylor To: topcat-announce@bristol.XXX topcat-user@bristol.XXX Cc: IVOA Applications WG Subject: TOPCAT/STILTS release Dear TOPCAT users, this is to announce the following releases: TOPCAT v3.8 http://www.starlink.ac.uk/topcat/ STILTS v2.3 http://www.starlink.ac.uk/stilts/ STIL v3.0-1 http://www.starlink.ac.uk/stil/ The headline item is TAP support: a TAP load dialogue in TOPCAT (see the Load Window or VO menu) and new commands tapquery and tapresume in STILTS. TAP is the VO Table Access Protocol, and allows you to perform SQL-like queries on remote databases. The other main noticeable change is that TOPCAT now starts a SAMP hub by default on startup, so you don't need to perform any explicit actions to start using SAMP. There are a number of other more minor enhancements and bug fixes, see here for details: TOPCAT: http://www.starlink.ac.uk/topcat/sun253/versions.html STILTS: http://www.starlink.ac.uk/stilts/sun256/versions.html STIL: http://www.starlink.ac.uk/stil/sun252/history.html Best wishes, Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jmvarela@iac.XXX Mon May 9 21:48:17 2011 Date: Mon, 09 May 2011 20:47:46 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: TOPCAT/STILTS release Hi Mark! Thanks a lot for these new versions! Jesús Mark Taylor ha escrito: > Dear TOPCAT users, > > this is to announce the following releases: > > TOPCAT v3.8 http://www.starlink.ac.uk/topcat/ > STILTS v2.3 http://www.starlink.ac.uk/stilts/ > STIL v3.0-1 http://www.starlink.ac.uk/stil/ > > The headline item is TAP support: a TAP load dialogue in TOPCAT > (see the Load Window or VO menu) and new commands tapquery and tapresume > in STILTS. TAP is the VO Table Access Protocol, and allows you to > perform SQL-like queries on remote databases. > > The other main noticeable change is that TOPCAT now starts a SAMP hub > by default on startup, so you don't need to perform any explicit > actions to start using SAMP. > > There are a number of other more minor enhancements and bug fixes, > see here for details: > > TOPCAT: http://www.starlink.ac.uk/topcat/sun253/versions.html > STILTS: http://www.starlink.ac.uk/stilts/sun256/versions.html > STIL: http://www.starlink.ac.uk/stil/sun252/history.html > > Best wishes, > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > -- **************************************************************** * Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * * * * Instituto de Astrofísica de Canarias * * --------------- www.iac.es ------------- * * * * C/ Vía Láctea s/n Tel:+34 922 605278 * * E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * **************************************************************** From m.b.taylor@bristol.XXX Mon May 9 23:39:18 2011 Date: Mon, 9 May 2011 23:39:17 +0100 (BST) From: Mark Taylor To: Ian Crossfield Cc: topcat-user@sympa.XXX Subject: Re: Saving synthetic column expressions Thanks Jesús, to confirm: there is currently no way to save expressions rather than values in a spreadsheet-like way. It's something which has been asked about a few times and it might be possible to introduce it at some stage if there is wide demand. Using STILTS in the way that Jesús outlines is the recommended way to handle this at the moment. Mark On Mon, 9 May 2011, jmvarela@iac.XXX wrote: > Hi Ian, > > I'm not sure about the possibility of storing the expressions through TOPCAT. > However, you can use STILTS (~ the command line of TOPCAT) as a workaround. > For example, to add a new column (newcol1) in table "Table1.fits" using > expression "expression1" and loading it into TOPCAT you can use the command: > > stilts tpipe cmd='addcol newcol1 expression1' in="Tabl1.fits" omode=topcat > > You can keep this command (or several commands) in a text file (AddNewColumn) > and do: > > source AddNewColumn (this works using tcsh as shell, not sure with bash) > > each time you want to load the table with the new column. > > You can put as many "cmd='...'" as you need, so you can add many columns. > However, I think that this doesn't keep the expression to be modified within > TOPCAT :( > > Of course, you need to have stilts installed since, I think, it is not > install with TOPCAT. Actually the STILTS code comes for free if you have TOPCAT - you can write 'topcat -stilts' rather than 'stilts' if you want. > > Hope this helps, > > Jesús > > Ian Crossfield ha escrito: > > > Hello, > > > > First-time question from a new TOPCAT user: after creating a synthetic > > column using an algebraic expression that is dependent on other > > columns, is there any way to save the current table so that the > > algebraic expression is saved (instead of saving the calculated > > values)? Documentation says only that "the new column and its contents > > will be written to the new output table," but none of the formats I > > have tried (various VOT, FITS, etc.) seem to implement the > > functionality I want: they save the calculated value, not the > > expression that computes it. > > > > This means that anytime I close TOPCAT and reopen my data, updating any > > of my low-level values requires me to re-enter the (long) algebraic > > expression that calculates the higher-level values. Any spreadsheet > > program can do this, but for a host of reasons I'd much rather be able > > to do it in TOPCAT. How can I do so? > > > > Thanks, > > > > -- > > Ian Crossfield > > UCLA Astronomy > > KH 3-145J > > http://www.astro.ucla.edu/~ianc/ > > > > -- > **************************************************************** > * Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * > * * > * Instituto de Astrofísica de Canarias * > * --------------- www.iac.es ------------- * > * * > * C/ Vía Láctea s/n Tel:+34 922 605278 * > * E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * > **************************************************************** > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From pierre.didelon@cea.XXX Tue May 10 07:56:46 2011 Date: Tue, 10 May 2011 08:56:18 +0200 From: Pierre Didelon To: Mark Taylor Cc: Ian Crossfield , topcat-user@sympa.XXX Subject: Re: Saving synthetic column expressions Mark Taylor wrote: >Thanks Jesús, > >to confirm: there is currently no way to save expressions rather than >values in a spreadsheet-like way. It's something which has been asked >about a few times and it might be possible to introduce it at some >stage if there is wide demand. > I vote for it! both hand ;-) Having a button recalculate column (1/n/all/...) would really be very nice... even if we can live without, I feel that it will be a great improvement for data "manipulators" :-) But don't worry you will not loose any "client" we are TOPCAT addict now :-( ;-) Thanks for all, Pierre >Using STILTS in the way that Jesús >outlines is the recommended way to handle this at the moment. > >Mark > >On Mon, 9 May 2011, jmvarela@iac.XXX wrote: > > > >>Hi Ian, >> >> I'm not sure about the possibility of storing the expressions through TOPCAT. >>However, you can use STILTS (~ the command line of TOPCAT) as a workaround. >>For example, to add a new column (newcol1) in table "Table1.fits" using >>expression "expression1" and loading it into TOPCAT you can use the command: >> >>stilts tpipe cmd='addcol newcol1 expression1' in="Tabl1.fits" omode=topcat >> >>You can keep this command (or several commands) in a text file (AddNewColumn) >>and do: >> >>source AddNewColumn (this works using tcsh as shell, not sure with bash) >> >>each time you want to load the table with the new column. >> >> You can put as many "cmd='...'" as you need, so you can add many columns. >>However, I think that this doesn't keep the expression to be modified within >>TOPCAT :( >> >> Of course, you need to have stilts installed since, I think, it is not >>install with TOPCAT. >> >> > >Actually the STILTS code comes for free if you have TOPCAT - you can >write 'topcat -stilts' rather than 'stilts' if you want. > > > >> Hope this helps, >> >>Jesús >> >>Ian Crossfield ha escrito: >> >> >> >>>Hello, >>> >>>First-time question from a new TOPCAT user: after creating a synthetic >>>column using an algebraic expression that is dependent on other >>>columns, is there any way to save the current table so that the >>>algebraic expression is saved (instead of saving the calculated >>>values)? Documentation says only that "the new column and its contents >>>will be written to the new output table," but none of the formats I >>>have tried (various VOT, FITS, etc.) seem to implement the >>>functionality I want: they save the calculated value, not the >>>expression that computes it. >>> >>>This means that anytime I close TOPCAT and reopen my data, updating any >>>of my low-level values requires me to re-enter the (long) algebraic >>>expression that calculates the higher-level values. Any spreadsheet >>>program can do this, but for a host of reasons I'd much rather be able >>>to do it in TOPCAT. How can I do so? >>> >>>Thanks, >>> >>>-- >>>Ian Crossfield >>>UCLA Astronomy >>>KH 3-145J >>>http://www.astro.ucla.edu/~ianc/ >>> >>> >> >>-- >>**************************************************************** >>* Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * >>* * >>* Instituto de Astrofísica de Canarias * >>* --------------- www.iac.es ------------- * >>* * >>* C/ Vía Láctea s/n Tel:+34 922 605278 * >>* E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * >>**************************************************************** >> >> >> > >-- >Mark Taylor Astronomical Programmer Physics, Bristol University, UK >m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > -- Pierre ------------------------------------------------------------------- DIDELON :@: pdidelon_at_cea.fr Phone : 33 (0)1 69 08 58 89 CEA SACLAY - Service d'Astrophysique 91191 Gif-Sur-Yvette Cedex ------------------------------------------------------------------- Aidez les enfants Tibétains : http://www.a-e-t.org/actions/ d'autres : http://www.sosesf.org/ ou trouvez une autre ONG -------------------------------------------------------------------- Seule compte la démarche. Car c'est elle qui dure et non le but qui n'est qu'illusion du voyageur... Saint Exupery - Citadelle -------------------------------------------------------------------- From mara.salvato@ipp.XXX Fri May 13 11:06:03 2011 Date: Fri, 13 May 2011 11:05:38 +0200 From: Mara Salvato To: topcat-user@mlist2.XXX Subject: Maybe a bug in the matching? Hi Mark, I encountered something funny in TOPCAT, while making a match in coordinates. Hope to hbe able to explain it clearly. If in the first file there are duplicated lines, the final output of the match will have a match only for the first one of the duplicated lines and will have NULL for the second one. This does not make sense to me, as if I do not know that there are duplications in my catalog, I assume that the non matched sources do not have a counterpart, and not that they are duplicated. Ideally TOPCAT should make the match for all the sources and at the same time inform that there are XXX duplications.. I understand that we should first run the internal match, clean the catalog and then make a pair match... but it takes longer... don't you agree? Thanks for your work, Mara PS: if you want to try by your self, i can send you a cutout of my files... From m.b.taylor@bristol.XXX Fri May 13 10:23:39 2011 Date: Fri, 13 May 2011 10:23:39 +0100 (BST) From: Mark Taylor To: Mara Salvato Cc: topcat-user@bristol.XXX Subject: Re: Maybe a bug in the matching? Mara, On Fri, 13 May 2011, Mara Salvato wrote: > Hi Mark, > > I encountered something funny in TOPCAT, while making a match in coordinates. > Hope to hbe able to explain it clearly. > > If in the first file there are duplicated lines, the final output of the > match will have a match only for the first one of the duplicated lines and > will have NULL for the second one. > > This does not make sense to me, as if I do not know that there are > duplications in my catalog, I assume that the non matched sources do not have > a counterpart, and not that they are duplicated. > > Ideally TOPCAT should make the match for all the sources and at the same time > inform that there are XXX duplications.. > I understand that we should first run the internal match, clean the catalog > and then make a pair match... but it takes longer... don't you agree? You may be able to get the result that you want by selecting "All Matches" rather than "Best Match Only" in the Output Rows panel at the bottom of the match window. However, if some of the objects in the second file are close together (closer than the matching distance) this may give you more matches than you want. See item 3.12 in the FAQ for more discussion: http://www.starlink.ac.uk/topcat/faq.html#xmatchIssues I'm hoping to introduce an asymmetric best match mode (for each item in table 1, tell me the closest item in table 2) in the next few months. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From deil.christoph@googlemail.XXX Fri May 13 11:28:16 2011 Date: Fri, 13 May 2011 11:27:35 +0200 From: Christoph Deil To: Mark Taylor Cc: Mara Salvato , topcat-user@bristol.XXX Subject: Re: Maybe a bug in the matching? Hi Mara, Mark, I also need the asymmetric best match mode you mentioned below frequently. Simply doing it with TOPCAT would be great! Thanks for implementing it. Christoph On May 13, 2011, at 11:23 AM, Mark Taylor wrote: > Mara, > > On Fri, 13 May 2011, Mara Salvato wrote: > >> Hi Mark, >> >> I encountered something funny in TOPCAT, while making a match in coordinates. >> Hope to hbe able to explain it clearly. >> >> If in the first file there are duplicated lines, the final output of the >> match will have a match only for the first one of the duplicated lines and >> will have NULL for the second one. >> >> This does not make sense to me, as if I do not know that there are >> duplications in my catalog, I assume that the non matched sources do not have >> a counterpart, and not that they are duplicated. >> >> Ideally TOPCAT should make the match for all the sources and at the same time >> inform that there are XXX duplications.. >> I understand that we should first run the internal match, clean the catalog >> and then make a pair match... but it takes longer... don't you agree? > > You may be able to get the result that you want by selecting > "All Matches" rather than "Best Match Only" in the Output Rows > panel at the bottom of the match window. However, if some of > the objects in the second file are close together (closer than > the matching distance) this may give you more matches than you want. > See item 3.12 in the FAQ for more discussion: > > http://www.starlink.ac.uk/topcat/faq.html#xmatchIssues > > I'm hoping to introduce an asymmetric best match mode (for each item > in table 1, tell me the closest item in table 2) in the next > few months. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From mara.salvato@ipp.XXX Fri May 13 11:28:20 2011 Date: Fri, 13 May 2011 11:28:04 +0200 From: Mara Salvato To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Maybe a bug in the matching? Hi Mark, thanks for the suggestion. I was indeed using it as temporary solution. But my catalogs are crowded and as you said, with the "all matches" I get to much. I think that the best is first to clean the catalog with an internal match and then make the pair match. Thanks a lot Mara > Mara, > > On Fri, 13 May 2011, Mara Salvato wrote: > >> Hi Mark, >> >> I encountered something funny in TOPCAT, while making a match in coordinates. >> Hope to hbe able to explain it clearly. >> >> If in the first file there are duplicated lines, the final output of the >> match will have a match only for the first one of the duplicated lines and >> will have NULL for the second one. >> >> This does not make sense to me, as if I do not know that there are >> duplications in my catalog, I assume that the non matched sources do not have >> a counterpart, and not that they are duplicated. >> >> Ideally TOPCAT should make the match for all the sources and at the same time >> inform that there are XXX duplications.. >> I understand that we should first run the internal match, clean the catalog >> and then make a pair match... but it takes longer... don't you agree? > > You may be able to get the result that you want by selecting > "All Matches" rather than "Best Match Only" in the Output Rows > panel at the bottom of the match window. However, if some of > the objects in the second file are close together (closer than > the matching distance) this may give you more matches than you want. > See item 3.12 in the FAQ for more discussion: > > http://www.starlink.ac.uk/topcat/faq.html#xmatchIssues > > I'm hoping to introduce an asymmetric best match mode (for each item > in table 1, tell me the closest item in table 2) in the next > few months. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ -- ******************************************************************************** Mara Salvato Max-Planck-Institut fuer Plasma Physik Bolzmann Strasse 2 85741 Garching GERMANY Phone: (+49)(89) 3299 1553 Email: mara.salvato@ipp.XXX ******************************************************************************** From jtlz2@astro.XXX Fri May 13 15:19:04 2011 Date: Fri, 13 May 2011 10:18:21 -0400 From: Jonathan Zwart To: Mark Taylor Cc: Ian Crossfield , topcat-user@sympa.XXX Subject: Re: Saving synthetic column expressions Hi Mark, On 9 May 2011 18:39, Mark Taylor wrote: > Thanks Jesús, > > to confirm: there is currently no way to save expressions rather than > values in a spreadsheet-like way. It's something which has been asked > about a few times and it might be possible to introduce it at some > stage if there is wide demand. Using STILTS in the way that Jesús > outlines is the recommended way to handle this at the moment. > As a stop-gap, what about writing the expressions to some FITS (of course assuming it's FITS output..) comment fields, so that at least then they could be cut and pasted back? But I don't which other output formats might support something like this. Jon On Mon, 9 May 2011, jmvarela@iac.XXX wrote: > > > Hi Ian, > > > > I'm not sure about the possibility of storing the expressions through > TOPCAT. > > However, you can use STILTS (~ the command line of TOPCAT) as a > workaround. > > For example, to add a new column (newcol1) in table "Table1.fits" using > > expression "expression1" and loading it into TOPCAT you can use the > command: > > > > stilts tpipe cmd='addcol newcol1 expression1' in="Tabl1.fits" > omode=topcat > > > > You can keep this command (or several commands) in a text file > (AddNewColumn) > > and do: > > > > source AddNewColumn (this works using tcsh as shell, not sure with > bash) > > > > each time you want to load the table with the new column. > > > > You can put as many "cmd='...'" as you need, so you can add many > columns. > > However, I think that this doesn't keep the expression to be modified > within > > TOPCAT :( > > > > Of course, you need to have stilts installed since, I think, it is not > > install with TOPCAT. > > Actually the STILTS code comes for free if you have TOPCAT - you can > write 'topcat -stilts' rather than 'stilts' if you want. > > > > > Hope this helps, > > > > Jesús > > > > Ian Crossfield ha escrito: > > > > > Hello, > > > > > > First-time question from a new TOPCAT user: after creating a synthetic > > > column using an algebraic expression that is dependent on other > > > columns, is there any way to save the current table so that the > > > algebraic expression is saved (instead of saving the calculated > > > values)? Documentation says only that "the new column and its contents > > > will be written to the new output table," but none of the formats I > > > have tried (various VOT, FITS, etc.) seem to implement the > > > functionality I want: they save the calculated value, not the > > > expression that computes it. > > > > > > This means that anytime I close TOPCAT and reopen my data, updating any > > > of my low-level values requires me to re-enter the (long) algebraic > > > expression that calculates the higher-level values. Any spreadsheet > > > program can do this, but for a host of reasons I'd much rather be able > > > to do it in TOPCAT. How can I do so? > > > > > > Thanks, > > > > > > -- > > > Ian Crossfield > > > UCLA Astronomy > > > KH 3-145J > > > http://www.astro.ucla.edu/~ianc/ > > > > > > > > -- > > **************************************************************** > > * Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * > > * * > > * Instituto de Astrofísica de Canarias * > > * --------------- www.iac.es ------------- * > > * * > > * C/ Vía Láctea s/n Tel:+34 922 605278 * > > * E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * > > **************************************************************** > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Fri May 13 15:26:18 2011 Date: Fri, 13 May 2011 15:26:18 +0100 (BST) From: Mark Taylor To: Jonathan Zwart Cc: Ian Crossfield , topcat-user@sympa.XXX Subject: Re: Saving synthetic column expressions On Fri, 13 May 2011, Jonathan Zwart wrote: > Hi Mark, > > On 9 May 2011 18:39, Mark Taylor wrote: > > > Thanks Jesús, > > > > to confirm: there is currently no way to save expressions rather than > > values in a spreadsheet-like way. It's something which has been asked > > about a few times and it might be possible to introduce it at some > > stage if there is wide demand. Using STILTS in the way that Jesús > > outlines is the recommended way to handle this at the moment. > > > > As a stop-gap, what about writing the expressions to some FITS (of course > assuming it's FITS output..) comment fields, so that at least then they > could be cut and pasted back? But I don't which other output formats might > support something like this. actually, it does that already - you can see the expression in the Description column of the Columns window. Works for FITS (variants) and VOTable, though if the expression is too long to fit in a FITS header it's omitted in FITS-basic. FITS-plus includes it regardless of length. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From margudo@iaa.XXX Tue May 17 11:50:08 2011 Date: Tue, 17 May 2011 11:49:39 +0200 (CEST) From: Mamen Argudo To: topcat-user@mlist2.XXX Subject: Class shape > 1 Hello, I would like to know if topcat is able to change column format class float[] with shape > 1 without open the original table with a text editor (too large). In particular, I need to change a column class float[5] in 5 independent columns, for example a column "mag" in format (u,g,r,i.z) to "mag_u", "mag_g", "mag_r", "mag_i", "mag_z". Thanks! Mamen From jmvarela@iac.XXX Tue May 17 12:58:17 2011 Date: Tue, 17 May 2011 11:57:51 +0100 From: jmvarela@iac.XXX To: topcat-user@sympa.XXX Subject: Re: Class shape > 1 Hi Mamen, you can use "stilts" (~ topcat command-line) with the following command: stilts tpipe cmd="explodecols multi_dimension_column; colmeta -name newname1 multi_dimension_column_1; colmeta -name newname2 multi_dimension_column_2;....,colmeta -name newname_N multi_dimension_column_N" in=input.fits out=output.fits where "multi_dimension_column" is the identification of the column (in your example "mag") and newname is the names of the new columns ("mag_u","mag_g",...). In summary, you should run the command: stilts tpipe cmd="explodecols mag; colmeta -name mag_u mag_1; colmeta -name mag_g mag_2; colmeta -name mag_r mag_3; colmeta -name mag_i mag_4;; colmeta -name mag_z mag_5" in=input.fits out=out.fits Maybe the "colmeta" commands can be compacted in just one but I'm not sure. Check the syntax because maybe there is something wrong but I think you get the idea. Cheers, Jesús Mamen Argudo ha escrito: > Hello, > > I would like to know if topcat is able to change column format class > float[] with shape > 1 without open the original table with a text editor > (too large). In particular, I need to change a column class float[5] in 5 > independent columns, for example a column "mag" in format (u,g,r,i.z) to > "mag_u", "mag_g", "mag_r", "mag_i", "mag_z". > > Thanks! > Mamen > > > -- **************************************************************** * Dr. Jesús Varela López e-mail: jmvarela@iac.XXX * * * * Instituto de Astrofísica de Canarias * * --------------- www.iac.es ------------- * * * * C/ Vía Láctea s/n Tel:+34 922 605278 * * E38205 - La Laguna (Tenerife). España Fax:+34 922 605210 * **************************************************************** --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From sn.fotop@gmail.XXX Tue May 17 14:01:48 2011 Date: Tue, 17 May 2011 14:02:41 +0200 From: Sotiria Fotopoulou To: topcat-user@sympa.XXX Subject: Re: Class shape > 1 Hi Mamen, the solution of Jesús' is possible also inside topcat. Load your table and from "Views -> Column Info" select the column you want to expand. Then go to "Columns-> Explode Array Column". Don't forget to save the new table. Cheers, Sotiria On 05/17/2011 12:57 PM, jmvarela@iac.XXX wrote: > Hi Mamen, > > you can use "stilts" (~ topcat command-line) with the following > command: > > stilts tpipe cmd="explodecols multi_dimension_column; colmeta -name > newname1 multi_dimension_column_1; colmeta -name newname2 > multi_dimension_column_2;....,colmeta -name newname_N > multi_dimension_column_N" in=input.fits out=output.fits > > where "multi_dimension_column" is the identification of the column > (in your example "mag") and newname is the names of the new columns > ("mag_u","mag_g",...). > > In summary, you should run the command: > > stilts tpipe cmd="explodecols mag; colmeta -name mag_u mag_1; colmeta > -name mag_g mag_2; colmeta -name mag_r mag_3; colmeta -name mag_i > mag_4;; colmeta -name mag_z mag_5" in=input.fits out=out.fits > > Maybe the "colmeta" commands can be compacted in just one but I'm > not sure. > > Check the syntax because maybe there is something wrong but I think > you get the idea. > > Cheers, > > Jesús > > Mamen Argudo ha escrito: > >> Hello, >> >> I would like to know if topcat is able to change column format class >> float[] with shape > 1 without open the original table with a text >> editor >> (too large). In particular, I need to change a column class float[5] >> in 5 >> independent columns, for example a column "mag" in format (u,g,r,i.z) to >> "mag_u", "mag_g", "mag_r", "mag_i", "mag_z". >> >> Thanks! >> Mamen >> >> >> > > > -- Sotiria http://ath-muc.blogspot.com From lobo@astro.XXX Thu May 19 12:38:53 2011 Date: Thu, 19 May 2011 10:38:28 +0100 From: Catarina Lobo To: topcat-user@bristol.XXX Subject: exact matching of 2 files using the values of 2 columns in each file Hi Mark and TOPCAT/STILTS users, I am trying to match 2 files using the values listed in 2 columns of each file (exact value matching). For that I use STILTS with tmatch2 (guess I cannot do this with TOPCAT, right?) but can't seem to make it... SOS?? What I have is: - in file1 (about 100 objects): I will use columns ID11 and ID12 as identifiers - in file2 (MANY objects including the 100 from file1): I will use columns ID21 and ID22 I wish to select from file2 the rows matching my ~100 file1 objects, by using identifiers such that ID21=ID11 and ID22=ID12 I tried: java -Xmx1024M -jar stilts.jar tmatch2 \ in1=file1.fits in2=file2.fits matcher=exact \ values1='concat(toString(ID11), ";", toString(ID12))' \ values2='concat(toString(ID21), ";", toString(ID22))' \ join=1and2 out=test_matched.fits But I get error message: Bad value for parameter values1: Wrong number of words - wanted 1, got 4 Usage: values1= NOTE: Using the 'meta' cmd, I see that my columns may not have the same exact format? Output info is: Class Format Code ID11 and ID12 Integer J ID11 and ID12 Short I Maybe this is messing things up?....................... Thanks for any help on this!! Catarina From m.b.taylor@bristol.XXX Thu May 19 14:57:07 2011 Date: Thu, 19 May 2011 14:57:07 +0100 (BST) From: Mark Taylor To: Catarina Lobo Cc: topcat-user@bristol.XXX Subject: Re: exact matching of 2 files using the values of 2 columns in each file Catarina, you're on the right lines, but getting quoting right can be very difficult when there are strings involved. I think syntax like this on the command line will do it: values1='concat(toString(ID11),\";\",toString(ID12))' This will work in TOPCAT too if you choose Exact matching (actually, the quoting is less problematic in TOPCAT, because you don't need to worry about what the shell does to the quotes). There are other possibilities too; you could maybe use (sec 7.1.8 of the manual): matcher=exact+exact values1='ID11 ID12' values2='ID21 ID22' or matcher=exact values1='array(ID11,ID12)' values2='array(ID21,ID22)' though those two might fail if the types aren't the same. Mark On Thu, 19 May 2011, Catarina Lobo wrote: > Hi Mark and TOPCAT/STILTS users, > > I am trying to match 2 files using the values listed in 2 columns of each file > (exact value matching). For that I use STILTS with tmatch2 (guess I cannot do > this with TOPCAT, right?) but can't seem to make it... SOS?? > > What I have is: > > - in file1 (about 100 objects): I will use columns ID11 and ID12 as > identifiers > - in file2 (MANY objects including the 100 from file1): I will use columns > ID21 and ID22 > > I wish to select from file2 the rows matching my ~100 file1 objects, by using > identifiers such that ID21=ID11 and ID22=ID12 > > I tried: > > java -Xmx1024M -jar stilts.jar tmatch2 \ > in1=file1.fits in2=file2.fits matcher=exact \ > values1='concat(toString(ID11), ";", toString(ID12))' \ > values2='concat(toString(ID21), ";", toString(ID22))' \ > join=1and2 out=test_matched.fits > > But I get error message: > > Bad value for parameter values1: Wrong number of words - wanted 1, got 4 > Usage: values1= > > > NOTE: Using the 'meta' cmd, I see that my columns may not have the same exact > format? Output info is: > > Class Format Code > ID11 and ID12 Integer J > ID11 and ID12 Short I > > Maybe this is messing things up?....................... > > Thanks for any help on this!! > Catarina > > > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From lobo@astro.XXX Fri May 20 00:28:18 2011 Date: Thu, 19 May 2011 22:27:59 +0100 From: Catarina Lobo To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: exact matching of 2 files using the values of 2 columns in each file It worked! It was indeed a matter of syntax... many thanks, Mark!!! Catarina On 19-05-2011 14:57, Mark Taylor wrote: > Catarina, > > you're on the right lines, but getting quoting right can be very difficult > when there are strings involved. I think syntax like this on the command > line will do it: > > values1='concat(toString(ID11),\";\",toString(ID12))' > > This will work in TOPCAT too if you choose Exact matching > (actually, the quoting is less problematic in TOPCAT, because you > don't need to worry about what the shell does to the quotes). > > There are other possibilities too; you could maybe use (sec 7.1.8 of > the manual): > > matcher=exact+exact values1='ID11 ID12' values2='ID21 ID22' > > or > > matcher=exact values1='array(ID11,ID12)' values2='array(ID21,ID22)' > > though those two might fail if the types aren't the same. > > Mark > > On Thu, 19 May 2011, Catarina Lobo wrote: > >> Hi Mark and TOPCAT/STILTS users, >> >> I am trying to match 2 files using the values listed in 2 columns of each file >> (exact value matching). For that I use STILTS with tmatch2 (guess I cannot do >> this with TOPCAT, right?) but can't seem to make it... SOS?? >> >> What I have is: >> >> - in file1 (about 100 objects): I will use columns ID11 and ID12 as >> identifiers >> - in file2 (MANY objects including the 100 from file1): I will use columns >> ID21 and ID22 >> >> I wish to select from file2 the rows matching my ~100 file1 objects, by using >> identifiers such that ID21=ID11 and ID22=ID12 >> >> I tried: >> >> java -Xmx1024M -jar stilts.jar tmatch2 \ >> in1=file1.fits in2=file2.fits matcher=exact \ >> values1='concat(toString(ID11), ";", toString(ID12))' \ >> values2='concat(toString(ID21), ";", toString(ID22))' \ >> join=1and2 out=test_matched.fits >> >> But I get error message: >> >> Bad value for parameter values1: Wrong number of words - wanted 1, got 4 >> Usage: values1= >> >> >> NOTE: Using the 'meta' cmd, I see that my columns may not have the same exact >> format? Output info is: >> >> Class Format Code >> ID11 and ID12 Integer J >> ID11 and ID12 Short I >> >> Maybe this is messing things up?....................... >> >> Thanks for any help on this!! >> Catarina >> >> >> >> >> >> >> > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ipf@iac.XXX Fri May 20 20:45:42 2011 Date: Fri, 20 May 2011 19:45:16 +0100 From: Ismael Perez Fournon To: topcat-user@sympa.XXX Mark Taylor Cc: Ismael Perez Fournon Subject: activation actions in TOPCAT Hi Mark, I'm using a lot the Activation Action in TOPCAT, which I find very useful. Have you thought of the possibility to have more than one activation action defined at the same time, for instance for cases with more than one active link column in the tables that one would like to activate simultaneously without having to change the "Web page location column"? Thanks, Ismael --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protección de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From m.b.taylor@bristol.XXX Mon May 23 09:41:24 2011 Date: Mon, 23 May 2011 09:41:23 +0100 (BST) From: Mark Taylor To: Ismael Perez Fournon Cc: topcat-user@sympa.XXX Subject: Re: activation actions in TOPCAT Ismael, On Fri, 20 May 2011, Ismael Perez Fournon wrote: > Hi Mark, > > I'm using a lot the Activation Action in TOPCAT, which I find very useful. > > Have you thought of the possibility to have more than one activation action > defined at the same time, for instance for cases with more than one active > link > column in the tables that one would like to activate simultaneously without > having > to change the "Web page location column"? thanks for the suggestion, I'll think about it. I don't have a very clear idea how widely and in what ways the activation action thing is used, so although I agree it's a bit clunky at the moment, I'm not sure about putting efforts in to make improvements. Anyway, I'll give it some thought. In the mean time it is possible, though a bit fiddly, to do two things per activation. If you select "Execute Custom Code" and enter an expression which runs a script you supply, then you can put any code you want in that script. For instance if you have columns URL1 and URL2 then you could use the custom code expression exec("./foo.sh", URL1, URL2) and write a script foo.sh like #!/bin/sh firefox $1 firefox $2 it would show the URLs for both columns in a web browser when you activate the row. That may or may not help depending on what actions you want to take for each row. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From lobo@astro.XXX Mon May 23 15:54:59 2011 Date: Mon, 23 May 2011 14:53:49 +0100 From: Catarina Lobo To: topcat-user@bristol.XXX Subject: draw a line on a scatter plot Hi again Mark, I have another question - probably it's straightforward but haven't found a quick-just-click-on-the-button way to do it... I have a scatter plot of (x,y) where x and y are the values in 2 columns of one data file. I would now like to draw a separation line of the form y=mx+b, m and b being known constant values. Can that be done?? Thanks! Catarina From jtlz2@astro.XXX Mon May 23 15:18:54 2011 Date: Mon, 23 May 2011 10:18:13 -0400 From: Jonathan Zwart To: Catarina Lobo Cc: topcat-user@bristol.XXX Subject: Re: draw a line on a scatter plot Hi Catarina, On 23 May 2011 09:53, Catarina Lobo wrote: > Hi again Mark, > > I have another question - probably it's straightforward but haven't found a > quick-just-click-on-the-button way to do it... > > I have a scatter plot of (x,y) where x and y are the values in 2 columns of > one data file. I would now like to draw a separation line of the form > y=mx+b, m and b being known constant values. Can that be done?? > You might generate a separate column with the y values calculated from y=mx+b for each of your x values, then join them by a dot-to-dot line in TOPCAT. I've asked Mark before about being able to overplot arbitrary functions but at that time it was outside the scope of TOPCAT. In the Plot Style Editor you can plot a best-fit straight line which would calculate m and b for you anyway. Jon From bianca.garilli@oamp.XXX Mon May 23 16:25:49 2011 Date: Mon, 23 May 2011 16:24:26 +0200 From: Bianca Garilli To: topcat-user@sympa.XXX Subject: Re: draw a line on a scatter plot Hi Catarina if m and b are known values, you can simply create a second tab in the scattered plot windoww, where yiu put your x variable as abscissa, and in the odrinate you write your formula m*x+b with your known values for m and b Then choose dot-todot I usually do that Bianca On Monday, May 23, 2011 04:18 PM Jonathan Zwart wrote > Hi Catarina, > > On 23 May 2011 09:53, Catarina Lobo wrote: > > Hi again Mark, > > > > I have another question - probably it's straightforward but haven't found > > a quick-just-click-on-the-button way to do it... > > > > I have a scatter plot of (x,y) where x and y are the values in 2 columns > > of one data file. I would now like to draw a separation line of the form > > y=mx+b, m and b being known constant values. Can that be done?? > > You might generate a separate column with the y values calculated from > y=mx+b for each of your x values, then join them by a dot-to-dot line in > TOPCAT. > > I've asked Mark before about being able to overplot arbitrary functions but > at that time it was outside the scope of TOPCAT. In the Plot Style Editor > you can plot a best-fit straight line which would calculate m and b for you > anyway. > > Jon --------TEMPORARY ADDRESS--------------------------------- Bianca Garilli; Laboratoire d'Astrophysique de Marseille 38, rue Frédéric Joliot-Curie 13388 Marseille cedex 13, France E-mail: bianca.garilli@oamp.XXX phone: 33 (0)491 056912 --------PERMANENT ADRESS------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- From yannick.roehlly@oamp.XXX Mon May 23 21:32:50 2011 Date: Mon, 23 May 2011 22:32:03 +0200 From: Yannick Roehlly To: topcat-user@sympa.XXX Subject: Bug in the sky matching algorithm Hi everybody, Hi Mark, A while ago, I encountered a bug in Topcat while testing the sky matching algorithm of an information system of ours. I finally take time to report it here (Is it the right place?). When doing a sky position based cross-match between two table, if the points are near the RA=0 line (when absolute value of RA is less than radius) you may not get all the matching points. For instance, attached to this message is a FITS-plus file containing three tables. test-data.csv is a dummy catalogue with points around (0d,+90d) and (0d,+45d). test-target.csv contains (0d,+90d) and (0d,+45d). If you cross-match the two tables with sky algorithm, 1.0 degree max error and asking for all matches, you get the match(1,2) table. Doing this, you should get two disc shaped set of points. But as you can see by doing a "spherical polar scatter plot", you only get 1/2 of the data matching (0d,+45d) - those with positive RA - and 1/6 for (0d,+90d). Thanks again for this wonderful tool. Regards, Yannick -- En science, ce qui est démontrable ne doit pas être admis sans démonstration. -+- Richard Dedekind -+- [ Part 2: "Attached Text" ] [ The following attachment was DELETED when this message was saved: ] [ A Application/X-GZIP segment of about 968,958 bytes. ] From lobo@astro.XXX Tue May 24 00:08:08 2011 Date: Mon, 23 May 2011 23:07:39 +0100 From: Catarina Lobo To: Bianca Garilli , jtlz2@astro.XXX Cc: topcat-user@sympa.XXX Subject: Re: draw a line on a scatter plot Hi Bianca and Jon, Both ways work fine and simple, thanks!! Catarina On 23-05-2011 15:24, Bianca Garilli wrote: > Hi Catarina > if m and b are known values, > you can simply create a second tab in the scattered plot windoww, where yiu > put your x variable as abscissa, and in the odrinate you write your formula > m*x+b > with your known values for m and b > > > Then choose dot-todot > > I usually do that > > Bianca > > > > > On Monday, May 23, 2011 04:18 PM Jonathan Zwart wrote >> Hi Catarina, >> >> On 23 May 2011 09:53, Catarina Lobo wrote: >>> Hi again Mark, >>> >>> I have another question - probably it's straightforward but haven't found >>> a quick-just-click-on-the-button way to do it... >>> >>> I have a scatter plot of (x,y) where x and y are the values in 2 columns >>> of one data file. I would now like to draw a separation line of the form >>> y=mx+b, m and b being known constant values. Can that be done?? >> You might generate a separate column with the y values calculated from >> y=mx+b for each of your x values, then join them by a dot-to-dot line in >> TOPCAT. >> >> I've asked Mark before about being able to overplot arbitrary functions but >> at that time it was outside the scope of TOPCAT. In the Plot Style Editor >> you can plot a best-fit straight line which would calculate m and b for you >> anyway. >> >> Jon > --------TEMPORARY ADDRESS--------------------------------- > Bianca Garilli; > Laboratoire d'Astrophysique de Marseille > 38, rue Frédéric Joliot-Curie > 13388 Marseille cedex 13, France > E-mail: bianca.garilli@oamp.XXX > phone: 33 (0)491 056912 > --------PERMANENT ADRESS------------------------------------- > Bianca Garilli; INAF, IASF-Milano > via Bassini 15; 20133 Milano, Italy > E-mail: bianca@lambrate.XXX > phone: 39-02-23699313 > Fax: 39-02-2666017 > -------------------------------------------------------------------------- > "Le signorine del french-cancan e della spaccata sono > le belle addormentate di una felicità perduta" > Léon-Paul Fargue > > "Science demands dogged work. Work in spite of everything and > continuous observation" > V.Van Gogh (adapted from) > > "In a world where science is treated as a branch of magic, > the rewards often go to those who behave as illusionists" > ------------------------------------------------------------------------- > From m.b.taylor@bristol.XXX Tue May 24 15:19:06 2011 Date: Tue, 24 May 2011 15:19:06 +0100 (BST) From: Mark Taylor To: Yannick Roehlly Cc: topcat-user@sympa.XXX Subject: Re: Bug in the sky matching algorithm Yannick, thank you very much for taking the trouble to make this clear report. You are quite right that it's a bug, the first that's been reported in the crossmatching code for several years, so I take it quite seriously. The error is a consequence of some optimisation code in which I wasn't carefully enough handling the extent of sub-regions in which sky matches might take place. Specifically, if *all* of the points in one table fall on one side of the line RA=0, then matches on the other side of that line may be missed. It won't be very common that all points in a dataset have RA>=0 (or RA<=360) while some are close enough to RA=0 that matches exist on the other side, but as you've demonstrated, it can happen. A related issue is that if the range of RA is not normalised consistently (e.g. if one table uses the range 0..360 degrees and the other uses -180..180 degrees) matches may be missed as well; again, this is probably not common. Sky-coordinate crossmatching in STILTS is affected in the same way as for TOPCAT. I have fixed the error, and you can find a pre-release version which works correctly at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ The next release will certainly include this fix. I'm not sure if it's sufficiently serious that I ought to make a new public release containing the fix now; I'm inclined to think that since it's been there for 5+ years a few more weeks/months won't make much difference, but I could be persuaded. Apologies to anybody (hopefully few) who may have got the wrong crossmatch results while using this code (hopefully few), and thanks again to Yannick. Mark On Mon, 23 May 2011, Yannick Roehlly wrote: > Hi everybody, > Hi Mark, > > A while ago, I encountered a bug in Topcat while testing the sky > matching algorithm of an information system of ours. I finally take > time to report it here (Is it the right place?). > > When doing a sky position based cross-match between two table, if the > points are near the RA=0 line (when absolute value of RA is less than > radius) you may not get all the matching points. > > For instance, attached to this message is a FITS-plus file containing > three tables. test-data.csv is a dummy catalogue with points around > (0d,+90d) and (0d,+45d). test-target.csv contains (0d,+90d) and > (0d,+45d). If you cross-match the two tables with sky algorithm, > 1.0 degree max error and asking for all matches, you get the match(1,2) > table. > > Doing this, you should get two disc shaped set of points. But as you > can see by doing a "spherical polar scatter plot", you only get 1/2 of > the data matching (0d,+45d) - those with positive RA - and 1/6 for > (0d,+90d). > > Thanks again for this wonderful tool. > > Regards, > > Yannick -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From yannick.roehlly@oamp.XXX Tue May 24 20:13:34 2011 Date: Tue, 24 May 2011 20:12:54 +0200 From: Yannick Roehlly To: topcat-user@sympa.XXX Subject: Re: Bug in the sky matching algorithm Hi Mark, Le Tue, 24 May 2011 15:19:06 +0100 (BST), Mark Taylor a écrit : > thank you very much for taking the trouble to make this clear report. > You are quite right that it's a bug, the first that's been reported > in the crossmatching code for several years, so I take it quite > seriously. I also think this bug should affect very few people. I only noticed it while testing the cone search of an information system of ours, to check if our 'pre-boxing' was working around RA=0 and around the pole. ;-) Nevertheless, thanks for you prompt answer and for correcting the bug. Best regards, Yannick From arnabc74@gmail.XXX Sun Jun 19 11:21:43 2011 Date: Sat, 11 Jun 2011 17:23:39 -0400 From: Arnab Chakraborty To: topcat-user@bristol.XXX Subject: Topcat VO table problem Hello, I am developing a SAMP-aware web service called VOStat, and found a little problem with VO tables broadcast by Topcat. Sometimes the table has an ampersand character inside an attribute. This happens typically in the href attribute of a LINK element where the URL has ampersands in it. XML does not allow dangling ampersands in an attribute (no, not even insie double quotes). It must be escaped as & As a result standard XML readers based on libxml produce error messages. Is it that the VO table is a "lenient" version of XML, where such departures are allowed? Or is it a bug in Topcat? -- Arnab From rebecap@cab.XXX Sun Jun 19 12:11:45 2011 Date: Wed, 15 Jun 2011 09:50:01 +0200 From: Rebeca Pulido To: fran@cab.XXX question@simbad.XXX topcat-user@bristol.XXX Subject: Problem with Topcat-Simbad Hello, using Topcat to perform a Multiple Cone Search (verbosity 3 maximum)of SIMBAD (http://simbad.u-strasbg.fr/simbad/sim-cone?) we have found some problems because Topcat does not provide the information displayed in SIMBAD webpage. In particular, few weeks ago, the Topcat's result the spectral type is not given for all sources with spectral type in SIMBAD, and the number of references is not given. And today, We have tried it once agian and Topcat does not give any spectral type. Is there something that I can do to get the spectral type and the number of references using Topcat? Thank you very much Rebeca. From berthier@imcce.XXX Wed Jun 15 17:47:01 2011 Date: Wed, 15 Jun 2011 18:34:47 +0200 From: jberthier To: topcat-user@bristol.XXX Cc: Frédéric Vachier Subject: arrows in 2d plots Hi, I would like to represent vectors in 2-D plots, between 2 points, for which the (x,y) coordinates of the point of reference and the point of interest are given. For example, each line is: date, xref, yref, xc, yc Each vector should start at coordinates xref, yref and end at xc, yc, the arrow being pointed toward xc,yc. Until now I found how to create an ellipse, but it does not provide the direction (see attached figure). I cannot find neither in stilts nor topcat how to draw arrows. Is it possible to do that (and how)? If not, could it be a future improvement of stilts/Topcat? cheers, jerome -- --- Dr. Jerome Berthier ------------------ Phone: +33 (0)14051 2261 -- Institut de mecanique celeste Fax: +33 (0)14633 2834 -- 77 av. Denfert RochereauMailto:jerome.berthier@imcce.XXX -- --- 75014 Paris - France ---------------------http://www.imcce.fr/ -- [ Part 2: "Attached Text" ] [ The following attachment was DELETED when this message was saved: ] [ A Image/PNG (Name="omc_graph.png") segment of about 16,560 bytes. ] From m.b.taylor@bristol.XXX Mon Jun 20 10:53:14 2011 Date: Mon, 20 Jun 2011 10:53:14 +0100 (BST) From: Mark Taylor To: Arnab Chakraborty Cc: topcat-user@bristol.XXX Subject: Re: Topcat VO table problem On Sat, 11 Jun 2011, Arnab Chakraborty wrote: > Hello, > > I am developing a SAMP-aware web service called VOStat, and found a > little problem with VO tables broadcast by Topcat. Sometimes the table has > an ampersand character inside an attribute. This happens typically in the > href attribute of a LINK element where the URL has ampersands in it. > > XML does not allow dangling ampersands in an attribute (no, not even insie > double quotes). It must be escaped as & > > As a result standard XML readers based on libxml produce error messages. > > Is it that the VO table is a "lenient" version of XML, where such > departures are allowed? Or is it a bug in Topcat? Hi Arnab, an unescaped ampersand in an attribute value is certainly a bug, but without a bit more detail I'm not sure where it's arising. Can you show me the XML where this happens, and tell me where the table being broadcast came from in the first place? You can send me the table off-list if it's large. thanks Mark PS sorry for the slow response, I was on holiday last week. The reason that your message took a week to appear on the mailing list is that messages from *unsubscribed* users require my moderation to appear - if you subscribe to the topcat-user list your messages appear straight away. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Jun 20 11:28:00 2011 Date: Mon, 20 Jun 2011 11:28:00 +0100 (BST) From: Mark Taylor To: Rebeca Pulido Cc: fran@cab.XXX question@simbad.XXX topcat-user@bristol.XXX Subject: Re: Problem with Topcat-Simbad Rebeca, On Wed, 15 Jun 2011, Rebeca Pulido wrote: > Hello, > > using Topcat to perform a Multiple Cone Search (verbosity 3 maximum)of > SIMBAD (http://simbad.u-strasbg.fr/simbad/sim-cone?) we have found some > problems because Topcat does not provide the information > displayed in SIMBAD webpage. In particular, few weeks ago, the Topcat's result > the > spectral type is not given for all sources with spectral type in SIMBAD, > and the number of references is not given. And today, We have tried it once > agian and Topcat does not give any spectral type. > > Is there something that I can do to get the spectral type and the number > of references using Topcat? other than setting the verbosity to 3, there is no action that TOPCAT can take to fix this, it's down to how the SIMBAD cone search service is implemented. I would expect that verbosity=3 would include all columns but it seems not. There may be some way of modifying the cone search URL (e.g. adding some more &name=value parameters to the end of it) which will get SIMBAD to be more verbose - the CDS people will have to advise on this. A non-TOPCAT solution would be to use the Query By List option on the SIMBAD web page. Mark PS sorry for the slow response, I was on holiday last week. The reason that your message took several days to appear on the mailing list is that messages from *unsubscribed* users require my moderation to appear - if you subscribe to the topcat-user list your messages appear straight away. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Jun 20 12:55:38 2011 Date: Mon, 20 Jun 2011 12:55:34 +0100 (BST) From: Mark Taylor To: jberthier Cc: topcat-user@bristol.XXX Frédéric Vachier Subject: Re: arrows in 2d plots Hi Jerome, On Wed, 15 Jun 2011, jberthier wrote: > Hi, > > I would like to represent vectors in 2-D plots, between 2 points, for which > the (x,y) coordinates of the point of reference and the point of interest are > given. For example, each line is: > > date, xref, yref, xc, yc > > Each vector should start at coordinates xref, yref and end at xc, yc, the > arrow being pointed toward xc,yc. Until now I found how to create an ellipse, > but it does not provide the direction (see attached figure). > > I cannot find neither in stilts nor topcat how to draw arrows. Is it possible > to do that (and how)? If not, could it be a future improvement of > stilts/Topcat? > > cheers, > jerome it is possible to draw vertical or horizontal arrows by hijacking the error bar mechanism, but not arrows with arbitrary orientation as you suggest. One or two other people have asked for this in the past. I hope to add this capability at some point - as always, user requests like this give me more incentive to do it. thanks Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From marc.wenger@astro.XXX Fri Jun 24 11:50:05 2011 Date: Fri, 24 Jun 2011 11:33:38 +0200 From: wenger To: Rebeca Pulido Cc: CDS Helpdesk , fran , topcat-user@bristol.XXX Subject: Re: Fwd: Fwd: Re: Problem with Topcat-Simbad Hello, I tried your example, and the resulting table in TOPCAT contains only the columns set in SIMBAD for VERB=2, even when VERB=3 is set in the TOPCAT window. The bug must be in TOPCAT. If I do the cone search directly in Simbad with the following URL (corresponding to your first entry HIP 1008): http://simbad.u-strasbg.fr/simbad/sim-cone?RA=3.1401&DEC=44.59074&SR=0.1&VERB=3 the resulting votable file contains all the parameters I have set for VERB=3, including the spectral type, and even now, on your request, the number of references. Sincerely, Marc. ----------------------------- > > Hello, > > I send you a table that I use for you try it, but with verb=3 in the Cone Search of TOPCAT, I am not the spectral type. > > It would be very useful to put the number of reference. > > Thank you > > Rebeca -- *** Please note my email address change: marc.wenger@astro.XXX =================================================================================== Marc Wenger CDS Centre de donnees astronomique de Strasbourg mailto:marc.wenger@astro.XXX Observatoire Astronomique de Strasbourg Tel: +33-(0)3 68 85 24 33 11, rue de l'Universite Fax: +33-(0)3 68 85 24 25 F-67000 Strasbourg (France) ----------------------------------------------------------------------------------- From fran.jimenez-esteban@cab.XXX Fri Jun 24 12:01:42 2011 Date: Fri, 24 Jun 2011 11:51:40 +0200 From: Fran Jimenez-Esteban To: wenger Cc: Rebeca Pulido , CDS Helpdesk , fran , topcat-user@bristol.XXX Subject: Re: Fwd: Fwd: Re: Problem with Topcat-Simbad Thank you very much (specially for including the number of references). So, we will wait for the answer from Topcat's people. Regards, Fran. On 24/06/11 11:33, wenger wrote: > Hello, > > I tried your example, and the resulting table in TOPCAT contains only > the columns set in SIMBAD for VERB=2, even > when VERB=3 is set in the TOPCAT window. The bug must be in TOPCAT. > > If I do the cone search directly in Simbad with the following URL > (corresponding to your first entry HIP 1008): > > http://simbad.u-strasbg.fr/simbad/sim-cone?RA=3.1401&DEC=44.59074&SR=0.1&VERB=3 > > > the resulting votable file contains all the parameters I have set for > VERB=3, including the spectral type, > and even now, on your request, the number of references. > > Sincerely, > > Marc. > ----------------------------- >> >> Hello, >> >> I send you a table that I use for you try it, but with verb=3 in the >> Cone Search of TOPCAT, I am not the spectral type. >> >> It would be very useful to put the number of reference. >> >> Thank you >> >> Rebeca > > -- +++++++++++++++++++++++++++++++++ Francisco Jiménez-Esteban Spanish Virtual Observatory Centro de Astrobiología (INTA-CSIC) ESA P.O. Box 78 28691 Villanueva de la Cañada Madrid, Spain Tel: +34 - 91 813 11 93 Fax: +34 - 91 813 11 60 +++++++++++++++++++++++++++++++++ From m.b.taylor@bristol.XXX Fri Jun 24 11:16:13 2011 Date: Fri, 24 Jun 2011 11:16:13 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Cc: wenger , Rebeca Pulido , CDS Helpdesk , fran Subject: Re: Fwd: Fwd: Re: Problem with Topcat-Simbad Aargh! You are right this is a TOPCAT bug. Because of a silly error the verbosity was being ignored if it was 3 or higher. A fix will be in the next public release. In the mean time you can find a pre-release version containing this fix at: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Many apologies to Rebeca and Fran for the time lost and to Marc for casting aspersions on Simbad's cone search service. Mark (Topcat's people) On Fri, 24 Jun 2011, Fran Jimenez-Esteban wrote: > Thank you very much (specially for including the number of references). > > So, we will wait for the answer from Topcat's people. > > Regards, > Fran. > > On 24/06/11 11:33, wenger wrote: > > Hello, > > > > I tried your example, and the resulting table in TOPCAT contains only the > > columns set in SIMBAD for VERB=2, even > > when VERB=3 is set in the TOPCAT window. The bug must be in TOPCAT. > > > > If I do the cone search directly in Simbad with the following URL > > (corresponding to your first entry HIP 1008): > > > > http://simbad.u-strasbg.fr/simbad/sim-cone?RA=3.1401&DEC=44.59074&SR=0.1&VERB=3 > > > > the resulting votable file contains all the parameters I have set for > > VERB=3, including the spectral type, > > and even now, on your request, the number of references. > > > > Sincerely, > > > > Marc. > > ----------------------------- > > > > > > Hello, > > > > > > I send you a table that I use for you try it, but with verb=3 in the Cone > > > Search of TOPCAT, I am not the spectral type. > > > > > > It would be very useful to put the number of reference. > > > > > > Thank you > > > > > > Rebeca > > > > > > -- > +++++++++++++++++++++++++++++++++ > Francisco Jiménez-Esteban > Spanish Virtual Observatory > Centro de Astrobiología (INTA-CSIC) > ESA > P.O. Box 78 > 28691 Villanueva de la Cañada > Madrid, Spain > Tel: +34 - 91 813 11 93 > Fax: +34 - 91 813 11 60 > +++++++++++++++++++++++++++++++++ -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Jul 4 17:38:40 2011 Date: Mon, 4 Jul 2011 17:38:39 +0100 (BST) From: Mark Taylor To: Seb Oliver Cc: topcat-user@bristol.XXX Subject: Re: weird bug On Mon, 14 Mar 2011, Seb Oliver wrote: > Dear All, > > I appear to have found a weird bug. If I create some subsets of a table and > then look at the statistics of the subsets and then change the expressions for > the subsets then go back and look at the statistics it seems to remember the > previous expression so that not only does it get the Row statistics wrong but > it also resets the value of the "Size" in the Row Subset window > > > > See in these screen grabs. Originally ID_8 is not of ID _6 and has Size 2955. > In the first I've changed ID _8 so that it is the same as ID _3 with size = > 5108 after a short delay ... I've made a fix which should improve matters here. The bug in question should have disappeared, and subset membership is a bit more responsive when you change the expressions of other subsets or columns. Fixed pre-release version at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ for anybody who would like to try it out. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From marquett@iap.XXX Tue Jul 5 12:56:18 2011 Date: Tue, 5 Jul 2011 13:55:46 +0200 From: Jean-Baptiste Marquette To: topcat-user@bristol.XXX Subject: A replaceval issue in JyStilts on VOTable Hi all, Using a Jython script, I would like to change the values in a column in the 2nd table of 201,000 VOTables created by Emmanuel Bertin's SExtractor. In other words, a zero time by a MJD epoch. My code is: import glob import os import stilts as st DirDate = '/Volumes/pepperland/erosdata/datesmjd/' DirCat = '/Volumes/pepperland/erosdata/sexcatcalib/' for Dates in glob.glob(DirDate + '*.date'): Obj, Field = os.path.basename(Dates)[0:2], os.path.basename(Dates)[0:5] Data = st.tread(Dates, fmt='ascii', random=True) print os.path.basename(Dates), 'processing ...' Rank = 0 while Rank < len(Data): Code, MJD = Data.coldata('Code')[Rank], Data.coldata('MJD')[Rank] for Cam in ('0', '1'): Cat = DirCat + Obj + '/' + Field + '/' + Field + Cam + '/' + Field + Cam + '_' + Code + '.cat' if not os.path.exists(Cat): continue Table = st.tread(Cat+"#1", fmt='votable', random=True) Table.cmd_replaceval(0.0, MJD, 'Epoch') Table.write() exit(0) But the column Epoch appears unchanged: +-----------+------------+----------+----------+---------+-------------+-------------+---------------+---------------+-----------+---------------+-------------+-------+------+-------------+ | Extension | Date | Time | Duration | NDetect | NSextracted | Image_Ident | Background_Me | Background_St | Threshold | Weight_Scalin | Pixel_Scale | Epoch | Gain | Satur_Level | +-----------+------------+----------+----------+---------+-------------+-------------+---------------+---------------+-----------+---------------+-------------+-------+------+-------------+ | 1 | 02/07/2010 | 15:25:41 | 2.0 | 6707 | 6705 | beta scu | 0.00103222 | 0.0175993 | 0.0 | 0.026399 | 1.0 | 0.0 | 6.2 | 30000.0 | | 2 | 02/07/2010 | 15:25:47 | 6.0 | 10700 | 9658 | beta scu | 2087.47 | 138.843 | 0.0 | 208.265 | 1.0 | 0.0 | 5.6 | 30000.0 | | 3 | 02/07/2010 | 15:25:52 | 5.0 | 14594 | 12941 | beta scu | 936.666 | 102.11 | 0.0 | 153.164 | 1.0 | 0.0 | 6.7 | 30000.0 | | 4 | 02/07/2010 | 15:25:56 | 4.0 | 13689 | 12254 | beta scu | 550.438 | 114.203 | 0.0 | 171.304 | 1.0 | 0.0 | 6.2 | 30000.0 | | 5 | 02/07/2010 | 15:26:02 | 6.0 | 11093 | 11068 | beta scu | 0.00572755 | 0.032932 | 0.0 | 0.049398 | 1.0 | 0.0 | 6.2 | 30000.0 | | 6 | 02/07/2010 | 15:26:06 | 4.0 | 14330 | 12949 | beta scu | 1496.53 | 99.5118 | 0.0 | 149.268 | 1.0 | 0.0 | 6.2 | 30000.0 | | 7 | 02/07/2010 | 15:26:11 | 5.0 | 12659 | 11134 | beta scu | 950.955 | 89.1277 | 0.0 | 133.692 | 1.0 | 0.0 | 6.3 | 30000.0 | | 8 | 02/07/2010 | 15:26:15 | 4.0 | 11119 | 10149 | beta scu | 241.868 | 73.9349 | 0.0 | 110.902 | 1.0 | 0.0 | 6.0 | 30000.0 | +-----------+------------+----------+----------+---------+-------------+-------------+---------------+---------------+-----------+---------------+-------------+-------+------+-------------+ Any hint welcome. Cheers Jean-Baptiste From m.b.taylor@bristol.XXX Tue Jul 5 13:02:50 2011 Date: Tue, 5 Jul 2011 13:02:50 +0100 (BST) From: Mark Taylor To: Jean-Baptiste Marquette Cc: topcat-user@bristol.XXX Subject: Re: A replaceval issue in JyStilts on VOTable Hi Jean-Baptiste, the cmd_* filters do not alter tables in place, they are functions which return the modified table. So instead of this: > Table.cmd_replaceval(0.0, MJD, 'Epoch') you want this: > Table = Table.cmd_replaceval(0.0, MJD, 'Epoch') Mark On Tue, 5 Jul 2011, Jean-Baptiste Marquette wrote: > Hi all, > > Using a Jython script, I would like to change the values in a column in the 2nd table of 201,000 VOTables created by Emmanuel Bertin's SExtractor. In other words, a zero time by a MJD epoch. > > My code is: > > import glob > import os > import stilts as st > > DirDate = '/Volumes/pepperland/erosdata/datesmjd/' > DirCat = '/Volumes/pepperland/erosdata/sexcatcalib/' > > for Dates in glob.glob(DirDate + '*.date'): > Obj, Field = os.path.basename(Dates)[0:2], os.path.basename(Dates)[0:5] > Data = st.tread(Dates, fmt='ascii', random=True) > print os.path.basename(Dates), 'processing ...' > Rank = 0 > while Rank < len(Data): > Code, MJD = Data.coldata('Code')[Rank], Data.coldata('MJD')[Rank] > for Cam in ('0', '1'): > Cat = DirCat + Obj + '/' + Field + '/' + Field + Cam + '/' + Field + Cam + '_' + Code + '.cat' > if not os.path.exists(Cat): continue > Table = st.tread(Cat+"#1", fmt='votable', random=True) > Table.cmd_replaceval(0.0, MJD, 'Epoch') > Table.write() > exit(0) > > But the column Epoch appears unchanged: > > +-----------+------------+----------+----------+---------+-------------+-------------+---------------+---------------+-----------+---------------+-------------+-------+------+-------------+ > | Extension | Date | Time | Duration | NDetect | NSextracted | Image_Ident | Background_Me | Background_St | Threshold | Weight_Scalin | Pixel_Scale | Epoch | Gain | Satur_Level | > +-----------+------------+----------+----------+---------+-------------+-------------+---------------+---------------+-----------+---------------+-------------+-------+------+-------------+ > | 1 | 02/07/2010 | 15:25:41 | 2.0 | 6707 | 6705 | beta scu | 0.00103222 | 0.0175993 | 0.0 | 0.026399 | 1.0 | 0.0 | 6.2 | 30000.0 | > | 2 | 02/07/2010 | 15:25:47 | 6.0 | 10700 | 9658 | beta scu | 2087.47 | 138.843 | 0.0 | 208.265 | 1.0 | 0.0 | 5.6 | 30000.0 | > | 3 | 02/07/2010 | 15:25:52 | 5.0 | 14594 | 12941 | beta scu | 936.666 | 102.11 | 0.0 | 153.164 | 1.0 | 0.0 | 6.7 | 30000.0 | > | 4 | 02/07/2010 | 15:25:56 | 4.0 | 13689 | 12254 | beta scu | 550.438 | 114.203 | 0.0 | 171.304 | 1.0 | 0.0 | 6.2 | 30000.0 | > | 5 | 02/07/2010 | 15:26:02 | 6.0 | 11093 | 11068 | beta scu | 0.00572755 | 0.032932 | 0.0 | 0.049398 | 1.0 | 0.0 | 6.2 | 30000.0 | > | 6 | 02/07/2010 | 15:26:06 | 4.0 | 14330 | 12949 | beta scu | 1496.53 | 99.5118 | 0.0 | 149.268 | 1.0 | 0.0 | 6.2 | 30000.0 | > | 7 | 02/07/2010 | 15:26:11 | 5.0 | 12659 | 11134 | beta scu | 950.955 | 89.1277 | 0.0 | 133.692 | 1.0 | 0.0 | 6.3 | 30000.0 | > | 8 | 02/07/2010 | 15:26:15 | 4.0 | 11119 | 10149 | beta scu | 241.868 | 73.9349 | 0.0 | 110.902 | 1.0 | 0.0 | 6.0 | 30000.0 | > +-----------+------------+----------+----------+---------+-------------+-------------+---------------+---------------+-----------+---------------+-------------+-------+------+-------------+ > > Any hint welcome. > > Cheers > Jean-Baptiste -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From marquett@iap.XXX Tue Jul 5 14:58:02 2011 Date: Tue, 5 Jul 2011 14:57:30 +0200 From: Jean-Baptiste Marquette To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: A replaceval issue in JyStilts on VOTable Le 5 juil. 2011 à 14:02, Mark Taylor a écrit : > he cmd_* filters do not alter tables in place, they are functions > which return the modified table. So instead of this: > >> Table.cmd_replaceval(0.0, MJD, 'Epoch') > > you want this: > >> Table = Table.cmd_replaceval(0.0, MJD, 'Epoch') > Hi Mark, Of course! Now I have another question: how to reinsert Table as Cat#1, in other words as the 2nd table in a VOTable ? Table was read as Table = st.tread(Cat+"#1", fmt='votable', random=True) Table.write(Cat+"#1", fmt='votable') doesn't work, it creates a new table named Cat#1 Cheers JB From m.b.taylor@bristol.XXX Tue Jul 5 14:12:57 2011 Date: Tue, 5 Jul 2011 14:12:56 +0100 (BST) From: Mark Taylor To: Jean-Baptiste Marquette Cc: topcat-user@bristol.XXX Subject: Re: A replaceval issue in JyStilts on VOTable On Tue, 5 Jul 2011, Jean-Baptiste Marquette wrote: > > Le 5 juil. 2011 à 14:02, Mark Taylor a écrit : > > > he cmd_* filters do not alter tables in place, they are functions > > which return the modified table. So instead of this: > > > >> Table.cmd_replaceval(0.0, MJD, 'Epoch') > > > > you want this: > > > >> Table = Table.cmd_replaceval(0.0, MJD, 'Epoch') > > > > > Hi Mark, > > Of course! > Now I have another question: how to reinsert Table as Cat#1, in other words as the 2nd table in a VOTable ? Table was read as > > Table = st.tread(Cat+"#1", fmt='votable', random=True) > > Table.write(Cat+"#1", fmt='votable') doesn't work, it creates a new table named Cat#1 You can't edit an existing (single- or multi-) table file on disk, you can only write a new one. So if you want to do this, you will have to read all the tables in the input file, adjust some or all of them as required, and then write them all back to a new output file. You can use the multi-table I/O functions treads() and twrites() for this. So you'd do something like: tables = stilts.treads('tables.vot', fmt='votable') tables[1] = tables[1].cmd_replaceval(0.0, MJD, 'Epoch') stilts.twrites(tables, 'tables_new.vot', fmt='votable') Note that this won't leave the VOTable content of either table identical to the input - getting read into and written out from STILTS may lose some details of the metadata, for instance GROUP structures. You said in your earlier message that you have 201,000 tables. If you mean these are all in the same file, you may encounter memory problems. Since twrites() takes an iterable rather than (necessarily) an array, you can probably get round this by treating the list in a cleverer way than what I've written above - I can advise if necessary. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From marquett@iap.XXX Tue Jul 5 16:04:11 2011 Date: Tue, 5 Jul 2011 16:03:57 +0200 From: Jean-Baptiste Marquette To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: A replaceval issue in JyStilts on VOTable Hi Mark, > You can use the multi-table I/O functions treads() and twrites() for this. > So you'd do something like: > > tables = stilts.treads('tables.vot', fmt='votable') > tables[1] = tables[1].cmd_replaceval(0.0, MJD, 'Epoch') > stilts.twrites(tables, 'tables_new.vot', fmt='votable') This does the job perfectly. > > Note that this won't leave the VOTable content of either table > identical to the input - getting read into and written out from > STILTS may lose some details of the metadata, for instance GROUP > structures. I don't think this has important consequences. > > You said in your earlier message that you have 201,000 tables. > If you mean these are all in the same file, you may encounter memory > problems. I have 201,000 individual VOTables, each containing 3 tables, the second one to be modified. Thus no problem of memory. Cheers JB From Nick.Ball@nrc-cnrc.XXX Fri Jul 15 00:48:32 2011 Date: Thu, 14 Jul 2011 15:51:16 -0700 From: "Ball, Nick" To: "topcat-user@sympa.XXX" Subject: No handler for URL Hi, I'm running Topcat 3.8 on Mac OS X 10.6. When I start up the GUI from the command line it says (3:23pm)nball(151)> topcatgui 2048 & [1] 18532 (3:23pm)nball(152)> WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) WARNING: Hub connection attempt failed (org.astrogrid.samp.client.HubConnector.setActive) WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) WARNING: SAMP connection attempt failed: org.astrogrid.samp.client.SampException: 404 No handler for URL (uk.ac.starlink.topcat.interop.SampCommunicator.setActive) WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) ... and just keeps repeating the 404 line. The GUI works but SAMP doesn't. topcatgui is an alias to java -jar -Xmx"!*"M /Applications/Topcat/TOPCAT.app/Contents/Resources/Java/topcat-full.jar . It does it from more than different machine. Does anyone know what the cause is? Thanks, Nick -- Nick Ball Herzberg Institute of Astrophysics 5071 West Saanich Road Victoria, BC V9E 2E7 Canada From m.b.taylor@bristol.XXX Fri Jul 15 09:20:12 2011 Date: Fri, 15 Jul 2011 09:20:12 +0100 (BST) From: Mark Taylor To: "Ball, Nick" Cc: "topcat-user@sympa.XXX" Subject: Re: No handler for URL Nick, this looks a bit like you may have an existing SAMP hub running on the machine which has somehow or other only partially shut down. Since v3.8 TOPCAT starts its own hub if none is already running, but it should cope either way. However, the fact that you're seeing the same thing on multiple machines makes that sound less likely - unless you have the same (cross-mounted) home director on all the machines, is that the case? If the problem is something like that,at's it, you could try deleting the ~/.samp file, if there is one, and try again (that might leave a harmless running server process somewhere - you could try to track it down and kill it using 'ps aux | grep -i java' or similar). If that's not it, could you run topcat with the flags "-verbose -verbose" and show me the output. The -verbose flags go after the jar file; or you can use the supplied script, i.e. /Applications/Topcat/bin/topcat -verbose -verbose thanks for the report, Mark (PS if anybody else is seeing what looks like the same problem, please let me know). On Thu, 14 Jul 2011, Ball, Nick wrote: > Hi, > > I'm running Topcat 3.8 on Mac OS X 10.6. When I start up the GUI from the command line it says > > (3:23pm)nball(151)> topcatgui 2048 & > [1] 18532 > (3:23pm)nball(152)> WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) > WARNING: Hub connection attempt failed (org.astrogrid.samp.client.HubConnector.setActive) > WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) > WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) > WARNING: SAMP connection attempt failed: org.astrogrid.samp.client.SampException: 404 No handler for URL (uk.ac.starlink.topcat.interop.SampCommunicator.setActive) > WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) > WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) > WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) > ... > > and just keeps repeating the 404 line. The GUI works but SAMP doesn't. > > topcatgui is an alias to java -jar -Xmx"!*"M /Applications/Topcat/TOPCAT.app/Contents/Resources/Java/topcat-full.jar . > > It does it from more than different machine. Does anyone know what the cause is? > > Thanks, > > Nick > > -- > Nick Ball > Herzberg Institute of Astrophysics > 5071 West Saanich Road > Victoria, BC V9E 2E7 > Canada > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Nick.Ball@nrc-cnrc.XXX Sat Jul 16 00:38:52 2011 Date: Fri, 15 Jul 2011 15:41:38 -0700 From: "Ball, Nick" To: Mark Taylor Cc: "topcat-user@sympa.XXX" Subject: Re: No handler for URL Hi Mark, Thanks for the reply. I don't have cross-mounted directories, but removing the .samp file on both machines appears to have solved the problem. They're not identical, but both are Mac OS 10.6, so perhaps the same thing happened twice. Nick On 2011-07-15, at 1:20 AM, Mark Taylor wrote: > Nick, > > this looks a bit like you may have an existing SAMP hub running on the > machine which has somehow or other only partially shut down. Since > v3.8 TOPCAT starts its own hub if none is already running, but it > should cope either way. However, the fact that you're seeing the > same thing on multiple machines makes that sound > less likely - unless you have the same (cross-mounted) home director > on all the machines, is that the case? If the problem is something > like that,at's it, you could try deleting the ~/.samp file, if there is > one, and try again (that might leave a harmless running server process > somewhere - you could try to track it down and kill it using > 'ps aux | grep -i java' or similar). > > If that's not it, could you run topcat with the flags "-verbose -verbose" > and show me the output. The -verbose flags go after the jar file; > or you can use the supplied script, i.e. > > /Applications/Topcat/bin/topcat -verbose -verbose > > thanks for the report, > > Mark > > (PS if anybody else is seeing what looks like the same problem, > please let me know). > > > On Thu, 14 Jul 2011, Ball, Nick wrote: > >> Hi, >> >> I'm running Topcat 3.8 on Mac OS X 10.6. When I start up the GUI from the command line it says >> >> (3:23pm)nball(151)> topcatgui 2048 & >> [1] 18532 >> (3:23pm)nball(152)> WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) >> WARNING: Hub connection attempt failed (org.astrogrid.samp.client.HubConnector.setActive) >> WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) >> WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) >> WARNING: SAMP connection attempt failed: org.astrogrid.samp.client.SampException: 404 No handler for URL (uk.ac.starlink.topcat.interop.SampCommunicator.setActive) >> WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) >> WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) >> WARNING: POST /xmlrpc --> 404 No handler for URL (org.astrogrid.samp.httpd.HttpServer.serveRequest) >> ... >> >> and just keeps repeating the 404 line. The GUI works but SAMP doesn't. >> >> topcatgui is an alias to java -jar -Xmx"!*"M /Applications/Topcat/TOPCAT.app/Contents/Resources/Java/topcat-full.jar . >> >> It does it from more than different machine. Does anyone know what the cause is? >> >> Thanks, >> >> Nick >> >> -- >> Nick Ball >> Herzberg Institute of Astrophysics >> 5071 West Saanich Road >> Victoria, BC V9E 2E7 >> Canada >> >> >> >> > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Tue Jul 26 01:02:36 2011 Date: Tue, 26 Jul 2011 01:02:04 +0200 From: Ivan Zolotukhin To: topcat-user@sympa.XXX Subject: bitwise operators in algebraic expressions Hi Mark, I have recently tried to find bitwise operators in TOPCAT's documentation and was pretty surprised it's not there. Of course one can often emulate them with the modulus operator, but it's not that straightforward (e.g. compare expressions: "col1 & 32" and "(col1 % 64) >= 32"). Is this already on the todo list or I'm just the first person asked? (or it exists somewhere and I didn't find it?) -- With best regards, Ivan From m.b.taylor@bristol.XXX Tue Jul 26 09:15:36 2011 Date: Tue, 26 Jul 2011 09:15:36 +0100 (BST) From: Mark Taylor To: Ivan Zolotukhin Cc: topcat-user@sympa.XXX Subject: Re: bitwise operators in algebraic expressions Hi Ivan, On Tue, 26 Jul 2011, Ivan Zolotukhin wrote: > Hi Mark, > > I have recently tried to find bitwise operators in TOPCAT's > documentation and was pretty surprised it's not there. Of course one > can often emulate them with the modulus operator, but it's not that > straightforward (e.g. compare expressions: "col1 & 32" and "(col1 % > 64) >= 32"). the expression language includes all the basic Java syntax, so the bitwise operators are present and correct. Looks like I just didn't document them because I thought they were rather too special interest, though it's kind of covered where I say "Unfortunately a complete tutorial on writing Java is beyond the scope of this document". If you're not already aware of it you might find the stilts calc command useful, e.g.: % stilts calc '63 & 32' 32 > Is this already on the todo list or I'm just the first person asked? > (or it exists somewhere and I didn't find it?) I think you're the first person to look at the documentation before trying it out - you deserve some sort of award for that. I'll add the bitwise operators to the list in the manual. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From cgp@star.XXX Tue Jul 26 10:28:28 2011 Date: Tue, 26 Jul 2011 09:27:56 +0100 From: Clive Page To: topcat-user@sympa.XXX Subject: Re: bitwise operators in algebraic expressions On 26/07/2011 09:15, Mark Taylor wrote: > I'll add the bitwise operators to the list in the manual. It would be good to have all the operators documented in one place, for those of us who use Java but rarely. I think I discovered one of the bitwise operators by accident :-) when assuming that x^y would do exponentiation, as it does in many other languages. I suppose it isn't possible to add an exponentiation operator to TOPCAT/STILTS syntax - there are so many cases when one is badly needed? -- Clive Page Dept of Physics & Astronomy, University of Leicester. From ivan.zolotukhin@gmail.XXX Tue Jul 26 14:08:40 2011 Date: Tue, 26 Jul 2011 14:08:23 +0200 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user@sympa.XXX Subject: Re: bitwise operators in algebraic expressions Hi Mark, I still don't get it quite well: can I make e.g. a row subset in TOPCAT using bitwise operators? Expressions like col1 & 32 in this dialogue give me an error: Can't convert int to boolean Making row subset based on a bitmask column (of integer type) was my original motivation of using bitwise operators and of course I tried all the possible combinations before looking in the docs, so I'm afraid I don't deserve the award you mention. -- With best regards, Ivan On Tue, Jul 26, 2011 at 10:15 AM, Mark Taylor wrote: > Hi Ivan, > > On Tue, 26 Jul 2011, Ivan Zolotukhin wrote: > >> Hi Mark, >> >> I have recently tried to find bitwise operators in TOPCAT's >> documentation and was pretty surprised it's not there. Of course one >> can often emulate them with the modulus operator, but it's not that >> straightforward (e.g. compare expressions: "col1 & 32" and "(col1 % >> 64) >= 32"). > > the expression language includes all the basic Java syntax, so the > bitwise operators are present and correct.  Looks like I just didn't > document them because I thought they were rather too special interest, > though it's kind of covered where I say "Unfortunately a complete > tutorial on writing Java is beyond the scope of this document". > If you're not already aware of it you might find the stilts calc > command useful, e.g.: > >    % stilts calc '63 & 32' >       32 > >> Is this already on the todo list or I'm just the first person asked? >> (or it exists somewhere and I didn't find it?) > > I think you're the first person to look at the documentation before > trying it out - you deserve some sort of award for that. > > I'll add the bitwise operators to the list in the manual. > > Mark > > -- > Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From m.b.taylor@bristol.XXX Tue Jul 26 13:17:01 2011 Date: Tue, 26 Jul 2011 13:17:00 +0100 (BST) From: Mark Taylor To: Ivan Zolotukhin Cc: topcat-user@sympa.XXX Subject: Re: bitwise operators in algebraic expressions On Tue, 26 Jul 2011, Ivan Zolotukhin wrote: > Hi Mark, > > I still don't get it quite well: can I make e.g. a row subset in > TOPCAT using bitwise operators? Expressions like col1 & 32 in this > dialogue give me an error: > > Can't convert int to boolean you're almost there: the operators like & and | have integer operands and an integer result. boolean is the type you need to define a subset, and in java (hence the TOPCAT expression language) boolean and integer are distinct types, with no automatic conversion between them. So you just need to convert explicitly from integer to boolean, e.g. by testing for equality with zero. This should do it: (col1 & 32) != 0 -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Tue Jul 26 14:25:17 2011 Date: Tue, 26 Jul 2011 14:25:01 +0200 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user@sympa.XXX Subject: Re: bitwise operators in algebraic expressions On Tue, Jul 26, 2011 at 2:17 PM, Mark Taylor wrote: > On Tue, 26 Jul 2011, Ivan Zolotukhin wrote: > >> Hi Mark, >> >> I still don't get it quite well: can I make e.g. a row subset in >> TOPCAT using bitwise operators? Expressions like col1 & 32 in this >> dialogue give me an error: >> >> Can't convert int to boolean > > you're almost there: the operators like & and | have integer operands > and an integer result.  boolean is the type you need to define a > subset, and in java (hence the TOPCAT expression language) boolean and > integer are distinct types, with no automatic conversion between them. > So you just need to convert explicitly from integer to boolean, > e.g. by testing for equality with zero.  This should do it: > >   (col1 & 32) != 0 Amazingly, it works! Thanks for your continuous support, Mark. From elle@oal.XXX Wed Jul 27 19:51:27 2011 Date: Wed, 27 Jul 2011 19:40:45 +0200 From: elle To: topcat-user@bristol.XXX Subject: finding multiple matches in two columns Hi everybody there, I am a new topcat user and I have this "problem". I have to columns $1 and $2, which contains strings and they are related (they are the same things with different names). I have to find if there is any multiplicity, i.e. if a string which appears in $2 is repeated in its column (that means that different strings in $1 would correspond to the same string in $2). Does anybody know which function can do that? thanks! cheers elle From m.b.taylor@bristol.XXX Thu Jul 28 09:42:45 2011 Date: Thu, 28 Jul 2011 09:42:44 +0100 (BST) From: Mark Taylor To: elle Cc: topcat-user@bristol.XXX Subject: Re: finding multiple matches in two columns Hallo elle, The Internal Match window (use the Joins|Internal Match menu item in the Control Window) can do this for you. http://www.starlink.ac.uk/topcat/sun253/MatchWindow1.html Its job is to find matching entries within a single table. Select "Exact Value" in the Algorithm selector, and then select your Table and use $1 for the Matched Value Column. Exactly what it does when it's identified the matching entries depends on how you fill in the Action box - see: http://www.starlink.ac.uk/topcat/sun253/matchInternalAction.html Mark On Wed, 27 Jul 2011, elle wrote: > Hi everybody there, I am a new topcat user and I have this "problem". I > have to columns $1 and $2, which contains strings and they are related > (they are the same things with different names). I have to find if there is > any multiplicity, i.e. if a string which appears in $2 is repeated in its > column (that means that different strings in $1 would correspond to the > same string in $2). Does anybody know which function can do that? > > thanks! > cheers > > elle -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From p.w.lucas@herts.XXX Thu Jul 28 18:01:40 2011 Date: Thu, 28 Jul 2011 16:50:43 +0100 From: p.w.lucas@herts.XXX To: topcat-user@sympa.XXX Subject: Re: Re: No handler for URL Dear Nick and Mark, Thanks for flagging this up Nick and supplying the fix Mark. I have also had the same problem on my Mac with Snow Leopard and other users at Hertfordshire have noticed it too. Deleting .samp seems to have fixed it. Phil Lucas, University of Hertfordshire From m.b.taylor@bristol.XXX Thu Jul 28 18:16:41 2011 Date: Thu, 28 Jul 2011 18:16:40 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Re: No handler for URL On Thu, 28 Jul 2011, p.w.lucas@herts.XXX wrote: > Dear Nick and Mark, > > Thanks for flagging this up Nick and supplying the fix Mark. I have also > had the same problem on my Mac with Snow Leopard and other users > at Hertfordshire have noticed it too. Deleting .samp seems to have fixed > it. Hmm, it's regrettable if this is a common problem, since it probably means that most people will just end up having SAMP switched off because it's not clear what's wrong or how to fix it. A quick fix while TOPCAT is running is to click the hub connection icon at the bottom right of the Control Window, which will start a new hub (overwriting the duff .samp file at the same time). I'll think about whether I should modify the startup sequence so that this works without user interventions. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From nrm@astro.XXX Fri Jul 29 13:38:35 2011 Date: Fri, 29 Jul 2011 12:36:21 +0100 From: Neil Mawson To: topcat-user@mlist2.XXX Subject: STILTS help/potential bug Dear Mark Taylor I'm currently having some issues with your STILTS software, I'm trying to use the addcol command to add an extra column as as identifier using the values from two other columns and some text. This is the command I'm trying: concat("Skycam_",formatDecimal(col1_1_1,2),"_",formatDecimal(col2_1_1,"+0.###;-0.###"))' This command works in Topcat, however when I try to run it in STILTS I get error messages: Unused arguments: ',formatDecimal(col1_1_1,2),_' ',formatDecimal(col2_1_1,+0.###;-0.###' '))' Command was: addcol skyref concat("Skycam_",formatDecimal(col1_1_1,2),"_",formatDecimal(col2_1_1,"+0.###;-0.###")) Usage: addcol [-after | -before ] [-units ] [-ucd ] [-desc ] (Unused arguments: ',formatDecimal(col1_1_1,2),_' ',formatDecimal(col2_1_1,+0.###;-0.###' '))') I've tried many connotations of this command with no joy, and have been left scratching my head. Is this an expected limitation with STILTS or have I missed something out. Any help you could provide would be greatly appreciated Best regards Neil Mawson -- Neil Mawson Astrophysics Research Institute Liverpool John Moores University Twelve Quays House, Egerton Wharf, Birkenhead, Wirral, CH41 1LD UK Tel: +44 151 231 2923 From m.b.taylor@bristol.XXX Fri Jul 29 13:29:00 2011 Date: Fri, 29 Jul 2011 13:28:59 +0100 (BST) From: Mark Taylor To: Neil Mawson Cc: topcat-user@mlist2.XXX Subject: Re: STILTS help/potential bug Hi Neil, I'm afraid you are experiencing quoting misery. STILTS and the un*x shell each have their own quoting and special character rules and they don't get on very well together. The upshot is that it can be extremely painful to specify some expressions on the command line, especially if they involve literal strings. Getting it to work is a bit of a black art. Avoiding spaces, which you are already doing, is good. The other thing that is sometimes necessary is escaping individual problematic characters (including quotes, hashes, semicolons) with a backslash. I can get the following command to work from the C shell: stilts tpipe in=... cmd='addcol x concat(\"Skycam_\",formatDecimal(col1_1_1,2),\"_\",formatDecimal(col2_1_1,\"+0.\#\#\#\;-0.\#\#\#\"))' My apologies that this is so ugly. If you like python it's worth noting that these problems mostly go away in JyStilts, the jython interface to STILTS (http://www.starlink.ac.uk/stilts/sun256/jystilts.html). You could do it like this in jython: >>> import stilts >>> a = stilts.tread('in.txt', fmt='ascii') >>> b = a.cmd_addcol('x', ... 'concat("Skycam_", formatDecimal(col1_1_1, 2), "_", \ ... formatDecimal(col2_1_1, "+0.###;-0.###"))') >>> b.write('out.txt', fmt='ascii') Mark On Fri, 29 Jul 2011, Neil Mawson wrote: > Dear Mark Taylor > > I'm currently having some issues with your STILTS software, I'm trying to use > the addcol command to add an extra column as as identifier using the values > from two other columns and some text. This is the command I'm trying: > > concat("Skycam_",formatDecimal(col1_1_1,2),"_",formatDecimal(col2_1_1,"+0.###;-0.###"))' > > This command works in Topcat, however when I try to run it in STILTS I get > error messages: > > Unused arguments: ',formatDecimal(col1_1_1,2),_' > ',formatDecimal(col2_1_1,+0.###;-0.###' '))' > > Command was: addcol skyref > concat("Skycam_",formatDecimal(col1_1_1,2),"_",formatDecimal(col2_1_1,"+0.###;-0.###")) > > Usage: addcol [-after | -before ] > [-units ] [-ucd ] [-desc ] > > (Unused arguments: ',formatDecimal(col1_1_1,2),_' > ',formatDecimal(col2_1_1,+0.###;-0.###' '))') > > > > I've tried many connotations of this command with no joy, and have been left > scratching my head. Is this an expected limitation with STILTS or have I > missed something out. > > Any help you could provide would be greatly appreciated > > Best regards > Neil Mawson > > -- > Neil Mawson > Astrophysics Research Institute > Liverpool John Moores University > Twelve Quays House, > Egerton Wharf, > Birkenhead, > Wirral, > CH41 1LD > UK > > Tel: +44 151 231 2923 > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Nick.Ball@nrc-cnrc.XXX Mon Aug 1 22:11:27 2011 Date: Mon, 1 Aug 2011 13:59:37 -0700 From: "Ball, Nick" To: "topcat-user@sympa.XXX" Subject: Re: No handler for URL I got the problem again after doing a force quit when Topcat ran out of memory. Deleting ~/.samp works. Nick -- Nick Ball Herzberg Institute of Astrophysics 5071 West Saanich Road Victoria, BC V9E 2E7 Canada From m.b.taylor@bristol.XXX Tue Aug 2 15:32:55 2011 Date: Tue, 2 Aug 2011 15:32:55 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: No handler for URL On Mon, 1 Aug 2011, Ball, Nick wrote: > I got the problem again after doing a force quit when Topcat ran out of memory. Deleting ~/.samp works. I'm not surprised that a forced termination of the process causes this to happen. Anyway it seems that this is not a one-off, so I've fixed it to avoid this problem in the future: on startup, TOPCAT will check to see whether the .samp file represents a moribund hub, and will overwrite it if it does. That should mean that this issue won't cause trouble in the future. This fix will be in the next public release of TOPCAT - a pre-release version is at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ if anyone would like a go (jar file only, no dmg though). Thanks to Nick and Phil for reporting this. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Aug 8 18:14:38 2011 Date: Mon, 8 Aug 2011 18:14:37 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Additional pair matching options Hi all, I've added some new options to the pair matching capabilities in TOPCAT and STILTS. As well as the existing find=BEST/ALL options for the tmatch2 and tskymatch2 STILTS commands there are now new BEST1 and BEST2 options (the corresponding thing in TOPCAT is the Match Selection selector in the Output Rows panel of the Pair Match window). This should make life easier for some common crossmatching scenarios in the presence of crowded fields; although the default symmetric crossmatching represented by the BEST option is well-defined, it doesn't always do what's intuitively expected when objects in one or both input tables are closer to each other than the match radius. The new BEST1/BEST2 options correspond (I believe) to what Aladin does. In many (probably most) cases you can ignore the distinction between BEST, BEST1 and BEST2, but in others it has caused confusion, and so is addressed in the FAQ: http://www.starlink.ac.uk/topcat/faq.html#xmatchIssues These options will be available in the next public releases of TOPCAT and STILTS. In the mean time if anybody would like to try them out, you can get pre-release versions from the usual places: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ Here is the new documentation for the FIND parameter in the STILTS tmatch2 (and tskymatch2) commands: % stilts tmatch2 help=find Help for parameter FIND in task TMATCH2 --------------------------------------- Name: find Usage: [find=all|best|best1|best2] Summary: Which pair matches to include Description: Determines what happens when a row in one table can be matched by more than one row in the other table. The options are: * all: All matches. Every match between the two tables is included in the result. Rows from both of the input tables may appear multiple times in the result. * best: Best match, symmetric. The best pairs are selected in a way which treats the two tables symmetrically. Any input row which appears in one result pair is disqualified from appearing in any other result pair, so each row from both input tables will appear in at most one row in the result. * best1: Best match for each table 1 row. For each row in table 1, only the best match from table 2 will appear in the result. Each row from table 1 will appear a maximum of once in the result, but rows from table 2 may appear multiple times. * best2: Best match for each table 2 row. For each row in table 2, only the best match from table 1 will appear in the result. Each row from table 2 will appear a maximum of once in the result, but rows from table 1 may appear multiple times. The differences between best, best1 and best2 are a bit subtle. In cases where it's obvious which object in each table is the best match for which object in the other, choosing betwen these options will not affect the result. However, in crowded fields (where the distance between objects within one or both tables is typically similar to or smaller than the specified match radius) it will make a difference. In this case one of the asymmetric options (best1 or best2) is usually more appropriate than best, but you'll have to think about which of them suits your requirements. The performance (time and memory usage) of the match may also differ between these options, especially if one table is much bigger than the other. Default: best Comments, as always, very welcome. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From dabrusco@gmail.XXX Tue Aug 30 17:05:37 2011 Date: Tue, 30 Aug 2011 11:54:09 -0400 From: Raffaele D'Abrusco To: topcat-user@bristol.XXX Subject: Discarding multiple tables at once It seems to me that, right now, is not possible to discard multiple tables at once in Topcat. Mark, do you think this feature could be implemented in one of the next Topcat releases? Cheers, Raffaele From m.b.taylor@bristol.XXX Tue Aug 30 17:08:56 2011 Date: Tue, 30 Aug 2011 17:08:55 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Discarding multiple tables at once On Tue, 30 Aug 2011, Raffaele D'Abrusco wrote: > It seems to me that, right now, is not possible to discard multiple tables > at once in Topcat. correct. > Mark, do you think this feature could be implemented in one of the next > Topcat releases? yes, it's on The List, I will try to do it sooner rather than later. Thanks for the request. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Deil.Christoph@googlemail.XXX Wed Sep 21 02:40:52 2011 Date: Wed, 21 Sep 2011 02:40:19 +0200 From: Christoph Deil To: topcat-user@bristol.XXX Subject: Rescale column to median 1 with STILTS? Hi, I've got a column which I'd like to divide by the median value of that column. Is this possible with STILTS? Thanks! Christoph From m.b.taylor@bristol.XXX Thu Sep 22 09:15:19 2011 Date: Thu, 22 Sep 2011 09:15:19 +0100 (BST) From: Mark Taylor To: Christoph Deil Cc: topcat-user@bristol.XXX Subject: Re: Rescale column to median 1 with STILTS? On Wed, 21 Sep 2011, Christoph Deil wrote: > Hi, > > I've got a column which I'd like to divide by the median value of that column. > Is this possible with STILTS? you can't do it directly. You can use the stats output mode or filter to calculate the median, and then use that value as a constant in a subsequent command using the addcol filter. This would be quite fiddly to do without human intervention from command-line stilts, but in JyStilts using the stats filter not too bad. If it's not obvious how to, I can elucidate. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From mtibbetts@cfa.XXX Fri Sep 23 19:57:18 2011 Date: Fri, 23 Sep 2011 18:46:33 +0100 From: mtibbetts@cfa.XXX To: topcat-user@sympa.XXX Subject: stil: Accessing TAP query status 'overflow' Hi, The TAP recommendation state: If an overflow occurs (result exceeds MAXREC), the service must close the table and append another INFO element to the RESOURCE (after the TABLE) with name=”QUERY_STATUS” and the value=”OVERFLOW”. I've noted that the TableStreamer throws a SuccessfulCompletionException once the TABLE element is closed. If I am interested in reading the INFO element following the TABLE to check for over flow, is there any ways to do this using the classes in the library? Thanks, Mike From m.b.taylor@bristol.XXX Mon Sep 26 15:35:20 2011 Date: Mon, 26 Sep 2011 15:35:20 +0100 (BST) From: Mark Taylor To: mtibbetts@cfa.XXX Cc: topcat-user@sympa.XXX Subject: Re: stil: Accessing TAP query status 'overflow' Mike, On Fri, 23 Sep 2011, mtibbetts@cfa.XXX wrote: > Hi, > > The TAP recommendation state: > If an overflow occurs (result exceeds MAXREC), the service must close the > table and append another > INFO element to the RESOURCE (after the TABLE) with name=”QUERY_STATUS” > and the > value=”OVERFLOW”. > > I've noted that the TableStreamer throws a SuccessfulCompletionException once > the TABLE element is closed. If I am interested in reading the INFO element > following the TABLE to check for over flow, is there any ways to do this using > the classes in the library? You're right, TableStreamer, which is used by VOTableFactory, and hence for reading VOTables in a generic way, packs up and goes home once it has reached the end of the table rows, and hence misses any INFO elements after the end of the TABLE element. If you want to be able to do something more specific with a VOTable document than the one-size-fits-all read provided by VOTableFactory, such as examine post-TABLE INFO elements, then you need to look at the VOTable document structure itself, which happens at a lower level. To do this, you can use uk.ac.starlink.votable.VOElementFactory, which constructs a DOM from an input stream (it does this in an efficient way, so you don't need to worry about it building a gigantic data structure full of TD elements). You can then navigate this DOM in the usual way. There is one special trick that you need to know: once you've identified a TABLE element in this DOM, you can turn it into a StarTable by using the VOStarTable(TableElement) constructor. There's an example of this in section 7.3.2 of the STIL manual: http://www.starlink.ac.uk/stil/sun252/votableDom.html or as part of the TOPCAT/STILTS code base, in the readResultVOTable method of uk.ac.starlink.vo.TapQuery at: http://starjava.jach.hawaii.edu/viewvc/trunk/vo/src/main/uk/ac/starlink/vo/TapQuery.java?view=co If you prefer SAX, you can see sec 7.3.3 of the STIL manual. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From dvanstone@cfa.XXX Tue Oct 4 20:18:20 2011 Date: Tue, 04 Oct 2011 19:17:28 +0100 From: dvanstone@cfa.XXX To: topcat-user@sympa.XXX Subject: starTable.getCell(row,col) and NaN Hello, I recently upgraded from 2.8 to 3.0 and noticed a change of behavior I wanted to make sure was correct. I read in a StarTable from an xml file with a null double value (). In version 2.8, the object returned from starTable.getCell() was null. In version 3.0, the object returned is Double.NaN. Is this the correct behavior? Thanks, David From m.b.taylor@bristol.XXX Wed Oct 5 10:29:19 2011 Date: Wed, 5 Oct 2011 10:29:19 +0100 (BST) From: Mark Taylor To: dvanstone@cfa.XXX Cc: topcat-user@sympa.XXX Subject: Re: starTable.getCell(row,col) and NaN David, On Tue, 4 Oct 2011, dvanstone@cfa.XXX wrote: > Hello, > > I recently upgraded from 2.8 to 3.0 and noticed a change of behavior I wanted > to make sure was correct. > > I read in a StarTable from an xml file with a null double value (). > > In version 2.8, the object returned from starTable.getCell() was null. > > In version 3.0, the object returned is Double.NaN. > > Is this the correct behavior? I'm not sure exactly what I did that changed the behaviour, but as far as the data model of both STIL and VOTable goes, I would say either of those is a reasonable result. STIL blurs the distinction between null and NaN for floating point values, so it's not advisable to rely on the difference between the two (from a Java point of view, this is because a double precision floating point value may either be represented as a Double object or as a double primitive, depending on exactly which bits of implementation it's passing through, and in the latter case no null is possible, so NaN is co-opted to stand for either). This lack of distinction is implicit in the VOTable data model as well (inherited from FITS, which does the same). As it happens, it's a bit of a hot topic in VOTable just now. If the above behaviour is problematic for you, rather than just surprising, I'd be interested to hear a bit more about why, since it may feed into current discussions about how VOTable is specified. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From dvanstone@cfa.XXX Wed Oct 5 16:39:57 2011 Date: Wed, 05 Oct 2011 15:39:22 +0100 From: dvanstone@cfa.XXX To: m.b.taylor@bristol.XXX topcat-user@sympa.XXX Subject: Re: Re: starTable.getCell(row,col) and NaN >This lack of distinction is implicit in the VOTable data model as >well (inherited from FITS, which does the same). As it happens, >it's a bit of a hot topic in VOTable just now. If the above >behaviour is problematic for you, rather than just surprising, >I'd be interested to hear a bit more about why, since it may feed >into current discussions about how VOTable is specified. I'm not quite sure I want to give the knowledge to my client program that "Double.NaN" is a representation of null. If an integer value has a null representation of "-32768" in a VOTable, I wouldn't want the stil interface to respond to getCell() with the Integer value of -32768, since then I'd have to query the column and ask it what its null representation is. (Assuming I could do so.) I hope that helps. -David- From m.b.taylor@bristol.XXX Wed Oct 5 16:06:00 2011 Date: Wed, 5 Oct 2011 16:06:00 +0100 (BST) From: Mark Taylor To: dvanstone@cfa.XXX Cc: topcat-user@sympa.XXX Subject: Re: Re: starTable.getCell(row,col) and NaN On Wed, 5 Oct 2011, dvanstone@cfa.XXX wrote: > >This lack of distinction is implicit in the VOTable data model as > >well (inherited from FITS, which does the same). As it happens, > >it's a bit of a hot topic in VOTable just now. If the above > >behaviour is problematic for you, rather than just surprising, > >I'd be interested to hear a bit more about why, since it may feed > >into current discussions about how VOTable is specified. > > I'm not quite sure I want to give the knowledge to my client program that > "Double.NaN" is a representation of null. > > If an integer value has a null representation of "-32768" in a VOTable, I > wouldn't want the stil interface to respond to getCell() with the Integer value > of -32768, since then I'd have to query the column and ask it what its null > representation is. (Assuming I could do so.) STIL will give you a null and not -32768 in this case, it is only floating point values where this arises. So thinking about it more, my earlier excuse about why STIL works this way doesn't seem to be valid; if it can preserve nulls for integer types (which it can) then it should be able to do the same for floating point types too. If you'd like me to try to restore the previous behaviour, let me know and I'll put it on the list. However, for the case of VOTables as the VOTable standard currently stands (and not everybody is happy about this) the two things are really not well distinguished; quoting from the discussion of the Double Precision data type VOTable 1.2 section 6, 'The IEEE NaN pattern is used to represent null values. ... ... the default representation of a null value is an empty cell ..., and the special values "+Inf", "-Inf", and "NaN" are accepted.' To put it another way: > I'm not quite sure I want to give the knowledge to my client program that > "Double.NaN" is a representation of null. that knowledge is basically a fact about VOTable's data model. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From francois@cdsarc.XXX Wed Oct 5 18:27:29 2011 Date: Wed, 05 Oct 2011 18:27:11 +0200 From: "Francois Ochsenbein (ext.52429)" To: Mark Taylor Cc: dvanstone@cfa.XXX topcat-user@mlist2.XXX Subject: Re: starTable.getCell(row,col) and NaN Hi Mark, Well, when I wrote the VOTable document, I had in mind that, in the xml serialization, NaN, and +/-Inf, are accepted as a representation of "null", as does the blank (or empty cell). In other terms, floating-point numbers are acceptable in just 2 categories: either "valid numbers" (in practice, numbers that can be represented by digits), or "null". In this context, the infinity is not a "valid number" -- then it should be considered as a "null". I agree that the text is somewhat ambiguous... One could argue about the infinity, but FITS does not specify anything about the infinity in binary tables; and about NaN, FITS (and VOTable) specify that the NaN values do represent the "null": it is not allowed to specify some "valid number" as a representation of "null" for floating-point columns. Cheers, francois >On Wed, 5 Oct 2011, dvanstone@cfa.XXX wrote: > >> >This lack of distinction is implicit in the VOTable data model as >> >well (inherited from FITS, which does the same). As it happens, >> >it's a bit of a hot topic in VOTable just now. If the above >> >behaviour is problematic for you, rather than just surprising, >> >I'd be interested to hear a bit more about why, since it may feed >> >into current discussions about how VOTable is specified. >> >> I'm not quite sure I want to give the knowledge to my client program that >> "Double.NaN" is a representation of null. >> >> If an integer value has a null representation of "-32768" in a VOTable, I >> wouldn't want the stil interface to respond to getCell() with the Integer value >> of -32768, since then I'd have to query the column and ask it what its null >> representation is. (Assuming I could do so.) > >STIL will give you a null and not -32768 in this case, >it is only floating point values where this arises. > >So thinking about it more, my earlier excuse about why STIL works this >way doesn't seem to be valid; if it can preserve nulls for integer types >(which it can) then it should be able to do the same for floating point >types too. If you'd like me to try to restore the previous behaviour, >let me know and I'll put it on the list. > >However, for the case of VOTables as the VOTable standard currently >stands (and not everybody is happy about this) the two things are >really not well distinguished; quoting from the discussion of the >Double Precision data type VOTable 1.2 section 6, > > 'The IEEE NaN pattern is used to represent null values. ... > ... the default representation of a null value is an empty cell ..., > and the special values "+Inf", "-Inf", and "NaN" are accepted.' > >To put it another way: > >> I'm not quite sure I want to give the knowledge to my client program that >> "Double.NaN" is a representation of null. > >that knowledge is basically a fact about VOTable's data model. > >Mark > >-- >Mark Taylor Astronomical Programmer Physics, Bristol University, UK ======================================================================= Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 Email: francois@astro.XXX (France) Fax: +33-(0)368 85 24 17 ======================================================================= From m.b.taylor@bristol.XXX Thu Oct 6 11:12:54 2011 Date: Thu, 6 Oct 2011 11:12:52 +0100 (BST) From: Mark Taylor To: Francois Ochsenbein Cc: topcat-user@bristol.XXX Subject: Re: starTable.getCell(row,col) and NaN Francois, I should say that my comments are not intended as a criticism of VOTable, my personal feeling is that using NaN as the only way to designate a missing value as per FITS is a reasonable thing to do. However this does depend on one's perspective and intended uses, and people used to the data models used by RDBMS tend to find it uncomfortable. Concerning +/-Inf: I don't think the intention that they are synonymous with NaN is clear from the VOTable text, and I also don't think it's the right thing to do. FITS uses IEEE 754 for single/double floating point values, and IEEE 754 does accommodate representation of positive and negative infinity (see Table E.2 of the FITS v3.0 standard). STIL treats +/-Inf contents of TDs distinctly from NaN in accordance with this. Mark On Wed, 5 Oct 2011, Francois Ochsenbein (ext.52429) wrote: > > Hi Mark, > > Well, when I wrote the VOTable document, I had in mind that, > in the xml serialization, NaN, and +/-Inf, are accepted as a > representation of "null", as does the blank (or empty cell). > In other terms, floating-point numbers are acceptable in just > 2 categories: either "valid numbers" (in practice, numbers that > can be represented by digits), or "null". In this context, the > infinity is not a "valid number" -- then it should be considered > as a "null". I agree that the text is somewhat ambiguous... > > One could argue about the infinity, but FITS does not specify > anything about the infinity in binary tables; and about NaN, > FITS (and VOTable) specify that the NaN values do represent the > "null": it is not allowed to specify some "valid number" as > a representation of "null" for floating-point columns. > > > Cheers, francois > > >On Wed, 5 Oct 2011, dvanstone@cfa.XXX wrote: > > > >> >This lack of distinction is implicit in the VOTable data model as > >> >well (inherited from FITS, which does the same). As it happens, > >> >it's a bit of a hot topic in VOTable just now. If the above > >> >behaviour is problematic for you, rather than just surprising, > >> >I'd be interested to hear a bit more about why, since it may feed > >> >into current discussions about how VOTable is specified. > >> > >> I'm not quite sure I want to give the knowledge to my client program that > >> "Double.NaN" is a representation of null. > >> > >> If an integer value has a null representation of "-32768" in a VOTable, I > >> wouldn't want the stil interface to respond to getCell() with the Integer value > >> of -32768, since then I'd have to query the column and ask it what its null > >> representation is. (Assuming I could do so.) > > > >STIL will give you a null and not -32768 in this case, > >it is only floating point values where this arises. > > > >So thinking about it more, my earlier excuse about why STIL works this > >way doesn't seem to be valid; if it can preserve nulls for integer types > >(which it can) then it should be able to do the same for floating point > >types too. If you'd like me to try to restore the previous behaviour, > >let me know and I'll put it on the list. > > > >However, for the case of VOTables as the VOTable standard currently > >stands (and not everybody is happy about this) the two things are > >really not well distinguished; quoting from the discussion of the > >Double Precision data type VOTable 1.2 section 6, > > > > 'The IEEE NaN pattern is used to represent null values. ... > > ... the default representation of a null value is an empty cell ..., > > and the special values "+Inf", "-Inf", and "NaN" are accepted.' > > > >To put it another way: > > > >> I'm not quite sure I want to give the knowledge to my client program that > >> "Double.NaN" is a representation of null. > > > >that knowledge is basically a fact about VOTable's data model. > > > >Mark > > > >-- > >Mark Taylor Astronomical Programmer Physics, Bristol University, UK > ======================================================================= > Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg > 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 > Email: francois@astro.XXX (France) Fax: +33-(0)368 85 24 17 > ======================================================================= > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From francois@cdsarc.XXX Thu Oct 6 16:24:58 2011 Date: Thu, 06 Oct 2011 16:21:44 +0200 From: "Francois Ochsenbein (ext.52429)" To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: starTable.getCell(row,col) and NaN Mark, As discussed on another thread, RDBMS are not happy with these peculiar floating-point values: no standard syntax in SQL to deal with +/-Inf or NaN, no standard handling defined (at least as far as I know), the results may be wrong or generate nasty errors. Better anyway to avoid the +/-Inf if it is expected to keep the correctness of the data when these are moved between RDBMSs, FITS tables, and VO tools... Cheers, francois > >Francois, > >I should say that my comments are not intended as a criticism of >VOTable, my personal feeling is that using NaN as the only way to >designate a missing value as per FITS is a reasonable thing to do. >However this does depend on one's perspective and intended uses, >and people used to the data models used by RDBMS tend to find it >uncomfortable. > >Concerning +/-Inf: I don't think the intention that they are synonymous >with NaN is clear from the VOTable text, and I also don't think it's >the right thing to do. FITS uses IEEE 754 for single/double floating >point values, and IEEE 754 does accommodate representation of positive >and negative infinity (see Table E.2 of the FITS v3.0 standard). >STIL treats +/-Inf contents of TDs distinctly from NaN in accordance >with this. > >Mark > > >On Wed, 5 Oct 2011, Francois Ochsenbein (ext.52429) wrote: > >> >> Hi Mark, >> >> Well, when I wrote the VOTable document, I had in mind that, >> in the xml serialization, NaN, and +/-Inf, are accepted as a >> representation of "null", as does the blank (or empty cell). >> In other terms, floating-point numbers are acceptable in just >> 2 categories: either "valid numbers" (in practice, numbers that >> can be represented by digits), or "null". In this context, the >> infinity is not a "valid number" -- then it should be considered >> as a "null". I agree that the text is somewhat ambiguous... >> >> One could argue about the infinity, but FITS does not specify >> anything about the infinity in binary tables; and about NaN, >> FITS (and VOTable) specify that the NaN values do represent the >> "null": it is not allowed to specify some "valid number" as >> a representation of "null" for floating-point columns. >> >> >> Cheers, francois >> >> >On Wed, 5 Oct 2011, dvanstone@cfa.XXX wrote: >> > >> >> >This lack of distinction is implicit in the VOTable data model as >> >> >well (inherited from FITS, which does the same). As it happens, >> >> >it's a bit of a hot topic in VOTable just now. If the above >> >> >behaviour is problematic for you, rather than just surprising, >> >> >I'd be interested to hear a bit more about why, since it may feed >> >> >into current discussions about how VOTable is specified. >> >> >> >> I'm not quite sure I want to give the knowledge to my client program that >> >> "Double.NaN" is a representation of null. >> >> >> >> If an integer value has a null representation of "-32768" in a VOTable, I >> >> wouldn't want the stil interface to respond to getCell() with the Integer value >> >> of -32768, since then I'd have to query the column and ask it what its null >> >> representation is. (Assuming I could do so.) >> > >> >STIL will give you a null and not -32768 in this case, >> >it is only floating point values where this arises. >> > >> >So thinking about it more, my earlier excuse about why STIL works this >> >way doesn't seem to be valid; if it can preserve nulls for integer types >> >(which it can) then it should be able to do the same for floating point >> >types too. If you'd like me to try to restore the previous behaviour, >> >let me know and I'll put it on the list. >> > >> >However, for the case of VOTables as the VOTable standard currently >> >stands (and not everybody is happy about this) the two things are >> >really not well distinguished; quoting from the discussion of the >> >Double Precision data type VOTable 1.2 section 6, >> > >> > 'The IEEE NaN pattern is used to represent null values. ... >> > ... the default representation of a null value is an empty cell ..., >> > and the special values "+Inf", "-Inf", and "NaN" are accepted.' >> > >> >To put it another way: >> > >> >> I'm not quite sure I want to give the knowledge to my client program that >> >> "Double.NaN" is a representation of null. >> > >> >that knowledge is basically a fact about VOTable's data model. >> > >> >Mark >> > >> >-- >> >Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> ======================================================================= >> Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg >> 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 >> Email: francois@astro.XXX (France) Fax: +33-(0)368 85 24 17 >> ======================================================================= >> > >-- >Mark Taylor Astronomical Programmer Physics, Bristol University, UK >m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ ======================================================================= Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 Email: francois@astro.XXX (France) Fax: +33-(0)368 85 24 17 ======================================================================= From Christoph.Deil@mpi-hd.XXX Mon Oct 10 12:28:32 2011 Date: Mon, 10 Oct 2011 12:01:06 +0200 From: Christoph Deil To: topcat-user@bristol.XXX Subject: Concatenate tables with STILTS Hi, in TOPCAT the concatenate tables dialog automatically pairs columns with the same names. As far as I can see this functionality is not included in STILTS tcat and tcatn ? Would it be possible to add this feature to STILTS, as an automatic pairing for identical names and with an option for explicit pairing when the names don't match up, e.g. colnames='Name:Source_Name,RA:RAJ2000,DEC:DEJ2000' ? Christoph From m.b.taylor@bristol.XXX Mon Oct 10 12:01:48 2011 Date: Mon, 10 Oct 2011 12:01:48 +0100 (BST) From: Mark Taylor To: Christoph Deil Cc: topcat-user@bristol.XXX Subject: Re: Concatenate tables with STILTS Christoph, On Mon, 10 Oct 2011, Christoph Deil wrote: > Hi, > > in TOPCAT the concatenate tables dialog automatically pairs columns with the same names. > As far as I can see this functionality is not included in STILTS tcat and tcatn ? > > Would it be possible to add this feature to STILTS, as an automatic pairing for identical names and with an option for explicit pairing when the names don't match up, e.g. colnames='Name:Source_Name,RA:RAJ2000,DEC:DEJ2000' ? it's a reasonable request but ... I think the answer is that I won't. In TOPCAT the auto-pairing is a default which you have the option to change, and it's clear from the GUI which columns do not have a pair. Mostly the defaults are intended for the case where the tables have exactly the same columns, which is served in STILTS by the tcat command. From a command-line interface it's much less obvious to the user what a non-exact automatic matching is actually doing, and the logic of which columns to match automatically and which to do by request may get tortuous (especially for >2 input files...) If the tables have exactly the same columns in the same order, you can just use tcat. For tables which are similar but not quite the same, probably the most straightforward idiom is to use tcatn with similar keepcols input filters for each table as in the third example at http://www.starlink.ac.uk/stilts/sun256/tcatn-examples.html. If you're in jystilts, you can automate this to some extent by getting the list of column names with an expression like map((lambda c: c.name), table.columns()) Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From elle@oal.XXX Wed Oct 12 17:42:52 2011 Date: Wed, 12 Oct 2011 17:31:57 +0200 From: elle To: topcat-user@bristol.XXX Subject: how to draw lines in a plot hi everybody there, I'd like to draw a line in a plot according a given equation which contains constants and variables ($X in a table). Can I do that? i tried to play with the row subsets a little bit but I didn't manage to. Can anybody please help me? Thanks! From pierre.didelon@cea.XXX Wed Oct 12 17:51:44 2011 Date: Wed, 12 Oct 2011 17:46:38 +0200 From: Pierre Didelon To: elle Cc: topcat-user@bristol.XXX Subject: Re: how to draw lines in a plot elle wrote: >hi everybody there, I'd like to draw a line in a plot according a given >equation which contains constants and variables ($X in a table). Can I do >that? i tried to play with the row subsets a little bit but I didn't manage >to. >Can anybody please help me? > >Thanks! > > > hello create a column ( c1 ) with the X value you want (entering them by hand if there is no other ways or importing them) then create a column ( c2 ) calculate from X with your formula plotting c2 vs c1 would give what you want? does it answer fully to your question? -- Pierre ------------------------------------------------------------------- DIDELON :@: pdidelon_at_cea.fr Phone : 33 (0)1 69 08 58 89 CEA SACLAY - Service d'Astrophysique 91191 Gif-Sur-Yvette Cedex ------------------------------------------------------------------- Aidez les enfants Tibétains : http://www.a-e-t.org/actions/ d'autres : http://www.sosesf.org/ ou trouvez une autre ONG ---------------------------------------------------------------------------- - Il n'est pas toujours possible de bien faire dans une situation difficile - La foi est la blessure que le savoir guérit -Le Dit d'Aka, Ursula Le Guin ---------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Wed Oct 12 17:26:44 2011 Date: Wed, 12 Oct 2011 17:26:43 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: how to draw lines in a plot On Wed, 12 Oct 2011, Pierre Didelon wrote: > elle wrote: > > > hi everybody there, I'd like to draw a line in a plot according a given > > equation which contains constants and variables ($X in a table). Can I do > > that? i tried to play with the row subsets a little bit but I didn't manage > > to. > > Can anybody please help me? > > > > Thanks! > > > > > hello > create a column ( c1 ) with the X value you want (entering them by hand if > there is no other ways or importing them) > then create a column ( c2 ) calculate from X with your formula > plotting c2 vs c1 would give what you want? > does it answer fully to your question? Thanks Pierre, that's correct. Usually it's not necessary to enter values by hand, since you can use the row number of any available table and scale it appropriately to give you linear coverage of whatever range you want. A convenient way to do that is to define a new column in terms of "$0", which is a magic value giving the row number. In the attached screenshot I've done it without defining a new column, (over)plotting the values X = ($0/875.)*14+8 Y = sin($0/875.)+9 875 is the number of rows in the table. Having said all that, TOPCAT isn't really optimised for plotting lines, but as you can see it is possible to persuade it to do that. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ [ Part 2: "screenshot of line overplotted on scatter plot" ] The following attachment was sent, but NOT saved in the Fcc copy: A Image/gif (Name="lineshot.gif") segment of about 92,149 bytes. From ivan.zolotukhin@gmail.XXX Sun Oct 16 21:42:14 2011 Date: Sun, 16 Oct 2011 21:31:17 +0200 From: Ivan Zolotukhin To: topcat-user Subject: long integer in select Hi Mark, Looks like I've just hit STILTS gotcha: > stilts tpipe table.fits cmd="select objid==588295841780007034" Error: Bad expression "objid==588295841780007034" (Encountered unexpected end of expression.) (Encountered unexpected end of expression.) The number in question is just a long integer, same as the column type. I managed to work it around only with this expression: > stilts tpipe table.fits cmd="select 'objid==parseLong(\"588295841780007034\")'" Am I doing something wrong? -- With best regards, Ivan From m.b.taylor@bristol.XXX Mon Oct 17 04:37:08 2011 Date: Mon, 17 Oct 2011 04:37:08 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: long integer in select On Sun, 16 Oct 2011, Ivan Zolotukhin wrote: > Hi Mark, > > Looks like I've just hit STILTS gotcha: > > > stilts tpipe table.fits cmd="select objid==588295841780007034" > Error: Bad expression "objid==588295841780007034" (Encountered > unexpected end of expression.) > (Encountered unexpected end of expression.) > > The number in question is just a long integer, same as the column > type. I managed to work it around only with this expression: > > > stilts tpipe table.fits cmd="select 'objid==parseLong(\"588295841780007034\")'" that works - the more straightforward (though not very obvious) way is "objid==588295841780007034L" - the appended L marks a long integer in java. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Sun Oct 23 00:52:36 2011 Date: Sun, 23 Oct 2011 04:11:34 +0530 From: Ivan Zolotukhin To: topcat-user Subject: string from array Hi Mark, I wonder if there's a way to transform an array column (e.g. float[]) to a string? This is needed for custom activation function where I'd like to easily pass row contents to external script without naming explicitly all the array elements. I did not find anything in the docs. -- With best regards, Ivan From m.b.taylor@bristol.XXX Sun Oct 23 12:42:32 2011 Date: Sun, 23 Oct 2011 12:42:32 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: string from array On Sun, 23 Oct 2011, Ivan Zolotukhin wrote: > Hi Mark, > > I wonder if there's a way to transform an array column (e.g. float[]) > to a string? This is needed for custom activation function where I'd > like to easily pass row contents to external script without naming > explicitly all the array elements. I did not find anything in the > docs. not sure I understand the question - do you mean [1.,2.,3.5] -> "1,2,3.5" or something else? -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Sun Oct 23 14:29:45 2011 Date: Sun, 23 Oct 2011 17:59:30 +0530 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: string from array On Sun, Oct 23, 2011 at 5:12 PM, Mark Taylor wrote: > On Sun, 23 Oct 2011, Ivan Zolotukhin wrote: > >> I wonder if there's a way to transform an array column (e.g. float[]) >> to a string? This is needed for custom activation function where I'd >> like to easily pass row contents to external script without naming >> explicitly all the array elements. I did not find anything in the >> docs. > > not sure I understand the question - do you mean > >   [1.,2.,3.5] -> "1,2,3.5" > > or something else? I mean is there any method of doing smth like toString(ARR) instead of using toString(ARR[0]) + "," + toString(ARR[1]) + "," + toString(ARR[2]) etc, where ARR is an array column from a table. From m.b.taylor@bristol.XXX Mon Oct 24 13:31:52 2011 Date: Mon, 24 Oct 2011 13:31:52 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: string from array On Sun, 23 Oct 2011, Ivan Zolotukhin wrote: > On Sun, Oct 23, 2011 at 5:12 PM, Mark Taylor wrote: > > On Sun, 23 Oct 2011, Ivan Zolotukhin wrote: > > > >> I wonder if there's a way to transform an array column (e.g. float[]) > >> to a string? This is needed for custom activation function where I'd > >> like to easily pass row contents to external script without naming > >> explicitly all the array elements. I did not find anything in the > >> docs. > > > > not sure I understand the question - do you mean > > > >   [1.,2.,3.5] -> "1,2,3.5" > > > > or something else? > > I mean is there any method of doing smth like toString(ARR) instead of > using toString(ARR[0]) + "," + toString(ARR[1]) + "," + > toString(ARR[2]) etc, where ARR is an array column from a table. Not in the application as it stands. This might be a suitable case for adding a simple extension function as described at: http://www.starlink.ac.uk/topcat/sun253/jelExtend.html Like this: % cat ArrayString.java import java.lang.reflect.Array; public class ArrayString { public static String array2string(Object array) { StringBuffer buf = new StringBuffer(); for (int i = 0; i < Array.getLength(array); i++) { if (i > 0) { buf.append(", "); } buf.append(Array.get(array, i)); } return buf.toString(); } } % javac ArrayString.java % stilts -classpath . -Djel.classes=ArrayString \ calc 'array2string(array(1,2,3))' 1.0, 2.0, 3.0 In this example the array that is being stringified is one constructed using the expression "array(1,2,3)", but it could equally be an array-valued variable/column. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From anthonysmith80@gmail.XXX Mon Oct 24 15:18:03 2011 Date: Mon, 24 Oct 2011 14:17:47 +0100 From: Anthony Smith To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: string from array Probably a one-line command in JyStilts :) Something like ','.join([str(value) for value in arr]) Cheers, Anthony On 24 October 2011 13:31, Mark Taylor wrote: > On Sun, 23 Oct 2011, Ivan Zolotukhin wrote: > > > On Sun, Oct 23, 2011 at 5:12 PM, Mark Taylor > wrote: > > > On Sun, 23 Oct 2011, Ivan Zolotukhin wrote: > > > > > >> I wonder if there's a way to transform an array column (e.g. float[]) > > >> to a string? This is needed for custom activation function where I'd > > >> like to easily pass row contents to external script without naming > > >> explicitly all the array elements. I did not find anything in the > > >> docs. > > > > > > not sure I understand the question - do you mean > > > > > > [1.,2.,3.5] -> "1,2,3.5" > > > > > > or something else? > > > > I mean is there any method of doing smth like toString(ARR) instead of > > using toString(ARR[0]) + "," + toString(ARR[1]) + "," + > > toString(ARR[2]) etc, where ARR is an array column from a table. > > Not in the application as it stands. This might be a suitable case for > adding a simple extension function as described at: > > http://www.starlink.ac.uk/topcat/sun253/jelExtend.html > > Like this: > > % cat ArrayString.java > import java.lang.reflect.Array; > public class ArrayString { > public static String array2string(Object array) { > StringBuffer buf = new StringBuffer(); > for (int i = 0; i < Array.getLength(array); i++) { > if (i > 0) { > buf.append(", "); > } > buf.append(Array.get(array, i)); > } > return buf.toString(); > } > } > > % javac ArrayString.java > % stilts -classpath . -Djel.classes=ArrayString \ > calc 'array2string(array(1,2,3))' > 1.0, 2.0, 3.0 > > In this example the array that is being stringified is one constructed > using the expression "array(1,2,3)", but it could equally be an > array-valued variable/column. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Mon Oct 24 15:23:37 2011 Date: Mon, 24 Oct 2011 18:53:21 +0530 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: string from array Hi Mark, Thanks for giving an example of user-defined function, but this is not an option for me as I always care about portability of what I do with TOPCAT (think tutorials for researchers) which immediately means custom plugins is not what I want. Can you please consider adding such a function to TOPCAT distribution itself in the future releases? To sound more convincing, I just give here an example of ugly custom activation function (okay, I didn't write that myself but generated with python, but still) I have to use now to plot 4 arrays using Google Charts API (yep, I'm extremely lucky that my arrays all have fixed dimensions, otherwise that would be simply impossible): displayBasicImage("Line profiles", "https://chart.googleapis.com/chart?cht=lc&chdl=Forbidden|Allowed|Forbidden_sm|Allowed_sm&chco=676767,0000FF,000000,00A5C6&chtt=Line+profiles|" + NAME + "&chd=t:" + toString(LINE_LOSVD_FORBIDDEN[0]) + "," + toString(LINE_LOSVD_FORBIDDEN[1]) + "," + toString(LINE_LOSVD_FORBIDDEN[2]) + "," + toString(LINE_LOSVD_FORBIDDEN[3]) + "," + toString(LINE_LOSVD_FORBIDDEN[4]) + "," + toString(LINE_LOSVD_FORBIDDEN[5]) + "," + toString(LINE_LOSVD_FORBIDDEN[6]) + "," + toString(LINE_LOSVD_FORBIDDEN[7]) + "," + toString(LINE_LOSVD_FORBIDDEN[8]) + "," + toString(LINE_LOSVD_FORBIDDEN[9]) + "," + toString(LINE_LOSVD_FORBIDDEN[10]) + "," + toString(LINE_LOSVD_FORBIDDEN[11]) + "," + toString(LINE_LOSVD_FORBIDDEN[12]) + "," + toString(LINE_LOSVD_FORBIDDEN[13]) + "," + toString(LINE_LOSVD_FORBIDDEN[14]) + "," + toString(LINE_LOSVD_FORBIDDEN[15]) + "," + toString(LINE_LOSVD_FORBIDDEN[16]) + "|" + toString(LINE_LOSVD_ALLOWED[0]) + "," + toString(LINE_LOSVD_ALLOWED[1]) + "," + toString(LINE_LOSVD_ALLOWED[2]) + "," + toString(LINE_LOSVD_ALLOWED[3]) + "," + toString(LINE_LOSVD_ALLOWED[4]) + "," + toString(LINE_LOSVD_ALLOWED[5]) + "," + toString(LINE_LOSVD_ALLOWED[6]) + "," + toString(LINE_LOSVD_ALLOWED[7]) + "," + toString(LINE_LOSVD_ALLOWED[8]) + "," + toString(LINE_LOSVD_ALLOWED[9]) + "," + toString(LINE_LOSVD_ALLOWED[10]) + "," + toString(LINE_LOSVD_ALLOWED[11]) + "," + toString(LINE_LOSVD_ALLOWED[12]) + "," + toString(LINE_LOSVD_ALLOWED[13]) + "," + toString(LINE_LOSVD_ALLOWED[14]) + "," + toString(LINE_LOSVD_ALLOWED[15]) + "," + toString(LINE_LOSVD_ALLOWED[16]) + "|" + toString(LINE_LOSVD_FORBIDDEN_SM[0]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[1]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[2]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[3]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[4]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[5]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[6]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[7]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[8]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[9]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[10]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[11]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[12]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[13]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[14]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[15]) + "," + toString(LINE_LOSVD_FORBIDDEN_SM[16]) + "|" + toString(LINE_LOSVD_ALLOWED_SM[0]) + "," + toString(LINE_LOSVD_ALLOWED_SM[1]) + "," + toString(LINE_LOSVD_ALLOWED_SM[2]) + "," + toString(LINE_LOSVD_ALLOWED_SM[3]) + "," + toString(LINE_LOSVD_ALLOWED_SM[4]) + "," + toString(LINE_LOSVD_ALLOWED_SM[5]) + "," + toString(LINE_LOSVD_ALLOWED_SM[6]) + "," + toString(LINE_LOSVD_ALLOWED_SM[7]) + "," + toString(LINE_LOSVD_ALLOWED_SM[8]) + "," + toString(LINE_LOSVD_ALLOWED_SM[9]) + "," + toString(LINE_LOSVD_ALLOWED_SM[10]) + "," + toString(LINE_LOSVD_ALLOWED_SM[11]) + "," + toString(LINE_LOSVD_ALLOWED_SM[12]) + "," + toString(LINE_LOSVD_ALLOWED_SM[13]) + "," + toString(LINE_LOSVD_ALLOWED_SM[14]) + "," + toString(LINE_LOSVD_ALLOWED_SM[15]) + "," + toString(LINE_LOSVD_ALLOWED_SM[16]) + "&chxt=x,y&chs=500x300&chds=a") -- With best regards, Ivan On Mon, Oct 24, 2011 at 6:01 PM, Mark Taylor wrote: > On Sun, 23 Oct 2011, Ivan Zolotukhin wrote: > >> On Sun, Oct 23, 2011 at 5:12 PM, Mark Taylor wrote: >> > On Sun, 23 Oct 2011, Ivan Zolotukhin wrote: >> > >> >> I wonder if there's a way to transform an array column (e.g. float[]) >> >> to a string? This is needed for custom activation function where I'd >> >> like to easily pass row contents to external script without naming >> >> explicitly all the array elements. I did not find anything in the >> >> docs. >> > >> > not sure I understand the question - do you mean >> > >> >   [1.,2.,3.5] -> "1,2,3.5" >> > >> > or something else? >> >> I mean is there any method of doing smth like toString(ARR) instead of >> using toString(ARR[0]) + "," + toString(ARR[1]) + "," + >> toString(ARR[2]) etc, where ARR is an array column from a table. > > Not in the application as it stands.  This might be a suitable case for > adding a simple extension function as described at: > >   http://www.starlink.ac.uk/topcat/sun253/jelExtend.html > > Like this: > >   % cat ArrayString.java >   import java.lang.reflect.Array; >   public class ArrayString { >       public static String array2string(Object array) { >           StringBuffer buf = new StringBuffer(); >           for (int i = 0; i < Array.getLength(array); i++) { >               if (i > 0) { >                   buf.append(", "); >               } >               buf.append(Array.get(array, i)); >           } >           return buf.toString(); >       } >   } > >   % javac ArrayString.java >   % stilts -classpath . -Djel.classes=ArrayString \ >            calc 'array2string(array(1,2,3))' >      1.0, 2.0, 3.0 > > In this example the array that is being stringified is one constructed > using the expression "array(1,2,3)", but it could equally be an > array-valued variable/column. > > Mark > > -- > Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From JJ.Kavelaars@nrc-cnrc.XXX Thu Oct 27 17:03:32 2011 Date: Thu, 27 Oct 2011 08:03:02 -0700 From: "Kavelaars, JJ" To: "topcat-user@bristol.XXX" Subject: TOCAT and VOSpace Is there a plan to put a VOSpace download/upload option into TOPCAT? Something like currently exists for MySpace? JJ From m.b.taylor@bristol.XXX Fri Oct 28 09:03:38 2011 Date: Fri, 28 Oct 2011 09:03:37 +0100 (BST) From: Mark Taylor To: topcat-announce@bristol.XXX topcat-user@bristol.XXX Cc: IVOA Applications WG Subject: TOPCAT/STILTS release Hi all, this is to announce the following releases: TOPCAT v3.9 http://www.starlink.ac.uk/topcat/ STILTS v2.4 http://www.starlink.ac.uk/stilts/ STIL v3.0-3 http://www.starlink.ac.uk/stil/ These releases feature significant improvements to crossmatching: asymmetric match options (often better for crowded fields than the existing Best Match Only option) and capabilities for matching general elliptical shapes on a plane or the sky. Some notable usability improvements include degree-based angular calculation functions (alongside the existing radian-based ones), ability to reorder columns by dragging them in the columns window, ability to discard multiple tables at once, ADQL syntax checking (thanks to Gregory Mantelet's ADQL library from CDS) ... and more! Finally, the SAMP hub launched by this version of TOPCAT has the new Web Profile turned on by default, allowing SAMP-enabled web pages to communicate with TOPCAT and other desktop tools. Full details of the changes, which include the usual crop of bug fixes and minor enhancements, can be found here: http://www.starlink.ac.uk/topcat/sun253/versions.html http://www.starlink.ac.uk/stilts/sun256/versions.html http://www.starlink.ac.uk/stil/sun252/history.html Thanks as ever to all the user feedback which has contributed to these updates. Please send any comments, bug reports, questions etc to the topcat-user list or to me. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Fri Oct 28 09:47:54 2011 Date: Fri, 28 Oct 2011 09:47:54 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: string from array Ivan, it's Christmas. On Mon, 24 Oct 2011, Ivan Zolotukhin wrote: > Thanks for giving an example of user-defined function, but this is not > an option for me as I always care about portability of what I do with > TOPCAT (think tutorials for researchers) which immediately means > custom plugins is not what I want. > > Can you please consider adding such a function to TOPCAT distribution > itself in the future releases? To sound more convincing, I just give ... TOPCAT 3.9 (released today) has: - Arrays.join() function to turn an array into a string - Multi-line SQL entry field in SQL load dialogue - JyStilts runnable from the topcat-*.jar files The Schlegel thing is there but experimental and not documented; if you want to use it I can tell you how. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Fri Oct 28 10:20:08 2011 Date: Fri, 28 Oct 2011 10:20:08 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Discarding multiple tables at once Raffaele, On Tue, 30 Aug 2011, Raffaele D'Abrusco wrote: > It seems to me that, right now, is not possible to discard multiple tables > at once in Topcat. > Mark, do you think this feature could be implemented in one of the next > Topcat releases? Done - in v3.9 if you select multiple tables in the list (ctrl-click on my OS) and then use the File|Discard Table(s) it will get rid of them all (subject to a confirmation dialogue). Thanks for the suggestion. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Fri Oct 28 10:23:30 2011 Date: Fri, 28 Oct 2011 10:23:30 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Reordering list of columns from the "Table Columns" window? On Wed, 30 Mar 2011, Raffaele D'Abrusco wrote: > Hi Mark, > I don't remember if this point has already been raised before in the > Topcat mailing list > (or somewhere else), but I would find very handy to be able to > re-order the columns of > a table from the "Table Columns" window instead of from the "Table > Browser" window. > This would be particularly useful when dealing with tables containing > many columns, in which > case the user is required to drag one column at a time from one side > to the other of the > visible portion of the table in the "Table Browser" window, then > scrolling the table, dragging the > column again, etc. > > Would implementing a similar mechanism in the "Table Columns" window > require a great effort? > By adding, for example, an additional column with "handles" (or > something similar) > to move the columns vertically in the window? this one's done too - in v3.9 you can grab the grey numeric row header at the left of the row in the columns window and drag it up and down. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Fri Oct 28 11:24:42 2011 Date: Fri, 28 Oct 2011 11:24:04 +0200 (CEST) From: Maciej Bilicki Cc: topcat-user@bristol.XXX Subject: Re: string from array Hi Mark, Do I undertsand well that you're adding Schlegel et al. maps to the tools? That would be Christmas for me. Cheers, Maciek > > The Schlegel thing is there but experimental and not documented; > if you want to use it I can tell you how. > From ivan.zolotukhin@gmail.XXX Fri Oct 28 14:16:46 2011 Date: Fri, 28 Oct 2011 14:16:32 +0200 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: string from array On Fri, Oct 28, 2011 at 10:47 AM, Mark Taylor wrote: > Ivan, it's Christmas. Thanks Santa, I was a good boy this year. > TOPCAT 3.9 (released today) has: > >   - Arrays.join() function to turn an array into a string >   - Multi-line SQL entry field in SQL load dialogue >   - JyStilts runnable from the topcat-*.jar files That's cool indeed. > The Schlegel thing is there but experimental and not documented; > if you want to use it I can tell you how. I personally don't need that at the moment, but as I promised there's strong interest from the user community, even in this thread, so I'd suggest you to briefly describe things on the mailing list, so that folks can start testing. -- With best regards, Ivan From ivan.zolotukhin@gmail.XXX Fri Oct 28 13:44:17 2011 Date: Fri, 28 Oct 2011 14:43:44 +0200 From: Ivan Zolotukhin To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: string from array >   - JyStilts runnable from the topcat-*.jar files It actually takes 2 to 2.5 sec longer to import stilts in jython having topcat-full.jar in CLASSPATH rather then stilts.jar on my laptop. From dabrusco@head.XXX Fri Oct 28 13:46:31 2011 Date: Fri, 28 Oct 2011 08:45:48 -0400 From: Raffaele D'Abrusco To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Discarding multiple tables at once Thanks, great! On Fri, Oct 28, 2011 at 5:20 AM, Mark Taylor wrote: > Raffaele, > > On Tue, 30 Aug 2011, Raffaele D'Abrusco wrote: > > > It seems to me that, right now, is not possible to discard multiple > tables > > at once in Topcat. > > Mark, do you think this feature could be implemented in one of the next > > Topcat releases? > > Done - in v3.9 if you select multiple tables in the list > (ctrl-click on my OS) and then use the File|Discard Table(s) it > will get rid of them all (subject to a confirmation dialogue). > Thanks for the suggestion. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > -- *Raffaele D'Abrusco Harvard-Smithsonian Center for Astrophysics 60 Garden Street, B 306A Cambridge, MA 02138 * *tel: +1 617 495 7201 rdabrusco@cfa.XXX http://hea-www.harvard.edu/~dabrusco other e-mail: **dabrusco@gmail.XXX skype: raffaele_dabrusco* *google profile: dabrusco@gmail.XXX* From dabrusco@head.XXX Fri Oct 28 13:46:45 2011 Date: Fri, 28 Oct 2011 08:46:09 -0400 From: Raffaele D'Abrusco To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Reordering list of columns from the "Table Columns" window? This is even greater! Thanks, Mark. On Fri, Oct 28, 2011 at 5:23 AM, Mark Taylor wrote: > On Wed, 30 Mar 2011, Raffaele D'Abrusco wrote: > > > Hi Mark, > > I don't remember if this point has already been raised before in the > > Topcat mailing list > > (or somewhere else), but I would find very handy to be able to > > re-order the columns of > > a table from the "Table Columns" window instead of from the "Table > > Browser" window. > > This would be particularly useful when dealing with tables containing > > many columns, in which > > case the user is required to drag one column at a time from one side > > to the other of the > > visible portion of the table in the "Table Browser" window, then > > scrolling the table, dragging the > > column again, etc. > > > > Would implementing a similar mechanism in the "Table Columns" window > > require a great effort? > > By adding, for example, an additional column with "handles" (or > > something similar) > > to move the columns vertically in the window? > > this one's done too - in v3.9 you can grab the grey numeric row header > at the left of the row in the columns window and drag it up and down. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > -- *Raffaele D'Abrusco Harvard-Smithsonian Center for Astrophysics 60 Garden Street, B 306A Cambridge, MA 02138 * *tel: +1 617 495 7201 rdabrusco@cfa.XXX http://hea-www.harvard.edu/~dabrusco other e-mail: **dabrusco@gmail.XXX skype: raffaele_dabrusco* *google profile: dabrusco@gmail.XXX* From m.b.taylor@bristol.XXX Fri Oct 28 14:06:21 2011 Date: Fri, 28 Oct 2011 14:06:21 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Schlegel dust extinction etc Ivan, Maciej and others, On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: > On Fri, Oct 28, 2011, Mark Taylor wrote: > > The Schlegel thing is there but experimental and not documented; > > if you want to use it I can tell you how. > > I personally don't need that at the moment, but as I promised there's > strong interest from the user community, even in this thread, so I'd > suggest you to briefly describe things on the mailing list, so that > folks can start testing. This is the story. Following a request from Ivan a while back, I have implemented within STILTS (though currently not TOPCAT) a capability to query the Galactic Dust Reddening and Extinction service hosted at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). This allows you to add columns for E(B-V) Reddening, 100 Micron Emission, and Dust Temperature to tables which already have sky positions - it's a bit like a multi-cone search, but not quite. This means it will be easy for people to fire off lots and lots of queries (one per row) to IPAC, and I don't want to annoy IPAC or their web servers, so I've asked them how they feel about it before publicising this in the interface. If they say it's OK I'll pass it on. In the mean time, I'm wondering if there are similar but different services along the same lines I should provide. It's not feasible to do the calculations within the TOPCAT directly, since the data files are a few tens of Mb per statistic, which is too much to bundle with the application. Are there other web services which provide this sort of information that people would like to see available from TOPCAT/STILTS? Alternatively, there seem to be a number of all-sky maps containing various data available as FITS files out there. Schlegel/Finkbeiner at http://astro.berkeley.edu/~marc/dust/ is one source; LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads of goodies (WMAP, HI, H-alpha, some of the Schlegel data) in HEALPix format. So another possibility would be a function that allowed TOPCAT/STILTS users to download one of these HEALPix files locally and query it by position (value at, or averaged round, a given sky position) to add new columns to a table which has sky positions. Compared to the web service option this would give more flexibility, be considerably faster, and avoid the problem of overloading remote web servers. I quite like this idea because it's efficient and opens up a lot of possibilities, but if the Schlegel extinction data is really the only dataset that people are going to be interested in, then it may not be worth the effort. I am only a humble programmer, and I don't understand the details of the science behind these data sets, and in particular what people would find it useful to be able to access in this way. So, any input on what people would like to see along these lines (including, is Ivan right that lots of people would like to be able to get Schlegel data in this way) is welcomed. Feel free to mail me off-list if you think it's more appropriate (e.g. "me too" comments). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Fri Oct 28 15:39:43 2011 Date: Fri, 28 Oct 2011 15:28:36 +0200 (CEST) From: Maciej Bilicki To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc I don't know what people would want, but I personally would welcome a feature to have Schlegel et al. reddening directly in TOPCAT for the objects I have with defined (l,b) coordinates. I could then do the extinction correction at the level of TOPCAT, not needing to import and export and import etc. my data. Maciek On Fri, 28 Oct 2011, Mark Taylor wrote: > Ivan, Maciej and others, > > On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: > >> On Fri, Oct 28, 2011, Mark Taylor wrote: >>> The Schlegel thing is there but experimental and not documented; >>> if you want to use it I can tell you how. >> >> I personally don't need that at the moment, but as I promised there's >> strong interest from the user community, even in this thread, so I'd >> suggest you to briefly describe things on the mailing list, so that >> folks can start testing. > > This is the story. Following a request from Ivan a while back, > I have implemented within STILTS (though currently not TOPCAT) a capability > to query the Galactic Dust Reddening and Extinction service hosted > at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). > This allows you to add columns for E(B-V) Reddening, > 100 Micron Emission, and Dust Temperature to tables which already > have sky positions - it's a bit like a multi-cone search, but not quite. > > This means it will be easy for people to fire off lots and lots of > queries (one per row) to IPAC, and I don't want to annoy IPAC or > their web servers, so I've asked them how they feel about it > before publicising this in the interface. If they say it's OK > I'll pass it on. > > In the mean time, I'm wondering if there are similar but different > services along the same lines I should provide. It's not feasible > to do the calculations within the TOPCAT directly, > since the data files are a few tens of Mb per statistic, which is > too much to bundle with the application. Are there other web > services which provide this sort of information that people would > like to see available from TOPCAT/STILTS? > > Alternatively, there seem to be a number of all-sky maps containing > various data available as FITS files out there. Schlegel/Finkbeiner > at http://astro.berkeley.edu/~marc/dust/ is one source; > LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads > of goodies (WMAP, HI, H-alpha, some of the Schlegel data) > in HEALPix format. So another possibility would be a function that > allowed TOPCAT/STILTS users to download one of these HEALPix files > locally and query it by position (value at, or averaged round, a > given sky position) to add new columns to a table which has sky > positions. Compared to the web service option this would give > more flexibility, be considerably faster, and avoid the problem > of overloading remote web servers. I quite like this idea > because it's efficient and opens up a lot of possibilities, > but if the Schlegel extinction data is really the only dataset > that people are going to be interested in, then it may not be > worth the effort. > > I am only a humble programmer, and I don't understand the details of > the science behind these data sets, and in particular what people > would find it useful to be able to access in this way. > So, any input on what people would like to see along these lines > (including, is Ivan right that lots of people would like to be > able to get Schlegel data in this way) is welcomed. > Feel free to mail me off-list if you think it's more appropriate > (e.g. "me too" comments). > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > From mara.salvato@ipp.XXX Fri Oct 28 15:35:12 2011 Date: Fri, 28 Oct 2011 15:36:28 +0200 From: Mara Salvato To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc > > I am only a humble programmer, Mark, you have no idea how happy we are for what you are! :) It is definitely useful to have the possibility to query for E(B-V) directly from TOPCAT; going to use it a lot! It sounds a bit exaggerating to ask for more options considering that you just did an upgrade of the system, but is there in a near or far future the idea of having maximum likelhood ratio method as option among the matches? Thanks, Mara From m.b.taylor@bristol.XXX Fri Oct 28 14:41:00 2011 Date: Fri, 28 Oct 2011 14:40:59 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc On Fri, 28 Oct 2011, Mara Salvato wrote: > you have no idea how happy we are for what you are! > :) thanks! > It sounds a bit exaggerating to ask for more options considering that you just did an upgrade of the system, but is there in a near or far future the idea of having maximum likelhood ratio method as option among the matches? maybe - if anyone else is enthusiastic for that, please send me an off-list message (with maybe a bit of background for requirements). -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From j.o.higson@durham.XXX Fri Oct 28 17:49:33 2011 Date: Fri, 28 Oct 2011 16:17:54 +0100 From: HIGSON J.O. To: topcat-user@bristol.XXX Subject: inserting filled column Hi, I am very new to this so if you can help me that would be great. I have managed to load up the various tables I need but I was wondering how to/if it is possible to copy an excel column into a table in topcat? Jack Higson Senior DSU Rep Hatfield College North Bailey Durham DH13RQ j.o.higson@durham.XXX 07928304274 From S.Oliver@sussex.XXX Fri Oct 28 17:59:27 2011 Date: Fri, 28 Oct 2011 16:58:48 +0100 From: Seb Oliver To: HIGSON J.O. Cc: topcat-user@bristol.XXX Subject: Re: inserting filled column inport/export via csv On 28 Oct 2011, at 16:17, HIGSON J.O. wrote: > Hi, > I am very new to this so if you can help me that would be great. > I have managed to load up the various tables I need but I was > wondering how to/if it is possible to copy an excel column into a > table in topcat? > > Jack Higson > Senior DSU Rep > > Hatfield College > North Bailey > Durham > DH13RQ > > j.o.higson@durham.XXX > 07928304274 > > From deil.christoph@googlemail.XXX Fri Oct 28 18:00:19 2011 Date: Fri, 28 Oct 2011 17:59:42 +0200 From: Christoph Deil To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc On Oct 28, 2011, at 3:06 PM, Mark Taylor wrote: > Alternatively, there seem to be a number of all-sky maps containing > various data available as FITS files out there. Schlegel/Finkbeiner > at http://astro.berkeley.edu/~marc/dust/ is one source; > LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads > of goodies (WMAP, HI, H-alpha, some of the Schlegel data) > in HEALPix format. So another possibility would be a function that > allowed TOPCAT/STILTS users to download one of these HEALPix files > locally and query it by position (value at, or averaged round, a > given sky position) to add new columns to a table which has sky > positions. Hi Mark, I frequently have the need to look up values in survey FITS images (also multi-extension FITS images and cubes) for a given catalog. I currently do this with python scripts. but if that functionality were available in TOPCAT and/or STILTS, I would heavily use it. It would make data exploration much more interactive, without having to export from TOPCAT, write and run a script, re-import and then look at the results. Most of the maps I use are CAR or AIR or MER projection, some are in HEALPIX format. Probably it would be a lot of work to implement all the FITS projections in TOPCAT and Aladin / ds9 already has that functionality, so maybe the already existing interop between TOPCAT and the imagers can be extended to implement image lookup? For me as a gamma-ray astronomer, having only the special case of dust extinction maps of a few other pre-defined maps like those from LAMBDA would not be useful. --- While I'm at it, can I add one more feature request: I often have catalogs with 50 or 100 columns, and want to look at the values for one given object, i.e. row. This is very tedious currently in TOPCAT because I have to scroll back and forth a lot. It would be *very* helpful if I could switch rows and columns, either for the whole table or at least look at the values for one row in a separate window. Thanks for the new awesome TOPCAT release, btw! Christoph From bilicki@camk.XXX Fri Oct 28 21:06:48 2011 Date: Fri, 28 Oct 2011 21:05:54 +0200 (CEST) From: Maciej Bilicki To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc Yet another idea. Excuse me if this has already been discussed. Is it possible to implement a prompt asking if the open table(s) should be saved when the X [e.g. in top-right corner] meaning Close is clicked? At the moment it seems that one may lose lots of processed data if they or their kid click(s) it and confirms to shut down TOPCAT. Maciek From m.b.taylor@bristol.XXX Mon Oct 31 11:19:03 2011 Date: Mon, 31 Oct 2011 11:19:03 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc On Fri, 28 Oct 2011, Mark Taylor wrote: > Ivan, Maciej and others, > > On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: > > > On Fri, Oct 28, 2011, Mark Taylor wrote: > > > The Schlegel thing is there but experimental and not documented; > > > if you want to use it I can tell you how. > > > > I personally don't need that at the moment, but as I promised there's > > strong interest from the user community, even in this thread, so I'd > > suggest you to briefly describe things on the mailing list, so that > > folks can start testing. > > This is the story. Following a request from Ivan a while back, > I have implemented within STILTS (though currently not TOPCAT) a capability > to query the Galactic Dust Reddening and Extinction service hosted > at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). > This allows you to add columns for E(B-V) Reddening, > 100 Micron Emission, and Dust Temperature to tables which already > have sky positions - it's a bit like a multi-cone search, but not quite. > > This means it will be easy for people to fire off lots and lots of > queries (one per row) to IPAC, and I don't want to annoy IPAC or > their web servers, so I've asked them how they feel about it > before publicising this in the interface. If they say it's OK > I'll pass it on. I've talked to the nice people at IPAC and they are fine about it, so if you want to use STILTS to add some or all of E(B-V) Reddening, 100 Micron Emission and Dust Temperature, you can do it by using a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". Its usage is (as determined from running it with the wrong arguments) is: % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' Error: No ra/dec specified Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help Usage: uk.ac.starlink.ttools.calc.SchlegelFilter [-results (reddening|emission|temperature)[,...]] [-stats (mean|ref_pixel|std|max|min)[,...]] i.e. you need to give the names of RA and Dec columns from the input table, and you can optionally specify which output columns you want. Simple usage might thus look something like: stilts tpipe in=cat.xml cmd='uk.ac.starlink.ttools.calc.SchlegelFilter RAJ2000 DEJ2000' out=catdust.xml The source of the data is the service documented here: http://irsa.ipac.caltech.edu/applications/DUST/ see instructions there for information on how to acknowledge this servie if it you use it for publication. The filter is not particularly fast (mostly my fault not IPAC's)! I suggest cmd=progress ahead of the cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the command line so you can see how it's getting on. If this feature becomes a documented part of the package I will hopefully speed it up and of course give it a snappier name. > In the mean time, I'm wondering if there are similar but different > services along the same lines I should provide. It's not feasible > to do the calculations within the TOPCAT directly, > since the data files are a few tens of Mb per statistic, which is > too much to bundle with the application. Are there other web > services which provide this sort of information that people would > like to see available from TOPCAT/STILTS? > > Alternatively, there seem to be a number of all-sky maps containing > various data available as FITS files out there. Schlegel/Finkbeiner > at http://astro.berkeley.edu/~marc/dust/ is one source; > LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads > of goodies (WMAP, HI, H-alpha, some of the Schlegel data) > in HEALPix format. So another possibility would be a function that > allowed TOPCAT/STILTS users to download one of these HEALPix files > locally and query it by position (value at, or averaged round, a > given sky position) to add new columns to a table which has sky > positions. Compared to the web service option this would give > more flexibility, be considerably faster, and avoid the problem > of overloading remote web servers. I quite like this idea > because it's efficient and opens up a lot of possibilities, > but if the Schlegel extinction data is really the only dataset > that people are going to be interested in, then it may not be > worth the effort. Following some of the feedback I've had, I will think about something along these lines for the future as well. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Oct 31 11:29:01 2011 Date: Mon, 31 Oct 2011 11:29:01 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc On Fri, 28 Oct 2011, Maciej Bilicki wrote: > Yet another idea. Excuse me if this has already been discussed. > > Is it possible to implement a prompt asking if the open table(s) should be > saved when the X [e.g. in top-right corner] meaning Close is clicked? At the > moment it seems that one may lose lots of processed data if they or their kid > click(s) it and confirms to shut down TOPCAT. Possible, but I'm inclined not to unless a lot of people disagree. I think the existing confirmation dialogue, which means you need at least two clicks to shut the thing down, is sufficient. I could make it a bit safer by setting the default option in the confirmation dialogue to Cancel rather than OK. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Nick.Ball@nrc-cnrc.XXX Mon Oct 31 23:25:16 2011 Date: Mon, 31 Oct 2011 16:24:00 -0700 From: "Ball, Nick" To: "topcat-user@bristol.XXX" Subject: Re: Schlegel dust extinction etc What is an order of magnitude largest reasonable usage of this? E.g., would sending 20 million galaxies be frowned upon? Nick On 2011-10-31, at 4:19 AM, Mark Taylor wrote: On Fri, 28 Oct 2011, Mark Taylor wrote: Ivan, Maciej and others, On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: On Fri, Oct 28, 2011, Mark Taylor > wrote: The Schlegel thing is there but experimental and not documented; if you want to use it I can tell you how. I personally don't need that at the moment, but as I promised there's strong interest from the user community, even in this thread, so I'd suggest you to briefly describe things on the mailing list, so that folks can start testing. This is the story. Following a request from Ivan a while back, I have implemented within STILTS (though currently not TOPCAT) a capability to query the Galactic Dust Reddening and Extinction service hosted at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). This allows you to add columns for E(B-V) Reddening, 100 Micron Emission, and Dust Temperature to tables which already have sky positions - it's a bit like a multi-cone search, but not quite. This means it will be easy for people to fire off lots and lots of queries (one per row) to IPAC, and I don't want to annoy IPAC or their web servers, so I've asked them how they feel about it before publicising this in the interface. If they say it's OK I'll pass it on. I've talked to the nice people at IPAC and they are fine about it, so if you want to use STILTS to add some or all of E(B-V) Reddening, 100 Micron Emission and Dust Temperature, you can do it by using a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". Its usage is (as determined from running it with the wrong arguments) is: % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' Error: No ra/dec specified Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help Usage: uk.ac.starlink.ttools.calc.SchlegelFilter [-results (reddening|emission|temperature)[,...]] [-stats (mean|ref_pixel|std|max|min)[,...]] i.e. you need to give the names of RA and Dec columns from the input table, and you can optionally specify which output columns you want. Simple usage might thus look something like: stilts tpipe in=cat.xml cmd='uk.ac.starlink.ttools.calc.SchlegelFilter RAJ2000 DEJ2000' out=catdust.xml The source of the data is the service documented here: http://irsa.ipac.caltech.edu/applications/DUST/ see instructions there for information on how to acknowledge this servie if it you use it for publication. The filter is not particularly fast (mostly my fault not IPAC's)! I suggest cmd=progress ahead of the cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the command line so you can see how it's getting on. If this feature becomes a documented part of the package I will hopefully speed it up and of course give it a snappier name. In the mean time, I'm wondering if there are similar but different services along the same lines I should provide. It's not feasible to do the calculations within the TOPCAT directly, since the data files are a few tens of Mb per statistic, which is too much to bundle with the application. Are there other web services which provide this sort of information that people would like to see available from TOPCAT/STILTS? Alternatively, there seem to be a number of all-sky maps containing various data available as FITS files out there. Schlegel/Finkbeiner at http://astro.berkeley.edu/~marc/dust/ is one source; LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads of goodies (WMAP, HI, H-alpha, some of the Schlegel data) in HEALPix format. So another possibility would be a function that allowed TOPCAT/STILTS users to download one of these HEALPix files locally and query it by position (value at, or averaged round, a given sky position) to add new columns to a table which has sky positions. Compared to the web service option this would give more flexibility, be considerably faster, and avoid the problem of overloading remote web servers. I quite like this idea because it's efficient and opens up a lot of possibilities, but if the Schlegel extinction data is really the only dataset that people are going to be interested in, then it may not be worth the effort. Following some of the feedback I've had, I will think about something along these lines for the future as well. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ -- Nick Ball Herzberg Institute of Astrophysics 5071 West Saanich Road Victoria, BC V9E 2E7 Canada https://sites.google.com/site/nickballastronomer nick.ball at nrc.cnrc.gc.ca From m.b.taylor@bristol.XXX Tue Nov 1 10:13:42 2011 Date: Tue, 1 Nov 2011 10:13:40 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc Nick, as I said, the current implementation is rather slow; I'm currently seeing of the order of 1 query per second. So, if you have 2e7 galaxies, the fastest way to get the answers is probably to wait until I provide a faster implementation. To answer the question a bit more directly - I should think it's OK to run a query all day or all week if you have the patience, but it will only get you a few hundred thousand results. Setting up 100 of those in parallel would probably be a bit faster, (not sure how much) but is probably bad form. Mark On Mon, 31 Oct 2011, Ball, Nick wrote: > What is an order of magnitude largest reasonable usage of this? E.g., would sending 20 million galaxies be frowned upon? > > Nick > > On 2011-10-31, at 4:19 AM, Mark Taylor wrote: > > On Fri, 28 Oct 2011, Mark Taylor wrote: > > Ivan, Maciej and others, > > On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: > > On Fri, Oct 28, 2011, Mark Taylor > wrote: > The Schlegel thing is there but experimental and not documented; > if you want to use it I can tell you how. > > I personally don't need that at the moment, but as I promised there's > strong interest from the user community, even in this thread, so I'd > suggest you to briefly describe things on the mailing list, so that > folks can start testing. > > This is the story. Following a request from Ivan a while back, > I have implemented within STILTS (though currently not TOPCAT) a capability > to query the Galactic Dust Reddening and Extinction service hosted > at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). > This allows you to add columns for E(B-V) Reddening, > 100 Micron Emission, and Dust Temperature to tables which already > have sky positions - it's a bit like a multi-cone search, but not quite. > > This means it will be easy for people to fire off lots and lots of > queries (one per row) to IPAC, and I don't want to annoy IPAC or > their web servers, so I've asked them how they feel about it > before publicising this in the interface. If they say it's OK > I'll pass it on. > > I've talked to the nice people at IPAC and they are fine about it, > so if you want to use STILTS to add some or all of E(B-V) Reddening, > 100 Micron Emission and Dust Temperature, you can do it by using > a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) > with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". > > Its usage is (as determined from running it with the wrong arguments) is: > > % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' > Error: No ra/dec specified > > Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help > > Usage: uk.ac.starlink.ttools.calc.SchlegelFilter > [-results (reddening|emission|temperature)[,...]] > [-stats (mean|ref_pixel|std|max|min)[,...]] > > > i.e. you need to give the names of RA and Dec columns from the input > table, and you can optionally specify which output columns you want. > Simple usage might thus look something like: > > stilts tpipe in=cat.xml > cmd='uk.ac.starlink.ttools.calc.SchlegelFilter RAJ2000 DEJ2000' > out=catdust.xml > > The source of the data is the service documented here: > > http://irsa.ipac.caltech.edu/applications/DUST/ > > see instructions there for information on how to acknowledge this > servie if it you use it for publication. > > The filter is not particularly fast (mostly my fault not IPAC's)! > I suggest cmd=progress ahead of the > cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the > command line so you can see how it's getting on. > If this feature becomes a documented part of the package I will > hopefully speed it up and of course give it a snappier name. > > In the mean time, I'm wondering if there are similar but different > services along the same lines I should provide. It's not feasible > to do the calculations within the TOPCAT directly, > since the data files are a few tens of Mb per statistic, which is > too much to bundle with the application. Are there other web > services which provide this sort of information that people would > like to see available from TOPCAT/STILTS? > > Alternatively, there seem to be a number of all-sky maps containing > various data available as FITS files out there. Schlegel/Finkbeiner > at http://astro.berkeley.edu/~marc/dust/ is one source; > LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads > of goodies (WMAP, HI, H-alpha, some of the Schlegel data) > in HEALPix format. So another possibility would be a function that > allowed TOPCAT/STILTS users to download one of these HEALPix files > locally and query it by position (value at, or averaged round, a > given sky position) to add new columns to a table which has sky > positions. Compared to the web service option this would give > more flexibility, be considerably faster, and avoid the problem > of overloading remote web servers. I quite like this idea > because it's efficient and opens up a lot of possibilities, > but if the Schlegel extinction data is really the only dataset > that people are going to be interested in, then it may not be > worth the effort. > > Following some of the feedback I've had, I will think about something > along these lines for the future as well. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > -- > Nick Ball > Herzberg Institute of Astrophysics > 5071 West Saanich Road > Victoria, BC V9E 2E7 > Canada > > https://sites.google.com/site/nickballastronomer > nick.ball at nrc.cnrc.gc.ca > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Tue Nov 1 11:29:06 2011 Date: Tue, 1 Nov 2011 11:28:48 +0000 From: Seb Oliver To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc Hi Nick, If you want to do 2e7 galaxies. Why not just down-load the Schelegel maps and read off the values. Schelegel provides IDL code to do this. This could probably run in a few minutes. Or in a VO context you could maybe try and persuade the Aladin people to do it, as I think it would be more appropriate for their app than TOPCAT. Seb. On 1 Nov 2011, at 10:13, Mark Taylor wrote: > Nick, > > as I said, the current implementation is rather slow; I'm currently > seeing of the order of 1 query per second. So, if you have 2e7 > galaxies, the fastest way to get the answers is probably to wait > until I provide a faster implementation. > > To answer the question a bit more directly - I should think it's > OK to run a query all day or all week if you have the patience, > but it will only get you a few hundred thousand results. > Setting up 100 of those in parallel would probably be a bit faster, > (not sure how much) but is probably bad form. > > Mark > > On Mon, 31 Oct 2011, Ball, Nick wrote: > >> What is an order of magnitude largest reasonable usage of this? >> E.g., would sending 20 million galaxies be frowned upon? >> >> Nick >> >> On 2011-10-31, at 4:19 AM, Mark Taylor wrote: >> >> On Fri, 28 Oct 2011, Mark Taylor wrote: >> >> Ivan, Maciej and others, >> >> On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: >> >> On Fri, Oct 28, 2011, Mark Taylor > >> wrote: >> The Schlegel thing is there but experimental and not documented; >> if you want to use it I can tell you how. >> >> I personally don't need that at the moment, but as I promised there's >> strong interest from the user community, even in this thread, so I'd >> suggest you to briefly describe things on the mailing list, so that >> folks can start testing. >> >> This is the story. Following a request from Ivan a while back, >> I have implemented within STILTS (though currently not TOPCAT) a >> capability >> to query the Galactic Dust Reddening and Extinction service hosted >> at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). >> This allows you to add columns for E(B-V) Reddening, >> 100 Micron Emission, and Dust Temperature to tables which already >> have sky positions - it's a bit like a multi-cone search, but not >> quite. >> >> This means it will be easy for people to fire off lots and lots of >> queries (one per row) to IPAC, and I don't want to annoy IPAC or >> their web servers, so I've asked them how they feel about it >> before publicising this in the interface. If they say it's OK >> I'll pass it on. >> >> I've talked to the nice people at IPAC and they are fine about it, >> so if you want to use STILTS to add some or all of E(B-V) Reddening, >> 100 Micron Emission and Dust Temperature, you can do it by using >> a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) >> with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". >> >> Its usage is (as determined from running it with the wrong >> arguments) is: >> >> % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' >> cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' >> Error: No ra/dec specified >> >> Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help >> >> Usage: uk.ac.starlink.ttools.calc.SchlegelFilter >> [-results (reddening|emission|temperature)[,...]] >> [-stats (mean|ref_pixel|std|max|min)[,...]] >> >> >> i.e. you need to give the names of RA and Dec columns from the input >> table, and you can optionally specify which output columns you want. >> Simple usage might thus look something like: >> >> stilts tpipe in=cat.xml >> cmd='uk.ac.starlink.ttools.calc.SchlegelFilter >> RAJ2000 DEJ2000' >> out=catdust.xml >> >> The source of the data is the service documented here: >> >> http://irsa.ipac.caltech.edu/applications/DUST/ >> >> see instructions there for information on how to acknowledge this >> servie if it you use it for publication. >> >> The filter is not particularly fast (mostly my fault not IPAC's)! >> I suggest cmd=progress ahead of the >> cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the >> command line so you can see how it's getting on. >> If this feature becomes a documented part of the package I will >> hopefully speed it up and of course give it a snappier name. >> >> In the mean time, I'm wondering if there are similar but different >> services along the same lines I should provide. It's not feasible >> to do the calculations within the TOPCAT directly, >> since the data files are a few tens of Mb per statistic, which is >> too much to bundle with the application. Are there other web >> services which provide this sort of information that people would >> like to see available from TOPCAT/STILTS? >> >> Alternatively, there seem to be a number of all-sky maps containing >> various data available as FITS files out there. Schlegel/Finkbeiner >> at http://astro.berkeley.edu/~marc/dust/ is one source; >> LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads >> of goodies (WMAP, HI, H-alpha, some of the Schlegel data) >> in HEALPix format. So another possibility would be a function that >> allowed TOPCAT/STILTS users to download one of these HEALPix files >> locally and query it by position (value at, or averaged round, a >> given sky position) to add new columns to a table which has sky >> positions. Compared to the web service option this would give >> more flexibility, be considerably faster, and avoid the problem >> of overloading remote web servers. I quite like this idea >> because it's efficient and opens up a lot of possibilities, >> but if the Schlegel extinction data is really the only dataset >> that people are going to be interested in, then it may not be >> worth the effort. >> >> Following some of the feedback I've had, I will think about something >> along these lines for the future as well. >> >> Mark >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol >> University, UK >> m.b.taylor@bris.XXX >> +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> >> >> -- >> Nick Ball >> Herzberg Institute of Astrophysics >> 5071 West Saanich Road >> Victoria, BC V9E 2E7 >> Canada >> >> https://sites.google.com/site/nickballastronomer >> nick.ball at nrc.cnrc.gc.ca >> >> > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, > UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/ > ~mbt/ > From jbrinchmann@gmail.XXX Tue Nov 1 12:59:44 2011 Date: Tue, 1 Nov 2011 13:59:08 +0100 From: Jarle Brinchmann To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc I think this is indeed the way to go - indeed there are less independent pixels in the SFD maps than 2x10^7. But I think this is a generic issue: Some types of queries are very frequent and there is little point doing them across the net one by one (even thought that might look appealing from the coding point of view). I quite like the idea of having a local cache of HEALpix'd maps that TOPCAT can query locally for those that have a large number of queries to do. This is not only about dust queries - stellar densities, Planck maps etc will all fall in the same category that they are not particularly large datasets but you might need frequent access to small parts of them. The same kind of infrastructure could of course be provided through VO - I think the reason why IPAC doesn't have a table upload facility is simply that they did not have user cases that implied that this would be useful - indeed on their pages they have a phrase: "<...> the table upload option on the form interface (which loops over a list of coordinates to produce all the results in one submit) is not necessary in program mode since programmers can perform this looping themselves." which is eminently true and useless in the case of multi-million queries :) It ought to be very easy for them to change that since the infrastructure is in place. Cheers Jarle. On Tue, Nov 1, 2011 at 12:28 PM, Seb Oliver wrote: > Hi Nick, > > If you want to do 2e7 galaxies.  Why not just down-load the Schelegel maps > and read off the values.  Schelegel provides IDL code to do this.  This > could probably run in a few minutes.  Or in a VO context you could maybe try > and persuade the Aladin people to do it, as I think it would be more > appropriate for their app than TOPCAT. > > Seb. > > On 1 Nov 2011, at 10:13, Mark Taylor wrote: > >> Nick, >> >> as I said, the current implementation is rather slow; I'm currently >> seeing of the order of 1 query per second.  So, if you have 2e7 >> galaxies, the fastest way to get the answers is probably to wait >> until I provide a faster implementation. >> >> To answer the question a bit more directly - I should think it's >> OK to run a query all day or all week if you have the patience, >> but it will only get you a few hundred thousand results. >> Setting up 100 of those in parallel would probably be a bit faster, >> (not sure how much) but is probably bad form. >> >> Mark >> >> On Mon, 31 Oct 2011, Ball, Nick wrote: >> >>> What is an order of magnitude largest reasonable usage of this? E.g., >>> would sending 20 million galaxies be frowned upon? >>> >>> Nick >>> >>> On 2011-10-31, at 4:19 AM, Mark Taylor wrote: >>> >>> On Fri, 28 Oct 2011, Mark Taylor wrote: >>> >>> Ivan, Maciej and others, >>> >>> On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: >>> >>> On Fri, Oct 28, 2011, Mark Taylor >>> > wrote: >>> The Schlegel thing is there but experimental and not documented; >>> if you want to use it I can tell you how. >>> >>> I personally don't need that at the moment, but as I promised there's >>> strong interest from the user community, even in this thread, so I'd >>> suggest you to briefly describe things on the mailing list, so that >>> folks can start testing. >>> >>> This is the story.  Following a request from Ivan a while back, >>> I have implemented within STILTS (though currently not TOPCAT) a >>> capability >>> to query the Galactic Dust Reddening and Extinction service hosted >>> at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). >>> This allows you to add columns for E(B-V) Reddening, >>> 100 Micron Emission, and Dust Temperature to tables which already >>> have sky positions - it's a bit like a multi-cone search, but not quite. >>> >>> This means it will be easy for people to fire off lots and lots of >>> queries (one per row) to IPAC, and I don't want to annoy IPAC or >>> their web servers, so I've asked them how they feel about it >>> before publicising this in the interface.  If they say it's OK >>> I'll pass it on. >>> >>> I've talked to the nice people at IPAC and they are fine about it, >>> so if you want to use STILTS to add some or all of E(B-V) Reddening, >>> 100 Micron Emission and Dust Temperature, you can do it by using >>> a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) >>> with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". >>> >>> Its usage is (as determined from running it with the wrong arguments) is: >>> >>>  % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' >>> cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' >>>  Error: No ra/dec specified >>> >>>  Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help >>> >>>  Usage: uk.ac.starlink.ttools.calc.SchlegelFilter >>>             [-results (reddening|emission|temperature)[,...]] >>>             [-stats (mean|ref_pixel|std|max|min)[,...]] >>>             >>> >>> i.e. you need to give the names of RA and Dec columns from the input >>> table, and you can optionally specify which output columns you want. >>> Simple usage might thus look something like: >>> >>>  stilts tpipe in=cat.xml >>>              cmd='uk.ac.starlink.ttools.calc.SchlegelFilter RAJ2000 >>> DEJ2000' >>>              out=catdust.xml >>> >>> The source of the data is the service documented here: >>> >>>  http://irsa.ipac.caltech.edu/applications/DUST/ >>> >>> see instructions there for information on how to acknowledge this >>> servie if it you use it for publication. >>> >>> The filter is not particularly fast (mostly my fault not IPAC's)! >>> I suggest cmd=progress ahead of the >>> cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the >>> command line so you can see how it's getting on. >>> If this feature becomes a documented part of the package I will >>> hopefully speed it up and of course give it a snappier name. >>> >>> In the mean time, I'm wondering if there are similar but different >>> services along the same lines I should provide.  It's not feasible >>> to do the calculations within the TOPCAT directly, >>> since the data files are a few tens of Mb per statistic, which is >>> too much to bundle with the application.  Are there other web >>> services which provide this sort of information that people would >>> like to see available from TOPCAT/STILTS? >>> >>> Alternatively, there seem to be a number of all-sky maps containing >>> various data available as FITS files out there.  Schlegel/Finkbeiner >>> at http://astro.berkeley.edu/~marc/dust/ is one source; >>> LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads >>> of goodies (WMAP, HI, H-alpha, some of the Schlegel data) >>> in HEALPix format.  So another possibility would be a function that >>> allowed TOPCAT/STILTS users to download one of these HEALPix files >>> locally and query it by position (value at, or averaged round, a >>> given sky position) to add new columns to a table which has sky >>> positions.  Compared to the web service option this would give >>> more flexibility, be considerably faster, and avoid the problem >>> of overloading remote web servers.  I quite like this idea >>> because it's efficient and opens up a lot of possibilities, >>> but if the Schlegel extinction data is really the only dataset >>> that people are going to be interested in, then it may not be >>> worth the effort. >>> >>> Following some of the feedback I've had, I will think about something >>> along these lines for the future as well. >>> >>> Mark >>> >>> -- >>> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK >>> m.b.taylor@bris.XXX +44-117-928-8776 >>> http://www.star.bris.ac.uk/~mbt/ >>> >>> >>> -- >>> Nick Ball >>> Herzberg Institute of Astrophysics >>> 5071 West Saanich Road >>> Victoria, BC  V9E 2E7 >>> Canada >>> >>> https://sites.google.com/site/nickballastronomer >>> nick.ball at nrc.cnrc.gc.ca >>> >>> >> >> -- >> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> > > > From Thomas.A.McGlynn@nasa.XXX Tue Nov 1 13:39:22 2011 Date: Tue, 1 Nov 2011 09:22:34 -0400 From: Tom McGlynn Cc: "topcat-user@bristol.XXX" Subject: Re: Schlegel dust extinction etc It seems like TOPCAT already has the basic framework required to implement the functionality that's being discussed here. Basically you want to be able to interpolate a map at a given RA,Dec point where the RA and Dec are given in the table. There is lots of software out there that does map interpolation -- a fair bit of it in Java. With that in hand one should be able to write a TOPCAT user defined function of the form map_interpolate(file/url, ra, dec) The internal logic would be something like if (first call or file/URL has changed) { read in file/URL } return interpolation(ra,dec) This simple logic would only work for single file image data sets, but that's enough for a lot of what people want to do (including the original case) and could extended to handle higher resolution tiled datasets as needed. I haven't actually build my own functions in TOPCAT, but Mark's description certainly makes it sound easy enough. Regards, Tom McGlyn Jarle Brinchmann wrote: > I think this is indeed the way to go - indeed there are less > independent pixels in the SFD maps than 2x10^7. But I think this is > a generic issue: Some types of queries are very frequent and there is > little point doing them across the net one by one (even thought that > might look appealing from the coding point of view). > > I quite like the idea of having a local cache of HEALpix'd maps that > TOPCAT can query locally for those that have a large number of queries > to do. This is not only about dust queries - stellar densities, Planck > maps etc will all fall in the same category that they are not > particularly large datasets but you might need frequent access to > small parts of them. > > The same kind of infrastructure could of course be provided through VO > - I think the reason why IPAC doesn't have a table upload facility is > simply that they did not have user cases that implied that this would > be useful - indeed on their pages they have a phrase: > > "<...> the table upload option on the form interface (which loops over > a list of coordinates to produce all the results in one submit) is not > necessary in program mode since programmers can perform this looping > themselves." > > which is eminently true and useless in the case of multi-million > queries :) It ought to be very easy for them to change that since the > infrastructure is in place. > > Cheers > Jarle. > > > On Tue, Nov 1, 2011 at 12:28 PM, Seb Oliver wrote: >> Hi Nick, >> >> If you want to do 2e7 galaxies. Why not just down-load the Schelegel maps >> and read off the values. Schelegel provides IDL code to do this. This >> could probably run in a few minutes. Or in a VO context you could maybe try >> and persuade the Aladin people to do it, as I think it would be more >> appropriate for their app than TOPCAT. >> >> Seb. >> >> On 1 Nov 2011, at 10:13, Mark Taylor wrote: >> >>> Nick, >>> >>> as I said, the current implementation is rather slow; I'm currently >>> seeing of the order of 1 query per second. So, if you have 2e7 >>> galaxies, the fastest way to get the answers is probably to wait >>> until I provide a faster implementation. >>> >>> To answer the question a bit more directly - I should think it's >>> OK to run a query all day or all week if you have the patience, >>> but it will only get you a few hundred thousand results. >>> Setting up 100 of those in parallel would probably be a bit faster, >>> (not sure how much) but is probably bad form. >>> >>> Mark >>> >>> On Mon, 31 Oct 2011, Ball, Nick wrote: >>> >>>> What is an order of magnitude largest reasonable usage of this? E.g., >>>> would sending 20 million galaxies be frowned upon? >>>> >>>> Nick >>>> >>>> On 2011-10-31, at 4:19 AM, Mark Taylor wrote: >>>> >>>> On Fri, 28 Oct 2011, Mark Taylor wrote: >>>> >>>> Ivan, Maciej and others, >>>> >>>> On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: >>>> >>>> On Fri, Oct 28, 2011, Mark Taylor >>>> > wrote: >>>> The Schlegel thing is there but experimental and not documented; >>>> if you want to use it I can tell you how. >>>> >>>> I personally don't need that at the moment, but as I promised there's >>>> strong interest from the user community, even in this thread, so I'd >>>> suggest you to briefly describe things on the mailing list, so that >>>> folks can start testing. >>>> >>>> This is the story. Following a request from Ivan a while back, >>>> I have implemented within STILTS (though currently not TOPCAT) a >>>> capability >>>> to query the Galactic Dust Reddening and Extinction service hosted >>>> at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). >>>> This allows you to add columns for E(B-V) Reddening, >>>> 100 Micron Emission, and Dust Temperature to tables which already >>>> have sky positions - it's a bit like a multi-cone search, but not quite. >>>> >>>> This means it will be easy for people to fire off lots and lots of >>>> queries (one per row) to IPAC, and I don't want to annoy IPAC or >>>> their web servers, so I've asked them how they feel about it >>>> before publicising this in the interface. If they say it's OK >>>> I'll pass it on. >>>> >>>> I've talked to the nice people at IPAC and they are fine about it, >>>> so if you want to use STILTS to add some or all of E(B-V) Reddening, >>>> 100 Micron Emission and Dust Temperature, you can do it by using >>>> a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) >>>> with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". >>>> >>>> Its usage is (as determined from running it with the wrong arguments) is: >>>> >>>> % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' >>>> cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' >>>> Error: No ra/dec specified >>>> >>>> Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help >>>> >>>> Usage: uk.ac.starlink.ttools.calc.SchlegelFilter >>>> [-results (reddening|emission|temperature)[,...]] >>>> [-stats (mean|ref_pixel|std|max|min)[,...]] >>>> >>>> >>>> i.e. you need to give the names of RA and Dec columns from the input >>>> table, and you can optionally specify which output columns you want. >>>> Simple usage might thus look something like: >>>> >>>> stilts tpipe in=cat.xml >>>> cmd='uk.ac.starlink.ttools.calc.SchlegelFilter RAJ2000 >>>> DEJ2000' >>>> out=catdust.xml >>>> >>>> The source of the data is the service documented here: >>>> >>>> http://irsa.ipac.caltech.edu/applications/DUST/ >>>> >>>> see instructions there for information on how to acknowledge this >>>> servie if it you use it for publication. >>>> >>>> The filter is not particularly fast (mostly my fault not IPAC's)! >>>> I suggest cmd=progress ahead of the >>>> cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the >>>> command line so you can see how it's getting on. >>>> If this feature becomes a documented part of the package I will >>>> hopefully speed it up and of course give it a snappier name. >>>> >>>> In the mean time, I'm wondering if there are similar but different >>>> services along the same lines I should provide. It's not feasible >>>> to do the calculations within the TOPCAT directly, >>>> since the data files are a few tens of Mb per statistic, which is >>>> too much to bundle with the application. Are there other web >>>> services which provide this sort of information that people would >>>> like to see available from TOPCAT/STILTS? >>>> >>>> Alternatively, there seem to be a number of all-sky maps containing >>>> various data available as FITS files out there. Schlegel/Finkbeiner >>>> at http://astro.berkeley.edu/~marc/dust/ is one source; >>>> LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads >>>> of goodies (WMAP, HI, H-alpha, some of the Schlegel data) >>>> in HEALPix format. So another possibility would be a function that >>>> allowed TOPCAT/STILTS users to download one of these HEALPix files >>>> locally and query it by position (value at, or averaged round, a >>>> given sky position) to add new columns to a table which has sky >>>> positions. Compared to the web service option this would give >>>> more flexibility, be considerably faster, and avoid the problem >>>> of overloading remote web servers. I quite like this idea >>>> because it's efficient and opens up a lot of possibilities, >>>> but if the Schlegel extinction data is really the only dataset >>>> that people are going to be interested in, then it may not be >>>> worth the effort. >>>> >>>> Following some of the feedback I've had, I will think about something >>>> along these lines for the future as well. >>>> >>>> Mark >>>> >>>> -- >>>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>>> m.b.taylor@bris.XXX +44-117-928-8776 >>>> http://www.star.bris.ac.uk/~mbt/ >>>> >>>> >>>> -- >>>> Nick Ball >>>> Herzberg Institute of Astrophysics >>>> 5071 West Saanich Road >>>> Victoria, BC V9E 2E7 >>>> Canada >>>> >>>> https://sites.google.com/site/nickballastronomer >>>> nick.ball at nrc.cnrc.gc.ca >>>> >>>> >>> >>> -- >>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >>> >> >> >> > From Nick.Ball@nrc-cnrc.XXX Tue Nov 1 20:13:49 2011 Date: Tue, 1 Nov 2011 13:12:56 -0700 From: "Ball, Nick" To: Seb Oliver Cc: Mark Taylor , "topcat-user@bristol.XXX" Subject: Re: Schlegel dust extinction etc Hi Seb, Mark, Yeah, this is actually what I do at the moment (well, the C code). But I've only done it once so haven't made it convenient yet, and was curious if Topcat would be better. Sounds like not at the moment, but worth keeping in mind at some level. Thanks for the suggestion, though. Nick On 2011-11-01, at 4:28 AM, Seb Oliver wrote: > Hi Nick, > > If you want to do 2e7 galaxies. Why not just down-load the Schelegel > maps and read off the values. Schelegel provides IDL code to do > this. This could probably run in a few minutes. Or in a VO context > you could maybe try and persuade the Aladin people to do it, as I > think it would be more appropriate for their app than TOPCAT. > > Seb. > > On 1 Nov 2011, at 10:13, Mark Taylor wrote: > >> Nick, >> >> as I said, the current implementation is rather slow; I'm currently >> seeing of the order of 1 query per second. So, if you have 2e7 >> galaxies, the fastest way to get the answers is probably to wait >> until I provide a faster implementation. >> >> To answer the question a bit more directly - I should think it's >> OK to run a query all day or all week if you have the patience, >> but it will only get you a few hundred thousand results. >> Setting up 100 of those in parallel would probably be a bit faster, >> (not sure how much) but is probably bad form. >> >> Mark >> >> On Mon, 31 Oct 2011, Ball, Nick wrote: >> >>> What is an order of magnitude largest reasonable usage of this? >>> E.g., would sending 20 million galaxies be frowned upon? >>> >>> Nick >>> >>> On 2011-10-31, at 4:19 AM, Mark Taylor wrote: >>> >>> On Fri, 28 Oct 2011, Mark Taylor wrote: >>> >>> Ivan, Maciej and others, >>> >>> On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: >>> >>> On Fri, Oct 28, 2011, Mark Taylor >>>> wrote: >>> The Schlegel thing is there but experimental and not documented; >>> if you want to use it I can tell you how. >>> >>> I personally don't need that at the moment, but as I promised there's >>> strong interest from the user community, even in this thread, so I'd >>> suggest you to briefly describe things on the mailing list, so that >>> folks can start testing. >>> >>> This is the story. Following a request from Ivan a while back, >>> I have implemented within STILTS (though currently not TOPCAT) a >>> capability >>> to query the Galactic Dust Reddening and Extinction service hosted >>> at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). >>> This allows you to add columns for E(B-V) Reddening, >>> 100 Micron Emission, and Dust Temperature to tables which already >>> have sky positions - it's a bit like a multi-cone search, but not >>> quite. >>> >>> This means it will be easy for people to fire off lots and lots of >>> queries (one per row) to IPAC, and I don't want to annoy IPAC or >>> their web servers, so I've asked them how they feel about it >>> before publicising this in the interface. If they say it's OK >>> I'll pass it on. >>> >>> I've talked to the nice people at IPAC and they are fine about it, >>> so if you want to use STILTS to add some or all of E(B-V) Reddening, >>> 100 Micron Emission and Dust Temperature, you can do it by using >>> a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) >>> with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". >>> >>> Its usage is (as determined from running it with the wrong >>> arguments) is: >>> >>> % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' >>> cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' >>> Error: No ra/dec specified >>> >>> Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help >>> >>> Usage: uk.ac.starlink.ttools.calc.SchlegelFilter >>> [-results (reddening|emission|temperature)[,...]] >>> [-stats (mean|ref_pixel|std|max|min)[,...]] >>> >>> >>> i.e. you need to give the names of RA and Dec columns from the input >>> table, and you can optionally specify which output columns you want. >>> Simple usage might thus look something like: >>> >>> stilts tpipe in=cat.xml >>> cmd='uk.ac.starlink.ttools.calc.SchlegelFilter >>> RAJ2000 DEJ2000' >>> out=catdust.xml >>> >>> The source of the data is the service documented here: >>> >>> http://irsa.ipac.caltech.edu/applications/DUST/ >>> >>> see instructions there for information on how to acknowledge this >>> servie if it you use it for publication. >>> >>> The filter is not particularly fast (mostly my fault not IPAC's)! >>> I suggest cmd=progress ahead of the >>> cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the >>> command line so you can see how it's getting on. >>> If this feature becomes a documented part of the package I will >>> hopefully speed it up and of course give it a snappier name. >>> >>> In the mean time, I'm wondering if there are similar but different >>> services along the same lines I should provide. It's not feasible >>> to do the calculations within the TOPCAT directly, >>> since the data files are a few tens of Mb per statistic, which is >>> too much to bundle with the application. Are there other web >>> services which provide this sort of information that people would >>> like to see available from TOPCAT/STILTS? >>> >>> Alternatively, there seem to be a number of all-sky maps containing >>> various data available as FITS files out there. Schlegel/Finkbeiner >>> at http://astro.berkeley.edu/~marc/dust/ is one source; >>> LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads >>> of goodies (WMAP, HI, H-alpha, some of the Schlegel data) >>> in HEALPix format. So another possibility would be a function that >>> allowed TOPCAT/STILTS users to download one of these HEALPix files >>> locally and query it by position (value at, or averaged round, a >>> given sky position) to add new columns to a table which has sky >>> positions. Compared to the web service option this would give >>> more flexibility, be considerably faster, and avoid the problem >>> of overloading remote web servers. I quite like this idea >>> because it's efficient and opens up a lot of possibilities, >>> but if the Schlegel extinction data is really the only dataset >>> that people are going to be interested in, then it may not be >>> worth the effort. >>> >>> Following some of the feedback I've had, I will think about something >>> along these lines for the future as well. >>> >>> Mark >>> >>> -- >>> Mark Taylor Astronomical Programmer Physics, Bristol >>> University, UK >>> m.b.taylor@bris.XXX >>> +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >>> >>> >>> -- >>> Nick Ball >>> Herzberg Institute of Astrophysics >>> 5071 West Saanich Road >>> Victoria, BC V9E 2E7 >>> Canada >>> >>> https://sites.google.com/site/nickballastronomer >>> nick.ball at nrc.cnrc.gc.ca >>> >>> >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, >> UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/ >> ~mbt/ >> > From m.b.taylor@bristol.XXX Tue Nov 1 23:15:56 2011 Date: Tue, 1 Nov 2011 23:15:55 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc Thanks to everyone for the useful input on this. I think I will take a look in the relatively near future at adding support for taking samples from locally-held HEALPix, but likely not otherwise arranged, sky maps. Watch this space. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Tue Nov 1 23:25:12 2011 Date: Tue, 1 Nov 2011 23:25:12 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: TOCAT and VOSpace On Thu, 27 Oct 2011, Kavelaars, JJ wrote: > Is there a plan to put a VOSpace download/upload option into TOPCAT? Something like currently exists for MySpace? JJ, I have rather taken my eye off the ball with respect to VOSpace over the last few years. Yes, given the fact that CADC at least are making major use of VOSpace these days, access via TOPCAT would seem to make sense. So, thanks for the reminder, I'll try to take a look at this. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Wed Nov 2 10:41:03 2011 Date: Wed, 2 Nov 2011 10:40:26 +0000 From: Seb Oliver To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc hmm. Not sure the whole astronomical community uses HEALPix.. I never have. If you are going to this effort I am sure that there are java libraries that would allow you to read any recognised WCS from a fits image. Seb. On 1 Nov 2011, at 23:15, Mark Taylor wrote: > Thanks to everyone for the useful input on this. > I think I will take a look in the relatively near future > at adding support for taking samples from locally-held HEALPix, > but likely not otherwise arranged, sky maps. Watch this space. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, > UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/ > ~mbt/ > From m.b.taylor@bristol.XXX Fri Nov 4 22:03:02 2011 Date: Fri, 4 Nov 2011 22:03:00 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc Seb, I appreciate that HEALPix is not a very common general purpose format for image data. However the Schlegel data in particular exists in that format, and it seems popular for the kind of all-sky continuum data set which I'd expect this sort of thing to be most useful for. I wouldn't expect pulling a flux value at a sky position from e.g. a telescope observation image (as a kind of poor man's source extraction) would be that useful, though I admit I've already heard one opinion that disagrees. Also: I believe the implementation would be relatively easy for HEALPix, since I already have the classes to hand. If there are comprehensive FITS-WCS libraries I don't know about them, though I admit I haven't really looked. So I'll investigate HEALPix first, if I get it working and it stimulates demand for other coordinate systems I'll consider that. Mark On Wed, 2 Nov 2011, Seb Oliver wrote: > hmm. Not sure the whole astronomical community uses HEALPix.. I never have. > If you are going to this effort I am sure that there are java libraries that > would allow you to read any recognised WCS from a fits image. > > Seb. > > On 1 Nov 2011, at 23:15, Mark Taylor wrote: > > > Thanks to everyone for the useful input on this. > > I think I will take a look in the relatively near future > > at adding support for taking samples from locally-held HEALPix, > > but likely not otherwise arranged, sky maps. Watch this space. > > > > Mark > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From sebastien.derriere@astro.XXX Sun Nov 6 14:40:03 2011 Date: Sun, 06 Nov 2011 15:39:14 +0100 From: Sebastien Derriere To: topcat-user@mlist2.XXX Subject: Re: Schlegel dust extinction etc On 11/04/2011 11:03 PM, Mark Taylor wrote: > Seb, > > I appreciate that HEALPix is not a very common general purpose > format for image data. However the Schlegel data in particular exists > in that format, and it seems popular for the kind of all-sky > continuum data set which I'd expect this sort of thing to be > most useful for. I wouldn't expect pulling a flux value at a > sky position from e.g. a telescope observation image (as a kind > of poor man's source extraction) would be that useful, though > I admit I've already heard one opinion that disagrees. > > Also: I believe the implementation would be relatively easy for HEALPix, > since I already have the classes to hand. If there are comprehensive > FITS-WCS libraries I don't know about them, though I admit I haven't > really looked. So I'll investigate HEALPix first, if I get it working > and it stimulates demand for other coordinate systems I'll consider > that. Hello Mark, I can only support you on this topic: HEALPix is becoming widely used for all-sky studies. It is the case for Planck for example. We also use it extensively in Aladin for all-sky image surveys, catalogue coverage and density maps, etc. Sebastien -- ((( Sebastien Derriere sebastien.derriere@astro.XXX (. .) Observatoire de Strasbourg Phone +33 (0) 368 852 444 (( v )) 11, rue de l'universite Telefax +33 (0) 368 852 417 ---m-m--- F-67000 Strasbourg France From m.b.taylor@bristol.XXX Fri Nov 11 17:02:58 2011 Date: Fri, 11 Nov 2011 17:02:58 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc On Fri, 28 Oct 2011, Mark Taylor wrote: > On Fri, 28 Oct 2011, Mara Salvato wrote: > > > It sounds a bit exaggerating to ask for more options considering that you just did an upgrade of the system, but is there in a near or far future the idea of having maximum likelhood ratio method as option among the matches? > > maybe - if anyone else is enthusiastic for that, please send me an > off-list message (with maybe a bit of background for requirements). My thanks to the people who have helped me to understand this requirement a bit better. I will put it on the list, but from what I can see it's going to be moderately complicated to make a good job of it, so this functionality will probably not make it into a release in the very near future. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From dvanstone@cfa.XXX Mon Nov 21 17:24:00 2011 Date: Mon, 21 Nov 2011 17:23:33 +0000 From: dvanstone@cfa.XXX To: topcat-user@sympa.XXX Subject: table.load.votable message: name Hello, I am working on a client program (seleste) which allows a user to send the output of a query (a VOTable) to TOPCAT via SAMP. I'm allowing the user to set the name of the table using the SAMP table.load.votable message: samp.params: { name: samplename table-id: sampletable url: [URL of VOtable] } I was expecting TOPCAT to use the name ("samplename") for the table's label. (This behavior worked as I had expected in an earlier version of TOPCAT.) Instead, it looks like TOPCAT is reading the
field from the VOTable. Is this the correct behavior? Thanks, David Van Stone From m.b.taylor@bristol.XXX Mon Nov 21 18:08:16 2011 Date: Mon, 21 Nov 2011 18:08:16 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: table.load.votable message: name Hi David, On Mon, 21 Nov 2011, dvanstone@cfa.XXX wrote: > Hello, > > I am working on a client program (seleste) which allows a user to send the > output of a query (a VOTable) to TOPCAT via SAMP. I'm allowing the user to set > the name of the table using the SAMP table.load.votable message: > > samp.params: { > name: samplename > table-id: sampletable > url: [URL of VOtable] > } > > I was expecting TOPCAT to use the name ("samplename") for the table's label. > (This behavior worked as I had expected in an earlier version of TOPCAT.) > Instead, it looks like TOPCAT is reading the
field from the > VOTable. > > Is this the correct behavior? No it's not, it's a bug, and moreover one that Tom McGlynn has already pointed out to me. This time I've got round to fixing it (top tip: reporting bugs on this list is more likely to shame me into doing something about it than a private mail!) The fix will be in the next release, meantime you can find a version that honours the name parameter correctly at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Thanks for the report, Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bianca.garilli@oamp.XXX Tue Nov 22 13:04:12 2011 Date: Tue, 22 Nov 2011 14:00:01 +0100 From: Bianca Garilli To: topcat-user@bristol.XXX Subject: stilts an jystilts usage Hy Mark, I need to execute stilts on a number of files, changing the value of one constant. Thus I built a shell script like the following stilts tpipe temp1.fits ifmt=fits omode=out ofmt=fits cmd='replacecol OBJ_START OBJ_START-$shift' out=temp2.fits where $shift has been perviously defined in some way (for a simple example, set shift=2.5) but I fell into the c-sh+stilts syntax trap, and find no way to really pass the value of $shift to stilts Error: Bad expression "(OBJ_START-$shift): The name "$shift" is unknown. (The name "$shift" is unknown.) Is there a way to do it? Alternatively, I tried with jython but when trying to import stilts I get the errors message import stilts Traceback (most recent call last): File "", line 1, in File "stilts.py", line 20, in import uk.ac.starlink.table.ColumnInfo as _ColumnInfo ImportError: No module named uk What did I do wrong? Thanks Bianca --------TEMPORARY ADDRESS--------------------------------- Bianca Garilli; Laboratoire d'Astrophysique de Marseille 38, rue Frédéric Joliot-Curie 13388 Marseille cedex 13, France E-mail: bianca.garilli@oamp.XXX phone: 33 (0)491 056912 --------PERMANENT ADRESS------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Tue Nov 22 13:11:21 2011 Date: Tue, 22 Nov 2011 13:11:21 +0000 (GMT) From: Mark Taylor To: Bianca Garilli Cc: topcat-user@bristol.XXX Subject: Re: stilts an jystilts usage On Tue, 22 Nov 2011, Bianca Garilli wrote: > Hy Mark, > I need to execute stilts on a number of files, changing the value of one > constant. Thus I built a shell script like the following > > > stilts tpipe temp1.fits ifmt=fits omode=out ofmt=fits cmd='replacecol > OBJ_START OBJ_START-$shift' out=temp2.fits > > where $shift has been perviously defined in some way > (for a simple example, set shift=2.5) > > but I fell into the c-sh+stilts syntax trap, and find no way to really pass > the value of $shift to stilts > > Error: Bad expression "(OBJ_START-$shift): The name "$shift" is unknown. > (The name "$shift" is unknown.) > > > Is there a way to do it? Yes - use double quotes not single quotes: stilts tpipe temp1.fits ifmt=fits omode=out ofmt=fits cmd="replacecol OBJ_START OBJ_START-$shift" out=temp2.fits In csh (and most other *sh) shell variable references are expanded within double quoted strings but not in single quoted ones. > Alternatively, I tried with jython but when trying to import stilts I get the > errors message Good plan - JyStilts will execute considerably faster in this case because you don't need to restart the JVM for every iteration, and it will be easier to get working if you have more compliated string substitution (etc) requirements in the future. > import stilts > Traceback (most recent call last): > File "", line 1, in > File "stilts.py", line 20, in > import uk.ac.starlink.table.ColumnInfo as _ColumnInfo > ImportError: No module named uk > > > What did I do wrong? don't know - can you tell me (maybe off-list) how you started up JyStilts? Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jbrinchmann@gmail.XXX Mon Nov 28 12:17:51 2011 Date: Mon, 28 Nov 2011 13:02:03 +0100 From: Jarle Brinchmann To: topcat-user@bristol.XXX Subject: Connecting to MySQL via sockets Hi I'm using topcat in teaching and I'd like for the students to be able to connect to mysql running on their own computers. When running mysqld locally you get a local socket and to start up mysql you then normally do something like: mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root However I can't find a way to specify the socket when opening mysql connections in topcat - is it possible somehow? Cheers, Jarle. From m.b.taylor@bristol.XXX Mon Nov 28 12:34:15 2011 Date: Mon, 28 Nov 2011 12:34:15 +0000 (GMT) From: Mark Taylor To: Jarle Brinchmann Cc: topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets Hi Jarle, On Mon, 28 Nov 2011, Jarle Brinchmann wrote: > Hi > > I'm using topcat in teaching and I'd like for the students to be able > to connect to mysql running on their own computers. When running > mysqld locally you get a local socket and to start up mysql you then > normally do something like: > > mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root > > However I can't find a way to specify the socket when opening mysql > connections in topcat - is it possible somehow? I haven't tried this, but from the MySQL Connector/J documentation (http://dev.mysql.com/doc/refman/5.5/en/connector-j-reference-configuration-properties.html) it looks like you should be able to specify the port number after a colon following the hostname. So entering "localhost:1001" rather than "localhost" in the SQL Query dialogue (http://www.starlink.ac.uk/topcat/sun253/SQLTableLoadDialog.html) to use, e.g., port 1001 ought to do it. You can do something similar if you're using a JDBC URL from the command line. Note that if you're using that dialogue you're advised to use the most recent TOPCAT version (v3.9), since in previous versions you only get a tiny one-line entry field for the SQL. Let me know if you do/don't get it working. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Mon Nov 28 13:25:34 2011 Date: Mon, 28 Nov 2011 14:24:57 +0100 From: Ivan Zolotukhin To: Jarle Brinchmann Cc: topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets Hi Jarle, I'm not an expert in Java, but Google says there's no unix domain sockets support in JDBC, so you'd better launch your MySQL server to accept TCP/IP connections from localhost. Otherwise it is probably possible to plug http://code.google.com/p/junixsocket/ into MySQL JDBC driver somehow. On Mon, Nov 28, 2011 at 1:02 PM, Jarle Brinchmann wrote: > Hi > > I'm using topcat in teaching and I'd like for the students to be able > to connect to mysql running on their own computers. When running > mysqld locally you get a local socket and to start up mysql you then > normally do something like: > > mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root > > However I can't find a way to specify the socket when opening mysql > connections in topcat - is it possible somehow? > >   Cheers, >        Jarle. > > From m.b.taylor@bristol.XXX Mon Nov 28 13:30:14 2011 Date: Mon, 28 Nov 2011 13:30:14 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets Er thanks Ivan for actually answering the question Jarle asked rather than answering a question a bit like it that I thought he'd asked. Jarle: please ignore my message and listen to Ivan instead. On Mon, 28 Nov 2011, Ivan Zolotukhin wrote: > Hi Jarle, > > I'm not an expert in Java, but Google says there's no unix domain > sockets support in JDBC, so you'd better launch your MySQL server to > accept TCP/IP connections from localhost. Otherwise it is probably > possible to plug http://code.google.com/p/junixsocket/ into MySQL JDBC > driver somehow. > > On Mon, Nov 28, 2011 at 1:02 PM, Jarle Brinchmann wrote: > > Hi > > > > I'm using topcat in teaching and I'd like for the students to be able > > to connect to mysql running on their own computers. When running > > mysqld locally you get a local socket and to start up mysql you then > > normally do something like: > > > > mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root > > > > However I can't find a way to specify the socket when opening mysql > > connections in topcat - is it possible somehow? > > > >   Cheers, > >        Jarle. > > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jbrinchmann@gmail.XXX Mon Nov 28 16:37:30 2011 Date: Mon, 28 Nov 2011 17:37:01 +0100 From: Jarle Brinchmann To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets It seems to work fine actually. Well it does under linux at least. I tried in my Mac and for some reason it did not work under OS X - but the complaint turned out, after reading a bit more carefully, to be that it did not allow me to connect with an empty password (something that _does_ work under Linux). So the summary is: socket access is fine, no problem, but for some reason mysql refuses connections when providing empty passwords on Mac OS X (defining a password makes that problem go away, so it is not a real problem). Ah well, sorry for that - I should have tested on some more machines! Cheers, Jarle. On Mon, Nov 28, 2011 at 1:34 PM, Mark Taylor wrote: > Hi Jarle, > > On Mon, 28 Nov 2011, Jarle Brinchmann wrote: > >> Hi >> >> I'm using topcat in teaching and I'd like for the students to be able >> to connect to mysql running on their own computers. When running >> mysqld locally you get a local socket and to start up mysql you then >> normally do something like: >> >> mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root >> >> However I can't find a way to specify the socket when opening mysql >> connections in topcat - is it possible somehow? > > I haven't tried this, but from the MySQL Connector/J documentation > (http://dev.mysql.com/doc/refman/5.5/en/connector-j-reference-configuration-properties.html) > it looks like you should be able to specify the port number after a > colon following the hostname. > So entering "localhost:1001" rather than "localhost" in the SQL Query dialogue > (http://www.starlink.ac.uk/topcat/sun253/SQLTableLoadDialog.html) > to use, e.g., port 1001 ought to do it. > You can do something similar if you're using a JDBC URL from the > command line. > > Note that if you're using that dialogue you're advised to use > the most recent TOPCAT version (v3.9), since in previous > versions you only get a tiny one-line entry field for the SQL. > > Let me know if you do/don't get it working. > > Mark > > -- > Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > From jbrinchmann@gmail.XXX Mon Nov 28 16:48:20 2011 Date: Mon, 28 Nov 2011 17:47:20 +0100 From: Jarle Brinchmann To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets Just in case someone would like to do the same, here is what does work for me: Do once - define a local mysql database. > mysql_install_db --datadir=/home/brinchmann/MySQL/data --defaults-file=/home/brinchmann/MySQL/my.cnf where my.cnf contains: # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 [mysqld_safe] datadir=/home/brinchmann/MySQL/data socket=/home/brinchmann/MySQL/mysql.sock log-error=/home/brinchmann/MySQL/mysqld.log pid-file=/home/brinchmann/MySQL/mysqld.pid I then launch mysqld: > mysqld_safe --defaults-file=/home/brinchmann/my.cnf & I create a database in mysql (not sure I could do that with topcat): > mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root mysql> create database Test1; Query OK, 1 row affected (1.63 sec) On the Mac I also needed to set a password (well, I defined a new user account with a password and suitable access privileges), but on the linux machine this seems unnecessary. Then I followed Mark's instructions in the topcat document to download the connector and start topcat as: > java -classpath topcat-full-3.9.jar:mysql-connector-java-5.1.18-bin.jar -Djdbc.drivers=com.mysql.jdbc.Driver uk.ac.starlink.topcat.Driver Now it seems to all work very nicely. It seems it is unnecessary to do anything further which is a bit surprising. This is of course totally useless for production systems, but it is quite practical for students to play around with as you do not have to worry (much) about security issues and they get to be root on their databases. Cheers, J. From ivan.zolotukhin@gmail.XXX Mon Nov 28 17:15:34 2011 Date: Mon, 28 Nov 2011 18:14:48 +0100 From: Ivan Zolotukhin To: Jarle Brinchmann Cc: Mark Taylor , topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets Please also share the connection strings you use in TOPCAT to complete the example. On Mon, Nov 28, 2011 at 5:47 PM, Jarle Brinchmann wrote: > Just in case someone would like to do the same, here is what does work for me: > > Do once - define a local mysql database. >> mysql_install_db --datadir=/home/brinchmann/MySQL/data  --defaults-file=/home/brinchmann/MySQL/my.cnf > > where my.cnf contains: > > # Disabling symbolic-links is recommended to prevent assorted security risks > symbolic-links=0 > [mysqld_safe] > datadir=/home/brinchmann/MySQL/data > socket=/home/brinchmann/MySQL/mysql.sock > log-error=/home/brinchmann/MySQL/mysqld.log > pid-file=/home/brinchmann/MySQL/mysqld.pid > > I then launch mysqld: > >> mysqld_safe --defaults-file=/home/brinchmann/my.cnf & > > I create a database in mysql (not sure I could do that with topcat): > >> mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root > mysql> create database Test1; > Query OK, 1 row affected (1.63 sec) > > On the Mac I also needed to set a password (well, I defined a new user > account with a password and suitable access privileges), but on the > linux machine this seems unnecessary. > > >  Then I followed Mark's instructions in the topcat document to > download the connector and start topcat as: > >> java -classpath topcat-full-3.9.jar:mysql-connector-java-5.1.18-bin.jar -Djdbc.drivers=com.mysql.jdbc.Driver uk.ac.starlink.topcat.Driver > > Now it seems to all work very nicely. It seems it is unnecessary to do > anything further which is a bit surprising. > > This is of course totally useless for production systems, but it is > quite practical for students to play around with as you do not have to > worry (much) about security issues and they get to be root on their > databases. > >       Cheers, >          J. > > From jbrinchmann@gmail.XXX Mon Nov 28 18:37:10 2011 Date: Mon, 28 Nov 2011 19:36:54 +0100 From: Jarle Brinchmann To: Ivan Zolotukhin Cc: Mark Taylor , topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets Good point, but I didn't do anything special. The attached images show the windows for load and save just before submitting a successful query on Fedora. The settings were then: protocol: mysql, host: localhost, then the rest are database name and user name. Cheers, Jarle. On Mon, Nov 28, 2011 at 6:14 PM, Ivan Zolotukhin wrote: > Please also share the connection strings you use in TOPCAT to complete > the example. > > On Mon, Nov 28, 2011 at 5:47 PM, Jarle Brinchmann wrote: >> Just in case someone would like to do the same, here is what does work for me: >> >> Do once - define a local mysql database. >>> mysql_install_db --datadir=/home/brinchmann/MySQL/data  --defaults-file=/home/brinchmann/MySQL/my.cnf >> >> where my.cnf contains: >> >> # Disabling symbolic-links is recommended to prevent assorted security risks >> symbolic-links=0 >> [mysqld_safe] >> datadir=/home/brinchmann/MySQL/data >> socket=/home/brinchmann/MySQL/mysql.sock >> log-error=/home/brinchmann/MySQL/mysqld.log >> pid-file=/home/brinchmann/MySQL/mysqld.pid >> >> I then launch mysqld: >> >>> mysqld_safe --defaults-file=/home/brinchmann/my.cnf & >> >> I create a database in mysql (not sure I could do that with topcat): >> >>> mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root >> mysql> create database Test1; >> Query OK, 1 row affected (1.63 sec) >> >> On the Mac I also needed to set a password (well, I defined a new user >> account with a password and suitable access privileges), but on the >> linux machine this seems unnecessary. >> >> >>  Then I followed Mark's instructions in the topcat document to >> download the connector and start topcat as: >> >>> java -classpath topcat-full-3.9.jar:mysql-connector-java-5.1.18-bin.jar -Djdbc.drivers=com.mysql.jdbc.Driver uk.ac.starlink.topcat.Driver >> >> Now it seems to all work very nicely. It seems it is unnecessary to do >> anything further which is a bit surprising. >> >> This is of course totally useless for production systems, but it is >> quite practical for students to play around with as you do not have to >> worry (much) about security issues and they get to be root on their >> databases. >> >>       Cheers, >>          J. >> >> > [ Part 2: "Attached Text" ] [ The following attachment was DELETED when this message was saved: ] [ A Image/PNG (Name="Screenshot1.png") segment of about 14,991 bytes. ] [ Part 3: "Attached Text" ] [ The following attachment was DELETED when this message was saved: ] [ A Image/PNG (Name="Screenshot2.png") segment of about 15,024 bytes. ] From jbrinchmann@gmail.XXX Mon Nov 28 18:44:32 2011 Date: Mon, 28 Nov 2011 19:44:00 +0100 From: Jarle Brinchmann To: topcat-user@bristol.XXX Subject: Char columns in tables - mysql woes & how to convert Hi, To continue my pesky mysql usage, I have one problem that crops up when I try to save some tables to mysql (as a specific example J_A+A_512_A63_m33h2 from Vizier). When I try to save this to mysql, I get: Error saving table to M33Abundances Data truncation: Data too long for column 'n_Seq' at row 1 Normally saving works without a hitch. What sets this table off is that the 'n_Seq' column has Class 'Character'. I do not know how this is mapped to mysql types but it seems not to work well. Then I thought I'd just change the type of the column or create a new column with format String, but I must admit that I do not know how to do that - all the string functions seem to refuse working on Character columns and the conversion functions too. I am probably overlooking something for both problems - if you have any suggestions it would be very welcome! Cheers, Jarle. From ivan.zolotukhin@gmail.XXX Mon Nov 28 18:57:52 2011 Date: Mon, 28 Nov 2011 19:57:35 +0100 From: Ivan Zolotukhin To: Jarle Brinchmann Cc: Mark Taylor , topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets Sorry for being nerd, but you're doing your connection through TCP/IP, not through Unix domain socket, looking at your connection string. Anyhow, nice that it works. On Mon, Nov 28, 2011 at 7:36 PM, Jarle Brinchmann wrote: > Good point, but I didn't do anything special. > > The attached images show the windows for load and save just before > submitting a successful query on Fedora. The settings were then: > protocol: mysql, host: localhost, then the rest are database name and > user name. > >      Cheers, >         Jarle. > > > > > On Mon, Nov 28, 2011 at 6:14 PM, Ivan Zolotukhin > wrote: >> Please also share the connection strings you use in TOPCAT to complete >> the example. >> >> On Mon, Nov 28, 2011 at 5:47 PM, Jarle Brinchmann wrote: >>> Just in case someone would like to do the same, here is what does work for me: >>> >>> Do once - define a local mysql database. >>>> mysql_install_db --datadir=/home/brinchmann/MySQL/data  --defaults-file=/home/brinchmann/MySQL/my.cnf >>> >>> where my.cnf contains: >>> >>> # Disabling symbolic-links is recommended to prevent assorted security risks >>> symbolic-links=0 >>> [mysqld_safe] >>> datadir=/home/brinchmann/MySQL/data >>> socket=/home/brinchmann/MySQL/mysql.sock >>> log-error=/home/brinchmann/MySQL/mysqld.log >>> pid-file=/home/brinchmann/MySQL/mysqld.pid >>> >>> I then launch mysqld: >>> >>>> mysqld_safe --defaults-file=/home/brinchmann/my.cnf & >>> >>> I create a database in mysql (not sure I could do that with topcat): >>> >>>> mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root >>> mysql> create database Test1; >>> Query OK, 1 row affected (1.63 sec) >>> >>> On the Mac I also needed to set a password (well, I defined a new user >>> account with a password and suitable access privileges), but on the >>> linux machine this seems unnecessary. >>> >>> >>>  Then I followed Mark's instructions in the topcat document to >>> download the connector and start topcat as: >>> >>>> java -classpath topcat-full-3.9.jar:mysql-connector-java-5.1.18-bin.jar -Djdbc.drivers=com.mysql.jdbc.Driver uk.ac.starlink.topcat.Driver >>> >>> Now it seems to all work very nicely. It seems it is unnecessary to do >>> anything further which is a bit surprising. >>> >>> This is of course totally useless for production systems, but it is >>> quite practical for students to play around with as you do not have to >>> worry (much) about security issues and they get to be root on their >>> databases. >>> >>>       Cheers, >>>          J. >>> >>> >> > From jbrinchmann@gmail.XXX Mon Nov 28 22:34:59 2011 Date: Mon, 28 Nov 2011 23:34:39 +0100 From: Jarle Brinchmann To: Ivan Zolotukhin Cc: Mark Taylor , topcat-user@bristol.XXX Subject: Re: Connecting to MySQL via sockets Yes, that is correct - my apologies for not being clear about that. I had not realized that that even was a possibility before you pointed it out in your earlier email. But indeed, it was quite clear that since it worked without me specifying the socket it had to be using TCP/IP, I just forgot to note that in my reply. Cheers, Jarle. On Mon, Nov 28, 2011 at 7:57 PM, Ivan Zolotukhin wrote: > Sorry for being nerd, but you're doing your connection through TCP/IP, > not through Unix domain socket, looking at your connection string. > Anyhow, nice that it works. > > On Mon, Nov 28, 2011 at 7:36 PM, Jarle Brinchmann wrote: >> Good point, but I didn't do anything special. >> >> The attached images show the windows for load and save just before >> submitting a successful query on Fedora. The settings were then: >> protocol: mysql, host: localhost, then the rest are database name and >> user name. >> >>      Cheers, >>         Jarle. >> >> >> >> >> On Mon, Nov 28, 2011 at 6:14 PM, Ivan Zolotukhin >> wrote: >>> Please also share the connection strings you use in TOPCAT to complete >>> the example. >>> >>> On Mon, Nov 28, 2011 at 5:47 PM, Jarle Brinchmann wrote: >>>> Just in case someone would like to do the same, here is what does work for me: >>>> >>>> Do once - define a local mysql database. >>>>> mysql_install_db --datadir=/home/brinchmann/MySQL/data  --defaults-file=/home/brinchmann/MySQL/my.cnf >>>> >>>> where my.cnf contains: >>>> >>>> # Disabling symbolic-links is recommended to prevent assorted security risks >>>> symbolic-links=0 >>>> [mysqld_safe] >>>> datadir=/home/brinchmann/MySQL/data >>>> socket=/home/brinchmann/MySQL/mysql.sock >>>> log-error=/home/brinchmann/MySQL/mysqld.log >>>> pid-file=/home/brinchmann/MySQL/mysqld.pid >>>> >>>> I then launch mysqld: >>>> >>>>> mysqld_safe --defaults-file=/home/brinchmann/my.cnf & >>>> >>>> I create a database in mysql (not sure I could do that with topcat): >>>> >>>>> mysql --socket=/home/brinchmann/MySQL/mysql.sock -u root >>>> mysql> create database Test1; >>>> Query OK, 1 row affected (1.63 sec) >>>> >>>> On the Mac I also needed to set a password (well, I defined a new user >>>> account with a password and suitable access privileges), but on the >>>> linux machine this seems unnecessary. >>>> >>>> >>>>  Then I followed Mark's instructions in the topcat document to >>>> download the connector and start topcat as: >>>> >>>>> java -classpath topcat-full-3.9.jar:mysql-connector-java-5.1.18-bin.jar -Djdbc.drivers=com.mysql.jdbc.Driver uk.ac.starlink.topcat.Driver >>>> >>>> Now it seems to all work very nicely. It seems it is unnecessary to do >>>> anything further which is a bit surprising. >>>> >>>> This is of course totally useless for production systems, but it is >>>> quite practical for students to play around with as you do not have to >>>> worry (much) about security issues and they get to be root on their >>>> databases. >>>> >>>>       Cheers, >>>>          J. >>>> >>>> >>> >> > From m.b.taylor@bristol.XXX Tue Nov 29 10:43:14 2011 Date: Tue, 29 Nov 2011 10:43:14 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Char columns in tables - mysql woes & how to convert On Mon, 28 Nov 2011, Jarle Brinchmann wrote: > Hi, > > To continue my pesky mysql usage, I have one problem that crops up > when I try to save some tables to mysql (as a specific example > J_A+A_512_A63_m33h2 from Vizier). When I try to save this to mysql, I > get: > > Error saving table to M33Abundances > Data truncation: Data too long for column 'n_Seq' at row 1 Bug. I've made a fix which appears to get it working, for that table in MySQL at least. I don't *think* this fix will have broken anything else, but JDBC adjustments are always a bit hairy because there are lots of ways it can go wrong, and lots of different databases which I don't test with. If you'd like to try it out, grab a prerelease from: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ if you see any (especially, but not only, new) unexpected behaviour with this version in relation to writing SQL tables, please let me know. > Normally saving works without a hitch. What sets this table off is > that the 'n_Seq' column has Class 'Character'. I do not know how this > is mapped to mysql types but it seems not to work well. Then I thought > I'd just change the type of the column or create a new column with > format String, but I must admit that I do not know how to do that - > all the string functions seem to refuse working on Character columns > and the conversion functions too. I am probably overlooking something The not-very-obvious solution to this (which as you say is a way you can do the write with the existing release) is to make a new column using the expression: "" + n_Seq (or, you could just hide that column before the save if you don't need it). Thanks for the report, Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From drodrigu@das.XXX Tue Nov 29 17:31:40 2011 Date: Tue, 29 Nov 2011 14:17:08 -0300 From: David Rodriguez To: topcat-user@bristol.XXX Subject: If..then..else in topcat? Greetings, I've been using Topcat for a some time now, but recently ran into a problem. I have two functions I've fit to my data, but neither applies to the whole range. Basically if x=a function 2 applies. I want to have a column with the final result, but I'm not sure how to do so. I was looking for something like if X then Y else Z, but didn't see anything in the function list. I can think of a way to do this using a script and stilts, but perhaps this already exists within topcat. I know broken power laws commonly appear in papers so I wonder if this is a worthwhile addition. Any thoughts? David R. ----------------------------------------------- David R. Rodriguez Department of Astronomy Universidad de Chile drodrigu@ das.uchile.cl http://www.das.uchile.cl/~drodrigu ----------------------------------------------- From fap@head.XXX Tue Nov 29 17:31:53 2011 Date: Tue, 29 Nov 2011 12:31:12 -0500 From: Frank Primini To: David Rodriguez Cc: topcat-user@bristol.XXX Subject: Re: If..then..else in topcat? Try the conditional switch "?", i.e., new column=x Greetings, > > I've been using Topcat for a some time now, but recently ran into a problem. > I have two functions I've fit to my data, but neither applies to the whole range. Basically if x=a function 2 applies. > I want to have a column with the final result, but I'm not sure how to do so. I was looking for something like if X then Y else Z, but didn't see anything in the function list. > I can think of a way to do this using a script and stilts, but perhaps this already exists within topcat. I know broken power laws commonly appear in papers so I wonder if this is a worthwhile addition. > > Any thoughts? > > David R. > > ----------------------------------------------- > David R. Rodriguez > Department of Astronomy > Universidad de Chile > drodrigu@das.XXX > http://www.das.uchile.cl/~drodrigu > ----------------------------------------------- > From drodrigu@das.XXX Tue Nov 29 17:49:15 2011 Date: Tue, 29 Nov 2011 14:36:32 -0300 From: David Rodriguez To: Frank Primini Cc: topcat-user@bristol.XXX Subject: Re: If..then..else in topcat? Excellent, that works perfectly: x>0? c1:c2 where x, c1, and c2 are column names. I had a feeling it was something really simple. Thanks, David R. On Tue, Nov 29, 2011 at 2:31 PM, Frank Primini wrote: > Try the conditional switch "?", i.e., > > new column=x > I believe that's the right syntax, but you should double-check. > > Cheers, > > Frank > > On Nov 29, 2011, at 12:17 PM, David Rodriguez wrote: > > > Greetings, > > > > I've been using Topcat for a some time now, but recently ran into a > problem. > > I have two functions I've fit to my data, but neither applies to the > whole range. Basically if x=a function 2 > applies. > > I want to have a column with the final result, but I'm not sure how to > do so. I was looking for something like if X then Y else Z, but didn't see > anything in the function list. > > I can think of a way to do this using a script and stilts, but perhaps > this already exists within topcat. I know broken power laws commonly appear > in papers so I wonder if this is a worthwhile addition. > > > > Any thoughts? > > > > David R. > > > > ----------------------------------------------- > > David R. Rodriguez > > Department of Astronomy > > Universidad de Chile > > drodrigu@das.XXX > > http://www.das.uchile.cl/~drodrigu > > ----------------------------------------------- > > > > From m.b.taylor@bristol.XXX Thu Dec 8 14:42:44 2011 Date: Thu, 8 Dec 2011 14:42:43 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc Hi all, On Tue, 1 Nov 2011, Mark Taylor wrote: > Thanks to everyone for the useful input on this. > I think I will take a look in the relatively near future > at adding support for taking samples from locally-held HEALPix, > but likely not otherwise arranged, sky maps. Watch this space. I have now implemented this as a new, somewhat experimental, STILTS command "pixsample". You can grab a pre-release copy from ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ with documentation including examples at ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/sun256/pixsample.html This will work for at least some of the Schlegel data, pixel files available from: http://lambda.gsfc.nasa.gov/product/foreground/f_products.cfm Once the pixel file is downloaded, the speed is no longer a concern, unlike the other Schlegel filter I advertised a while back (it now runs at a few 10^5 samples per second). (Actually the same functionality is available as a filter "addpixsample" as well, but I think this is less appropriate and I may withdraw it unless anyone thinks it's useful). I'd be grateful for any feedback, including whether the documentation adquately explains what it's for and how to use it, whether I should be pointing to other sources of HEALPix files in the docs, whether other statistical sampling modes (e.g. min/max/stdev) or some other configurability options would be useful etc. I don't have immediate plans to extend this to sampling from non-HEALPix image sources (because it would be harder to do) but I'll consider it for the future if there's much demand. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Fri Dec 9 16:50:46 2011 Date: Fri, 9 Dec 2011 16:50:46 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Addressing tables in Multi-extension FITS by EXTNAME I've just made a minor update following the suggestion made a while back by Bianca Garilli: you can now address tables in Multi-Extension FITS files by name (value of the EXTNAME header) rather than by HDU index if you prefer, e.g. data.fits#UV_DATA rather than data.fits#5 Pre-release versions in the usual places: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ Thanks Bianca for the suggestion. Mark On Thu, 1 Apr 2010, Mark Taylor wrote: > On Thu, 1 Apr 2010, Bianca Garilli wrote: > > > Hi Mark, I just tried your usefull suggestion given to Jorge, > > but it does not work exactly as you said > > I have a fits file with an empty primary header, and 2 binary tables > > in it. I load it into topcat as FITS, but the only table I see is the first > > one. Still, using ftools, I correctly see the 2 tables in the file (i.e. they > > ARE there) > > I'm using topcat 3.4-1, might that be the reason? > > yes, multiple table load has only recently been introduced. > You will need TOPCAT v3.5 or later for this to work. > > > Also, you might consider to use the EXTNAME k/w in the fits header > > to name tables in topcat. I'm not sure it is a compulsory k/w, but > > IF it is present, it can be used > > Yes, that looks appropriate, thanks for the suggestion. > I will see about using EXTNAME header cards as table names in future > releases. > > Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From fran@cab.XXX Wed Dec 14 11:06:28 2011 Date: Wed, 14 Dec 2011 12:03:45 +0100 From: fran To: topcat-user@bristol.XXX Subject: Multiple Cone Search Hi, we have been using the "multiple cone search" for cross-matching a list of sources with Tycho catalogue (http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&). And we have detected that for one object (there may be more), Topcat does not find the Tycho counterpart, even if it is very close to our coordinates. I attach a table with the query coordinates of this object. For the cross-match we have used 1arcsec. Curiously, if one increases the match radios to 1.1arcsec, Topcat finds the counterpart, with an angular separation between query position and result position of only 3.86264261101561E-6 degrees = 0.014 arcsec!!. So why did not Topcat find the counterpart before? What was wrong? Regards, Francisco. -- +++++++++++++++++++++++++++++++++ Francisco Jiménez-Esteban Spanish Virtual Observatory Centro de Astrobiología (INTA-CSIC) LAEFF Campus. POB 78 28691 Villanueva de la Cañada Madrid, Spain Tel: +34 - 91 813 11 93 Fax: +34 - 91 813 11 60 +++++++++++++++++++++++++++++++++ [ Part 2: "Attached Text" ]
114189_0.5 346.86965 21.13425
From m.b.taylor@bristol.XXX Wed Dec 14 11:40:50 2011 Date: Wed, 14 Dec 2011 11:40:50 +0000 (GMT) From: Mark Taylor To: fran Cc: topcat-user@bristol.XXX Subject: Re: Multiple Cone Search On Wed, 14 Dec 2011, fran wrote: > Hi, > > we have been using the "multiple cone search" for cross-matching a list of > sources with Tycho catalogue > (http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&). And we have > detected that for one object (there may be more), Topcat does not find the > Tycho counterpart, even if it is very close to our coordinates. I attach a > table with the query coordinates of this object. For the cross-match we have > used 1arcsec. > > Curiously, if one increases the match radios to 1.1arcsec, Topcat finds the > counterpart, with an angular separation between query position and result > position of only 3.86264261101561E-6 degrees = 0.014 arcsec!!. So why did not > Topcat find the counterpart before? What was wrong? It's an issue with the service not the client I think. If I issue the following Cone Search queries to VizieR: http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&RA=346.86965&DEC=21.13425&SR=0.00031 http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&RA=346.86965&DEC=21.13425&SR=0.00027 the first one returns a result within 3.9e-6 degrees and the second one returns no result. It may be to do with slightly different coordinate systems used for the matching and position reporting, but I believe that TOPCAT is doing the right thing here (submitted position and returned columns marked POS_EQ_{RA,DEC}_MAIN in ICRS). Is somebody from VizieR listening and able to comment? Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From thomas.boch@astro.XXX Wed Dec 14 13:44:46 2011 Date: Wed, 14 Dec 2011 14:44:27 +0100 From: Thomas Boch To: Mark Taylor Cc: fran , topcat-user@bristol.XXX Landais Gilles , Francois Ochsenbein Subject: Re: Multiple Cone Search Hi, I think I understood what is going wrong in VizieR. It looks for counterparts taking into account the original columns (RA(ICRS) and DE(ICRS)) in epoch J1991.25 whereas in the output, the columns marked POS_EQ_{RA,DEC}_MAIN have been computed in the epoch J2000.0, taking into account the proper motions. This explains the issue I think. I have put Gilles and François in CC, so that they can advertise this list when the problem is fixed. Shameless plug : in the meantime, you can use the CDS cross-match service ( http://cdsxmatch.u-strasbg.fr/ ) which is not affected by this problem Cheers, and thank you for finding this issue, Thomas Mark Taylor wrote: > On Wed, 14 Dec 2011, fran wrote: > > >> Hi, >> >> we have been using the "multiple cone search" for cross-matching a list of >> sources with Tycho catalogue >> (http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&). And we have >> detected that for one object (there may be more), Topcat does not find the >> Tycho counterpart, even if it is very close to our coordinates. I attach a >> table with the query coordinates of this object. For the cross-match we have >> used 1arcsec. >> >> Curiously, if one increases the match radios to 1.1arcsec, Topcat finds the >> counterpart, with an angular separation between query position and result >> position of only 3.86264261101561E-6 degrees = 0.014 arcsec!!. So why did not >> Topcat find the counterpart before? What was wrong? >> > > It's an issue with the service not the client I think. If I issue > the following Cone Search queries to VizieR: > > http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&RA=346.86965&DEC=21.13425&SR=0.00031 > http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&RA=346.86965&DEC=21.13425&SR=0.00027 > > the first one returns a result within 3.9e-6 degrees and the second > one returns no result. It may be to do with slightly different > coordinate systems used for the matching and position reporting, > but I believe that TOPCAT is doing the right thing here (submitted > position and returned columns marked POS_EQ_{RA,DEC}_MAIN in ICRS). > > Is somebody from VizieR listening and able to comment? > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > -- Thomas Boch Ingénieur de Recherche CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 F-67000 Strasbourg Email : thomas.boch@astro.XXX France http://cdsweb.u-strasbg.fr/~boch From francois@cdsarc.XXX Thu Dec 15 09:20:37 2011 Date: Thu, 15 Dec 2011 09:40:17 +0100 From: "Francois Ochsenbein (ext.52429)" To: Mark Taylor Cc: Thomas Boch , fran , topcat-user@bristol.XXX Landais Gilles , Francois Ochsenbein Subject: Re: Multiple Cone Search Hi Mark, Yes Thomas is right -- and the VOTable explains exactly the situation, the results contains the following: Distance from center (RA(ICRS)=346.86965000, DE(ICRS)= 21.13425000) at Epoch=J1991.25 There is no way currently in ConeSearch to specify an epoch for the search; vizier chooses the epoch of the catalog, which is the simplest: the actual position stored in vizier corresponds to the epoch of the catalog, using a different epoch would need to increase signifcantly the search radius to match the large proper motion objects. This behaviour could be changed in the near future in vizier... What is your own point of view on this matter ? Cheers, Francois > > Hi, > >I think I understood what is going wrong in VizieR. It looks for >counterparts taking into account the original columns (RA(ICRS) and >DE(ICRS)) in epoch J1991.25 whereas in the output, the columns marked >POS_EQ_{RA,DEC}_MAIN have been computed in the epoch J2000.0, taking >into account the proper motions. >This explains the issue I think. > >I have put Gilles and François in CC, so that they can advertise this >list when the problem is fixed. >Shameless plug : in the meantime, you can use the CDS cross-match >service ( http://cdsxmatch.u-strasbg.fr/ ) which is not affected by this >problem > >Cheers, and thank you for finding this issue, > >Thomas > >Mark Taylor wrote: >> On Wed, 14 Dec 2011, fran wrote: >> >> >>> Hi, >>> >>> we have been using the "multiple cone search" for cross-matching a list of >>> sources with Tycho catalogue >>> (http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&). And we have >>> detected that for one object (there may be more), Topcat does not find the >>> Tycho counterpart, even if it is very close to our coordinates. I attach a >>> table with the query coordinates of this object. For the cross-match we have >>> used 1arcsec. >>> >>> Curiously, if one increases the match radios to 1.1arcsec, Topcat finds the >>> counterpart, with an angular separation between query position and result >>> position of only 3.86264261101561E-6 degrees = 0.014 arcsec!!. So why did not >>> Topcat find the counterpart before? What was wrong? >>> >> >> It's an issue with the service not the client I think. If I issue >> the following Cone Search queries to VizieR: >> >> http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&RA=346.86965&DEC=21.13425&SR=0.00031 >> http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&RA=346.86965&DEC=21.13425&SR=0.00027 >> >> the first one returns a result within 3.9e-6 degrees and the second >> one returns no result. It may be to do with slightly different >> coordinate systems used for the matching and position reporting, >> but I believe that TOPCAT is doing the right thing here (submitted >> position and returned columns marked POS_EQ_{RA,DEC}_MAIN in ICRS). >> >> Is somebody from VizieR listening and able to comment? >> >> Mark >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ >> >> > > >-- >Thomas Boch >Ingénieur de Recherche > > CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 > 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 > F-67000 Strasbourg Email : thomas.boch@astro.XXX > France http://cdsweb.u-strasbg.fr/~boch > ======================================================================= Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 Email: francois@cdsarc.XXX (France) Fax: +33-(0)368 85 24 17 ======================================================================= From m.b.taylor@bristol.XXX Thu Dec 15 09:39:56 2011 Date: Thu, 15 Dec 2011 09:39:56 +0000 (GMT) From: Mark Taylor To: "Francois Ochsenbein (ext.52429)" Cc: Thomas Boch , fran , topcat-user@bristol.XXX Landais Gilles , Francois Ochsenbein Subject: Re: Multiple Cone Search Francois, I think the behaviour of the VizieR cone search services has to be resolved with reference to the Simple Cone Search standard (Section 2): http://www.ivoa.net/Documents/latest/ConeSearch.html The standard says that the search parameters (RA and DEC parameters of the HTTP request) and the columns marked with UCDs POS_EQ_{RA,DEC}_MAIN in the result must all be in ICRS coordinates. My understanding of celestial coordinate systems is patchy (does ICRS imply a particular epoch?), but at the least it seems that the input RA/DEC and output POS_EQ_RA/DEC_MAIN columns ought to be consistent with each other, and given the results that I quoted earlier, they are not. Presumably the answer from an implementation point of view is to convert from ICRS to catalogue-friendly coordinates in the service implementation before the query against the catalogue query itself is made. There is no provision in the Cone Search standard for making requests based on different coordinate systems (which seems like a good thing to me - coordinate conversions can be carried out by clients and services rather than having to pass it in the protocol). Mark On Thu, 15 Dec 2011, Francois Ochsenbein (ext.52429) wrote: > > Hi Mark, > > Yes Thomas is right -- and the VOTable explains exactly the > situation, the results contains the following: > > > > > Distance from center (RA(ICRS)=346.86965000, DE(ICRS)= 21.13425000) at Epoch=J1991.25 > > > > > There is no way currently in ConeSearch to specify an epoch for > the search; vizier chooses the epoch of the catalog, which is the > simplest: the actual position stored in vizier corresponds to the > epoch of the catalog, using a different epoch would need to > increase signifcantly the search radius to match the large > proper motion objects. > > This behaviour could be changed in the near future in vizier... > What is your own point of view on this matter ? > > Cheers, > Francois > > > > > > Hi, > > > >I think I understood what is going wrong in VizieR. It looks for > >counterparts taking into account the original columns (RA(ICRS) and > >DE(ICRS)) in epoch J1991.25 whereas in the output, the columns marked > >POS_EQ_{RA,DEC}_MAIN have been computed in the epoch J2000.0, taking > >into account the proper motions. > >This explains the issue I think. > > > >I have put Gilles and François in CC, so that they can advertise this > >list when the problem is fixed. > >Shameless plug : in the meantime, you can use the CDS cross-match > >service ( http://cdsxmatch.u-strasbg.fr/ ) which is not affected by this > >problem > > > >Cheers, and thank you for finding this issue, > > > >Thomas > > > >Mark Taylor wrote: > >> On Wed, 14 Dec 2011, fran wrote: > >> > >> > >>> Hi, > >>> > >>> we have been using the "multiple cone search" for cross-matching a list of > >>> sources with Tycho catalogue > >>> (http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&). And we have > >>> detected that for one object (there may be more), Topcat does not find the > >>> Tycho counterpart, even if it is very close to our coordinates. I attach a > >>> table with the query coordinates of this object. For the cross-match we have > >>> used 1arcsec. > >>> > >>> Curiously, if one increases the match radios to 1.1arcsec, Topcat finds the > >>> counterpart, with an angular separation between query position and result > >>> position of only 3.86264261101561E-6 degrees = 0.014 arcsec!!. So why did not > >>> Topcat find the counterpart before? What was wrong? > >>> > >> > >> It's an issue with the service not the client I think. If I issue > >> the following Cone Search queries to VizieR: > >> > >> http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&RA=346.86965&DEC=21.13425&SR=0.00031 > >> http://vizier.u-strasbg.fr/viz-bin/votable/-A?-source=I/239&RA=346.86965&DEC=21.13425&SR=0.00027 > >> > >> the first one returns a result within 3.9e-6 degrees and the second > >> one returns no result. It may be to do with slightly different > >> coordinate systems used for the matching and position reporting, > >> but I believe that TOPCAT is doing the right thing here (submitted > >> position and returned columns marked POS_EQ_{RA,DEC}_MAIN in ICRS). > >> > >> Is somebody from VizieR listening and able to comment? > >> > >> Mark > >> > >> -- > >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK > >> m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > >> > >> > > > > > >-- > >Thomas Boch > >Ingénieur de Recherche > > > > CDS/Observatoire Astronomique Phone : 33 (0)3 68 85 24 42 > > 11, rue de l'Universite Fax : 33 (0)3 68 85 24 17 > > F-67000 Strasbourg Email : thomas.boch@astro.XXX > > France http://cdsweb.u-strasbg.fr/~boch > > > ======================================================================= > Francois Ochsenbein ------ Observatoire Astronomique de Strasbourg > 11, rue de l'Universite 67000 STRASBOURG Phone: +33-(0)368 85 24 29 > Email: francois@cdsarc.XXX (France) Fax: +33-(0)368 85 24 17 > ======================================================================= > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From sean.farrell@sydney.XXX Fri Jan 6 01:17:59 2012 Date: Fri, 06 Jan 2012 12:17:03 +1100 From: Sean Farrell To: topcat-user@bristol.XXX Subject: Topcat activation action not working on Linux Hi everyone, I downloaded the latest version of Topcat this week (v3.9) and have encountered a problem relating to the activation actions. I¹m working with a table that has a column including a unique URL for each row, and previously I¹ve set it up so that when I click on the row it will open Firefox and take me to the specified link. This has worked flawlessly with previous versions of Topcat, but now when I click on the row it opens the web browser and takes me straight to the Google home page. I¹m running Topcat using the full standalone jar file on a linux machine running Redhat Enterprise 5 with the latest Java runtime environment. To set up the activation action I have selected ³View URL as Web Page² with the system browser as the ³browser type², and the URL link column as the ³Web page location column². I should note that when I tried this with exactly the same setup but on my Macbook Pro laptop it worked fine. Does anyone know whether there is a bug that causes this to not work in a Linux environment, or is this more likely to be a problem with my Java runtime environment? Cheers, Sean ---------------------------------------------------------------------------- Dr Sean Farrell ARC Postdoctoral Fellow School of Physics A29 The University of Sydney T +61 (0)2 9351 7688 NSW 2006, Australia E sean.farrell@sydney.XXX ---------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Fri Jan 6 10:28:32 2012 Date: Fri, 6 Jan 2012 10:28:32 +0000 (GMT) From: Mark Taylor To: Sean Farrell Cc: topcat-user@bristol.XXX Subject: Re: Topcat activation action not working on Linux Sean, from a quick look through the version history, the last time I did something I'd expect to change this behaviour seems to be at version 3.4 (2008). If the version that you had working before was more recent than that, I'd suspect that something else in your local setup has changed. But I might be wrong - if you can confirm that a post-3.4 version behaves differently for you in this respect, please let me know (you can find old versions at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/) If that's not it (i.e. it's broken in all recent versions) ... well the System Browser launching is a bit of a black art, which I delegate to another library (http://browserlaunch2.sourceforge.net/), so there could be strange system-dependent things going wrong. What browser are you using? Mark On Fri, 6 Jan 2012, Sean Farrell wrote: > Hi everyone, > > I downloaded the latest version of Topcat this week (v3.9) and have > encountered a problem relating to the activation actions. I¹m working with a > table that has a column including a unique URL for each row, and previously > I¹ve set it up so that when I click on the row it will open Firefox and take > me to the specified link. This has worked flawlessly with previous versions > of Topcat, but now when I click on the row it opens the web browser and > takes me straight to the Google home page. > > I¹m running Topcat using the full standalone jar file on a linux machine > running Redhat Enterprise 5 with the latest Java runtime environment. To set > up the activation action I have selected ³View URL as Web Page² with the > system browser as the ³browser type², and the URL link column as the ³Web > page location column². I should note that when I tried this with exactly the > same setup but on my Macbook Pro laptop it worked fine. > > Does anyone know whether there is a bug that causes this to not work in a > Linux environment, or is this more likely to be a problem with my Java > runtime environment? > > Cheers, > > Sean > > > ---------------------------------------------------------------------------- > Dr Sean Farrell > ARC Postdoctoral Fellow > School of Physics A29 > The University of Sydney T +61 (0)2 9351 7688 > NSW 2006, Australia E sean.farrell@sydney.XXX > ---------------------------------------------------------------------------- > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From phyqpwl@herts.XXX Thu Jan 19 15:15:32 2012 Date: Thu, 19 Jan 2012 15:01:52 +0000 From: Philip Lucas To: topcat-user@bristol.XXX Subject: changing axis label sizes in Topcat Dear Mark, I'm writing up a largely Topcat-generated paper at the moment and one thing that's a bit awkward is the axis labels generally appear too small to be included in a publication. In the past I have made them blank and then added labels with other software (always a bit tedious), but I was wondering whether you have plans to allow axis labels to be made bigger? Thanks, Phil Lucas, U Hertfordshire From m.b.taylor@bristol.XXX Thu Jan 19 15:24:04 2012 Date: Thu, 19 Jan 2012 15:24:04 +0000 (GMT) From: Mark Taylor To: Philip Lucas Cc: topcat-user@bristol.XXX Subject: Re: changing axis label sizes in Topcat Phil, funny you should ask. I have just started a major overhaul of TOPCAT graphics, and configurability of axis labels is one of the many things I plan to improve. So, watch this space, but it's quite a big job, so don't expect results too soon. In the mean time, one trick that might help is resizing the window to be smaller before you do the export - the text stays the same size, but the plot is smaller, so the labels look relatively bigger by the time the whole thing is resized to fit on a given area of journal page. In most cases the quality of vector format (EPS/PDF) graphics output won't suffer from doing this (but try it to check). STILTS also lets you change font sizes. Mark On Thu, 19 Jan 2012, Philip Lucas wrote: > Dear Mark, > > I'm writing up a largely Topcat-generated paper at the moment > and one thing that's a bit awkward is the axis labels generally > appear too small to be included in a publication. In the past > I have made them blank and then added labels with other software > (always a bit tedious), but I was wondering whether you have > plans to allow axis labels to be made bigger? > > Thanks, > > Phil Lucas, > U Hertfordshire > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From mara.salvato@ipp.XXX Fri Jan 27 19:39:36 2012 Date: Fri, 27 Jan 2012 20:38:21 +0100 From: Mara Salvato To: topcat-user@bristol.XXX Subject: history Hi Mark, is there the possibility for TOPCAT saving the history of the commands used in the previous sessions? Something like IDL, where you can see the last XXX executed commands. SM even managed to keep the history of the commands executed in a given directory . Would it be to much work ? Of course histories of matches or plots would make no sense, but at least to have memory of which values where for example used for certain subsamples selection would be great.. Thanks, Mara From m.b.taylor@bristol.XXX Mon Jan 30 15:47:34 2012 Date: Mon, 30 Jan 2012 15:47:34 +0000 (GMT) From: Mark Taylor To: Mara Salvato Cc: topcat-user@bristol.XXX Subject: Re: history Mara, the trouble is, the operations that TOPCAT performs do not currently exist in a form that can be stored or at least viewed as history, i.e. as text. Instead they are data structures in memory or sequences of mouse gestures, depending on how you look at it. So, it's not at all straightforward. Most of the operations in TOPCAT can alternatively be performed in STILTS however, and STILTS commands do have a textual representation. I do have in mind to implement better integration between TOPCAT and STILTS, so that it will be possible to get a TOPCAT window to report what it's doing as a STILTS command. If I do that, then plots and matches as well as newly created columns, subsets etc could be logged. I hope to try to do this later this year, though I'm not sure yet exactly what form the result will take. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ On Fri, 27 Jan 2012, Mara Salvato wrote: > Hi Mark, > > is there the possibility for TOPCAT saving the history of the commands used in > the previous sessions? > > Something like IDL, where you can see the last XXX executed commands. > SM even managed to keep the history of the commands executed in a given > directory . Would it be to much work ? > > Of course histories of matches or plots would make no sense, but at least to > have memory of which values where for example used for certain subsamples > selection would be great.. > > Thanks, > Mara From m.azimlu@gmail.XXX Tue Jan 31 21:20:55 2012 Date: Tue, 31 Jan 2012 21:09:28 +0000 From: m.azimlu@gmail.XXX To: topcat-user@sympa.XXX Subject: Why Best match only and All matches for exact value gives different answers? Comparing WISE sources with 2MASS key and WISE_X_2MASS from CDSXmatch: Weird! I match the exact value of “JNAME” from WISE_X_2MASS CDSXmatch with “designation” which is the source ID in WISE catalog with 2MASS keys for a large sample of sources (~750,000). When I do the best match only 831 sources do not match and when I do all matches, only 28 does not match. I checked the real names and coordinates. Those which has not been matched in "all match" do not really have a match in the other catalog. But the rest 831-28=803 objects actually have exactly the same name and coordinates and other properties! It is weird how the "best match" for exact value is different from the "all match" and why the "best match" fails to pick up the exact same values for 803 sources! From m.b.taylor@bristol.XXX Wed Feb 1 10:36:25 2012 Date: Wed, 1 Feb 2012 10:36:25 +0000 (GMT) From: Mark Taylor To: m.azimlu@gmail.XXX Cc: topcat-user@sympa.XXX Subject: Re: Why Best match only and All matches for exact value gives different answers? Dear m., can you take a look at this entry in the FAQ: http://www.starlink.ac.uk/topcat/faq.html#xmatchIssues and see if it addresses your issue? I suspect that the symmetric Best match is not appropriate for what you're trying to do and you should be using one of the asymmetric ones instead; you may need to upgrade to a more recent version for this. If you still think there's an issue, please get back in touch - ideally I'd like to see the data, but don't send 0.75Mrows to the mailing list. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ On Tue, 31 Jan 2012, m.azimlu@gmail.XXX wrote: > Comparing WISE sources with 2MASS key and WISE_X_2MASS from CDSXmatch: > > Weird! I match the exact value of “JNAME” from WISE_X_2MASS CDSXmatch with > “designation” which is the source ID in WISE catalog with 2MASS keys for a > large sample of sources (~750,000). When I do the best match only 831 sources > do not match and when I do all matches, only 28 does not match. I checked the > real names and coordinates. Those which has not been matched in "all match" do > not really have a match in the other catalog. But the rest 831-28=803 objects > actually have exactly the same name and coordinates and other properties! It > is weird how the "best match" for exact value is different from the "all match" > and why the "best match" fails to pick up the exact same values for 803 > sources! > From david.murphy@durham.XXX Tue Feb 7 18:17:15 2012 Date: Tue, 7 Feb 2012 15:15:29 -0300 From: David Murphy To: topcat-user@mlist2.XXX Subject: fits table concatenation with STILTS tcatn Hi all, When concatenating two .fits files with STILTS it appears that the order of the columns in the two files matters. Should this be the case? I would hope that "ra" in file1 would match to "ra" in file2. I appreciate that for concatenating ascii-type files, this should be different. As I'm only a recent convert from TOPCAT (due to flaky X-forwarding connections), I assume the following is correct: stilts tcatn nin=2 in1=file1.fits in2=file2.fits out=out.fits Cheers, David ------------------------------------------------------------------------------------------- David N. A. Murphy                david.murphy@durham.XXX Department of Physics           http://astro.dur.ac.uk/~dmurphy/ Durham University                  +44(0) 191 33 43787 ------------------------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Tue Feb 7 20:05:57 2012 Date: Tue, 7 Feb 2012 20:05:57 +0000 (GMT) From: Mark Taylor To: David Murphy Cc: topcat-user@bristol.XXX Subject: Re: fits table concatenation with STILTS tcatn On Tue, 7 Feb 2012, David Murphy wrote: > Hi all, > > When concatenating two .fits files with STILTS it appears that the > order of the columns in the two files matters. > > Should this be the case? I would hope that "ra" in file1 would match > to "ra" in file2. I appreciate that for concatenating ascii-type > files, this should be different. Yes, that's how it's supposed to work, it doesn't do anything as clever as rearranging columns by name. From the docs (http://www.starlink.ac.uk/stilts/sun256/tcatn.html): For this concatenation to make sense, each column of T1 must be compatible with the corresponding column of T2 - they must have compatible types and, presumably, meanings. If this is not the case for the tables that you wish to concatenate, for instance the columns are in different orders, or the units differ between a column in T1 and its opposite number in T2, you can use the icmd1 and/or icmd2 parameters to manipulate the input tables so that the column sequences are compatible. See Appendix B.16.2 for some examples. Like (almost) all STILTS commands, it behaves just the same regardless of input table format, so ASCII and FITS work the same way. > As I'm only a recent convert from TOPCAT (due to flaky X-forwarding > connections), I assume the following is correct: > > stilts tcatn nin=2 in1=file1.fits in2=file2.fits out=out.fits Yes, that's OK, but as noted above the columns have to be ordered correctly for the result to make sense. As a trivial example, if you just want to extract the RA and DEC columns from both those tables regardless of input table column order, and discarding any other columns, you could do: stilts tcatn nin=2 in1=file1.fits icmd1='keepcols "RA DEC"' \ in2=file2.fits icmd2='keepcols "RA DEC"' \ out=out.fits Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Feb 20 14:36:09 2012 Date: Mon, 20 Feb 2012 14:36:09 +0000 (GMT) From: Mark Taylor To: Seb Oliver Cc: topcat-user@bristol.XXX Subject: Re: Character and String Hi Seb, On Mon, 20 Feb 2012, Seb Oliver wrote: > Hi Mark et al, > > I have a catalogue down-loaded from Aladin J/A+A/449/951 which has data types including "string" and "character". "character" appears to be a type of string with one character. > > If I try and do functions on the characters e.g. contains(l_IR36mag,"<") topcat complains ... > > Function "contains" exists but parameters "contains(char,class java.lang.String)" can not be accepted by it. > > I've tried various things including converting the "character" to a string using toString but topcat toString doesn't seem to accept characters either. > > I'm sure I'm doing something really dumb, in fact I'm not sure this isn't a problem I've come across and resolved before, but I've run out of ideas and can't find any help on the topic. Not dumb at all. The strikingly non-obvious answer to how to do what you want is to use an expression like: l_IR36mag == '<' note the single quotes. A Character is a different data type than a String in java (hence the TOPCAT/STILTS expression language), it represents a single character, and it is effectively an integer type rather than (like String) an array-of-integers type, so the way you have to handle it is using the numeric-type expressions like == for equals. Character literals are represented with single quotes, unlike String literals which are represented with double quotes. It is clearly not a very happy situation that the hapless user has to know/worry about this stuff. I'll see if I can think of some way to make it work in a less obscure fashion - thanks for the question which will hopefully stimulate an improvement. If you still have questions get back to me. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Feb 20 17:42:18 2012 Date: Mon, 20 Feb 2012 17:42:18 +0000 (GMT) From: Mark Taylor To: Seb Oliver Cc: topcat-user@bristol.XXX Subject: Re: Character and String Catering for people who read the documentation eh? It's crazy, but it just might work. I'll certainly do at least one of these. Mark On Mon, 20 Feb 2012, Seb Oliver wrote: > HI Mark, > > Thanks for that, it fixed it. > > If you can make it work in a more obvious way that would of course be a valuable improvement. > > However, a quick win might be just to put something in the help documentation that explains this. E.g. a sub-section under > 7 Algebraic Expression Syntax > (perhaps in between 7.3 referencing Table parameters and 7.4 Null Values) > which listed the known data types and their foibles - and pointing to conversion functions in 7.6). > > Seb. > > > > > On 20 Feb 2012, at 14:36, Mark Taylor wrote: > > > Hi Seb, > > > > On Mon, 20 Feb 2012, Seb Oliver wrote: > > > >> Hi Mark et al, > >> > >> I have a catalogue down-loaded from Aladin J/A+A/449/951 which has data types including "string" and "character". "character" appears to be a type of string with one character. > >> > >> If I try and do functions on the characters e.g. contains(l_IR36mag,"<") topcat complains ... > >> > >> Function "contains" exists but parameters "contains(char,class java.lang.String)" can not be accepted by it. > >> > >> I've tried various things including converting the "character" to a string using toString but topcat toString doesn't seem to accept characters either. > >> > >> I'm sure I'm doing something really dumb, in fact I'm not sure this isn't a problem I've come across and resolved before, but I've run out of ideas and can't find any help on the topic. > > > > Not dumb at all. The strikingly non-obvious answer to how to do what > > you want is to use an expression like: > > > > l_IR36mag == '<' > > > > note the single quotes. > > > > A Character is a different data type than a String in java (hence the > > TOPCAT/STILTS expression language), it represents a single character, > > and it is effectively an integer type rather than (like String) > > an array-of-integers type, so the way you have to handle it is > > using the numeric-type expressions like == for equals. > > Character literals are represented with single quotes, unlike String > > literals which are represented with double quotes. > > > > It is clearly not a very happy situation that the hapless user has > > to know/worry about this stuff. I'll see if I can think of some way > > to make it work in a less obscure fashion - thanks for the question > > which will hopefully stimulate an improvement. If you still have > > questions get back to me. > > > > Mark > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Feb 22 09:28:39 2012 Date: Wed, 22 Feb 2012 09:28:39 +0000 (GMT) From: Mark Taylor To: Mattia Vaccari Cc: topcat-user@bristol.XXX Subject: Re: Character and String On Tue, 21 Feb 2012, Mattia Vaccari wrote: > hi mark, > > i was similarly stumbling on something i think i may have solved a while ago > and that, while simple, i can't find in the docs under arrays. > > i've got a 5-element column array called mag_aper_1 and i'd like to add > the same (or different numbers) to each of its five element, but i don't > know how to refer to array elements by index or wildcard. You can refer to a single array element by dereferencing it using square brackets, e.g. MAGS[0] is the first element of the cell in column MAGS (this is very tersely alluded to at the bottom of the section on operators, http://www.starlink.ac.uk/topcat/sun253/sec7.5.html). There's no way to apply an operation to all elements of an array at once, but you can do it by hand for relatively short arrays, e.g.: array(x[0]+1, x[1]+1, x[2]+1, x[3]+1, x[4]+1) gives you a result which is a 5-element array with elements the same as the elements of an existing 5-element array x all incremented by 1 (though the data type may get changed) - see http://www.starlink.ac.uk/topcat/sun253/Arrays.html. It's not really feasible to introduce a comprehensive suite of vector arithmetic functions just because you'd need a lot of them to cover all cases, and I doubt if many people want to do this. But it would be pretty easy to write a small java class that implements certain custom functions along these lines for a specific job (http://www.starlink.ac.uk/topcat/sun253/jelExtend.html). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Feb 22 11:12:49 2012 Date: Wed, 22 Feb 2012 11:12:49 +0000 (GMT) From: Mark Taylor To: Hugo Buddelmeijer Cc: topcat-user@bristol.XXX Subject: Re: TOPCAT does not enlarge string columns for fits files Hugo, On Tue, 21 Feb 2012, Hugo Buddelmeijer wrote: > Dear Mark, > > TOPCAT does not enlarge the size of a string column when saving a table as a > fits file. > > For example, col2 of the attached fits table is of class String with element > size 5. The size is essentially hidden information, although it can be made > vissible. These strings can be changed to longer values in the table browser. > However, the strings are cropped back to their original length upon saving the > table as fits again. Saving the table as csv does store the strings in their > proper length. > > This can potentially cause data loss, since at no point is there a > notification that the strings are cropped. Therefore, it seems it is a bug. > Personally, I can work around it, since I now know about it. > > Tested with the webstart version, so probably the latest TOPCAT. > > Greetings, > Hugo You are quite right. It turns out there is a similar bug in STILTS that applies to the replaceval and replacecol filters. I've made a fix to all these which will be in future releases - in the mean time a pre-release with the fix is at: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ Thanks for this report, and apologies to you or anyone else who's had trouble from this. Mark > P.S., planned to send this to the mailing list, but didn't register yet and > still have to receive the password. (so I posted it for you) -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From palau@aliga.XXX Wed Feb 22 16:26:43 2012 Date: Wed, 22 Feb 2012 17:19:28 +0100 From: Aina Palau To: TOPCAT user Subject: TOPCAT: reloading a session Hello I'm starting to use TOPCAT (3.6) and I realised that after saving a session I do not recover the algebraic expression used to define a new column. I read the help and says: "Note however that the reloaded state is not identical to that before saving. One important difference is that columns and row subsets which were defined algebraically are saved and restored as fixed values, so it is no longer possible to edit the expressions." I wanted to know if this issue has been fixed in the new TOPCAT releases. If not, I strongly encourage to implement this in future releases (ie, make possible the re-edition of algebraic formulae defining a column when reloading the session), as this would be really useful (saving you the time of re-typing again the formulae if you made a mistake or you want to make new tests). Best Aina Palau -- Institut de Ciencies de l'Espai (CSIC-IEEC) Campus UAB, Facultat de Ciencies, Torre C-5 parell E-08193 Bellaterra, Barcelona, Spain Phone: +34 93 581 4375 From m.b.taylor@bristol.XXX Wed Feb 22 16:43:29 2012 Date: Wed, 22 Feb 2012 16:43:28 +0000 (GMT) From: Mark Taylor To: Aina Palau Cc: TOPCAT user Subject: Re: TOPCAT: reloading a session Aina, The current version behaves the same. I'll think about improving it. In the mean time note that for columns, though not subsets, you can see the expression that was used to calculate them by looking in the "Description" entry in the Columns window (http://www.starlink.ac.uk/topcat/sun253/ColumnInfoWindow.html) Mark On Wed, 22 Feb 2012, Aina Palau wrote: > Hello > > I'm starting to use TOPCAT (3.6) and I realised that after saving a session I > do not recover the algebraic expression used to define a new column. I read > the help and says: > > "Note however that the reloaded state is not identical to that before saving. > One important difference is that columns and row subsets which were defined > algebraically are saved and restored as fixed values, so it is no longer > possible to edit the expressions." > > I wanted to know if this issue has been fixed in the new TOPCAT releases. If > not, I strongly encourage to implement this in future releases (ie, make > possible the re-edition of algebraic formulae defining a column when reloading > the session), as this would be really useful (saving you the time of re-typing > again the formulae if you made a mistake or you want to make new tests). > > Best > > Aina Palau > > -- > Institut de Ciencies de l'Espai (CSIC-IEEC) > Campus UAB, Facultat de Ciencies, Torre C-5 parell > E-08193 Bellaterra, Barcelona, Spain > Phone: +34 93 581 4375 -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Tue Mar 20 10:36:17 2012 Date: Tue, 20 Mar 2012 10:36:17 +0000 (GMT) From: Mark Taylor To: Raffaele D'Abrusco Cc: topcat-user@bristol.XXX Subject: Re: Column name search box and multiple files loading On Mon, 19 Mar 2012, Raffaele D'Abrusco wrote: > Hi Mark, > > I have a couple of questions/suggestions. > > I was wondering whether it could be feasible to add a search box in the > Topcat "Display Column Metadata" window, to search for a column by its name > (and, hopefully, use regular expressions to search for column names > substrings etc.). I have been finding myself wandering more or less > aimlessly through very long list of columns looking for one specific column > or for any column with "flux", "mag" or "time" in the name. Sorting > alphabetically the name of the columns can help in few lucky cases, but > having a more general interactive search mechanism would be much handier. > Do you think this could be implemented in the future? I'll think about that one, thanks for the suggestion. > On a different note, I would also lobby for the implementation of multiple > files loading. Sometime it can be useful to be able to select more than one > files at the same time in the same folder (same format as well). This doesn't exactly exist, but you can do something which is almost as good. All the load dialogue windows have a Keep Window Open toggle button top left in the toolbar. If you click this on, then you can load lots of files without having to keep opening the load dialogue. For many of the load dialogues, you can just double-click a filename to load it. Note this doesn't work for the System Browser dialogue though - that one is controlled by the OS and there's nothing much I can do about that. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From dabrusco@gmail.XXX Tue Mar 20 13:32:17 2012 Date: Tue, 20 Mar 2012 09:20:02 -0400 From: Raffaele D'Abrusco To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Column name search box and multiple files loading Hi Mark, > > On a different note, I would also lobby for the implementation of > multiple > > files loading. Sometime it can be useful to be able to select more than > one > > files at the same time in the same folder (same format as well). > > This doesn't exactly exist, but you can do something which is almost > as good. All the load dialogue windows have a Keep Window Open > toggle button top left in the toolbar. If you click this on, > then you can load lots of files without having to keep opening > the load dialogue. For many of the load dialogues, you can just > double-click a filename to load it. > > Thanks! I didn't notice that. Raffaele From mara.salvato@ipp.XXX Thu May 3 11:13:11 2012 Date: Thu, 3 May 2012 11:11:25 +0200 From: Mara Salvato To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: WARNING: Attempt to work around Mac OSX JTable bug (uk.ac.starlink.topcat.TopcatModel$1.getColumn) Hi Mark, I am trying to do something very trivial with TOPCAT installed on a Mac Lion (full java installation) create a new column such that $277>?1:0 I get the following error message: WARNING: Attempt to work around Mac OSX JTable bug (uk.ac.starlink.topcat.TopcatModel$1.getColumn) with the new column incomplete… Did someone else encountered the same problem and/or knows a solution ? Thanks, Mara From m.b.taylor@bristol.XXX Thu May 3 10:26:41 2012 Date: Thu, 3 May 2012 10:26:40 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: WARNING: Attempt to work around Mac OSX JTable bug (uk.ac.starlink.topcat.TopcatModel$1.getColumn) On Thu, 3 May 2012, Mara Salvato wrote: > Hi Mark, > > I am trying to do something very trivial with TOPCAT installed on a Mac Lion (full java installation) > > create a new column such that $277>?1:0 that syntax is not correct (do you mean "$277>0?1:0"?, but that may not be the problem. > I get the following error message: > > WARNING: Attempt to work around Mac OSX JTable bug (uk.ac.starlink.topcat.TopcatModel$1.getColumn) Getting this message itself isn't something that you necessarily need to worry about, it's just logging that TOPCAT is doing something questionable. However, it may indicate the cause of other things that are going wrong though. > with the new column incomplete… Can you explain what you mean by an incomplete column? Since this may have to do with bugs in the Java VM, it may be hard to fix. If you want to work round it, you can probably do so by duplicating the table, e.g. File|Duplicate Table menu, or save/reload. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From mara@mpe.XXX Thu May 3 11:59:01 2012 Date: Thu, 3 May 2012 11:37:54 +0200 From: Mara Salvato To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: WARNING: Attempt to work around Mac OSX JTable bug (uk.ac.starlink.topcat.TopcatModel$1.getColumn) Hi Mark, thanks for the quick reply, as usual. >> >> >> create a new column such that $277>?1:0 > > that syntax is not correct (do you mean "$277>0?1:0"?, but that may > not be the problem. sorry, I mistyped . Of course the syntax is as you wrote ( otherwise I would have had a syntax error message) > > Can you explain what you mean by an incomplete column? column 277 has either an integer number or is NULL. for those rows with a value, the new column reports a "1" as desired, while for the rows that were NULL in $277 remain NULL. mhhh….. I reversed the syntax with : NULL_$277?0:1 and it worked! Thanks, Mara From mara@mpe.XXX Thu May 3 11:39:36 2012 Date: Thu, 3 May 2012 11:37:54 +0200 From: Mara Salvato To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: WARNING: Attempt to work around Mac OSX JTable bug (uk.ac.starlink.topcat.TopcatModel$1.getColumn) Hi Mark, thanks for the quick reply, as usual. >> >> >> create a new column such that $277>?1:0 > > that syntax is not correct (do you mean "$277>0?1:0"?, but that may > not be the problem. sorry, I mistyped . Of course the syntax is as you wrote ( otherwise I would have had a syntax error message) > > Can you explain what you mean by an incomplete column? column 277 has either an integer number or is NULL. for those rows with a value, the new column reports a "1" as desired, while for the rows that were NULL in $277 remain NULL. mhhh….. I reversed the syntax with : NULL_$277?0:1 and it worked! Thanks, Mara From jgarrido@iaa.XXX Tue May 8 08:45:26 2012 Date: Tue, 8 May 2012 08:44:45 +0200 From: Julián Garrido To: topcat-user@bristol.XXX Subject: char functions and incorrect votables Dear users and Mark, In the framework of the wf4ever project, the AMIGA group [1] is implementing stilts-based functionalities in Taverna. We found out that some catalogs provide votable with incorrect fields when dealing with char datatypes (missing arraysize property), and we know this can be worked around using the system property votable.strict. However, we also realized that there are no functions to work with chars. For example, it seems sensible to us that string function such as concat accept both strings and chars as parameters. It would be great if this casting was automatically done, but it would be enough if a toString(char) function is implemented. Now toString function only accepts floats. Best regards, Julián Garrido. [1] http://amiga.iaa.es/p/8-technical-development.htm -- Julián Garrido Instituto de Astrofísica de Andalucía (IAA-CSIC) Glorieta de la Astronomía s/n, E-18008, Granada, Spain From m.b.taylor@bristol.XXX Tue May 8 10:35:30 2012 Date: Tue, 8 May 2012 10:35:29 +0100 (BST) From: Mark Taylor To: Julián Garrido Cc: topcat-user@bristol.XXX Subject: Re: char functions and incorrect votables Julián, good suggestion, thanks. A pre-release version with a toString(char) function (and a few other changes to toString that I've made recently) is available at: ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ However, there is another way to do this which will work with the existing public release. The "+" operator can be used as an alternative to concat for concatenating strings with other things. So if you have a string CS and a character column CC you can write CS+CC instead of concat(CS,toString(CC)) Mark On Tue, 8 May 2012, Julián Garrido wrote: > Dear users and Mark, > > In the framework of the wf4ever project, the AMIGA group [1] is > implementing stilts-based functionalities in Taverna. > > We found out that some catalogs provide votable with incorrect fields when > dealing with char datatypes (missing arraysize property), and we know this > can be worked around using the system property votable.strict. > > However, we also realized that there are no functions to work with chars. > For example, it seems sensible to us that string function such as concat > accept both strings and chars as parameters. > > It would be great if this casting was automatically done, but it would be > enough if a toString(char) function is implemented. Now toString function > only accepts floats. > > Best regards, Julián Garrido. > > [1] http://amiga.iaa.es/p/8-technical-development.htm > > -- > Julián Garrido > Instituto de Astrofísica de Andalucía (IAA-CSIC) > Glorieta de la Astronomía s/n, E-18008, Granada, Spain > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From jgarrido@iaa.XXX Wed May 9 17:11:56 2012 Date: Wed, 9 May 2012 17:11:03 +0200 From: Julián Garrido To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: char functions and incorrect votables Thank you!! I will take a look at that. regads, Julián. 2012/5/8 Mark Taylor > Julián, > > good suggestion, thanks. A pre-release version with a toString(char) > function (and a few other changes to toString that I've made recently) > is available at: > > ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ > > However, there is another way to do this which will work with the > existing public release. The "+" operator can be used as an alternative > to concat for concatenating strings with other things. So if you > have a string CS and a character column CC you can write > > CS+CC > > instead of > > concat(CS,toString(CC)) > > Mark > > On Tue, 8 May 2012, Julián Garrido wrote: > > > Dear users and Mark, > > > > In the framework of the wf4ever project, the AMIGA group [1] is > > implementing stilts-based functionalities in Taverna. > > > > We found out that some catalogs provide votable with incorrect fields > when > > dealing with char datatypes (missing arraysize property), and we know > this > > can be worked around using the system property votable.strict. > > > > However, we also realized that there are no functions to work with chars. > > For example, it seems sensible to us that string function such as concat > > accept both strings and chars as parameters. > > > > It would be great if this casting was automatically done, but it would be > > enough if a toString(char) function is implemented. Now toString function > > only accepts floats. > > > > Best regards, Julián Garrido. > > > > [1] http://amiga.iaa.es/p/8-technical-development.htm > > > > -- > > Julián Garrido > > Instituto de Astrofísica de Andalucía (IAA-CSIC) > > Glorieta de la Astronomía s/n, E-18008, Granada, Spain > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ -- Julián Garrido Instituto de Astrofísica de Andalucía (IAA-CSIC) Glorieta de la Astronomía s/n, E-18008, Granada, Spain From Daryl.Seibel@FalconJet.XXX Fri May 25 17:03:06 2012 Date: Fri, 25 May 2012 10:46:31 -0500 From: Daryl.Seibel@FalconJet.XXX To: topcat-user@bristol.XXX Subject: Linear regression fit through the zero point How do I force the linear regression fit through 0,0 or origin of a scatter plot? Thank you, Daryl From dottori@if.XXX Mon May 28 11:22:40 2012 Date: Mon, 28 May 2012 02:59:58 +0100 From: dottori@if.XXX To: topcat-user@sympa.XXX Subject: save table subset of rows Hi: 1- How should I procede to save a table subset of rows? 2- how can define a subset of rows through two conditions on a column?: V1 To: Daryl.Seibel@FalconJet.XXX Cc: topcat-user@bristol.XXX Subject: Re: Linear regression fit through the zero point On Fri, 25 May 2012, Daryl.Seibel@FalconJet.XXX wrote: > How do I force the linear regression fit through 0,0 or origin of a > scatter plot? Daryl, there is no way to impose additional constraints on the linear regression fit - TOPCAT is not a very sophisticated tool as far as statistical analysis goes. However, you could do something like add a new column using the expression Y/X (where Y and X are the quantities on the Y and X axes) and use the Statistics window to calculate the mean - this would give you the slope of a line through the origin and trying to pass through all the points. Whether that's the figure you are after depends on exactly what it is you're trying to achieve. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon May 28 10:28:50 2012 Date: Mon, 28 May 2012 10:28:48 +0100 (BST) From: Mark Taylor To: dottori@if.XXX Cc: topcat-user@sympa.XXX Subject: Re: save table subset of rows On Mon, 28 May 2012, dottori@if.XXX wrote: > Hi: > 1- How should I procede to save a table subset of rows? Once you have created a suitable subset, select it as the Current Row Subset in the main Control Window. Then, saving the table will include only the rows in that subset. Read more in sections 3 and 3.1 of the manual: http://www.star.bristol.ac.uk/~mbt/topcat/sun253/apparent.html http://www.star.bristol.ac.uk/~mbt/topcat/sun253/subsets.html > 2- how can define a subset of rows through two conditions on a column?: > V1 subset of rows. the "&&" operator combines two conditions in this way (use "||" for or). So the expression you'd use would be V1 To: ngodbole@berkeley.XXX Cc: topcat-user@bristol.XXX Subject: Re: Deleting Rows in tabe data in TOPCAT? On Fri, 1 Jun 2012, ngodbole@berkeley.XXX wrote: > Good morning, > I understand that this is the appropriate destination to address TOPCAT > related questions. > > I have a .fits file that I load into TOPCAT ver 3.9 as a table, and I > need to be able to delete certain rows of data before I can use the > file. I open the 'Display table cell data' window and pull down the > 'Subsets' menu. From here, I m not sure of how I can delete rows in the > table. You're on the right track: you need to create a Row Subset containing only the rows you want to keep and then choose that as the Current Subset by selecting it in the main Control Window. Most subsequent actions (including save table) then work on only those rows. If what you want to do is identify rows by eye to delete them, the most straightforward way is to select rows to delete (or to keep) in the table view window by clicking or ctrl-clicking on them, and then using the Subset From (Un)Selected Rows toolbar button (though I believe there's a bug in some versions of OS X java which means ctrl-click doesn't work in some cases on a Mac). That can get fiddly if you need to select lots of rows to delete/keep, so an alternative is to create a Boolean column (simply add a new column as described at http://www.star.bristol.ac.uk/~mbt/topcat/sun253/SyntheticColumnQueryWindow.html with the value "false" (or "true")) and click on it in the table view to set it true/false for different rows. You can then add a new subset with the name of that column which will only contain the rows for which that column is true (has a check in the checkbox). These two techniques are mentioned in the "Selecting rows in the browser" and "Boolean columns" items in the manual section http://www.star.bristol.ac.uk/~mbt/topcat/sun253/subsetDef.html That section also lists other ways you can define a subset, which might be more appropriate. Hope that helps. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From JJ.Kavelaars@nrc-cnrc.XXX Sat Jun 2 00:59:41 2012 Date: Fri, 1 Jun 2012 15:57:55 -0700 From: "Kavelaars, JJ" To: "" Cc: "topcat-user@bristol.XXX" Subject: Re: Deleting Rows in tabe data in TOPCAT? >From inside the interface you click on the icon that is a magenta oval with a red oval inside. This brings up the 'Row Subsets' menu. (you can get there form the 'Views' menu item also. Then click the green 'plus' to open dialogue. Inside the dialogue give a name to your subset and then a rule like ' A > 50' where 'A' is the name of one of the columns. There are lots of expression you can use, click the 'function' icon to see those. hope that helps. JJ On 2012-06-01, at 10:42 AM, wrote: > Good morning, > I understand that this is the appropriate destination to address TOPCAT > related questions. > > I have a .fits file that I load into TOPCAT ver 3.9 as a table, and I > need to be able to delete certain rows of data before I can use the > file. I open the 'Display table cell data' window and pull down the > 'Subsets' menu. From here, I m not sure of how I can delete rows in the > table. > > Any help would be appreciated. > Thank you, > Niharika Godbole > ngodbole@berkeley.XXX > > From m.b.taylor@bristol.XXX Tue Jun 5 18:16:50 2012 Date: Tue, 5 Jun 2012 18:16:50 +0100 (BST) From: Mark Taylor To: acava@fis.XXX Cc: topcat-user@sympa.XXX Subject: Re: Maximum number of columns Antonio, On Tue, 5 Jun 2012, acava@fis.XXX wrote: > Dear Mark and all list users, > > I'm working with STILTS and TOPCAT on tables containing large numbers of > columns and after some matching I obtain tables with >1000 columns getting the > following error: > > Error: Keyword too long > (Keyword too long) > > that seems to be related to the maximum allowed number of columns in a table > (1000?). > Is there any way to overcome this limitation and allow to work with tables with > >1000 columns? > > Btw, in TOPCAT the error comes up only when I try to save the table and it > gives the following additional warnings: > > ava.io.IOException: Keyword too long > at > uk.ac.starlink.fits.AbstractFitsTableWriter.writeTableHDU(AbstractFitsTableWriter.java:140) how interesting, as far as I know nobody has come up against this limit before. It's a restriction imposed by the FITS BINTABLE (binary table) format. In BINTABLE you have to define columns using header cards with names like TTYPEnnn for column nnn, and since FITS header card keyword names are limited to 8 characters, there is no way to define more than 999 columns. Output to non-FITS formats, such as VOTable, will (I hope) work. For now this is your only option, but VOTable is considerably less efficient than FITS for use in TOPCAT with large files (VOTables need to get read in their entirety when loaded, FITS data is just mapped in memory). I don't know how large your files are in number of rows, but they certainly have a lot of columns. I'd be interested to know if other people think this is something that needs addressing - are 1000+-column tables common, or likely to become so? I can think of a couple of possible solutions: - a CDF table handler may be coming up for STIL in the next few months; this would hopefully have similar performance characteristics to the FITS handler, and probably would not suffer the 999-column limit. - I could fix the STIL FITS BINTABLE I/O handlers to read/write TTYPnnnn rather than TTYPEnnn if the column count rolls over 999. This would not be standard FITS BINTABLE, and hence such FITS files wouldn't be readable by other FITS applications, but it would allow TOPCAT/STILTS etc to handle such tables. Any opinions? (me-toos can mail me off list if they prefer) Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bianca@lambrate.XXX Wed Jun 6 08:41:54 2012 Date: Wed, 6 Jun 2012 08:41:05 +0200 From: Bianca Garilli To: topcat-user@mlist2.XXX Subject: Re: Maximum number of columns Hi Mark > - I could fix the STIL FITS BINTABLE I/O handlers to read/write > TTYPnnnn rather than TTYPEnnn if the column count rolls over > 999. This would not be standard FITS BINTABLE, and hence such > FITS files wouldn't be readable by other FITS applications, but > it would allow TOPCAT/STILTS etc to handle such tables. > although this would fix the case of >999 columns, it would be very uncofortobale for the (more frequent) case of <1000 columns, as you say other applications would not access tables any more. I would suggest to have a double modality: fully fits compliant, with n. of columns limited to 999, and 'extended fits mode' which allows for more columns, but other programs may not be able to access the table Cheers Bianca ------------------------------------------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- From acava@fis.XXX Wed Jun 6 08:35:25 2012 Date: Wed, 6 Jun 2012 09:34:20 +0200 From: Antonio Cava To: Bianca Garilli Cc: topcat-user@mlist2.XXX Subject: Re: Maximum number of columns Bianca's suggestion would actually represent a good practical compromise, allowing the flexibility of extending the bintable names format TTYPnnn to TTYPnnnn only when strictly necessary and leaving untouched the compatibility of the standard FITS format wrt other FITS applications in all other cases (<999cols). On the other hand, probably it is time for our favourite standard fits format to be updated allowing to handle such large number of columns that is becoming more and more usual in the current epoch. -- -------------------------------------------------- Antonio Cava Universidad Complutense de Madrid Facultad de Ciencias Físicas Dpto. de Astrofísica y CC. de la Atmósfera Madrid 28040 SPAIN Phone #: +34 913944456 Fax #: +34 913944635 -------------------------------------------------- 2012/6/6 Bianca Garilli > Hi Mark > > > - I could fix the STIL FITS BINTABLE I/O handlers to read/write > > TTYPnnnn rather than TTYPEnnn if the column count rolls over > > 999. This would not be standard FITS BINTABLE, and hence such > > FITS files wouldn't be readable by other FITS applications, but > > it would allow TOPCAT/STILTS etc to handle such tables. > > > > although this would fix the case of >999 columns, it would be very > uncofortobale for the (more frequent) case of <1000 columns, as you say > other > applications would not access tables any more. > I would suggest to have a double modality: fully fits compliant, with n. of > columns limited to 999, and 'extended fits mode' which allows for more > columns, but other programs may not be able to access the table > > Cheers > > Bianca > > > ------------------------------------------------------------------------- > Bianca Garilli; INAF, IASF-Milano > via Bassini 15; 20133 Milano, Italy > E-mail: bianca@lambrate.XXX > phone: 39-02-23699313 > Fax: 39-02-2666017 > -------------------------------------------------------------------------- > "Le signorine del french-cancan e della spaccata sono > le belle addormentate di una felicità perduta" > Léon-Paul Fargue > > "Science demands dogged work. Work in spite of everything and > continuous observation" > V.Van Gogh (adapted from) > > "In a world where science is treated as a branch of magic, > the rewards often go to those who behave as illusionists" > ------------------------------------------------------------------------- > > From bianca@lambrate.XXX Wed Jun 6 09:43:54 2012 Date: Wed, 6 Jun 2012 09:42:58 +0200 From: Bianca Garilli To: topcat-user@mlist2.XXX Cc: Antonio Cava , topcat-user@mlist2.XXX Subject: Re: Maximum number of columns > On the other hand, probably it is time for our favourite standard fits > format to be updated allowing to handle such large number of columns that > is becoming more and more usual in the current epoch. > I'll start to ask the question to the" IAU fits commission vice convenor" (or whatever is the name) who is in our institute Bianca ------------------------------------------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Wed Jun 6 10:10:52 2012 Date: Wed, 6 Jun 2012 10:10:51 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Maximum number of columns On Wed, 6 Jun 2012, Bianca Garilli wrote: > Hi Mark > > > - I could fix the STIL FITS BINTABLE I/O handlers to read/write > > TTYPnnnn rather than TTYPEnnn if the column count rolls over > > 999. This would not be standard FITS BINTABLE, and hence such > > FITS files wouldn't be readable by other FITS applications, but > > it would allow TOPCAT/STILTS etc to handle such tables. > > > > although this would fix the case of >999 columns, it would be very > uncofortobale for the (more frequent) case of <1000 columns, as you say other > applications would not access tables any more. > I would suggest to have a double modality: fully fits compliant, with n. of > columns limited to 999, and 'extended fits mode' which allows for more > columns, but other programs may not be able to access the table yes, that is what I had in mind. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From acava@fis.XXX Wed Jun 6 11:19:30 2012 Date: Wed, 6 Jun 2012 11:18:31 +0200 From: Antonio Cava To: topcat-user@mlist2.XXX Subject: Re: Maximum number of columns 2012/6/6 Bianca Garilli > > > On the other hand, probably it is time for our favourite standard fits > > format to be updated allowing to handle such large number of columns that > > is becoming more and more usual in the current epoch. > > > > I'll start to ask the question to the" IAU fits commission vice convenor" > (or > whatever is the name) who is in our institute > > great! thank you, let's see what they opine From cgp@leicester.XXX Wed Jun 6 11:39:32 2012 Date: Wed, 06 Jun 2012 10:33:26 +0100 From: Clive Page To: topcat-user@sympa.XXX Subject: Re: Maximum number of columns On 06/06/2012 10:10, Mark Taylor wrote: >>> - I could fix the STIL FITS BINTABLE I/O handlers to read/write >>> TTYPnnnn rather than TTYPEnnn if the column count rolls over >>> 999. I just wanted to note, since nobody else has so far, that for binary tables it is not just TTYPEnnn but also TFORMnnn for which an abbreviated form needs to be devised. If tables of 1000 or more columns use physical units or scaling then TUNITnnn, TSCALnnn, and TZEROnnn will also be affected. The same issue also applies to ASCII tables, although it may be that nobody creates new ones in that format any more (apart from CDS, which does so to provide the most accurate possible representation of catalogues submitted to them in plain text form). An alternative might be to allow the "nnn" value to be extended to "nnnn" where necessary. This would also fall foul of the FITS Standard, which requires character posistions 9 and 10 to be "= ". Forcing the equals sign in to position 10 might still be accepted by some other FITS-reading packages I suppose even if non-standard. It would save having to invent abbreviated forms for the all these keywords. The only question in my mind is whether, say, "TTYPE1234=" would be taken to refer to column 123 in a program not expecting a 4-digit value; the "=" in the wrong character position ought to be enough to prevent mistakes, but it is hard to be sure how all other FITS-reading software would cope with this. In the longer term, I agree, the FITS Standard needs to be updated. I've been generating tables with over 300 columns for some years - the 999-column limit hasn't affected me so far, but the limited headroom is of some concern. Regards -- Clive Page From m.b.taylor@bristol.XXX Wed Jun 6 10:50:03 2012 Date: Wed, 6 Jun 2012 10:50:03 +0100 (BST) From: Mark Taylor To: Clive Page Cc: topcat-user@sympa.XXX Subject: Re: Maximum number of columns On Wed, 6 Jun 2012, Clive Page wrote: > On 06/06/2012 10:10, Mark Taylor wrote: > > > > - I could fix the STIL FITS BINTABLE I/O handlers to read/write > > > > TTYPnnnn rather than TTYPEnnn if the column count rolls over > > > > 999. > > I just wanted to note, since nobody else has so far, that for binary tables it > is not just TTYPEnnn but also TFORMnnn for which an abbreviated form needs to > be devised. If tables of 1000 or more columns use physical units or scaling > then TUNITnnn, TSCALnnn, and TZEROnnn will also be affected. yes. > The same issue also applies to ASCII tables, although it may be that nobody > creates new ones in that format any more (apart from CDS, which does so to > provide the most accurate possible representation of catalogues submitted to > them in plain text form). > > An alternative might be to allow the "nnn" value to be extended to "nnnn" > where necessary. This would also fall foul of the FITS Standard, which > requires character posistions 9 and 10 to be "= ". Forcing the equals sign in > to position 10 might still be accepted by some other FITS-reading packages I > suppose even if non-standard. It would save having to invent abbreviated > forms for the all these keywords. The only question in my mind is whether, > say, "TTYPE1234=" would be taken to refer to column 123 in a program not > expecting a 4-digit value; the "=" in the wrong character position ought to be > enough to prevent mistakes, but it is hard to be sure how all other > FITS-reading software would cope with this. I hadn't thought of that, but it sounds a bit more dangerous to me - at least TTYPnnnn, TFORnnnn etc are legal FITS, even if not legal BINTABLE. My current feeling is that adopting those for use by STIL for columns where n>999 is a reasonable thing to do, with a view to its possible adoption as a de facto standard by other software and maybe in the distant future as part of FITS. I think I'll take the discussion over to the fitsbits list to see if the wider community of FITSerati have anything to say about it. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From cgp@leicester.XXX Wed Jun 6 12:58:23 2012 Date: Wed, 06 Jun 2012 10:58:15 +0100 From: Clive Page Cc: "topcat-user@sympa.XXX" Subject: Re: Maximum number of columns On 06/06/2012 10:50, Mark Taylor wrote: > I hadn't thought of that, but it sounds a bit more dangerous to me - > at least TTYPnnnn, TFORnnnn etc are legal FITS, even if not legal BINTABLE. That's a good point. I guess there's no point in breaking the Standard more badly than necessary, so I withdraw my suggestion. > I think I'll take the discussion over > to the fitsbits list to see if the wider community of FITSerati have > anything to say about it. That's also a good idea. Regards -- Clive Page From Pierre.Fernique@astro.XXX Wed Jun 6 13:25:44 2012 Date: Wed, 06 Jun 2012 13:25:08 +0200 From: Pierre Fernique To: topcat-user@mlist2.XXX Subject: Re: Maximum number of columns Concerning other FITS binary table reader. Just to say that I had a look in my source code, and I'm lucky, Aladin is already supporting TTYPnnnn keywords. The only constraint concerns the position of '=' character. Pierre Fernique Le 06/06/2012 11:58, Clive Page a écrit : > On 06/06/2012 10:50, Mark Taylor wrote: >> I hadn't thought of that, but it sounds a bit more dangerous to me - >> at least TTYPnnnn, TFORnnnn etc are legal FITS, even if not legal BINTABLE. > > That's a good point. I guess there's no point in breaking the Standard more > badly than necessary, so I withdraw my suggestion. > >> I think I'll take the discussion over >> to the fitsbits list to see if the wider community of FITSerati have >> anything to say about it. > > That's also a good idea. > > Regards > From m.b.taylor@bristol.XXX Wed Jun 6 12:33:38 2012 Date: Wed, 6 Jun 2012 12:33:38 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Maximum number of columns Interesting. Was that code speculative, or have you encountered 1000+ column FITS tables? I've opened the topic on the fitsbits list (http://listmgr.cv.nrao.edu/pipermail/fitsbits/2012-June/thread.html) so you might like to chip in there with your experience. On Wed, 6 Jun 2012, Pierre Fernique wrote: > Concerning other FITS binary table reader. Just to say that I had a look in my > source code, and I'm lucky, Aladin is already supporting TTYPnnnn keywords. > The only constraint concerns the position of '=' character. > Pierre Fernique > > Le 06/06/2012 11:58, Clive Page a écrit : > > On 06/06/2012 10:50, Mark Taylor wrote: > > > I hadn't thought of that, but it sounds a bit more dangerous to me - > > > at least TTYPnnnn, TFORnnnn etc are legal FITS, even if not legal > > > BINTABLE. > > > > That's a good point. I guess there's no point in breaking the Standard more > > badly than necessary, so I withdraw my suggestion. > > > > > I think I'll take the discussion over > > > to the fitsbits list to see if the wider community of FITSerati have > > > anything to say about it. > > > > That's also a good idea. > > > > Regards > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From Pierre.Fernique@astro.XXX Thu Jun 7 09:42:48 2012 Date: Thu, 07 Jun 2012 09:40:57 +0200 From: Pierre Fernique To: topcat-user@mlist2.XXX Subject: Re: Maximum number of columns In fact, I ignored that the binary tables were limited to NNN indice. Concretly I never encounter a table with so many columns, but as Aladin already supports them, it is quite logic that we did not have complains about this point. :-) Pierre Le 06/06/2012 13:33, Mark Taylor a écrit : > Interesting. Was that code speculative, or have you encountered > 1000+ column FITS tables? I've opened the topic on the fitsbits list > (http://listmgr.cv.nrao.edu/pipermail/fitsbits/2012-June/thread.html) > so you might like to chip in there with your experience. > > On Wed, 6 Jun 2012, Pierre Fernique wrote: > >> Concerning other FITS binary table reader. Just to say that I had a look in my >> source code, and I'm lucky, Aladin is already supporting TTYPnnnn keywords. >> The only constraint concerns the position of '=' character. >> Pierre Fernique >> >> Le 06/06/2012 11:58, Clive Page a écrit : >>> On 06/06/2012 10:50, Mark Taylor wrote: >>>> I hadn't thought of that, but it sounds a bit more dangerous to me - >>>> at least TTYPnnnn, TFORnnnn etc are legal FITS, even if not legal >>>> BINTABLE. >>> That's a good point. I guess there's no point in breaking the Standard more >>> badly than necessary, so I withdraw my suggestion. >>> >>>> I think I'll take the discussion over >>>> to the fitsbits list to see if the wider community of FITSerati have >>>> anything to say about it. >>> That's also a good idea. >>> >>> Regards >>> >> >> > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Jun 13 09:54:03 2012 Date: Wed, 13 Jun 2012 09:54:02 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Maximum number of columns On Wed, 6 Jun 2012, Mark Taylor wrote: > I hadn't thought of that, but it sounds a bit more dangerous to me - > at least TTYPnnnn, TFORnnnn etc are legal FITS, even if not legal BINTABLE. > My current feeling is that adopting those for use by STIL for columns > where n>999 is a reasonable thing to do, with a view to its possible > adoption as a de facto standard by other software and maybe in the > distant future as part of FITS. I think I'll take the discussion over > to the fitsbits list to see if the wider community of FITSerati have > anything to say about it. The discussion on fitsbits turned out to be moderately lively. To summarise, there was not wide approval for this solution as part of FITS. I think I'll sit on it for now (I'm quite busy with other things at the moment), and possibly implement some way of storing 1000+-column tables in FITS or FITS-like files at some point in the future. I haven't yet decided what would be the best way to do it. In the mean time: if you need to store the data in a wide table in a FITS file, the best thing might be to do it as two FITS tables, by using only a selection of the columns each time, to keep each individual table within 999 columns. Even better might be to identify a large number of columns that really you're never going to use and ditch those. If anybody else comes across this limit in the course of doing something they need/want to do, I'd be interested to hear about it; that would increase the priority of implementing some fix. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From mjones6@wisc.XXX Fri Jun 22 17:40:07 2012 Date: Fri, 22 Jun 2012 16:39:28 +0100 From: mjones6@wisc.XXX To: topcat-user@sympa.XXX Subject: TOPCAT can't load my CSV tables I'm a new at using TOPCAT, and I have encountered a problem with CSV files. I have gotten them to load before, but now I am getting error messages when trying to load them. The settings are the same to my knowledge, the data is fine (no empty cells or mismatched rows). Has anyone encountered this or know the correct settings to get the CSV table to load? I am using LibreOffice to edit them, if that makes any difference. Thanks, Megan From m.b.taylor@bristol.XXX Fri Jun 22 16:45:23 2012 Date: Fri, 22 Jun 2012 16:45:23 +0100 (BST) From: Mark Taylor To: mjones6@wisc.XXX Cc: topcat-user@sympa.XXX Subject: Re: TOPCAT can't load my CSV tables On Fri, 22 Jun 2012, mjones6@wisc.XXX wrote: > I'm a new at using TOPCAT, and I have encountered a problem with CSV files. I > have gotten them to load before, but now I am getting error messages when > trying to load them. The settings are the same to my knowledge, the data is > fine (no empty cells or mismatched rows). Has anyone encountered this or know > the correct settings to get the CSV table to load? I am using LibreOffice to > edit them, if that makes any difference. Hi Megan, you do need to tell TOPCAT that it's loading a CSV file, by setting the "Format" selector to "CSV" (the details differ a bit depending on what file loader window you're using) - it can't auto-detect CSV format just by looking at the file. If you've done that and you're having trouble still, can you mail me the file off-list and I'll take a look. CSV does sometimes mean slightly different things to different people/tools. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Fri Jun 22 16:48:20 2012 Date: Fri, 22 Jun 2012 17:46:12 +0200 From: Maciej Bilicki To: mjones6@wisc.XXX Cc: topcat-user@sympa.XXX Subject: Re: TOPCAT can't load my CSV tables Maybe try to open them in a text editor, save from there and see if the same will be happening? (by a text editor I mean some gedit or notepad) M 2012/6/22 > I'm a new at using TOPCAT, and I have encountered a problem with CSV > files. I > have gotten them to load before, but now I am getting error messages when > trying to load them. The settings are the same to my knowledge, the data is > fine (no empty cells or mismatched rows). Has anyone encountered this or > know > the correct settings to get the CSV table to load? I am using LibreOffice > to > edit them, if that makes any difference. > > Thanks, > Megan > > From m.azimlu@gmail.XXX Fri Jun 22 19:30:00 2012 Date: Fri, 22 Jun 2012 13:29:20 -0400 From: Mohaddesseh Azimlu To: topcat-user@sympa.XXX Cc: mjones6@wisc.XXX Subject: Re: TOPCAT can't load my CSV tables Hi Megan If you are saving your .csv files on Mac, there are two different options: Windows Comma Separated and MS-DOS Comma Separated. I have found that the MS-DOS Comma Separated and MS-DOS text format cannot be read by TOPCAT (I don't know why) and some other codes. You need to save your files in Windows format. -Mohaddesseh 2012/6/22 > >> I'm a new at using TOPCAT, and I have encountered a problem with CSV >> files. I >> have gotten them to load before, but now I am getting error messages when >> trying to load them. The settings are the same to my knowledge, the data >> is >> fine (no empty cells or mismatched rows). Has anyone encountered this or >> know >> the correct settings to get the CSV table to load? I am using LibreOffice >> to >> edit them, if that makes any difference. >> >> Thanks, >> Megan >> >> > From bianca@lambrate.XXX Tue Jun 26 16:54:32 2012 Date: Tue, 26 Jun 2012 16:53:46 +0200 From: Bianca Garilli To: topcat-user@bristol.XXX Subject: define subsets in stilts and save session? Hi Mark I have to define a number of subsets in a table and save the results I know very well how to do it interactively, and then save the 'session' But I have to repeat this exercise several times, thus I'd rather use stilts I know how to define subsets in stilts, and save the resulting subsets in different tables But is there a way in stilts to define several subsets and save the WHOLE table as a session, as I do interactively? Thanks Bianca ------------------------------------------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Tue Jun 26 16:09:31 2012 Date: Tue, 26 Jun 2012 16:09:31 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: define subsets in stilts and save session? On Tue, 26 Jun 2012, Bianca Garilli wrote: > Hi Mark > I have to define a number of subsets in a table and save the results > I know very well how to do it interactively, and then save the 'session' > > But I have to repeat this exercise several times, thus I'd rather use stilts > I know how to define subsets in stilts, and save the resulting subsets in > different tables > > But is there a way in stilts to define several subsets and save the WHOLE > table as a session, as I do interactively? There's no way (without working extremely hard) to write a table from STILTS that matches TOPCAT's session format. However, when TOPCAT loads a table, if it finds any boolean-valued columns, it automatically sets up a subset corresponding to each one. So probably you can achieve what you want to do by adding new boolean-valued columns in STILTS (something like cmd="addcol subset1 rmag-bmag<1.1"). When you load the resulting table into TOPCAT the corresponding subsets will be visible. There will also be a couple of boolean columns that you may not want, but you can always hide or ignore them. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From bianca@lambrate.XXX Tue Jun 26 17:13:11 2012 Date: Tue, 26 Jun 2012 17:12:10 +0200 From: Bianca Garilli To: topcat-user@mlist2.XXX Subject: Re: define subsets in stilts and save session? Yes, that will do perfectly THANKS B. On Tuesday, June 26, 2012 05:09 PM Mark Taylor wrote > On Tue, 26 Jun 2012, Bianca Garilli wrote: > > Hi Mark > > I have to define a number of subsets in a table and save the results > > I know very well how to do it interactively, and then save the 'session' > > > > But I have to repeat this exercise several times, thus I'd rather use > > stilts I know how to define subsets in stilts, and save the resulting > > subsets in different tables > > > > But is there a way in stilts to define several subsets and save the WHOLE > > table as a session, as I do interactively? > > There's no way (without working extremely hard) to write a table from > STILTS that matches TOPCAT's session format. > > However, when TOPCAT loads a table, if it finds any boolean-valued > columns, it automatically sets up a subset corresponding to each one. > > So probably you can achieve what you want to do by adding new > boolean-valued columns in STILTS (something like > cmd="addcol subset1 rmag-bmag<1.1"). When you load the resulting table > into TOPCAT the corresponding subsets will be visible. > There will also be a couple of boolean columns that you may not want, > but you can always hide or ignore them. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ ------------------------------------------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Wed Jun 27 21:39:34 2012 Date: Wed, 27 Jun 2012 21:39:34 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Compact syntax for multiple columns filtering & uneven binning for histograms On Wed, 27 Jun 2012, Raffaele D'Abrusco wrote: > Hi Mark, > > I have one question and one suggestion. > > Question: can users currently create subsets with boolean filters that > involve multiple columns > at the same time in a compact way? Let's say I have a table with N columns > (with different names), > containing different photometric information (mags/fluxes in different > filters). Missing data/NaN > in all column are indicated with the same number, let's say -999. > I can create a boolean filter: > > col1 > -999 & col2 > -999 .... colN > -999 > > but this can be lengthy when I have a very large number of columns. Is > there a way to use > a more compact syntax for the same purpose? For example: > > col* > -999 > > where I am selecting only the rows (sources) that have no -999 in all > columns at the same time. > I have gone through the documentation but at no avail, so far. Nice idea, but it would not be easy to accommodate syntax like that in the expression language. If I wanted to do something like this, I'd do it in JyStilts - you can get the list of columns by name, filter them using Python string manipulation, build up an expression and apply it (I can supply example code if it's not obvious what I mean). But I appreciate that's not always convenient. > Suggestion: I would find very useful to have implemented in one of the next > Topcat releases > an option to create unevenly binned histograms according to different > criteria. In particular, > I'd love to be able to pick the number of sources in the bin as the same > way now users > can choose the width of the bins. Do you think it would be possible (and > interesting for other > users as well) to implement this feature? I haven't heard that suggestion before, I'll bear it in mind. It may be feasible in a future release. Thanks for the input, Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ From nhurleywalker@cantab.XXX Fri Jul 20 08:05:19 2012 Date: Fri, 20 Jul 2012 14:04:37 +0800 (WST) From: Natasha Hurley-Walker Reply-To: nhw@icrar.XXX To: topcat-user@sympa.XXX Subject: VO Vizier query Hi all, I'm attempting to query Vizier using TopCat for some radio data. I am trying the 7C radio survey because I did this back in April and it worked fine, by selecting 'VO'->'Vizier Catalogue Service', ticking 'all rows', selecting 'Radio' wavelength, and using the keyword '7C'. Now I am stuck on 'Locating suitable catalogues (Found 1)'.... forever. I get an error in the terminal, which I have pasted at the end of this email. I have tried a few different servers but they all give the same error, except for http://data.bao.ac.cn/, which gives a 500 error. Does anyone have any idea how to fix this? I suspect it's not a problem with my set-up, rather with the external webservers, but if anyone else could run a quick check on their system I would appreciate it. Cheers, Natasha Dr Natasha Hurley-Walker Super-Science Fellow MWA Imaging and Calibration ICRAR / Curtin University Telephone: (+61) 8 9266 9178 Mobile: (+61) 4 2619 2677 Skype: nhurleywalker Email: nhw@icrar.XXX ICRAR: Discovering the hidden Universe through radio astronomy Web: http://www.icrar.org ICRAR's eNewsletter: http://www.icrar.org#subscribe ICRAR on Twitter: http://twitter.com/icrar ICRAR on Facebook: http://www.icrar.org/facebook Exception in thread "Thread-8" java.lang.NullPointerException at uk.ac.starlink.topcat.vizier.CatalogSaxHandler.endElement(CatalogSaxHandler.java:88) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:604) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1759) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2915) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:625) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:819) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:748) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1208) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:525) at javax.xml.parsers.SAXParser.parse(SAXParser.java:392) at javax.xml.parsers.SAXParser.parse(SAXParser.java:195) at uk.ac.starlink.topcat.vizier.SearchVizierMode.parseCatalogQuery(SearchVizierMode.java:351) at uk.ac.starlink.topcat.vizier.SearchVizierMode.access$1400(SearchVizierMode.java:49) at uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.attemptLoadCatalogs(SearchVizierMode.java:550) at uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.loadCatalogs(SearchVizierMode.java:505) at uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.run(SearchVizierMode.java:472) From m.b.taylor@bristol.XXX Fri Jul 20 10:58:24 2012 Date: Fri, 20 Jul 2012 10:58:24 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: VO Vizier query There may or may not be a fix from the VizieR end soon, but in the mean time you can grab a pre-release version of TOPCAT which will work with the service as it stands, from: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Mark On Fri, 20 Jul 2012, Mark Taylor wrote: > Natasha, > > thanks for this report. It seems the format of the VizieR output has > recently changed and TOPCAT is failing to cope with it. > I'm talking to the people at CDS, hopefully we can resolve it soon. > > Incidentally, the "By Category" and "By Keyword" tabs in that window > are independent, so the actions you describe will give you all > 7C-related catalogues not just the radio ones. > > Hopefully I can post an update soon. > > Mark > > On Fri, 20 Jul 2012, Natasha Hurley-Walker wrote: > > > > > Hi all, > > > > I'm attempting to query Vizier using TopCat for some radio data. I am > > trying the 7C radio survey because I did this back in April and it worked > > fine, by selecting 'VO'->'Vizier Catalogue Service', ticking 'all rows', > > selecting 'Radio' wavelength, and using the keyword '7C'. Now I am stuck > > on 'Locating suitable catalogues (Found 1)'.... forever. I get an error in > > the terminal, which I have pasted at the end of this email. I have tried a > > few different servers but they all give the same error, except for > > http://data.bao.ac.cn/, which gives a 500 error. > > > > Does anyone have any idea how to fix this? I suspect it's not a problem > > with my set-up, rather with the external webservers, but if anyone else > > could run a quick check on their system I would appreciate it. > > > > Cheers, > > Natasha > > > > Dr Natasha Hurley-Walker > > Super-Science Fellow > > MWA Imaging and Calibration > > ICRAR / Curtin University > > Telephone: (+61) 8 9266 9178 > > Mobile: (+61) 4 2619 2677 > > Skype: nhurleywalker > > Email: nhw@icrar.XXX > > > > ICRAR: Discovering the hidden Universe through radio astronomy > > Web: http://www.icrar.org > > ICRAR's eNewsletter: http://www.icrar.org#subscribe > > ICRAR on Twitter: http://twitter.com/icrar > > ICRAR on Facebook: http://www.icrar.org/facebook > > > > > > Exception in thread "Thread-8" java.lang.NullPointerException > > at > > uk.ac.starlink.topcat.vizier.CatalogSaxHandler.endElement(CatalogSaxHandler.java:88) > > at > > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:604) > > at > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1759) > > at > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2915) > > at > > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:625) > > at > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488) > > at > > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:819) > > at > > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:748) > > at > > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123) > > at > > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1208) > > at > > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:525) > > at javax.xml.parsers.SAXParser.parse(SAXParser.java:392) > > at javax.xml.parsers.SAXParser.parse(SAXParser.java:195) > > at > > uk.ac.starlink.topcat.vizier.SearchVizierMode.parseCatalogQuery(SearchVizierMode.java:351) > > at > > uk.ac.starlink.topcat.vizier.SearchVizierMode.access$1400(SearchVizierMode.java:49) > > at > > uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.attemptLoadCatalogs(SearchVizierMode.java:550) > > at > > uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.loadCatalogs(SearchVizierMode.java:505) > > at > > uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.run(SearchVizierMode.java:472) > > > > > > > > > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From dabrusco@gmail.XXX Fri Jul 20 18:54:05 2012 Date: Fri, 20 Jul 2012 09:53:10 -0700 From: Raffaele D'Abrusco To: topcat-user@sympa.XXX Subject: Histogram in STILTS? Hi all, I am trying to produce histograms (normalized counts in bins) using STILTS. I want to do exactly what plothist command does, except for the fact that I would like the counts in bin saved to a table file instead of a plot. tcube does something similar and can handle multi-dimensional histograms, but its output is an image, not a table. Am I overlooking something? Any suggestion would be appreciated. Thanks, Raffaele From sn.fotop@gmail.XXX Fri Jul 20 18:05:56 2012 Date: Fri, 20 Jul 2012 19:05:07 +0200 From: Sotiria Fotopoulou To: Raffaele D'Abrusco Cc: topcat-user@sympa.XXX Subject: Re: Histogram in STILTS? Hi Raffaele, is using TOPCAT an option? Create the histogram and then from the menu: export -> save as table Cheers, Sotiria 2012/7/20 Raffaele D'Abrusco > Hi all, > I am trying to produce histograms (normalized counts in bins) using STILTS. > I want to do exactly what plothist command does, except for the fact that > I would > like the counts in bin saved to a table file instead of a plot. tcube does > something > similar and can handle multi-dimensional histograms, but its output is an > image, > not a table. Am I overlooking something? Any suggestion would be > appreciated. > Thanks, > > Raffaele > From dabrusco@gmail.XXX Sat Jul 21 05:16:01 2012 Date: Sat, 21 Jul 2012 00:14:32 -0400 From: Raffaele D'Abrusco To: Sotiria Fotopoulou Cc: topcat-user@sympa.XXX Subject: Re: Histogram in STILTS? Hi Sotiria, unforunately no. I am writing a script to be run from command line, and don't want to use Topcat. Thanks for your suggestion, though. Raffaele On Fri, Jul 20, 2012 at 1:05 PM, Sotiria Fotopoulou wrote: > Hi Raffaele, > > is using TOPCAT an option? > > Create the histogram and then from the menu: > export -> save as table > > Cheers, > Sotiria > > > 2012/7/20 Raffaele D'Abrusco > >> Hi all, >> I am trying to produce histograms (normalized counts in bins) using >> STILTS. >> I want to do exactly what plothist command does, except for the fact that >> I would >> like the counts in bin saved to a table file instead of a plot. tcube >> does something >> similar and can handle multi-dimensional histograms, but its output is an >> image, >> not a table. Am I overlooking something? Any suggestion would be >> appreciated. >> Thanks, >> >> Raffaele >> > > From m.b.taylor@bristol.XXX Mon Jul 23 12:26:24 2012 Date: Mon, 23 Jul 2012 12:26:23 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Histogram in STILTS? On Fri, 20 Jul 2012, Raffaele D'Abrusco wrote: > Hi all, > I am trying to produce histograms (normalized counts in bins) using STILTS. > I want to do exactly what plothist command does, except for the fact that I > would > like the counts in bin saved to a table file instead of a plot. tcube does > something > similar and can handle multi-dimensional histograms, but its output is an > image, > not a table. Am I overlooking something? Any suggestion would be > appreciated. Sotiria's suggestion apart, you're not missing anything, this is not currently possible. A table output from tcube for 1D results might be a reasonable thing to expect, but I don't have time to implement it in the near future. The best I can suggest is doing it in JyStilts/Jython and using tcube, then using other FITS packages in Jython to make sense of the resulting 1D FITS image (though I'm not certain if there is a suitable jython-compatible FITS package). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed Jul 25 01:34:20 2012 Date: Wed, 25 Jul 2012 01:34:16 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: VO Vizier query Hi Natasha and others, the people at VizieR have made a partial fix for this (requests to the relevant service which can be identified as coming from TOPCAT are fed the old version of the format; any other client gets the new, incompatible format). For most users this should mean that the VizieR window will work as it used to again. However, this is not bulletproof. In particular if you are using TOPCAT from Java WebStart, it will be unable to talk to VizieR, displaying the error behaviour reported at the start of this thread. The same may apply if you are using an exotic Java implementation. In that case, you will have to use the pre-release version below, or wait until the next public release of TOPCAT. Mark On Fri, 20 Jul 2012, Mark Taylor wrote: > There may or may not be a fix from the VizieR end soon, > but in the mean time you can grab a pre-release version of TOPCAT > which will work with the service as it stands, from: > > ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ > > Mark > > On Fri, 20 Jul 2012, Mark Taylor wrote: > > > Natasha, > > > > thanks for this report. It seems the format of the VizieR output has > > recently changed and TOPCAT is failing to cope with it. > > I'm talking to the people at CDS, hopefully we can resolve it soon. > > > > Incidentally, the "By Category" and "By Keyword" tabs in that window > > are independent, so the actions you describe will give you all > > 7C-related catalogues not just the radio ones. > > > > Hopefully I can post an update soon. > > > > Mark > > > > On Fri, 20 Jul 2012, Natasha Hurley-Walker wrote: > > > > > > > > Hi all, > > > > > > I'm attempting to query Vizier using TopCat for some radio data. I am > > > trying the 7C radio survey because I did this back in April and it worked > > > fine, by selecting 'VO'->'Vizier Catalogue Service', ticking 'all rows', > > > selecting 'Radio' wavelength, and using the keyword '7C'. Now I am stuck > > > on 'Locating suitable catalogues (Found 1)'.... forever. I get an error in > > > the terminal, which I have pasted at the end of this email. I have tried a > > > few different servers but they all give the same error, except for > > > http://data.bao.ac.cn/, which gives a 500 error. > > > > > > Does anyone have any idea how to fix this? I suspect it's not a problem > > > with my set-up, rather with the external webservers, but if anyone else > > > could run a quick check on their system I would appreciate it. > > > > > > Cheers, > > > Natasha > > > > > > Dr Natasha Hurley-Walker > > > Super-Science Fellow > > > MWA Imaging and Calibration > > > ICRAR / Curtin University > > > Telephone: (+61) 8 9266 9178 > > > Mobile: (+61) 4 2619 2677 > > > Skype: nhurleywalker > > > Email: nhw@icrar.XXX > > > > > > ICRAR: Discovering the hidden Universe through radio astronomy > > > Web: http://www.icrar.org > > > ICRAR's eNewsletter: http://www.icrar.org#subscribe > > > ICRAR on Twitter: http://twitter.com/icrar > > > ICRAR on Facebook: http://www.icrar.org/facebook > > > > > > > > > Exception in thread "Thread-8" java.lang.NullPointerException > > > at > > > uk.ac.starlink.topcat.vizier.CatalogSaxHandler.endElement(CatalogSaxHandler.java:88) > > > at > > > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:604) > > > at > > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1759) > > > at > > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2915) > > > at > > > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:625) > > > at > > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488) > > > at > > > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:819) > > > at > > > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:748) > > > at > > > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123) > > > at > > > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1208) > > > at > > > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:525) > > > at javax.xml.parsers.SAXParser.parse(SAXParser.java:392) > > > at javax.xml.parsers.SAXParser.parse(SAXParser.java:195) > > > at > > > uk.ac.starlink.topcat.vizier.SearchVizierMode.parseCatalogQuery(SearchVizierMode.java:351) > > > at > > > uk.ac.starlink.topcat.vizier.SearchVizierMode.access$1400(SearchVizierMode.java:49) > > > at > > > uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.attemptLoadCatalogs(SearchVizierMode.java:550) > > > at > > > uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.loadCatalogs(SearchVizierMode.java:505) > > > at > > > uk.ac.starlink.topcat.vizier.SearchVizierMode$SearchWorker.run(SearchVizierMode.java:472) > > > > > > > > > > > > > > > > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From nthomas@head.XXX Thu Aug 2 18:10:23 2012 Date: Thu, 2 Aug 2012 12:07:54 -0400 From: nthomas@head.XXX To: topcat-user@bristol.XXX Subject: Export error Hello, I've found a problem with the 'Export as eps' scatter plot option. When I export eps files, the legend shows up as an empty black box in the exported figure. I do not have this problem exporting with any other file type. Do you know how to fix this problem? Thanks! Nancy Thomas From odysseus9672@gmail.XXX Thu Aug 9 06:08:34 2012 Date: Wed, 8 Aug 2012 21:07:32 -0700 From: Sean Lake To: topcat-user@bristol.XXX Subject: Bug in TOPCAT IPAC Table Parsing Hello all, There's a minor bug in TOPCAT's handling of IPAC tables. I call it minor because the program handles the tables correctly, but it incorrectly throws a warning. The warning is: WARNING: Unofficial IPAC data type "long" - assume long (uk.ac.starlink.table.formats.IpacReader.createColumnReader) This is incorrect, as the official page shows: http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html#datatypes Thanks, Sean Lake From m.b.taylor@bristol.XXX Thu Aug 9 09:53:54 2012 Date: Thu, 9 Aug 2012 09:53:54 +0100 (BST) From: Mark Taylor To: Sean Lake Cc: topcat-user@bristol.XXX Subject: Re: Bug in TOPCAT IPAC Table Parsing On Wed, 8 Aug 2012, Sean Lake wrote: > Hello all, > > There's a minor bug in TOPCAT's handling of IPAC tables. I call it minor because the program handles the tables correctly, but it incorrectly throws a warning. The warning is: > > WARNING: Unofficial IPAC data type "long" - assume long (uk.ac.starlink.table.formats.IpacReader.createColumnReader) > > This is incorrect, as the official page shows: http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html#datatypes Thanks Sean, it looks like the data format definition has changed since I last looked at it. I've removed the warning, and also now accept the abbreviation "l" for long. This fix will be in the next release, or you can find a pre-release at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From jrosvick@tru.XXX Fri Aug 17 23:28:11 2012 Date: Mon, 13 Aug 2012 23:23:47 +0100 From: jrosvick@tru.XXX To: topcat-user@sympa.XXX Subject: question regarding point labels Hello! I was wondering if you'd be able to help me with my issue of including point labels on my plot, please. I downloaded the dmg version of the software to my MacBook Pro (running Lion) and just opened it as an application, like usual. I produced a colour-colour diagram with the intrinsic line of black bodies as a subset of another table I loaded. I want to use the Point Labels capability in the scatter plot so that I can label each plotted point with the corresponding temperature (I only have 4 points so crowding isn't an issue). In the little menu at the bottom of the Scatter Plot window, I've "highlighted" the blackbody table, but when I click on the button for drawing labels on the plot, so that I can tell the window which column values to use as the text, all the data points disappear. The axis labels (and axes themselves) remain, i.e. the window itself doesn't disappear. If I click on the button again (to uncheck it, as it were) the plot returns. I get no error messages throughout this process. I haven't been able to find any info in the manual or online as to why this might be happening. If you could give me some advice as to what I might be doing wrong, I'd really appreciate it. Thanks very much, Joanne From m.b.taylor@bristol.XXX Fri Aug 17 23:41:36 2012 Date: Fri, 17 Aug 2012 23:41:36 +0100 (BST) From: Mark Taylor To: jrosvick@tru.XXX Cc: topcat-user@sympa.XXX Subject: Re: question regarding point labels On Mon, 13 Aug 2012, jrosvick@tru.XXX wrote: > Hello! I was wondering if you'd be able to help me with my issue of including > point labels on my plot, please. I downloaded the dmg version of the software > to my MacBook Pro (running Lion) and just opened it as an application, like > usual. > > I produced a colour-colour diagram with the intrinsic line of black bodies as a > subset of another table I loaded. I want to use the Point Labels capability in > the scatter plot so that I can label each plotted point with the corresponding > temperature (I only have 4 points so crowding isn't an issue). > > In the little menu at the bottom of the Scatter Plot window, I've "highlighted" > the blackbody table, but when I click on the button for drawing labels on the > plot, so that I can tell the window which column values to use as the text, all > the data points disappear. The axis labels (and axes themselves) remain, i.e. > the window itself doesn't disappear. If I click on the button again (to uncheck > it, as it were) the plot returns. I get no error messages throughout this > process. I haven't been able to find any info in the manual or online as to why > this might be happening. > > If you could give me some advice as to what I might be doing wrong, I'd really > appreciate it. Thanks very much, > > Joanne Joanne, your description of what you're doing sounds right, and this doesn't sound like any problem that I'm aware of. Can you send me (off-list) a screenshot of the window with(out) the missing points, including the bottom part with the controls, as well as the table you're trying to include label data from? Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From dabrusco@gmail.XXX Wed Sep 5 14:43:54 2012 Date: Wed, 5 Sep 2012 09:42:48 -0400 From: Raffaele D'Abrusco To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc Hi Mark, I have been using the galactic extinction service from STILTS lately and there seems to be a limit to the number of positions for which extinction data can be retrieved from a single file, set at 1025. Can you confirm this limit? Thanks, Raffaele On Mon, Oct 31, 2011 at 7:19 AM, Mark Taylor wrote: > On Fri, 28 Oct 2011, Mark Taylor wrote: > > > Ivan, Maciej and others, > > > > On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: > > > > > On Fri, Oct 28, 2011, Mark Taylor wrote: > > > > The Schlegel thing is there but experimental and not documented; > > > > if you want to use it I can tell you how. > > > > > > I personally don't need that at the moment, but as I promised there's > > > strong interest from the user community, even in this thread, so I'd > > > suggest you to briefly describe things on the mailing list, so that > > > folks can start testing. > > > > This is the story. Following a request from Ivan a while back, > > I have implemented within STILTS (though currently not TOPCAT) a > capability > > to query the Galactic Dust Reddening and Extinction service hosted > > at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). > > This allows you to add columns for E(B-V) Reddening, > > 100 Micron Emission, and Dust Temperature to tables which already > > have sky positions - it's a bit like a multi-cone search, but not quite. > > > > This means it will be easy for people to fire off lots and lots of > > queries (one per row) to IPAC, and I don't want to annoy IPAC or > > their web servers, so I've asked them how they feel about it > > before publicising this in the interface. If they say it's OK > > I'll pass it on. > > I've talked to the nice people at IPAC and they are fine about it, > so if you want to use STILTS to add some or all of E(B-V) Reddening, > 100 Micron Emission and Dust Temperature, you can do it by using > a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) > with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". > > Its usage is (as determined from running it with the wrong arguments) is: > > % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' > cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' > Error: No ra/dec specified > > Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help > > Usage: uk.ac.starlink.ttools.calc.SchlegelFilter > [-results (reddening|emission|temperature)[,...]] > [-stats (mean|ref_pixel|std|max|min)[,...]] > > > i.e. you need to give the names of RA and Dec columns from the input > table, and you can optionally specify which output columns you want. > Simple usage might thus look something like: > > stilts tpipe in=cat.xml > cmd='uk.ac.starlink.ttools.calc.SchlegelFilter RAJ2000 > DEJ2000' > out=catdust.xml > > The source of the data is the service documented here: > > http://irsa.ipac.caltech.edu/applications/DUST/ > > see instructions there for information on how to acknowledge this > servie if it you use it for publication. > > The filter is not particularly fast (mostly my fault not IPAC's)! > I suggest cmd=progress ahead of the > cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the > command line so you can see how it's getting on. > If this feature becomes a documented part of the package I will > hopefully speed it up and of course give it a snappier name. > > > In the mean time, I'm wondering if there are similar but different > > services along the same lines I should provide. It's not feasible > > to do the calculations within the TOPCAT directly, > > since the data files are a few tens of Mb per statistic, which is > > too much to bundle with the application. Are there other web > > services which provide this sort of information that people would > > like to see available from TOPCAT/STILTS? > > > > Alternatively, there seem to be a number of all-sky maps containing > > various data available as FITS files out there. Schlegel/Finkbeiner > > at http://astro.berkeley.edu/~marc/dust/ is one source; > > LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads > > of goodies (WMAP, HI, H-alpha, some of the Schlegel data) > > in HEALPix format. So another possibility would be a function that > > allowed TOPCAT/STILTS users to download one of these HEALPix files > > locally and query it by position (value at, or averaged round, a > > given sky position) to add new columns to a table which has sky > > positions. Compared to the web service option this would give > > more flexibility, be considerably faster, and avoid the problem > > of overloading remote web servers. I quite like this idea > > because it's efficient and opens up a lot of possibilities, > > but if the Schlegel extinction data is really the only dataset > > that people are going to be interested in, then it may not be > > worth the effort. > > Following some of the feedback I've had, I will think about something > along these lines for the future as well. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > From m.b.taylor@bristol.XXX Wed Sep 5 14:57:05 2012 Date: Wed, 5 Sep 2012 14:57:05 +0100 (BST) From: Mark Taylor To: Raffaele D'Abrusco Cc: topcat-user@bristol.XXX Subject: Re: Schlegel dust extinction etc Raffaele, I don't believe there is any such limit in STILTS, so if you're seeing one it is presumably imposed by the service. However, I suggest that if you're attempting to use this functionality for anything like large numbers of points, you use the much more efficient pixsample command in the pre-release version, as per my list message from December: https://sympa.bris.ac.uk/sympa/arc/topcat-user/2011-12/msg00000.html This is still not in the public STILTS release, since for one reason and another I haven't done a release for quite some time, but it should work OK. Mark On Wed, 5 Sep 2012, Raffaele D'Abrusco wrote: > Hi Mark, > > I have been using the galactic extinction service from STILTS lately and > there seems to be a limit to the number of positions for which extinction > data can be retrieved from a single file, set at 1025. > Can you confirm this limit? > Thanks, > > Raffaele > > On Mon, Oct 31, 2011 at 7:19 AM, Mark Taylor wrote: > > > On Fri, 28 Oct 2011, Mark Taylor wrote: > > > > > Ivan, Maciej and others, > > > > > > On Fri, 28 Oct 2011, Ivan Zolotukhin wrote: > > > > > > > On Fri, Oct 28, 2011, Mark Taylor wrote: > > > > > The Schlegel thing is there but experimental and not documented; > > > > > if you want to use it I can tell you how. > > > > > > > > I personally don't need that at the moment, but as I promised there's > > > > strong interest from the user community, even in this thread, so I'd > > > > suggest you to briefly describe things on the mailing list, so that > > > > folks can start testing. > > > > > > This is the story. Following a request from Ivan a while back, > > > I have implemented within STILTS (though currently not TOPCAT) a > > capability > > > to query the Galactic Dust Reddening and Extinction service hosted > > > at IPAC (http://irsa.ipac.caltech.edu/applications/DUST/). > > > This allows you to add columns for E(B-V) Reddening, > > > 100 Micron Emission, and Dust Temperature to tables which already > > > have sky positions - it's a bit like a multi-cone search, but not quite. > > > > > > This means it will be easy for people to fire off lots and lots of > > > queries (one per row) to IPAC, and I don't want to annoy IPAC or > > > their web servers, so I've asked them how they feel about it > > > before publicising this in the interface. If they say it's OK > > > I'll pass it on. > > > > I've talked to the nice people at IPAC and they are fine about it, > > so if you want to use STILTS to add some or all of E(B-V) Reddening, > > 100 Micron Emission and Dust Temperature, you can do it by using > > a filter (http://www.starlink.ac.uk/stilts/sun256/filterSteps.html) > > with the catchy name of "uk.ac.starlink.ttools.calc.SchlegelFilter". > > > > Its usage is (as determined from running it with the wrong arguments) is: > > > > % stilts tpipe in=/mbt/data/table/messier.xml cmd='head 5' > > cmd='uk.ac.starlink.ttools.calc.SchlegelFilter -help' > > Error: No ra/dec specified > > > > Command was: uk.ac.starlink.ttools.calc.SchlegelFilter -help > > > > Usage: uk.ac.starlink.ttools.calc.SchlegelFilter > > [-results (reddening|emission|temperature)[,...]] > > [-stats (mean|ref_pixel|std|max|min)[,...]] > > > > > > i.e. you need to give the names of RA and Dec columns from the input > > table, and you can optionally specify which output columns you want. > > Simple usage might thus look something like: > > > > stilts tpipe in=cat.xml > > cmd='uk.ac.starlink.ttools.calc.SchlegelFilter RAJ2000 > > DEJ2000' > > out=catdust.xml > > > > The source of the data is the service documented here: > > > > http://irsa.ipac.caltech.edu/applications/DUST/ > > > > see instructions there for information on how to acknowledge this > > servie if it you use it for publication. > > > > The filter is not particularly fast (mostly my fault not IPAC's)! > > I suggest cmd=progress ahead of the > > cmd=uk.ac.starlink.ttools.calc.SchlegelFilter argument on the > > command line so you can see how it's getting on. > > If this feature becomes a documented part of the package I will > > hopefully speed it up and of course give it a snappier name. > > > > > In the mean time, I'm wondering if there are similar but different > > > services along the same lines I should provide. It's not feasible > > > to do the calculations within the TOPCAT directly, > > > since the data files are a few tens of Mb per statistic, which is > > > too much to bundle with the application. Are there other web > > > services which provide this sort of information that people would > > > like to see available from TOPCAT/STILTS? > > > > > > Alternatively, there seem to be a number of all-sky maps containing > > > various data available as FITS files out there. Schlegel/Finkbeiner > > > at http://astro.berkeley.edu/~marc/dust/ is one source; > > > LAMBDA (http://lambda.gsfc.nasa.gov/) is another which has loads > > > of goodies (WMAP, HI, H-alpha, some of the Schlegel data) > > > in HEALPix format. So another possibility would be a function that > > > allowed TOPCAT/STILTS users to download one of these HEALPix files > > > locally and query it by position (value at, or averaged round, a > > > given sky position) to add new columns to a table which has sky > > > positions. Compared to the web service option this would give > > > more flexibility, be considerably faster, and avoid the problem > > > of overloading remote web servers. I quite like this idea > > > because it's efficient and opens up a lot of possibilities, > > > but if the Schlegel extinction data is really the only dataset > > > that people are going to be interested in, then it may not be > > > worth the effort. > > > > Following some of the feedback I've had, I will think about something > > along these lines for the future as well. > > > > Mark > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/ > > > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From odysseus9672@gmail.XXX Wed Sep 5 19:07:08 2012 Date: Wed, 5 Sep 2012 10:06:21 -0700 From: Sean Lake To: topcat-user@sympa.XXX Subject: Another IPAC tbl Bug Hello all, Topcat is not parsing a particular variant of the IPAC tbl format correctly. This variant uses hyphens ( - ) in the header lines instead of spaces ( ). From the definition: http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html Dashes can be used in place of the spaces in the header: |-----ra---|----dec---|---sao---|------v---|----sptype--------| 2.09708 29.09056 73765 2.06000 B8IVpMnHg The exact tables I'm trying to load are available from: http://irsa.ipac.caltech.edu/data/SPITZER/SDWFS/catalogs/combined_epochs/ Thanks, Sean From m.b.taylor@bristol.XXX Wed Sep 5 22:44:30 2012 Date: Wed, 5 Sep 2012 22:44:30 +0100 (BST) From: Mark Taylor To: Sean Lake Cc: topcat-user@sympa.XXX Subject: Re: Another IPAC tbl Bug On Wed, 5 Sep 2012, Sean Lake wrote: > Hello all, > > Topcat is not parsing a particular variant of the IPAC tbl format correctly. This variant uses hyphens ( - ) in the header lines instead of spaces ( ). From the definition: http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html > > Dashes can be used in place of the spaces in the header: > > |-----ra---|----dec---|---sao---|------v---|----sptype--------| > 2.09708 29.09056 73765 2.06000 B8IVpMnHg > > The exact tables I'm trying to load are available from: http://irsa.ipac.caltech.edu/data/SPITZER/SDWFS/catalogs/combined_epochs/ OK fixed in pre-release version: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ I wonder whether there's any good reason they introduced this variant - I'm pretty sure the format definition didn't used to allow dashes there. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From tgoldina@gmail.XXX Wed Sep 5 23:28:24 2012 Date: Wed, 5 Sep 2012 15:28:05 -0700 From: Tatiana Goldina To: Mark Taylor Cc: Sean Lake , topcat-user@sympa.XXX Subject: Re: Another IPAC tbl Bug Dashes are convenient to see spaces. But here at IPAC, we have forgotten this part of the specs too. Most of Spitzer software and Gator do not support it. We are considering removing dashes from the format definition. Sincerely, Tatiana Goldina ------------------------------------------------------------------ Programmer at Spitzer Science Center, IPAC On Wed, Sep 5, 2012 at 2:44 PM, Mark Taylor wrote: > On Wed, 5 Sep 2012, Sean Lake wrote: > > > Hello all, > > > > Topcat is not parsing a particular variant of the IPAC tbl format > correctly. This variant uses hyphens ( - ) in the header lines instead of > spaces ( ). From the definition: > http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html > > > > Dashes can be used in place of the spaces in the header: > > > > |-----ra---|----dec---|---sao---|------v---|----sptype--------| > > 2.09708 29.09056 73765 2.06000 B8IVpMnHg > > > > The exact tables I'm trying to load are available from: > http://irsa.ipac.caltech.edu/data/SPITZER/SDWFS/catalogs/combined_epochs/ > > OK fixed in pre-release version: > > ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ > > I wonder whether there's any good reason they introduced this variant - > I'm pretty sure the format definition didn't used to allow dashes there. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ > > From S.Oliver@sussex.XXX Wed Sep 5 23:45:45 2012 Date: Wed, 5 Sep 2012 23:45:28 +0100 From: Seb Oliver To: Tatiana Goldina Cc: Mark Taylor , Sean Lake , topcat-user@sympa.XXX Subject: Re: Another IPAC tbl Bug sounds like a case for some pre-processing with good old fashioned awk. I reckon you could change those "-" into " " with a 1-line awk command. Does anyone still use awk? Probably not, I think I am now officially an old fuddy duddy. On 5 Sep 2012, at 23:28, Tatiana Goldina wrote: > Dashes are convenient to see spaces. But here at IPAC, we have forgotten this part of the specs too. Most of Spitzer software and Gator do not support it. We are considering removing dashes from the format definition. > > Sincerely, > > Tatiana Goldina > ------------------------------------------------------------------ > Programmer at Spitzer Science Center, IPAC > > On Wed, Sep 5, 2012 at 2:44 PM, Mark Taylor wrote: > On Wed, 5 Sep 2012, Sean Lake wrote: > > > Hello all, > > > > Topcat is not parsing a particular variant of the IPAC tbl format correctly. This variant uses hyphens ( - ) in the header lines instead of spaces ( ). From the definition: http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html > > > > Dashes can be used in place of the spaces in the header: > > > > |-----ra---|----dec---|---sao---|------v---|----sptype--------| > > 2.09708 29.09056 73765 2.06000 B8IVpMnHg > > > > The exact tables I'm trying to load are available from: http://irsa.ipac.caltech.edu/data/SPITZER/SDWFS/catalogs/combined_epochs/ > > OK fixed in pre-release version: > > ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ > > I wonder whether there's any good reason they introduced this variant - > I'm pretty sure the format definition didn't used to allow dashes there. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ > > From m.b.taylor@bristol.XXX Thu Sep 6 00:20:49 2012 Date: Thu, 6 Sep 2012 00:20:48 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Another IPAC tbl Bug awk's OK but why resort to new-fangled technology? sed '/^|/s/-/ /g' will do the trick. On Wed, 5 Sep 2012, Seb Oliver wrote: > sounds like a case for some pre-processing with good old fashioned awk. I reckon you could change those "-" into " " with a 1-line awk command. > > Does anyone still use awk? Probably not, I think I am now officially an old fuddy duddy. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Thu Sep 6 09:59:21 2012 Date: Thu, 6 Sep 2012 10:58:22 +0200 From: Maciej Bilicki To: Seb Oliver Cc: Tatiana Goldina , Mark Taylor , Sean Lake , topcat-user@sympa.XXX Subject: Re: Another IPAC tbl Bug I admit having used awk more than once or twice! May I join the club now? Maciek 2012/9/6 Seb Oliver > sounds like a case for some pre-processing with good old fashioned awk. I > reckon you could change those "-" into " " with a 1-line awk command. > > Does anyone still use awk? Probably not, I think I am now officially an > old fuddy duddy. > > On 5 Sep 2012, at 23:28, Tatiana Goldina wrote: > > Dashes are convenient to see spaces. But here at IPAC, we have forgotten > this part of the specs too. Most of Spitzer software and Gator do not > support it. We are considering removing dashes from the format definition. > > Sincerely, > > Tatiana Goldina > ------------------------------------------------------------------ > Programmer at Spitzer Science Center, IPAC > > On Wed, Sep 5, 2012 at 2:44 PM, Mark Taylor wrote: > >> On Wed, 5 Sep 2012, Sean Lake wrote: >> >> > Hello all, >> > >> > Topcat is not parsing a particular variant of the IPAC tbl format >> correctly. This variant uses hyphens ( - ) in the header lines instead of >> spaces ( ). From the definition: >> http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html >> > >> > Dashes can be used in place of the spaces in the header: >> > >> > |-----ra---|----dec---|---sao---|------v---|----sptype--------| >> > 2.09708 29.09056 73765 2.06000 B8IVpMnHg >> > >> > The exact tables I'm trying to load are available from: >> http://irsa.ipac.caltech.edu/data/SPITZER/SDWFS/catalogs/combined_epochs/ >> >> OK fixed in pre-release version: >> >> ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ >> >> I wonder whether there's any good reason they introduced this variant - >> I'm pretty sure the format definition didn't used to allow dashes there. >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ >> >> > > From alexandra.truebenbach@colorado.XXX Thu Oct 4 18:03:27 2012 Date: Thu, 04 Oct 2012 17:02:32 +0100 From: alexandra.truebenbach@colorado.XXX To: topcat-user@sympa.XXX Subject: STILTS coneskymatch doesn't have find parameter Hi all, I'm interested in using coneskymatch to create a list of objects that are in my input table but not in a specified online cone service. For example, I'd like to do a search like: stilts coneskymatch \ ifmt=ascii in=first_test_mod.tbl ofmt=csv out=test.tbl \ ra='RA_deg' dec='Dec_deg' sr=1.0 \ serviceurl=http://archive.stsci.edu/hst/search.php \ join=1not2 However, there doesn't seem to be any sort of join parameter for this command. Do you have any recommendations/fixes for this problem? Thanks! From m.b.taylor@bristol.XXX Thu Oct 4 21:52:03 2012 Date: Thu, 4 Oct 2012 21:52:02 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX alexandra.truebenbach@colorado.XXX Subject: Re: STILTS coneskymatch doesn't have find parameter On Thu, 4 Oct 2012, alexandra.truebenbach@colorado.XXX wrote: > Hi all, > > I'm interested in using coneskymatch to create a list of objects that are in my > input table but not in a specified online cone service. For example, I'd like > to do a search like: > > stilts coneskymatch \ > ifmt=ascii in=first_test_mod.tbl ofmt=csv out=test.tbl \ > ra='RA_deg' dec='Dec_deg' sr=1.0 \ > serviceurl=http://archive.stsci.edu/hst/search.php \ > join=1not2 > > However, there doesn't seem to be any sort of join parameter for this command. Hmm, I see what you mean. I'll add it to the to-do list. In the mean time, one way to get that result is to do a coneskymatch first and follow it with a suitable match against the original table, keyed on an identifier like the row index. If your original table is in.csv, first you want to add a unique identifier to each row. If it already has one, you can use that. If not, add a column named "ID" with the value of the row number, which you can get using the magic value $0: stilts tpipe in=in.csv ifmt=csv cmd='addcol ID $0' out=in1.csv then do the coneskymatch: stilts coneskymatch in=in1.csv ifmt=csv \ ra=RA_deg dec=Dec_deg sr=1 \ serviceurl=http://archive.stsci.edu/hst/search.php \ out=coned.csv then match that against the input table with 2not1, doing an "exact"-type match with the ID value as the key: stilts tmatch2 in1=in1.csv in2=coned.csv ifmt1=csv ifmt2=csv \ matcher=exact values1=ID values2=ID join=1not2 \ progress=none out=not_coned.csv Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Thu Oct 4 22:08:52 2012 Date: Thu, 4 Oct 2012 22:08:51 +0100 (BST) From: Mark Taylor To: topcat-user@bristol.XXX alexandra.truebenbach@colorado.XXX Subject: Re: STILTS coneskymatch doesn't have find parameter On Thu, 4 Oct 2012, Mark Taylor wrote: > On Thu, 4 Oct 2012, alexandra.truebenbach@colorado.XXX wrote: > > > Hi all, > > > > I'm interested in using coneskymatch to create a list of objects that are in my > > input table but not in a specified online cone service. For example, I'd like > > to do a search like: > > > > stilts coneskymatch \ > > ifmt=ascii in=first_test_mod.tbl ofmt=csv out=test.tbl \ > > ra='RA_deg' dec='Dec_deg' sr=1.0 \ > > serviceurl=http://archive.stsci.edu/hst/search.php \ > > join=1not2 > > > > However, there doesn't seem to be any sort of join parameter for this command. > > Hmm, I see what you mean. I'll add it to the to-do list. In the mean > time, one way to get that result is to do a coneskymatch first > and follow it with a suitable match against the original table, > keyed on an identifier like the row index. Pah! Don't do it like that, there's a simpler way. If you use find=each, you will get exactly one output row for each input row. The ones without a match will have empty entries for the columns from the cone service, and also for the special "Separation" column which says how far away the input object was from the one found by the service. So, you just need to extract those rows with a blank value in the Separation column in the coneskymatch output. This is how: stilts coneskymatch in=in.csv ifmt=csv ra=RA_deg dec=Dec_deg \ serviceurl=http://archive.stsci.edu/hst/search.php \ sr=1 find=each \ ocmd='select NULL_Separation' out=nc.csv This will give you a table with a load of empty columns. You can get rid of those by adding the following parameters to the above command line: fixcols=all suffix0='' suffix1='_cone' \ ocmd='delcols "*_cone Separation"' I'll try to read the manual before answering next time :-). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX http://www.star.bris.ac.uk/~mbt/ From nestor.espinozap@gmail.XXX Sun Oct 14 00:08:10 2012 Date: Sat, 13 Oct 2012 18:07:28 -0400 From: Néstor Espinoza To: topcat-user@bristol.XXX Subject: Problem installing TOPCAT on Ubuntu-Linux Dear all, I'm trying to install TOPCAT on Ubuntu 12.04.1 LTS. I followed the instructions here: http://www.star.bris.ac.uk/~mbt/topcat/#install. First, I downloaded topcat-full.jar and the topcat shell script, then I did a "chmod +x topcat" and then ran the topcat shell script by doing ". topcat". However, I get the following message: Can't find topcat classes in /bin - looked for: ../lib/topcat/topcat.jar topcat-full.jar topcat-lite.jar However, I have the topcat-full.jar and topcat scripts on the same folder! I also tried to create lib folders just outside the folder where these files are, moving the topcat-full.jar file there but I get the same error. I also tried uncompressing the topcat-full.jar, and the same error. Finally, I tried downloading the full StarJava Installation, uncompressed it, entered and tried to run topcat from the script that is inside ( starjava/bin/topcat) but, again, got the same error. For completeness, when I do "java -version" on Terminal I get: java version "1.6.0_24" OpenJDK Runtime Environment (IcedTea6 1.11.4) (6b24-1.11.4-1ubuntu0.12.04.1) OpenJDK Client VM (build 20.0-b12, mixed mode, sharing) -- Néstor From nestor.espinozap@gmail.XXX Sun Oct 14 00:23:15 2012 Date: Sat, 13 Oct 2012 18:22:32 -0400 From: Néstor Espinoza To: topcat-user@bristol.XXX Subject: Re: Problem installing TOPCAT on Ubuntu-Linux Dear all, Nevermind, already solved it. It was just an user permission problem. 2012/10/13 Néstor Espinoza > Dear all, > > I'm trying to install TOPCAT on Ubuntu 12.04.1 LTS. I followed the > instructions here: http://www.star.bris.ac.uk/~mbt/topcat/#install. > First, I downloaded topcat-full.jar and the topcat shell script, then I did > a "chmod +x topcat" and then ran the topcat shell script by doing ". > topcat". However, I get the following message: > > Can't find topcat classes in /bin - looked for: ../lib/topcat/topcat.jar > topcat-full.jar topcat-lite.jar > > However, I have the topcat-full.jar and topcat scripts on the same folder! > I also tried to create lib folders just outside the folder where these > files are, moving the topcat-full.jar file there but I get the same error. > I also tried uncompressing the topcat-full.jar, and the same error. > Finally, I tried downloading the full StarJava Installation, uncompressed > it, entered and tried to run topcat from the script that is inside ( > starjava/bin/topcat) but, again, got the same error. > > For completeness, when I do "java -version" on Terminal I get: > > java version "1.6.0_24" > OpenJDK Runtime Environment (IcedTea6 1.11.4) > (6b24-1.11.4-1ubuntu0.12.04.1) > OpenJDK Client VM (build 20.0-b12, mixed mode, sharing) > > > -- > Néstor > -- Néstor From august.fly@gmail.XXX Tue Oct 23 18:49:57 2012 Date: Tue, 23 Oct 2012 12:49:09 -0400 From: "August (Gus) Muench" To: topcat-user@bristol.XXX Subject: Registry maheym, WISE. Version: TOPCAT Pre-release topcat-full.jar file from ftp:// (Dated: 9/27/2012) and tagged “TOPCAT Version 3.9+, STIL Version 3.0-3+, Starjava version 10308. Registry Mayhem. There are four registry entries for VO Cone searches: http://registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 (default) http://alt.registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 http://nvo.stsci.edu/vor10/ristandardservice.asmx http://registry.euro-vo.org/services/RegistrySearch Please not that #3 item *should actually be updated to*: http://vao.stsci.edu/directory/ristandardservice.asmx I am searching for: ivo://CDS.VizieR/II/311#wise on behalf of some folks on Twitter who asked me if I could figure out what is up. I cannot. :-) I know the service is live because I can find it on the Vizier page, and I can find it via Vizier Catalog search in TOPCAT. Further, the VAO Data Discovery Tool (http://vao.stsci.edu/portal/Mashup/Clients/Portal/DataDiscovery.html?useAV), queries it with no problem. Yet lookin for “WISE” under the “VO Cone search” option in TOPCAT *fails* for all four(five) registries in different ways: registry.astrogrid fails for all queries (dead?) alt.registry.astrogrid occasionally works, finds WISE related results (3) but NOT that entry nvo.stsci.edu (or vao.stsci.edu) fails for all queries (unexpected response?) registry.euro finds “WISE” results (5) but not that entry Of course that result (ivo://CDS.VizieR/II/311#wise) is in the http://vao.stsci.edu/directory/ristandardservice.asmx registry (http://vao.stsci.edu/directory/getRecord.aspx?id=ivo://CDS.VizieR/II/311 and try a search here: http://vao.stsci.edu/directory/keywordsearch.aspx ), which explains why the VAO DDT successfully finds WISE results. Yet as I said before TOPCAT isn’t being successful accessing the VAO registry at all. Why is that, anyway? What would it take for the vao.stsci.edu to become the default for TOPCAT? Thanks, - Gus -- August (Gus) Muench august.fly@gmail.XXX @augustmuench From bilicki@camk.XXX Tue Oct 23 18:43:45 2012 Date: Tue, 23 Oct 2012 19:42:46 +0200 From: Maciej Bilicki To: "August (Gus) Muench" Cc: topcat-user@bristol.XXX Subject: Re: Registry maheym, WISE. Dear August, Do I understand well that you're trying to get WISE data? I don't know if it's related, but I've been some having problems with getting WISE data straight from Vizier via web interface. I prefer to use IRSA Gator instead. Note that the WISE dataset is huge. The source catalog without images is 50 bzipped files 4.6 GB _each_ Best, Maciej 2012/10/23 August (Gus) Muench > Version: TOPCAT Pre-release topcat-full.jar file from ftp:// (Dated: > 9/27/2012) and tagged “TOPCAT Version 3.9+, STIL Version 3.0-3+, > Starjava version 10308. > > Registry Mayhem. > > There are four registry entries for VO Cone searches: > > http://registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 > (default) > > http://alt.registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 > http://nvo.stsci.edu/vor10/ristandardservice.asmx > http://registry.euro-vo.org/services/RegistrySearch > > Please not that #3 item *should actually be updated to*: > > http://vao.stsci.edu/directory/ristandardservice.asmx > > I am searching for: ivo://CDS.VizieR/II/311#wise on behalf of some > folks on Twitter who asked me if I could figure out what is up. I > cannot. :-) > > I know the service is live because I can find it on the Vizier page, > and I can find it via Vizier Catalog search in TOPCAT. Further, the > VAO Data Discovery Tool > ( > http://vao.stsci.edu/portal/Mashup/Clients/Portal/DataDiscovery.html?useAV > ), > queries it with no problem. > > Yet lookin for “WISE” under the “VO Cone search” option in TOPCAT > *fails* for all four(five) registries in different ways: > > registry.astrogrid fails for all queries (dead?) > alt.registry.astrogrid occasionally works, finds WISE related results > (3) but NOT that entry > nvo.stsci.edu (or vao.stsci.edu) fails for all queries (unexpected > response?) > registry.euro finds “WISE” results (5) but not that entry > > Of course that result (ivo://CDS.VizieR/II/311#wise) is in the > http://vao.stsci.edu/directory/ristandardservice.asmx registry > (http://vao.stsci.edu/directory/getRecord.aspx?id=ivo://CDS.VizieR/II/311 > and try a search here: > http://vao.stsci.edu/directory/keywordsearch.aspx ), which explains > why the VAO DDT successfully finds WISE results. > > Yet as I said before TOPCAT isn’t being successful accessing the VAO > registry at all. Why is that, anyway? What would it take for the > vao.stsci.edu to become the default for TOPCAT? > > Thanks, > > - Gus > > > -- > August (Gus) Muench > august.fly@gmail.XXX > @augustmuench > > From august.fly@gmail.XXX Tue Oct 23 19:56:43 2012 Date: Tue, 23 Oct 2012 13:56:07 -0400 From: "August (Gus) Muench" To: Maciej Bilicki Cc: topcat-user@bristol.XXX Subject: Re: Registry maheym, WISE. Thanks, Maciej, I think my concern is more narrowly focused on the problem with any/all VO registries currently being used by TOPCAT. They are all wrong. I was informed off thread in the past few minutes that at least the failure of the VAO registry is known: http://www.star.bris.ac.uk/~mbt/topcat/faq.html#Q4.1 this does not explain why the astrogrid registries are also dead/dying (but that is also effectively known) and euro-vo is not updated (no clue). given these circumstances we are kinda left in the lurch or as my offline contact said, 'death by single point failure'. seems to me someone ought to adapt/evolve here (and that may well be my colleagues). I think that any problems specific to Vizier should probably warrant a distinct ticket/conversation. The VAO DDT tool clearly shows that (today, a few moments ago), Vizier+WISE was working fine, as does the Vizier Catalog service in TOPCAT. Thanks, - Gus On Tue, Oct 23, 2012 at 1:42 PM, Maciej Bilicki wrote: > Dear August, > > Do I understand well that you're trying to get WISE data? > I don't know if it's related, but I've been some having problems with > getting WISE data straight from Vizier via web interface. I prefer to use > IRSA Gator instead. > Note that the WISE dataset is huge. The source catalog without images is 50 > bzipped files 4.6 GB _each_ > > Best, > Maciej > > 2012/10/23 August (Gus) Muench > >> Version: TOPCAT Pre-release topcat-full.jar file from ftp:// (Dated: >> 9/27/2012) and tagged “TOPCAT Version 3.9+, STIL Version 3.0-3+, >> Starjava version 10308. >> >> Registry Mayhem. >> >> There are four registry entries for VO Cone searches: >> >> >> http://registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 >> (default) >> >> http://alt.registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 >> http://nvo.stsci.edu/vor10/ristandardservice.asmx >> http://registry.euro-vo.org/services/RegistrySearch >> >> Please not that #3 item *should actually be updated to*: >> >> http://vao.stsci.edu/directory/ristandardservice.asmx >> >> I am searching for: ivo://CDS.VizieR/II/311#wise on behalf of some >> folks on Twitter who asked me if I could figure out what is up. I >> cannot. :-) >> >> I know the service is live because I can find it on the Vizier page, >> and I can find it via Vizier Catalog search in TOPCAT. Further, the >> VAO Data Discovery Tool >> >> (http://vao.stsci.edu/portal/Mashup/Clients/Portal/DataDiscovery.html?useAV), >> queries it with no problem. >> >> Yet lookin for “WISE” under the “VO Cone search” option in TOPCAT >> *fails* for all four(five) registries in different ways: >> >> registry.astrogrid fails for all queries (dead?) >> alt.registry.astrogrid occasionally works, finds WISE related results >> (3) but NOT that entry >> nvo.stsci.edu (or vao.stsci.edu) fails for all queries (unexpected >> response?) >> registry.euro finds “WISE” results (5) but not that entry >> >> Of course that result (ivo://CDS.VizieR/II/311#wise) is in the >> http://vao.stsci.edu/directory/ristandardservice.asmx registry >> (http://vao.stsci.edu/directory/getRecord.aspx?id=ivo://CDS.VizieR/II/311 >> and try a search here: >> http://vao.stsci.edu/directory/keywordsearch.aspx ), which explains >> why the VAO DDT successfully finds WISE results. >> >> Yet as I said before TOPCAT isn’t being successful accessing the VAO >> registry at all. Why is that, anyway? What would it take for the >> vao.stsci.edu to become the default for TOPCAT? >> >> Thanks, >> >> - Gus >> >> >> -- >> August (Gus) Muench >> august.fly@gmail.XXX >> @augustmuench >> > -- August (Gus) Muench august.fly@gmail.XXX @augustmuench From m.b.taylor@bristol.XXX Wed Oct 24 15:18:31 2012 Date: Wed, 24 Oct 2012 15:18:30 +0100 (BST) From: Mark Taylor To: "August (Gus) Muench" Cc: topcat-user@bristol.XXX Subject: Re: Registry maheym, WISE. Gus, registry provision is not great. The AstroGrid registries are maintained on a best efforts basis by former AstroGrid staff, but that project has been unfunded for several years. The Euro-VO registry is also suffering from poor institutional support, though I heard a rumour that might improve. The VAO registry as per that entry from the TOPCAT FAQ has for whatever reason not prioritised this issue over the last couple of years. I'm currently sitting in a talk suggesting that they are now planning to do something about it (thanks to complaints from a US-based project, I think). So possibly that will become fixed. Discussions are going on in the IVOA to rework registry provision altogether; it is acknowledged that there are problems that need solving. So this situation may change over the medium-term future. It may even improve :-). I'm considering the way TOPCAT does/should/will cope with all this, but currently it's not top of my to-do list (though I do agree it's important). Finally: I talked to the AstroGrid reg people today and as of now, that service appears to be working correctly (though it's possible that the content is not bang up to date, I think that may resolve itself as it performs updates over the next day or so(?)). Mark On Tue, 23 Oct 2012, August (Gus) Muench wrote: > Thanks, Maciej, > > I think my concern is more narrowly focused on the problem with > any/all VO registries currently being used by TOPCAT. They are all > wrong. I was informed off thread in the past few minutes that at least > the failure of the VAO registry is known: > > http://www.star.bris.ac.uk/~mbt/topcat/faq.html#Q4.1 > > this does not explain why the astrogrid registries are also dead/dying > (but that is also effectively known) and euro-vo is not updated (no > clue). given these circumstances we are kinda left in the lurch or as > my offline contact said, 'death by single point failure'. seems to me > someone ought to adapt/evolve here (and that may well be my > colleagues). > > I think that any problems specific to Vizier should probably warrant a > distinct ticket/conversation. The VAO DDT tool clearly shows that > (today, a few moments ago), Vizier+WISE was working fine, as does the > Vizier Catalog service in TOPCAT. > > Thanks, > > - Gus > > > > On Tue, Oct 23, 2012 at 1:42 PM, Maciej Bilicki wrote: > > Dear August, > > > > Do I understand well that you're trying to get WISE data? > > I don't know if it's related, but I've been some having problems with > > getting WISE data straight from Vizier via web interface. I prefer to use > > IRSA Gator instead. > > Note that the WISE dataset is huge. The source catalog without images is 50 > > bzipped files 4.6 GB _each_ > > > > Best, > > Maciej > > > > 2012/10/23 August (Gus) Muench > > > >> Version: TOPCAT Pre-release topcat-full.jar file from ftp:// (Dated: > >> 9/27/2012) and tagged “TOPCAT Version 3.9+, STIL Version 3.0-3+, > >> Starjava version 10308. > >> > >> Registry Mayhem. > >> > >> There are four registry entries for VO Cone searches: > >> > >> > >> http://registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 > >> (default) > >> > >> http://alt.registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 > >> http://nvo.stsci.edu/vor10/ristandardservice.asmx > >> http://registry.euro-vo.org/services/RegistrySearch > >> > >> Please not that #3 item *should actually be updated to*: > >> > >> http://vao.stsci.edu/directory/ristandardservice.asmx > >> > >> I am searching for: ivo://CDS.VizieR/II/311#wise on behalf of some > >> folks on Twitter who asked me if I could figure out what is up. I > >> cannot. :-) > >> > >> I know the service is live because I can find it on the Vizier page, > >> and I can find it via Vizier Catalog search in TOPCAT. Further, the > >> VAO Data Discovery Tool > >> > >> (http://vao.stsci.edu/portal/Mashup/Clients/Portal/DataDiscovery.html?useAV), > >> queries it with no problem. > >> > >> Yet lookin for “WISE” under the “VO Cone search” option in TOPCAT > >> *fails* for all four(five) registries in different ways: > >> > >> registry.astrogrid fails for all queries (dead?) > >> alt.registry.astrogrid occasionally works, finds WISE related results > >> (3) but NOT that entry > >> nvo.stsci.edu (or vao.stsci.edu) fails for all queries (unexpected > >> response?) > >> registry.euro finds “WISE” results (5) but not that entry > >> > >> Of course that result (ivo://CDS.VizieR/II/311#wise) is in the > >> http://vao.stsci.edu/directory/ristandardservice.asmx registry > >> (http://vao.stsci.edu/directory/getRecord.aspx?id=ivo://CDS.VizieR/II/311 > >> and try a search here: > >> http://vao.stsci.edu/directory/keywordsearch.aspx ), which explains > >> why the VAO DDT successfully finds WISE results. > >> > >> Yet as I said before TOPCAT isn’t being successful accessing the VAO > >> registry at all. Why is that, anyway? What would it take for the > >> vao.stsci.edu to become the default for TOPCAT? > >> > >> Thanks, > >> > >> - Gus > >> > >> > >> -- > >> August (Gus) Muench > >> august.fly@gmail.XXX > >> @augustmuench > >> > > > > > > -- > August (Gus) Muench > august.fly@gmail.XXX > @augustmuench > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Oct 29 15:32:16 2012 Date: Mon, 29 Oct 2012 15:32:15 +0000 (GMT) From: Mark Taylor To: "August (Gus) Muench" Cc: topcat-user@bristol.XXX Subject: Re: Registry maheym, WISE. On Thu, 25 Oct 2012, August (Gus) Muench wrote: > Thank you, Mark, esp for getting the registry.astrogrid instance back > up. With so many at IVOA the present is either a suboptimal time or a > perfect time to push on issues of registry search functionality. I > have always known that you maintain a well reasoned todo list wrt to > TOPCAT. At the same time, I remarked to someone yesterday that when > registry search dies, it is not simply analogous that the VO being > ill. When the registries die, the VO dies, IMHO. > > Yet I have to push on this some more because I continue to work with > users on Twitter, who lovingly discover new registry search > idiosynchronicities (in TOPCAT or elsewise). > > Consider the following: > > TOPCAT v3.9, Cone Search Services query using > http://registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 > > If you enter as a keyword search "SDSS" then the results FAIL to > include ivo://CDS.VizieR/II/294 or ivo://CDS.VizieR/II/306, although > the titles of those resources include "SDSS." A similar keyword > query of "II/294", for example, SUCCEEDS, correctly returning the > requested resource (perfect match on shortname). > > So is there some small metadata inconsistency that is causing this hiccup? I've investigated this and made a report to the AG people. I don't think it's a metadata hiccup, it looks like some paging or timeout issue (or maybe one of each) that means if a query would match large numbers of resources, then not all of the results are reported (and sometimes, none of them are). As you know AstroGrid registry maintenance is on a best-efforts basis, so we'll have to see whether a fix is forthcoming. I'll let you know what I hear. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From briceno@cida.XXX Thu Nov 8 16:14:04 2012 Date: Thu, 08 Nov 2012 11:13:00 -0500 From: Cesar Briceno To: topcat-user@sympa.XXX Subject: formatting columns when saving an ASCII table in Topcat Dear Mark and Topcat fans, Maybe this is a basic question, but I could not find the answer in the Topcat manual. Is there a way with Topcat to specify the output format when saving a table into ascii format? E.g. I would like to have a nice layout so that RA and DEC are both floating point with 7 decimal places (something like 10.7f, as you would in Perl or AWK), and magnitudes and their errors also in floating point but with 3 decimals (e.g. 8.3f), instead of some error values showing up as 8.745954564E-4! thanks! Cesar [ Part 2, Text/X-VCARD (Name: "briceno.vcf") ~541 bytes. ] [ Unable to print this part. ] From bilicki@camk.XXX Thu Nov 8 17:21:14 2012 Date: Thu, 8 Nov 2012 19:20:16 +0200 From: Maciej Bilicki To: Cesar Briceno Cc: topcat-user@sympa.XXX Subject: Re: formatting columns when saving an ASCII table in Topcat Dear Cesar, What I do to get an output format with n decimal places is roundDecimal(myQuantity,n) I suppose you have so many decimal places because of some conversions, is that so? Best, Maciek 2012/11/8 Cesar Briceno > > Dear Mark and Topcat fans, > > Maybe this is a basic question, but I could not find the answer in the > Topcat manual. > Is there a way with Topcat to specify the output format when saving a > table into ascii format? > E.g. I would like to have a nice layout so that RA and DEC are both > floating point with 7 decimal places > (something like 10.7f, as you would in Perl or AWK), and magnitudes and > their errors also in floating point but with 3 decimals (e.g. 8.3f), > instead of some error values showing up as 8.745954564E-4! > > thanks! > > Cesar > > > From m.b.taylor@bristol.XXX Thu Nov 8 19:06:46 2012 Date: Thu, 8 Nov 2012 19:06:46 +0000 (GMT) From: Mark Taylor To: Cesar Briceno Cc: topcat-user@sympa.XXX Subject: Re: formatting columns when saving an ASCII table in Topcat On Thu, 8 Nov 2012, Cesar Briceno wrote: > Dear Mark and Topcat fans, > > Maybe this is a basic question, but I could not find the answer in the Topcat > manual. > Is there a way with Topcat to specify the output format when saving a table > into ascii format? > E.g. I would like to have a nice layout so that RA and DEC are both floating > point with 7 decimal places > (something like 10.7f, as you would in Perl or AWK), and magnitudes and their > errors also in floating point but with 3 decimals (e.g. 8.3f), instead of some > error values showing up as 8.745954564E-4! Basic question, basic answer: No. Better go to awk/perl/python/etc for that part of the processing. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From briceno@cida.XXX Thu Nov 8 19:58:37 2012 Date: Thu, 08 Nov 2012 14:58:28 -0500 From: Cesar Briceno To: Mark Taylor Cc: topcat-user@mlist2.XXX Subject: Re: formatting columns when saving an ASCII table in Topcat Hi Mark, thanks. Yes, I regularly use AWK for that. I just wanted to make sure I was not missing anything. However, it would be a great feature to have in Topcat/STILTS. best regards, Cesar On 11/08/2012 02:06 PM, Mark Taylor wrote: > On Thu, 8 Nov 2012, Cesar Briceno wrote: > >> Dear Mark and Topcat fans, >> >> Maybe this is a basic question, but I could not find the answer in the Topcat >> manual. >> Is there a way with Topcat to specify the output format when saving a table >> into ascii format? >> E.g. I would like to have a nice layout so that RA and DEC are both floating >> point with 7 decimal places >> (something like 10.7f, as you would in Perl or AWK), and magnitudes and their >> errors also in floating point but with 3 decimals (e.g. 8.3f), instead of some >> error values showing up as 8.745954564E-4! > Basic question, basic answer: No. Better go to awk/perl/python/etc > for that part of the processing. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ [ Part 2: "Attached Text" ] [ The following attachment was DELETED when this message was saved: ] [ A Text/X-VCARD (Name="briceno.vcf") segment of about 526 bytes. ] From cbaorion2008@gmail.XXX Thu Nov 8 20:35:50 2012 Date: Thu, 08 Nov 2012 14:59:00 -0500 From: Cesar Briceño To: topcat-user@sympa.XXX Subject: Re: formatting columns when saving an ASCII table in Topcat Hi Maciek, thanks for the suggestion! I will certainly try it! cheers, Cesar On 11/08/2012 12:20 PM, Maciej Bilicki wrote: > Dear Cesar, > > What I do to get an output format with n decimal places is > roundDecimal(myQuantity,n) > > I suppose you have so many decimal places because of some conversions, > is that so? > > Best, > Maciek > > > > 2012/11/8 Cesar Briceno > > > > Dear Mark and Topcat fans, > > Maybe this is a basic question, but I could not find the answer in > the Topcat manual. > Is there a way with Topcat to specify the output format when > saving a table into ascii format? > E.g. I would like to have a nice layout so that RA and DEC are > both floating point with 7 decimal places > (something like 10.7f, as you would in Perl or AWK), and > magnitudes and their errors also in floating point but with 3 > decimals (e.g. 8.3f), instead of some error values showing up as > 8.745954564E-4! > > thanks! > > Cesar > > > -- ******************************************* * César Briceño * * ***************************************** * CIDA * * Av. Alberto Carnevali, via La Hechicera * * Mérida 5101, Venezuela * * Email: briceno@cida.XXX * * cbriceno@umich.XXX * * cbaorion2008@gmail.XXX * ******************************************* From Hauser@ARI.XXX Fri Nov 23 13:47:27 2012 Date: Fri, 23 Nov 2012 14:46:59 +0100 From: Marcus Hauser To: topcat-user@mlist2.XXX Subject: valid column names? Hi Mark et al, I've experienced problems when using the characters "[]-" as column names. Not for assigning the column name, but when using these names in expressions defining subsets. Is there an explicit reference somewhere what characters are valid in table column names? Secondly, there is an issue with long numbers: when selecting subsets on a column (here: "aaa" of type 'long') using a number with many digits, TOPCAT complains with an Expression Syntax Error: Syntax error in algebraic subset expression "aaa>77803200100352000": Encountered unexpected end of expression. Since I do not need the full resolution here, I can workaround by creating an auxiliary column "bbb" like "bbb=aaa/1.e9" and doing a select on "bbb>77803200", which is technically working. I this a know technical limitation, or some kind of bug? Regards, Marcus From m.b.taylor@bristol.XXX Fri Nov 23 13:58:30 2012 Date: Fri, 23 Nov 2012 13:58:30 +0000 (GMT) From: Mark Taylor To: Marcus Hauser Cc: topcat-user@bristol.XXX Subject: Re: valid column names? On Fri, 23 Nov 2012, Marcus Hauser wrote: > Hi Mark et al, > > I've experienced problems when using the characters "[]-" as column > names. Not for assigning the column name, but when using these names in > expressions defining subsets. > > Is there an explicit reference somewhere what characters are valid in > table column names? See this discussion: http://www.star.bristol.ac.uk/~mbt/topcat/sun253/jelCols.html It tells you both what is syntactically permitted in an algebraic expression (see the "By Name" section) and workarounds if you find yourself working with columns that don't follow these rules ("By $ID"). > Secondly, there is an issue with long numbers: when selecting subsets on > a column (here: "aaa" of type 'long') using a number with many digits, > TOPCAT complains with an > > Expression Syntax Error: Syntax error in algebraic subset expression > "aaa>77803200100352000": Encountered unexpected end of expression. > > Since I do not need the full resolution here, I can workaround by > creating an auxiliary column "bbb" like "bbb=aaa/1.e9" and doing a > select on "bbb>77803200", which is technically working. > > I this a know technical limitation, or some kind of bug? I think "gotcha" would be the technical term. The expression language uses Java syntax (for reasons beyond my control), and in Java, long integers need to have an "l" or "L" appended to them. Apologies for the obscureness. So "aaa>77803200100352000L" should work. Some other possibilites are: "aaa>77803200100352000.0" "aaa/1e9>77803200" "aaa>77803200e9" Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From gtr@physics.XXX Mon Nov 26 15:35:18 2012 Date: Mon, 26 Nov 2012 15:34:59 +0000 From: gtr@physics.XXX To: topcat-user@sympa.XXX Subject: Plotting array quantities If I have a FITS table that includes arrays (e.g., 5 SDSS mags in one column), is it possible to plot those values in TOPCAT? Right now when I try to make a plot, columns that are arrays don't even show up as a choice of values to plot. Thanks, Gordon From m.b.taylor@bristol.XXX Mon Nov 26 15:40:06 2012 Date: Mon, 26 Nov 2012 15:40:06 +0000 (GMT) From: Mark Taylor To: gtr@physics.XXX Cc: topcat-user@sympa.XXX Subject: Re: Plotting array quantities On Mon, 26 Nov 2012, gtr@physics.XXX wrote: > If I have a FITS table that includes arrays (e.g., 5 SDSS mags in one column), > is it possible to plot those values in TOPCAT? Right now when I try to > make a plot, columns that are arrays don't even show up as a choice of > values to plot. You can address these columns using array indices in square brackets: gmag[0] gmag[1] ... gmag[4] just type the relevant expression into the coordinate field in the plot window (or whereever), or you can create a new scalar column with that expression and then use that. An alternative for convenience is to use the Columns|Explode Array Column menu item (or equivalent toolbar button) in the Colums window: http://www.star.bristol.ac.uk/~mbt/topcat/sun253/ColumnInfoWindow.html which turns a highlighted array-valued column into N scalar ones - can save typing, but makes your wide table even wider than it was. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From chambers@ifa.XXX Wed Nov 28 09:24:31 2012 Date: Tue, 27 Nov 2012 14:03:58 -1000 From: Ken Chambers To: topcat-user@bristol.XXX Subject: setting aspect ratio for plots Is there a way to accurately set the aspect ratio for 2-d plots? This would be very useful for producing publication quality plots. Obviously one can do it roughly by sizing the window, but the best thing would be to specify it, especially in the Configure Axes and Title window. thanks for providing such a wonderful tool by the way!! Ken -- Dr. Kenneth C. Chambers (Ken) Director, PS1 System (Pan-STARRS Telescope #1) Institute for Astronomy, University of Hawaii Office: 808 956 9844 From m.b.taylor@bristol.XXX Wed Nov 28 10:41:11 2012 Date: Wed, 28 Nov 2012 10:41:10 +0000 (GMT) From: Mark Taylor To: Ken Chambers Cc: topcat-user@bristol.XXX Subject: Re: setting aspect ratio for plots On Tue, 27 Nov 2012, Ken Chambers wrote: > Is there a way to accurately set the aspect ratio for 2-d plots? > This would be very useful for producing publication quality plots. > Obviously one can do it roughly by sizing the window, but the best thing > would be to specify it, especially in the Configure Axes and Title window. I'm afraid you can't do that from TOPCAT. If you use STILTS plotting you can control this using the xpix and ypix parameters, though note that gives you the size of the full graphic rather than the region within the axes, so it's possible that several plots with the same xpix/ypix values would not look quite right next to each other. Publication-quality plots will be better supported in the next release. > thanks for providing such a wonderful tool by the way!! thanks for saying so! -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From maciek@ast.XXX Thu Nov 29 12:50:12 2012 Date: Thu, 29 Nov 2012 14:45:48 +0200 From: Maciej Bilicki To: Mark Taylor Cc: Ken Chambers , topcat-user@bristol.XXX Subject: Re: setting aspect ratio for plots Dear Mark, 2012/11/28 Mark Taylor > > > Publication-quality plots will be better supported in the next > release. > This is great news, I'm looking forward to it! > > > thanks for providing such a wonderful tool by the way!! > > thanks for saying so! > Let me reiterate Ken's thanks! Maciek > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > > From m.drinkwater@uq.XXX Thu Jan 10 07:28:47 2013 Date: Thu, 10 Jan 2013 07:28:24 +0000 From: Michael Drinkwater To: "topcat-user@bristol.XXX" Cc: Michael Drinkwater Subject: local file name format for Topcat Activation Action? Hi there, Is there a short way to specify local (relative) file names for the Activation Action? I have a table with the names of FITS spectrum files for each object (row). If I specify the full (absolute) path for each file in the input table then I can very nicely call up plots (using Starlink SPLAT) of selected objects from within TOPCAT. I would rather use some kind of relative path as the filename so that I can distribute the data (table plus FITS files) for others in my team to use without them having to edit the table to change the file paths. Any suggestions? regards, Michael Professor Michael Drinkwater School of Mathematics and Physics, University of Queensland, QLD 4072, Australia *** Currently on study leave until Jan 31, 2013 *** From m.b.taylor@bristol.XXX Thu Jan 10 10:36:11 2013 Date: Thu, 10 Jan 2013 10:36:10 +0000 (GMT) From: Mark Taylor To: Michael Drinkwater Cc: "topcat-user@bristol.XXX" Subject: Re: local file name format for Topcat Activation Action? Hi Michael. On Thu, 10 Jan 2013, Michael Drinkwater wrote: > Is there a short way to specify local (relative) file names for > the Activation Action? > > I have a table with the names of FITS spectrum files for each object > (row). If I specify the full (absolute) path for each file in the input > table then I can very nicely call up plots (using Starlink SPLAT) of > selected objects from within TOPCAT. I would rather use some kind of > relative path as the filename so that I can distribute the data (table > plus FITS files) for others in my team to use without them having to > edit the table to change the file paths. There is no strictly respectable way to get TOPCAT to send load-spectrum messages based on relative filenames in a table. The "View URL as Spectrum" activation action expects there to be a URL in the column you name, and it passes that in the URL field of the relevant SAMP message to SPLAT (or whoever else is listening). However, as it happens, both ends of that communication are a bit sloppy about the "URL", and if it happens to be a relative filename neither TOPCAT nor SPLAT will complain. So, to make it work, all you need to do is make sure that SPLAT's working directory is one from which the relative filenames make sense (note, TOPCAT's working directory makes no difference). The most straightforward way to do that is to start SPLAT from the directory in question (other ways include setting the system property user.dir from the command line). The same trick may or may not work for other spectrum viewers. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From sjkang@iastate.XXX Wed Jan 23 21:48:58 2013 Date: Wed, 23 Jan 2013 13:54:17 -0600 From: "Kang, Sung-Ju [PHYSA]" To: topcat-user@bristol.XXX Subject: Questions about matching files Dear whom it may concern, I am Sung-ju Kang who is a graduate student in Iowa State University in US. I have a quick question about merging(matching) tables. When I match the tables, I create new columns after match for example, I created the average RA from RA1 and RA2 but I wonder if there is no data in RA1, average RA is not created. If there is no data, I want average RA from only RA2. I am pretty sure there is a way to do it but I cannot figure it out. Can you help me this issue? Thanks Best Regards, Sung-ju Kang -- ------------------------------------------------------------ Sung-ju Kang From m.b.taylor@bristol.XXX Wed Jan 23 22:04:47 2013 Date: Wed, 23 Jan 2013 22:04:46 +0000 (GMT) From: Mark Taylor To: "Kang, Sung-Ju [PHYSA]" Cc: topcat-user@bristol.XXX Subject: Re: Questions about matching files On Wed, 23 Jan 2013, Kang, Sung-Ju [PHYSA] wrote: > Dear whom it may concern, > > I am Sung-ju Kang who is a graduate student in Iowa State University in US. > I have a quick question about merging(matching) tables. > When I match the tables, I create new columns after match > for example, I created the average RA from RA1 and RA2 > but I wonder if there is no data in RA1, average RA is not created. > If there is no data, I want average RA from only RA2. > I am pretty sure there is a way to do it but I cannot figure it out. there is a way to do it, but it's not so obvious. For most of the arithmetic operations in the expression language, if any of the inputs is blank, the output is blank too. So, as you've found out, an expression like (RA1+RA2)*0.5 will be blank if either RA1 or RA2 is blank. However some of the functions in the Arrays class (including mean) specifically look out for this and ignore blank values (http://www.star.bris.ac.uk/~mbt/topcat/sun253/Arrays.html). To use these you need to construct an array of the elements that you want to combine, which can be done with one of the array() functions. So, the answer is, use the expression: mean(array(RA1,RA2)) Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From elle@oal.XXX Fri Feb 15 14:40:21 2013 Date: Fri, 15 Feb 2013 14:39:51 +0000 From: elle To: topcat-user@bristol.XXX Subject: drawing equations on a plot Hi guys, maybe I already posted this question but I cannot find it in my mailbox. I have a 2D scatterplot and I have to draw a dashed line on it which is defined by an equation. How can I do that? Thanx. From m.b.taylor@bristol.XXX Fri Feb 15 14:57:16 2013 Date: Fri, 15 Feb 2013 14:57:15 +0000 (GMT) From: Mark Taylor To: elle Cc: topcat-user@bristol.XXX Subject: Re: drawing equations on a plot On Fri, 15 Feb 2013, elle wrote: > Hi guys, maybe I already posted this question but I cannot find it in my > mailbox. > I have a 2D scatterplot and I have to draw a dashed line on it which is > defined by an equation. > How can I do that? > Thanx. there's not a very elegant way of doing this, but it can be done. Add a new dataset (+ icon in control panel at bottom of plot). Use any old table with more than a few rows. For the X coordinate enter an expression that spans the range of the X axis in the coordinates you're looking at in the right order. Usually the easiest way to do this is to use the magic value "index" (a.k.a. $0), which is the row number of the table. If that spans a range which is too wide or too narrow, or if you need negative numbers, you can use a linear function of it (e.g. index*0.01). Then for the Y coordinate, enter an analytic expression based on it, e.g. sin(index*0.01). This will plot points giving that function. To turn it into a dotted line, open the relevant plot style editor (http://www.star.bristol.ac.uk/~mbt/topcat/sun253/MarkStyleEditor.html) and select "Hide Markers", "Dot to Dot", and a suitable Dash option. There will be a much nicer way of doing this in a (hopefully, near) future version of TOPCAT. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Mar 11 17:38:55 2013 Date: Mon, 11 Mar 2013 17:38:55 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Windows 8 On Mon, 4 Mar 2013, paul.rothnie@macquarie.XXX wrote: > Hi, > > I'm just about to purchase a new pc (laptop) and it's probably going to have > windows 8 on it. Has anyone had any experience with topcat and windows 8? Any > known issues? ... looks like the answer is no. All I'll say is I'd *expect* it to work as long as java is present on the machine. Let us know how you get on. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From frederic.meynadier@obspm.XXX Tue Mar 19 13:37:46 2013 Date: Tue, 19 Mar 2013 13:34:26 +0000 From: frederic.meynadier@obspm.XXX To: topcat-user@sympa.XXX Subject: Number of digits in topcat Hello, I would like to tweak the number of digits displayed in the table browser : one of the columns of my (ascii) catalogue is composed of floats that I want to display up to the 14th decimal digit. If I double-click on the cell, the full precision is displayed - but it would be really convenient for me to have it displayed directly for the whole column (and, more generally, set the display of any of my columns...) So, is there a mechanism to fix the display format of a column (for example, a C format string somewhere) ? Thanks for your help - and thanks for topcat ! -- Frédéric Meynadier From m.b.taylor@bristol.XXX Tue Mar 19 13:47:39 2013 Date: Tue, 19 Mar 2013 13:47:38 +0000 (GMT) From: Mark Taylor To: frederic.meynadier@obspm.XXX Cc: topcat-user@sympa.XXX Subject: Re: Number of digits in topcat On Tue, 19 Mar 2013, frederic.meynadier@obspm.XXX wrote: > Hello, > > I would like to tweak the number of digits displayed in the table browser : one > of the columns of my (ascii) catalogue is composed of floats that I want to > display up to the 14th decimal digit. If I double-click on the cell, the full > precision is displayed - but it would be really convenient for me to have it > displayed directly for the whole column (and, more generally, set the display > of any of my columns...) > > So, is there a mechanism to fix the display format of a column (for example, a > C format string somewhere) ? Sorry, currently there is not. One thing you can do is to add a new column (for display) which contains a string giving the value of an existing column formatted to your specifications. If your existing column is called XX, you can add (http://www.starlink.ac.uk/topcat/sun253/SyntheticColumnQueryWindow.html) a new column with the expression formatDecimal(XX, 14) See the Formats section in the Functions window (http://www.starlink.ac.uk/topcat/sun253/MethodWindow.html) for more details on how to use the variants of that formatting function. Bit of a hack, but better than nothing. Maybe I'll add configurable formatting of numeric columns for display one day. > Thanks for your help - and thanks for topcat ! You're welcome! Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From berthier@imcce.XXX Tue Mar 19 13:53:57 2013 Date: Tue, 19 Mar 2013 14:53:40 +0100 From: JBerthier To: topcat-user@sympa.XXX Subject: Re: Number of digits in topcat When loading votable, would it be possible that attributes such as width or precision are read to format numbers? cheers, jerome > On Tue, 19 Mar 2013, frederic.meynadier@obspm.XXX wrote: > >> Hello, >> >> I would like to tweak the number of digits displayed in the table browser : one >> of the columns of my (ascii) catalogue is composed of floats that I want to >> display up to the 14th decimal digit. If I double-click on the cell, the full >> precision is displayed - but it would be really convenient for me to have it >> displayed directly for the whole column (and, more generally, set the display >> of any of my columns...) >> >> So, is there a mechanism to fix the display format of a column (for example, a >> C format string somewhere) ? > Sorry, currently there is not. > > One thing you can do is to add a new column (for display) which contains > a string giving the value of an existing column formatted to your > specifications. If your existing column is called XX, you can add > (http://www.starlink.ac.uk/topcat/sun253/SyntheticColumnQueryWindow.html) > a new column with the expression > > formatDecimal(XX, 14) > > See the Formats section in the Functions window > (http://www.starlink.ac.uk/topcat/sun253/MethodWindow.html) > for more details on how to use the variants of that formatting function. > > Bit of a hack, but better than nothing. Maybe I'll add configurable > formatting of numeric columns for display one day. > z -- --- Dr. Jerome Berthier ------------------ Phone: +33 (0)14051 2261 -- Institut de mecanique celeste Fax: +33 (0)14633 2834 -- 77 av. Denfert Rochereau Mailto:jerome.berthier@imcce.XXX -- --- 75014 Paris - France --------------------- http://www.imcce.fr/ -- From m.b.taylor@bristol.XXX Thu Mar 21 14:04:18 2013 Date: Thu, 21 Mar 2013 14:04:16 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Number of digits in topcat On Tue, 19 Mar 2013, JBerthier wrote: > When loading votable, would it be possible that attributes such as width or > precision are read to format numbers? > cheers, > > jerome Hi Jerome, this obviously would be a logical thing to do; FITS has similar column attributes. I haven't implemented it up till now partly because the VOTable/FITS format for this metadata is FORTRAN-based, so implementation would probably be a bit fiddly, but mainly because (though I do not have, and have not attempted to gather, any evidence) I doubt that this information is present and correct in many existing tables. Maybe one day, but for now it is not high on the priority list. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Fri Mar 22 14:49:39 2013 Date: Fri, 22 Mar 2013 14:49:38 +0000 (GMT) From: Mark Taylor To: liskowski@gmail.XXX Cc: topcat-user@sympa.XXX Subject: Re: Encoding in TableWriters Hi liskowski. On Fri, 22 Mar 2013, liskowski@gmail.XXX wrote: > Hi, > > I'm writing to inquire whether any of you have ever encountered a problem with > encoding utf-8 characters in StarTables when using TextTableWriter or > HtmlTableWriter. > > Unfortunately, this code: > > TextTableWriter textTableWriter = new TextTableWriter(); > PrintStream fileOutputStream = new PrintStream(new FileOutputStream(file, > false), true, "UTF-8"); > > textTableWriter.writeStarTable(table, fileOutputStream); > > disrespects encoding settings and resulting file is full of junk characters. On > the other hand, CsvTableWriter works perfectly. > > Any ideas? TextTableWriter extends AbstractTextTableWriter, which features this code: (http://starjava.jach.hawaii.edu/viewvc/trunk/table/src/main/uk/ac/starlink/table/formats/AbstractTextTableWriter.java) /** * Returns a byte array corresponding to a given string. * * @param str string to decode */ protected static byte[] getBytes( String str ) { /* The decoding here is not that respectable (doesn't properly * handle Unicode), but it makes a big performance difference, * e.g. when writing out a table. * Leave it unless we find ourselves using much in the way of * unicode characters. * The correct way would be do use str.decode(). */ int leng = str.length(); byte[] buf = new byte[ leng ]; for ( int i = 0; i < leng; i++ ) { buf[ i ] = (byte) str.charAt( i ); } return buf; } It looks like overriding that method in the TextTableWriter you're using to do it right would almost fix this for you - except the method is static. Evidently whoever wrote that code (= me in 2003) didn't understand how overriding and static methods (don't) work together. Doh! Probably that's bad code anyway. I don't know now how serious the performance hit is, but it doesn't seem like a good enough excuse. I will try to do something about it. In the mean time, if you need something that works properly I could build you a copy of the library with that method non-static. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Thu Mar 28 17:56:59 2013 Date: Thu, 28 Mar 2013 17:56:59 +0000 (GMT) From: Mark Taylor To: topcat-announce@bristol.XXX topcat-user@bristol.XXX IVOA Applications WG Subject: Release: TOPCAT v4.0b, STILTS v2.5 Hi all, this is to announce version 4.0b of TOPCAT and version 2.5 of STILTS. The headline is a complete rewrite of the plotting windows in TOPCAT. You can see an introduction here: http://www.starlink.ac.uk/topcat/v4_graphics.html and full documentation here: http://www.starlink.ac.uk/topcat/sun253/plot2.html Note however this functionality is somewhat experimental in this release (hence the "b" on the version number). There are probably bugs and the user interface may undergo some revision based on user feedback. For this reason the new plot windows are available only from the Graphics menu in the main control window, and the existing plot windows are still in place with no changes, so you can continue to use the classic versions if you like. Feedback on the new functionality of any kind (even saying just that you're using it, or that you think it's rubbish) would be very welcome, either to me direct, or to the topcat-user mailing list. Nice screenshots gratefully accepted too. Other items of note: - TOPCAT gets a new logo for its tenth birthday! I never had any legal right to use the Hannah-Barbera cartoon image, and my conscience finally got to me. It's still a yellow cat. - Sky pixel functionality: TOPCAT and STILTS use MOCs to interrogate VizieR for multi-cone searches, which can make it a lot faster. Also new tools pixfoot (generates MOC files) and pixsample (samples from HEALPix files) in STILTS. - VOTable 1.3 fully implemented, including STILTS votlint validation - K-Correction code using Chilingarian/Zolotukhin algorithm - IPAC format can now be written as well as read These releases also contain multiple bugfixes and smaller enhancements; it's been more than a year since the last public releases. Sorry about the wait. See the web pages and version histories for details: http://www.star.bristol.ac.uk/~mbt/stilts/ http://www.star.bristol.ac.uk/~mbt/topcat/ Happy Easter! Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Fri Mar 29 12:57:29 2013 Date: Fri, 29 Mar 2013 12:57:28 +0000 (GMT) From: Mark Taylor To: topcat-announce@bristol.XXX topcat-user@bristol.XXX IVOA Applications WG Subject: TOPCAT v4.0b - fix for Macs On Thu, 28 Mar 2013, Mark Taylor wrote: > this is to announce version 4.0b of TOPCAT and version 2.5 of STILTS. Apologies - the TOPCAT release I made yesterday was missing some Mac-related classes, and the new plotting windows didn't work on OSX. I've made a fix - if you have a Mac please download the versions that are there now. Apologies for the inconvenience (and for crossposting, again). If anybody reported this thank you - I can't read my email at the moment. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From liskowski@gmail.XXX Fri Mar 22 13:22:00 2013 Date: Fri, 22 Mar 2013 14:21:40 +0000 From: liskowski@gmail.XXX To: topcat-user@sympa.XXX Subject: Encoding in TableWriters Hi, I'm writing to inquire whether any of you have ever encountered a problem with encoding utf-8 characters in StarTables when using TextTableWriter or HtmlTableWriter. Unfortunately, this code: TextTableWriter textTableWriter = new TextTableWriter(); PrintStream fileOutputStream = new PrintStream(new FileOutputStream(file, false), true, "UTF-8"); textTableWriter.writeStarTable(table, fileOutputStream); disrespects encoding settings and resulting file is full of junk characters. On the other hand, CsvTableWriter works perfectly. Any ideas? Cheers! From liskowski@gmail.XXX Sat Mar 23 06:38:40 2013 Date: Sat, 23 Mar 2013 08:38:32 +0100 From: Paweł Liskowski To: Mark Taylor Cc: topcat-user@sympa.XXX Subject: Re: Encoding in TableWriters On Mar 22, 2013, at 15:49 PM, Mark Taylor wrote: > Hi liskowski. > > On Fri, 22 Mar 2013, liskowski@gmail.XXX wrote: > >> Hi, >> >> I'm writing to inquire whether any of you have ever encountered a problem with >> encoding utf-8 characters in StarTables when using TextTableWriter or >> HtmlTableWriter. >> >> Unfortunately, this code: >> >> TextTableWriter textTableWriter = new TextTableWriter(); >> PrintStream fileOutputStream = new PrintStream(new FileOutputStream(file, >> false), true, "UTF-8"); >> >> textTableWriter.writeStarTable(table, fileOutputStream); >> >> disrespects encoding settings and resulting file is full of junk characters. On >> the other hand, CsvTableWriter works perfectly. >> >> Any ideas? > > TextTableWriter extends AbstractTextTableWriter, which features this > code: > > (http://starjava.jach.hawaii.edu/viewvc/trunk/table/src/main/uk/ac/starlink/table/formats/AbstractTextTableWriter.java) > > /** > * Returns a byte array corresponding to a given string. > * > * @param str string to decode > */ > protected static byte[] getBytes( String str ) { > > /* The decoding here is not that respectable (doesn't properly > * handle Unicode), but it makes a big performance difference, > * e.g. when writing out a table. > * Leave it unless we find ourselves using much in the way of > * unicode characters. > * The correct way would be do use str.decode(). */ > int leng = str.length(); > byte[] buf = new byte[ leng ]; > for ( int i = 0; i < leng; i++ ) { > buf[ i ] = (byte) str.charAt( i ); > } > return buf; > } > > It looks like overriding that method in the TextTableWriter you're > using to do it right would almost fix this for you - except the method > is static. Evidently whoever wrote that code (= me in 2003) didn't > understand how overriding and static methods (don't) work together. Doh! > > Probably that's bad code anyway. I don't know now how serious the > performance hit is, but it doesn't seem like a good enough excuse. > I will try to do something about it. In the mean time, if you > need something that works properly I could build you a copy of > the library with that method non-static. Mark, thank you for your response. I'm afraid your solution is only partial, since it won't fix problems in HtmlTableWriter as it uses direct str.getBytes(). Also note line 345 in the same class. Aside from the performance hit, I believe that Unicode support would be a truly anticipated feature. For the time being, I managed to write my own subclasses which simply wrap OutputStream with OutputStreamWriter. This basically solves the issue. If you manage to resolve this directly in the library, please let me know. Pawel From m.b.taylor@bristol.XXX Fri Mar 29 12:58:08 2013 Date: Fri, 29 Mar 2013 12:57:28 +0000 (GMT) From: Mark Taylor To: topcat-announce@bristol.XXX topcat-user@bristol.XXX IVOA Applications WG Subject: [topcat-announce] TOPCAT v4.0b - fix for Macs On Thu, 28 Mar 2013, Mark Taylor wrote: > this is to announce version 4.0b of TOPCAT and version 2.5 of STILTS. Apologies - the TOPCAT release I made yesterday was missing some Mac-related classes, and the new plotting windows didn't work on OSX. I've made a fix - if you have a Mac please download the versions that are there now. Apologies for the inconvenience (and for crossposting, again). If anybody reported this thank you - I can't read my email at the moment. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Wed Apr 10 18:11:08 2013 Date: Wed, 10 Apr 2013 19:09:55 +0200 From: Maciej Bilicki To: topcat-user@bristol.XXX Subject: Checking for double/multiple entries in a concatenated table Dear All, I have concatenated a couple of tables (with the same columns each) and now I would like to check if the resulting table has double or multiple entries, i.e. if the same objects, coming from tables that were concatenated, appear more than once in the resulting table. Is there any easy and quick way to do this, and to remove these doubled or multiplied entries? My tables contain millions of sources, so I need something automatic. Thanks Maciek From dabrusco@head.XXX Thu Apr 11 09:34:12 2013 Date: Wed, 10 Apr 2013 13:22:37 -0400 From: Raffaele D'Abrusco To: Maciej Bilicki Cc: topcat-user@bristol.XXX Subject: Re: Checking for double/multiple entries in a concatenated table Hi Maciek, have you tried "Internal Match" from the Joins menu? Cheers, Raffaele On Wed, Apr 10, 2013 at 1:09 PM, Maciej Bilicki wrote: > Dear All, > > I have concatenated a couple of tables (with the same columns each) and > now I would like to check if the resulting table has double or multiple > entries, i.e. if the same objects, coming from tables that were > concatenated, appear more than once in the resulting table. Is there any > easy and quick way to do this, and to remove these doubled or multiplied > entries? > > My tables contain millions of sources, so I need something automatic. > > Thanks > Maciek > -- *Raffaele D'Abrusco Harvard-Smithsonian Center for Astrophysics 60 Garden Street, B 306A Cambridge, MA 02138 * *tel: +1 617 495 7201 rdabrusco@cfa.XXX http://hea-www.harvard.edu/~dabrusco other e-mail: **dabrusco@gmail.XXX skype: raffaele_dabrusco* *google profile: dabrusco@gmail.XXX* From fap@head.XXX Wed Apr 10 18:35:00 2013 Date: Wed, 10 Apr 2013 13:33:57 -0400 From: Frank Primini To: Maciej Bilicki Cc: topcat-user@bristol.XXX Subject: Re: Checking for double/multiple entries in a concatenated table Hi, In the topcoat menu, click on "Joins" and then "Internal Match". You can then select the column to use for matching multiple entries. On Apr 10, 2013, at 1:09 PM, Maciej Bilicki wrote: > Dear All, > > I have concatenated a couple of tables (with the same columns each) and now I would like to check if the resulting table has double or multiple entries, i.e. if the same objects, coming from tables that were concatenated, appear more than once in the resulting table. Is there any easy and quick way to do this, and to remove these doubled or multiplied entries? > > My tables contain millions of sources, so I need something automatic. > > Thanks > Maciek From odysseus9672@gmail.XXX Sat Apr 20 00:06:54 2013 Date: Fri, 19 Apr 2013 16:05:54 -0700 From: Sean Lake To: topcat-user@sympa.XXX Subject: Feature Request: erf Hello all, Would it be possible and simple to expose the erf function in TOPCAT? I would imagine that it's part of the math library that TOPCAT uses, so adding it shouldn't be a lot of work. If not, that's Ok. Thanks, Sean Lake From M.B.Taylor@bristol.XXX Sun Apr 21 22:32:34 2013 Date: Sun, 21 Apr 2013 22:32:27 +0100 (BST) From: Mark Taylor To: Sean Lake Cc: topcat-user@bristol.XXX Subject: Re: Feature Request: erf On Fri, 19 Apr 2013, Sean Lake wrote: > Hello all, > > Would it be possible and simple to expose the erf function in TOPCAT? I would imagine that it's part of the math library that TOPCAT uses, so adding it shouldn't be a lot of work. If not, that's Ok. No, erf does not appear to be provided as part of the java standard libraries. Other people have written java erf functions, so I could add one, but I'll take this opportunity instead to explain how to use one of these external implementations (obviously, the same trick applies to other functions out there that somebody has written but are not available in topcat). A quick google shows up http://doodleproject.sourceforge.net/numerics/numerics4j/ which includes the functions erf(), erfc() and inverseErf() in package net.sf.doodleproject.numerics4j.special. There are other implementations which may differ in the details (e.g. accuracy) - pick one that suits. You can download the jar file associated with this project (download http://sourceforge.net/projects/doodleproject/files/numerics4j/1.3/numerics4j-1.3-bin.tar.gz/download and unpack it to find the numerics4j-1.3/numerics4j-1.3.jar file), put this jar file on the topcat class path at runtime, and tell topcat that you want to use functions from the relevant class, as explained at http://www.star.bris.ac.uk/~mbt/topcat/sun253/jelExtend.html If you have downloaded that jar file, you can do it like this: topcat -Djel.classes=net.sf.doodleproject.numerics4j.special.Erf -classpath /where/ever/numerics4j-1.3.jar Then bingo, you can use the erf function. If you look in the Available Functions window http://www.star.bris.ac.uk/~mbt/topcat/sun253/MethodWindow.html you'll see an entry for the new class with the functions you can use. Hope that helps Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From maciek@ast.XXX Thu May 16 23:12:32 2013 Date: Thu, 16 May 2013 21:00:22 +0200 From: Maciej Bilicki To: topcat-user@sympa.XXX Subject: Skewness and kurtosis Dear Mark, Which equations does Topcat use to calculate skewness and kurtosis? In particular the latter might be ambiguous - is it the so-called "excess kurtosis"? (e.g. http://en.wikipedia.org/wiki/Kurtosis#Pearson_moments ) Regards Maciej From M.B.Taylor@bristol.XXX Thu May 16 23:40:20 2013 Date: Fri, 17 May 2013 00:40:18 +0200 (CEST) From: Mark Taylor To: Maciej Bilicki Cc: topcat-user@sympa.XXX Subject: Re: Skewness and kurtosis On Thu, 16 May 2013, Maciej Bilicki wrote: > Dear Mark, > > Which equations does Topcat use to calculate skewness and kurtosis? In > particular the latter might be ambiguous - is it the so-called "excess > kurtosis"? > (e.g. http://en.wikipedia.org/wiki/Kurtosis#Pearson_moments ) You can see the source code at around line 1166 of StatsWindow.java: https://github.com/Starlink/starjava/blob/master/topcat/src/main/uk/ac/starlink/topcat/StatsWindow.java I must admit I don't know offhand whether that's excess kurtosis or something else. Perhaps you could summarise the answer to your own question on this list. If it's not clear to you from the source code though, I'll try to untangle it for you. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From odysseus9672@gmail.XXX Thu May 16 23:45:23 2013 Date: Thu, 16 May 2013 15:45:18 -0700 From: Sean Lake To: Mark Taylor Cc: Maciej Bilicki , topcat-user@sympa.XXX Subject: Re: Skewness and kurtosis Line 1178 there's a -3.0, that's excess kurtosis. Sean On May 16, 2013, at 15:40, Mark Taylor wrote: > On Thu, 16 May 2013, Maciej Bilicki wrote: > >> Dear Mark, >> >> Which equations does Topcat use to calculate skewness and kurtosis? In >> particular the latter might be ambiguous - is it the so-called "excess >> kurtosis"? >> (e.g. http://en.wikipedia.org/wiki/Kurtosis#Pearson_moments ) > > You can see the source code at around line 1166 of StatsWindow.java: > > https://github.com/Starlink/starjava/blob/master/topcat/src/main/uk/ac/starlink/topcat/StatsWindow.java > > I must admit I don't know offhand whether that's excess kurtosis > or something else. Perhaps you could summarise the answer to your > own question on this list. If it's not clear to you from the source > code though, I'll try to untangle it for you. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > From p.w.lucas@herts.XXX Fri May 17 08:43:59 2013 Date: Fri, 17 May 2013 08:42:38 +0100 From: "Lucas, Philip W" To: "topcat-user@sympa.XXX" Subject: STILTS procedure Dear Mark, A few of us at Hertfordshire are making a bunch of proper motion catalogues and doing the source matching with STILTS. We are wondering whether the basic rules for sky coordinate matches are documented somewhere. Specifically we want to be absolutely certain that each object at epoch 1 is always matched to the nearest object at epoch 2, before searching for more distant matches. This may seem obvious, but it's not clear exactly how it's implemented. Also, when there can be significant spatial offsets between epochs it is possible that e.g. binning of the data into area blocks might cause distant matches to be missed. Thanks, Phil Lucas From M.B.Taylor@bristol.XXX Sat May 18 20:56:50 2013 Date: Sat, 18 May 2013 20:56:43 +0100 (BST) From: Mark Taylor To: "Lucas, Philip W" Cc: "topcat-user@sympa.XXX" Subject: Re: STILTS procedure Hallo Phil. On Fri, 17 May 2013, Lucas, Philip W wrote: > A few of us at Hertfordshire are making a bunch of proper motion > catalogues and doing the source matching with STILTS. We are wondering > whether the basic rules for sky coordinate matches are documented > somewhere. Specifically we want to be absolutely certain that each > object at epoch 1 is always matched to the nearest object at epoch 2, > before searching for more distant matches. This may seem obvious, but > it's not clear It depends how you run it. If you're using the tskymatch2 or tmatch2 command, see the documentation of the "find" parameter (http://www.starlink.ac.uk/stilts/sun256/tskymatch2-usage.html). If you select find=all it will give you all matches within the criteria you give it, but "best", "best1" or "best2" gives only the best one (in most cases, including sky matching, that means the closest). I suggest you use best1 or best2 rather than best; they mostly work the same, but it's rather hard to understand the exact matching criteria for best, while the meaning of the best1/best2 options is pretty straightforward. > exactly how it's implemented. Also, when there can be significant spatial > offsets between epochs it is possible that > e.g. binning of the data into area blocks might cause distant matches to be > missed. The details of the implementation are not documented outside of the source code (if you're interested, see https://github.com/Starlink/starjava/blob/master/table/src/main/uk/ac/starlink/table/join/RowMatcher.java and related classes), although there is a short overview here: http://www.star.bris.ac.uk/~mbt/topcat/sun253/matchAlgorithm.html It does do binning, but of course it doesn't disregard the possibility that best matches might not be in the same bin. From a user's point of view, you just have to know that it does pick the closest counterpart if that's what you ask it to do. At least, if it doesn't, that's a bug. Of course I can't give you a cast iron guarantee it's 100% bug free, but it's had a lot of testing and I'm pretty confident that there are no serious bugs in the crossmatching. In recent years only two, fairly minor bugs have been reported in the pair matching: you can see them documented in the STILTS version history (http://www.starlink.ac.uk/stilts/sun256/versions.html) at v2.3-1 and v2.5. Overplotting the input and joined catalogues to see what it's done is a good way to convince yourself it's doing the right kind of thing, or to work out what's going on if you don't like the look of the results. It's fairly easy to do that manually, but I plan to add an option to display the results of crossmatches graphically at the click of a button in the future. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From phyqpwl@herts.XXX Mon May 20 10:54:57 2013 Date: Mon, 20 May 2013 10:54:39 +0100 From: Philip Lucas To: Mark Taylor Cc: "topcat-user@sympa.XXX" Subject: Re: STILTS procedure Many thanks Mark. That's encouraging and helpful. We are using tmatch2 and the default find=best, since we have two epochs of very similar data from the same facility (UKIRT). I think best is what we want, since it would be confusing to have multiple matches sometimes. We really just want the single closest match in symmetric fashion, since we are dealing with huge volumes of data. Best, Phil On Sat, 18 May 2013, Mark Taylor wrote: > Hallo Phil. > > On Fri, 17 May 2013, Lucas, Philip W wrote: > >> A few of us at Hertfordshire are making a bunch of proper motion >> catalogues and doing the source matching with STILTS. We are wondering >> whether the basic rules for sky coordinate matches are documented >> somewhere. Specifically we want to be absolutely certain that each >> object at epoch 1 is always matched to the nearest object at epoch 2, >> before searching for more distant matches. This may seem obvious, but >> it's not clear > > It depends how you run it. If you're using the tskymatch2 or tmatch2 > command, see the documentation of the "find" parameter > (http://www.starlink.ac.uk/stilts/sun256/tskymatch2-usage.html). > If you select find=all it will give you all matches within the > criteria you give it, but "best", "best1" or "best2" gives only > the best one (in most cases, including sky matching, that means the > closest). > > I suggest you use best1 or best2 rather than best; they mostly work > the same, but it's rather hard to understand the exact matching > criteria for best, while the meaning of the best1/best2 options > is pretty straightforward. > >> exactly how it's implemented. Also, when there can be significant spatial >> offsets between epochs it is possible that >> e.g. binning of the data into area blocks might cause distant matches to be >> missed. > > The details of the implementation are not documented outside of the > source code (if you're interested, see > > https://github.com/Starlink/starjava/blob/master/table/src/main/uk/ac/starlink/table/join/RowMatcher.java > > and related classes), although there is a short overview here: > > http://www.star.bris.ac.uk/~mbt/topcat/sun253/matchAlgorithm.html > > It does do binning, but of course it doesn't disregard the possibility > that best matches might not be in the same bin. From a user's point > of view, you just have to know that it does pick the closest counterpart > if that's what you ask it to do. At least, if it doesn't, that's a bug. > Of course I can't give you a cast iron guarantee it's 100% bug free, > but it's had a lot of testing and I'm pretty confident that there > are no serious bugs in the crossmatching. In recent years only two, > fairly minor bugs have been reported in the pair matching: you can > see them documented in the STILTS version history > (http://www.starlink.ac.uk/stilts/sun256/versions.html) > at v2.3-1 and v2.5. > > Overplotting the input and joined catalogues to see what it's done > is a good way to convince yourself it's doing the right kind of thing, > or to work out what's going on if you don't like the look of the > results. It's fairly easy to do that manually, but I plan to add > an option to display the results of crossmatches graphically at > the click of a button in the future. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > From Hauser@ari.XXX Fri May 24 09:48:07 2013 Date: Fri, 24 May 2013 10:46:05 +0200 From: Marcus Hauser To: topcat-user@mlist2.XXX Subject: topcat table label not saved/restored Hi Mark, hi list! I have a problem with saving and/or restoring the label of tables in topcat. According to the docs [1], the table label should be saved when performing "save session". However, after loading the saved FITS file again into topcat, *some* table labels are lost and replaced by "-". In my case, all tables which resulted from topcat-performed match operations kept their names ("match(1,3)" for example), but for 2 out of 4 tables originating from loading ASCII files, the label was replaced by "-" (when saving as FITS-plus; when saving as votable, 3/4 reproducible loose their name). Attached screenshots show the describes scenario. 'topcat1.png' is before saving the session in fits-plus format, 'topcat2.png' after re-starting topcat and loading the saved session. I tested this with topcat 3.9 and 4.0; both show the same behaviour. Any help would be very much appreciated! Marcus [1] http://www.star.bristol.ac.uk/~mbt/topcat/sun253/secA.7.1.3.html [ Part 2, Image/PNG (Name: "topcat1.png") 36 KB. ] [ Unable to print this part. ] [ Part 3, Image/PNG (Name: "topcat2.png") 39 KB. ] [ Unable to print this part. ] From rbarba@dfuls.XXX Tue May 28 09:07:38 2013 Date: Tue, 28 May 2013 09:06:46 +0100 From: rbarba@dfuls.XXX To: topcat-user@sympa.XXX Subject: Histogram of string values using TOPCAT Dear All, I'm trying to do a histogram plot of a column with string values like "O, B, A, F, G, K, M" using TOPCAT. Any hint about how to do that? Many thanks in advance. From m.b.taylor@bristol.XXX Wed May 29 10:14:54 2013 Date: Wed, 29 May 2013 10:14:54 +0100 (BST) From: Mark Taylor To: rbarba@dfuls.XXX Cc: topcat-user@sympa.XXX Subject: Re: Histogram of string values using TOPCAT Hi rbarba, On Tue, 28 May 2013, rbarba@dfuls.XXX wrote: > Dear All, > I'm trying to do a histogram plot of a column with string values like "O, B, A, > F, G, K, M" using TOPCAT. Any hint about how to do that? Many thanks in > advance. I'm afraid there is no real way to use non-numeric values on axes in TOPCAT as it stands. Functionality along these lines has been lurking on the to-do list for some while and may show up in the new layer plot windows at some point - I'll take this as a feature request. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From cgp@leicester.XXX Wed May 29 10:40:07 2013 Date: Wed, 29 May 2013 10:34:42 +0100 From: Clive Page To: topcat-user@sympa.XXX Subject: Re: Histogram of string values using TOPCAT On 29/05/2013 10:14, Mark Taylor wrote: >> I'm trying to do a histogram plot of a column with string values like "O, B, A, >> F, G, K, M" using TOPCAT. Any hint about how to do that? Many thanks in >> advance. It occurs to me that what would help here is a Java function to locate the position of a given character in a string, similar to the Fortran INDEX function, e.g. INDEX("OBAFGKM", mychar) which for example returns 3 if mychar happens to be A. That would allow the spectral type to be transformed into the integers 1 to 7. But I can't see a suitable function in the TOPCAT manual, unfortunately. -- Clive Page Dept of Physics & Astronomy, University of Leicester, Leicester, LE1 7RH, U.K. From m.b.taylor@bristol.XXX Wed May 29 10:48:34 2013 Date: Wed, 29 May 2013 10:48:34 +0100 (BST) From: Mark Taylor To: Clive Page Cc: topcat-user@sympa.XXX Subject: Re: Histogram of string values using TOPCAT On Wed, 29 May 2013, Clive Page wrote: > On 29/05/2013 10:14, Mark Taylor wrote: > > > I'm trying to do a histogram plot of a column with string values like "O, > > > B, A, > > > F, G, K, M" using TOPCAT. Any hint about how to do that? Many thanks in > > > advance. > > It occurs to me that what would help here is a Java function to locate the > position of a given character in a string, similar to the Fortran INDEX > function, e.g. > INDEX("OBAFGKM", mychar) > which for example returns 3 if mychar happens to be A. That would allow the > spectral type to be transformed into the integers 1 to 7. But I can't see a > suitable function in the TOPCAT manual, unfortunately. Well actually you can do that: "OBAFGKM".indexOf('F') returns the value 3 (these indices are zero-based), so it's not impossible to get a histogram with bars the right height, but the axis won't be labelled with the column values, only the relatively inscrutable index numbers. Note the two different quote character types in that example, you have to get them right. Other undocumented functions on String values (free from java, not introduced or documented by me) can be seen (if you know how to read them) e.g. here: http://docs.oracle.com/javase/6/docs/api/ Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed May 29 11:21:38 2013 Date: Wed, 29 May 2013 11:21:37 +0100 (BST) From: Mark Taylor To: Marcus Hauser Cc: topcat-user@bristol.XXX Subject: Re: topcat table label not saved/restored Hi Marcus, sorry for the delay, I was on holiday for a few days. A quick play doesn't reproduce this behaviour for me, though this functionality may not have been very heavily tested, so I wouldn't be too surprised to see it not working right. Would it be possible to send me (off-list) one of the offending session-saved files? thanks Mark On Fri, 24 May 2013, Marcus Hauser wrote: > Hi Mark, hi list! > > I have a problem with saving and/or restoring the label of tables in topcat. > > According to the docs [1], the table label should be saved when > performing "save session". However, after loading the saved FITS file > again into topcat, *some* table labels are lost and replaced by > "-". In my case, all tables which resulted > from topcat-performed match operations kept their names ("match(1,3)" > for example), but for 2 out of 4 tables originating from loading ASCII > files, the label was replaced by "-" (when > saving as FITS-plus; when saving as votable, 3/4 reproducible loose > their name). > Attached screenshots show the describes scenario. 'topcat1.png' is > before saving the session in fits-plus format, 'topcat2.png' after > re-starting topcat and loading the saved session. > > I tested this with topcat 3.9 and 4.0; both show the same behaviour. > > Any help would be very much appreciated! > Marcus > > > > [1] http://www.star.bristol.ac.uk/~mbt/topcat/sun253/secA.7.1.3.html > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Wed May 29 13:30:58 2013 Date: Wed, 29 May 2013 13:30:57 +0100 (BST) From: Mark Taylor To: Marcus Hauser Cc: topcat-user@bristol.XXX Subject: Re: topcat table label not saved/restored Test data gratefully received. It turns out this was a bug that kicked in only for single-column tables, of which there were a few in your session file. I've fixed this and you can find a pre-release that works properly at ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ The bug was on read and not write, so this will correctly read any session files you previously saved. Thanks for the report. Mark On Wed, 29 May 2013, Mark Taylor wrote: > Hi Marcus, > > sorry for the delay, I was on holiday for a few days. > > A quick play doesn't reproduce this behaviour for me, though this > functionality may not have been very heavily tested, so I wouldn't > be too surprised to see it not working right. Would it be possible > to send me (off-list) one of the offending session-saved files? > > thanks > > Mark > > On Fri, 24 May 2013, Marcus Hauser wrote: > > > Hi Mark, hi list! > > > > I have a problem with saving and/or restoring the label of tables in topcat. > > > > According to the docs [1], the table label should be saved when > > performing "save session". However, after loading the saved FITS file > > again into topcat, *some* table labels are lost and replaced by > > "-". In my case, all tables which resulted > > from topcat-performed match operations kept their names ("match(1,3)" > > for example), but for 2 out of 4 tables originating from loading ASCII > > files, the label was replaced by "-" (when > > saving as FITS-plus; when saving as votable, 3/4 reproducible loose > > their name). > > Attached screenshots show the describes scenario. 'topcat1.png' is > > before saving the session in fits-plus format, 'topcat2.png' after > > re-starting topcat and loading the saved session. > > > > I tested this with topcat 3.9 and 4.0; both show the same behaviour. > > > > Any help would be very much appreciated! > > Marcus > > > > > > > > [1] http://www.star.bristol.ac.uk/~mbt/topcat/sun253/secA.7.1.3.html > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From Hauser@ari.XXX Wed May 29 14:50:58 2013 Date: Wed, 29 May 2013 15:50:52 +0200 From: Marcus Hauser To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: topcat table label not saved/restored On 29.05.2013 14:30, Mark Taylor wrote: > It turns out this was a bug that kicked in only for single-column tables, > of which there were a few in your session file. I've fixed this and > you can find a pre-release that works properly at > > ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ > Hi Mark, it seems as if this indeed fixed. Thanks a lot for your quick help! Marcus From phyqpwl@herts.XXX Thu May 30 11:20:15 2013 Date: Thu, 30 May 2013 11:16:20 +0100 From: Philip Lucas To: topcat-user@bristol.XXX Subject: csv file loading problem Hi all, I have been unable to load csv files in the recent version of topcat, though the same files still load in the older releases that we have on our Starlink system. Has anyone else had this issue? Phil Lucas, Herts From M.B.Taylor@bristol.XXX Thu May 30 11:28:28 2013 Date: Thu, 30 May 2013 11:28:26 +0100 (BST) From: Mark Taylor To: Philip Lucas Cc: topcat-user@bristol.XXX Subject: Re: csv file loading problem On Thu, 30 May 2013, Philip Lucas wrote: > Hi all, > > I have been unable to load csv files in the recent version > of topcat, though the same files still load in the older releases > that we have on our Starlink system. > > Has anyone else had this issue? Phil and others, apologies for this. There is a regression bug in v4.0 which means that DOS-format CSV files (ones with CR+LF rather than just LF line breaks) no longer load on either DOS or un*x machines. This will be fixed in the next public release (which will come along much quicker than the last one). In the mean time you can get a pre-release version of TOPCAT/STILTS that should work OK with CSV files here: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ (running dos2unix or similar on the CSV files would get them to load in the existing version). If this doesn't seem to be the problem that you're seeing, please get back to me. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From Hauser@ari.XXX Mon Jun 10 12:04:23 2013 Date: Mon, 10 Jun 2013 13:02:44 +0200 From: Marcus Hauser To: topcat-user@mlist2.XXX Subject: STILTS and row subsets Hi Mark, hi list! I am aiming at defining row subsets of a FITS table using STILTS and having these subsets available once I load this table into TOPCAT. I have seen in the docs the description on how to use subsets for plotting (sect 8.1), but I fail to see how this would work in pure pipeline processing, to be able to use and combine these STILTS-created subsets in TOPCAT. Is this possible somehow, or am I looking for some feature that does not exist? Cheers, Marcus From m.b.taylor@bristol.XXX Mon Jun 10 12:14:10 2013 Date: Mon, 10 Jun 2013 12:14:09 +0100 (BST) From: Mark Taylor To: Marcus Hauser Cc: topcat-user@bristol.XXX Subject: Re: STILTS and row subsets Hi Marcus, On Mon, 10 Jun 2013, Marcus Hauser wrote: > I am aiming at defining row subsets of a FITS table using STILTS and having > these subsets available once I load this table into TOPCAT. > > I have seen in the docs the description on how to use subsets for plotting > (sect 8.1), but I fail to see how this would work in pure pipeline processing, > to be able to use and combine these STILTS-created subsets in TOPCAT. > > Is this possible somehow, or am I looking for some feature that does not > exist? Strictly speaking, subsets only exist inside TOPCAT and not in the kind of table that STILTS can produce (they do get serialized into topcat saved session files, but the format used for that is not currently documented or intended for public use). So you can't exactly do what you're asking. However, you can do something which has nearly the same effect. If a table has a boolean-valued column when TOPCAT loads it, it automatically creates a subset corresponding to it. So I suggest that you create a boolean column using STILTS, and when you load the resulting table into topcat you'll see the subset you've defined. The table will also have a boolean column which you may not otherwise be interested in. Adding a boolean column works just like adding any other column, just give it an expression with a true/false value, e.g. stilts tpipe ... cmd='addcol bright FLUX>100' Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From Hauser@ari.XXX Fri Jun 21 15:02:18 2013 Date: Fri, 21 Jun 2013 16:00:22 +0200 From: Marcus Hauser To: topcat-user@bristol.XXX Subject: TOPCAT -- link plot styles over different windows Hi Mark, is there a way to link the plot styles of row subsets in different plotting windows? Imagine I have a histogram and a scatter plot window open. Then I define a subset 'A' in the histogram view and change its colour to 'red'. I would like the subset 'A' in the scatter plot window (or any other plot window showing the same dataset) to automatically change its colour to match the histogram ones (in my example to also become 'red'). If that is not possible, please take this as a feature request for a future release. Thanks in advance! Marcus From m.b.taylor@bristol.XXX Fri Jun 21 15:07:30 2013 Date: Fri, 21 Jun 2013 15:07:30 +0100 (BST) From: Mark Taylor To: Marcus Hauser Cc: topcat-user@bristol.XXX Subject: Re: TOPCAT -- link plot styles over different windows On Fri, 21 Jun 2013, Marcus Hauser wrote: > Hi Mark, > > is there a way to link the plot styles of row subsets in different > plotting windows? > > Imagine I have a histogram and a scatter plot window open. Then I define > a subset 'A' in the histogram view and change its colour to 'red'. > I would like the subset 'A' in the scatter plot window (or any other > plot window showing the same dataset) to automatically change its colour > to match the histogram ones (in my example to also become 'red'). No, I'm afraid you can't do that. > If that is not possible, please take this as a feature request for a > future release. Noted. It's a reasonable request, but for various boring reasons not so easy to implement, so don't hold your breath. Thanks for the suggestion. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From andrew.obrien@uws.XXX Mon Jul 8 15:07:13 2013 Date: Mon, 08 Jul 2013 15:03:19 +0100 From: andrew.obrien@uws.XXX To: topcat-user@sympa.XXX Subject: User-defined function documentation Hi all, Is there any way to provide your own documentation for user-defined functions in TOPCAT? I've added some functions to the class path as per the TOPCAT documentation and they work wonderfully, however when viewed in the Available Functions window, there is nothing visible. This is expected, since nothing has been written about them! I noticed that the existing function documentation looked very similar to those created by javadoc. Is there a way to have TOPCAT read javadoc syntax, or is there a way I can add the documentation manually? Thanks! - Andrew From m.b.taylor@bristol.XXX Mon Jul 8 15:49:30 2013 Date: Mon, 8 Jul 2013 15:49:29 +0100 (BST) From: Mark Taylor To: andrew.obrien@uws.XXX Cc: topcat-user@sympa.XXX Subject: Re: User-defined function documentation On Mon, 8 Jul 2013, andrew.obrien@uws.XXX wrote: > Hi all, > > Is there any way to provide your own documentation for user-defined functions > in TOPCAT? > > I've added some functions to the class path as per the TOPCAT documentation and > they work wonderfully, however when viewed in the Available Functions window, > there is nothing visible. This is expected, since nothing has been written > about them! > > I noticed that the existing function documentation looked very similar to those > created by javadoc. Is there a way to have TOPCAT read javadoc syntax, or is > there a way I can add the documentation manually? Andrew, I didn't really design this to be doable by users, so it's not all that straightforward, but it can be done. You will have to generate the relevant HTML files from the javadocs and then put them on the classpath at runtime. You can generate HTML files in the right form with the right filenames using the HtmlDoclet buried in the topcat jar file, like this: javadoc -docletpath topcat-full.jar \ -doclet uk.ac.starlink.ttools.build.HtmlDoclet \ -d docdir \ Foo.java Bar.java Baz.java and jar them up like this: jar cf foodocs.jar -C docdir . then (if your compiled user-defined functions are in foo.jar) running topcat -classpath foo.jar:foodocs.jar -Djel.classes=Foo:Bar:Baz should give you documented user-defined functions. If you want to be perfectionist about it, you can take a look at the javadocs in the topcat sources to see what conventions I've used, e.g.: https://github.com/Starlink/starjava/blob/master/ttools/src/main/uk/ac/starlink/ttools/func/Arithmetic.java Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From Andrew.OBrien@uws.XXX Mon Jul 8 16:44:27 2013 Date: Mon, 8 Jul 2013 15:44:21 +0000 From: Andrew O'Brien To: Mark Taylor Cc: "topcat-user@sympa.XXX" Subject: Re: User-defined function documentation On 09/07/2013, at 12:49 AM, "Mark Taylor" wrote: > On Mon, 8 Jul 2013, andrew.obrien@uws.XXX wrote: > >> Hi all, >> >> Is there any way to provide your own documentation for user-defined functions >> in TOPCAT? >> >> I've added some functions to the class path as per the TOPCAT documentation and >> they work wonderfully, however when viewed in the Available Functions window, >> there is nothing visible. This is expected, since nothing has been written >> about them! >> >> I noticed that the existing function documentation looked very similar to those >> created by javadoc. Is there a way to have TOPCAT read javadoc syntax, or is >> there a way I can add the documentation manually? > > Andrew, > > I didn't really design this to be doable by users, so it's not all that > straightforward, but it can be done. > > You will have to generate the relevant HTML files from the javadocs > and then put them on the classpath at runtime. You can generate > HTML files in the right form with the right filenames using the > HtmlDoclet buried in the topcat jar file, like this: > > javadoc -docletpath topcat-full.jar \ > -doclet uk.ac.starlink.ttools.build.HtmlDoclet \ > -d docdir \ > Foo.java Bar.java Baz.java > > and jar them up like this: > > jar cf foodocs.jar -C docdir . > > then (if your compiled user-defined functions are in foo.jar) running > > topcat -classpath foo.jar:foodocs.jar -Djel.classes=Foo:Bar:Baz > > should give you documented user-defined functions. > > If you want to be perfectionist about it, you can take a look at the > javadocs in the topcat sources to see what conventions I've used, > e.g.: > > https://github.com/Starlink/starjava/blob/master/ttools/src/main/uk/ac/starlink/ttools/func/Arithmetic.java Mark, Thank you, that worked brilliantly! I didn't see that doclet. Putting the docs and the compiled source into a single jar also worked (and was a bit cleaner). Regards, Andrew From maciek@ast.XXX Thu Jul 11 13:05:36 2013 Date: Thu, 11 Jul 2013 12:51:09 +0100 From: Maciej Bilicki To: topcat-user@bristol.XXX Subject: Histogram normalisation Dear Mark, I'd like to know if it's possible to introduce an additional feature to histogram plots. Currently, what normalisation button does is that "each dataset will be normalised so that the sum of the counts of all its bars over the whole range of data is equal to one". I'd be happy to have the option of normalising all the histograms to the same maximum height - for instance of 1. Sometimes we are interested only in the relative shapes and offsets of the distributions. Normalising to equal area means however that what's broader, will usually have much lower maximum height. Is that doable? Best wishes Maciek From bianca@lambrate.XXX Thu Jul 11 13:09:47 2013 Date: Thu, 11 Jul 2013 14:07:13 +0200 From: Bianca Garilli To: topcat-user@mlist2.XXX Subject: Re: Histogram normalisation Hi Mark I support the request by Maciek: it would really be most useful Let me also thanks you for the new 'high quality plots' features, which is really extremely useful. It would be nice to have it also for histograms Best wishes Bianca On Thursday 11 July 2013 13:51 Maciej Bilicki wrote > Dear Mark, > > I'd like to know if it's possible to introduce an additional feature > to histogram plots. > > Currently, what normalisation button does is that "each dataset will > be normalised so that the sum of the counts of all its bars over the > whole range of data is equal to one". I'd be happy to have the option > of normalising all the histograms to the same maximum height - for > instance of 1. Sometimes we are interested only in the relative shapes > and offsets of the distributions. Normalising to equal area means > however that what's broader, will usually have much lower maximum > height. > > Is that doable? > > Best wishes > Maciek ------------------------------------------------------------------------- Bianca Garilli; INAF, IASF-Milano via Bassini 15; 20133 Milano, Italy E-mail: bianca@lambrate.XXX phone: 39-02-23699313 Fax: 39-02-2666017 -------------------------------------------------------------------------- "Le signorine del french-cancan e della spaccata sono le belle addormentate di una felicità perduta" Léon-Paul Fargue "Science demands dogged work. Work in spite of everything and continuous observation" V.Van Gogh (adapted from) "In a world where science is treated as a branch of magic, the rewards often go to those who behave as illusionists" ------------------------------------------------------------------------- From m.b.taylor@bristol.XXX Thu Jul 11 13:37:58 2013 Date: Thu, 11 Jul 2013 13:37:58 +0100 (BST) From: Mark Taylor To: Maciej Bilicki Cc: topcat-user@bristol.XXX Subject: Re: Histogram normalisation Maciek, On Thu, 11 Jul 2013, Maciej Bilicki wrote: > Dear Mark, > > I'd like to know if it's possible to introduce an additional feature > to histogram plots. > > Currently, what normalisation button does is that "each dataset will > be normalised so that the sum of the counts of all its bars over the > whole range of data is equal to one". I'd be happy to have the option > of normalising all the histograms to the same maximum height - for > instance of 1. Sometimes we are interested only in the relative shapes > and offsets of the distributions. Normalising to equal area means > however that what's broader, will usually have much lower maximum > height. > > Is that doable? thanks for the suggestion, but I probably will not get round to doing this in the histogram plot as it stands. The height of the tallest bar is not very robust, so it seems a bit of an arbitrary scaling possibility to add. I have in any case more or less stopped adding features to the old-style plot windows, in favour of working on the new-style layer plots (currently available only from the Graphics menu). I haven't implemented a new-style Histogram yet, so that doesn't help ... but when I do, I'll give consideration to more scaling options. In the mean time, a couple of hacks you might try in the existing histogram window are: - Use the Cumulative option as well as Normalisation, which may work better for comparing distributions, though the plots look a bit different. - Weight the histograms manually - don't use Normalisation, but add a Weight axis, and manually enter scaling values (just numbers) to one or more of the plots until they look most suited for comparison. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Thu Jul 11 17:36:38 2013 Date: Thu, 11 Jul 2013 17:36:37 +0100 (BST) From: Mark Taylor To: Bianca Garilli Cc: topcat-user@bristol.XXX Subject: Re: Histogram normalisation On Thu, 11 Jul 2013, Bianca Garilli wrote: > Let me also thanks you for the new 'high quality plots' features, which is > really extremely useful. It would be nice to have it also for histograms thanks Bianca - as I say, histograms are in the pipeline, but it may take a little while. I am continuing to work on the new plot types. Since I haven't had much in the way of bug reports, feedback etc I get the impression they are not so far being used very much, probably because they are so far not very prominent in the user interface. I know there are various things wrong with them which I want to improve as well as adding new capabilities. If you or anybody else who is actually using these has particular priorities or annoyances they'd like to see addressed, especially things that worked better in the old plot types than the new, I'd be very happy to hear either on- or off-list. thanks Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From pae9@leicester.XXX Fri Jul 12 08:57:47 2013 Date: Fri, 12 Jul 2013 08:55:23 +0100 From: Phil Evans To: topcat-user@bristol.XXX Subject: Memory/Disk usage Hi, I have a problem with Topcat's usage of disk space. I am trying to load a table, currently in csv format. It's a moderately large one: 168 columns, 150,000 rows. The file on disk is 211M (uncompressed). However, every time I try to load it, topcat creates a file on /tmp which grows rapidly, until it fills my 4GB /tmp partition and fails to load the table (but doesn't delete the temporary file....). If I invoke "topcat -memory" instead, topcat starts up with 3.5GB of memory, and loads the table no problem, without exceeding about 1GB of memory usage. I don't understand why there is such a large difference between the size of the table in memory and on disk. Is this symptomatic of a setup problem I have? Or do I need to have either a huge /tmp area or use -memory to load medium to large tables? Thanks Phil Evans -- ------------------------- Phil Evans, Swift Development Scientist X-ray and Observational Astronomy Group, University of Leicester Tel: +44 (0)116 252 5059 Mobile: +44 (0)7780 980240 (work) Mobile: +44 (0)7974 977723 (personal) pae9@leicester.XXX http://www.star.le.ac.uk/~pae9 http://www.swift.ac.uk Follow me as a Swift scientist on Twitter: @swift_phil http://www.star.le.ac.uk/~pae9/twitter From m.b.taylor@bristol.XXX Fri Jul 12 10:07:25 2013 Date: Fri, 12 Jul 2013 10:07:24 +0100 (BST) From: Mark Taylor To: Phil Evans Cc: topcat-user@bristol.XXX Subject: Re: Memory/Disk usage short answer: use FITS. Read on. On Fri, 12 Jul 2013, Phil Evans wrote: > I have a problem with Topcat's usage of disk space. I am trying to load a > table, currently in csv format. It's a moderately large one: 168 columns, > 150,000 rows. The file on disk is 211M (uncompressed). However, every time I > try to load it, topcat creates a file on /tmp which grows rapidly, until it > fills my 4GB /tmp partition and fails to load the table (but doesn't delete > the temporary file....). It ought to delete the temporary file at least when it exits, though if it exits abnormally (e.g. you have to kill -9 the process or it fails because of lack of memory) that might not happen. > If I invoke "topcat -memory" instead, topcat starts up with 3.5GB of memory, > and loads the table no problem, without exceeding about 1GB of memory usage. > > I don't understand why there is such a large difference between the size of I'm not sure I understand either, I'd expect memory usage to be similar for the two cases, though this might happen if your CSV file has String-valued columns in which a few of the entries are much longer than others. > the table in memory and on disk. Is this symptomatic of a setup problem I > have? Or do I need to have either a huge /tmp area or use -memory to load > medium to large tables? However, it's certainly possible to fix this so that it works well. The main thing that you want to do is to convert the CSV file to FITS format before you load it. You can do this using STILTS, TOPCAT's command-line evil twin: topcat -stilts tpipe in=foo.csv ifmt=csv out=foo.fits cmd=progress All your problems will then evaporate - TOPCAT will load foo.fits instantaneously and without using significant amounts of memory or disk. The STILTS conversion itself may take a few minutes, but it doesn't use much memory or disk space apart from the size of the output file. FWIW, if for some reason you do need to use TOPCAT in such a way that it's using a temporary file bigger than your /tmp can handle, you can tell it to put tmp files elsewhere with the flag -Djava.io.tmpdir=/my/big/scratchdir. Advanced: with 168 columns you might get somewhat better performance if you convert to colfits instead, which arranges the data on disk a column at a time rather than a row at a time (http://www.star.bristol.ac.uk/~mbt/topcat/sun253/inColfits.html). Just write "out=foo.colfits" in the above instead. However, this conversion does require scratch files, so you might need to do something like: topcat -Djava.io.tmpdir=. -stilts -verbose tpipe in=foo.csv ifmt=csv out=foo.colfits cmd=progress (the -verbose and cmd=progress business isn't essential, it's just so you can see it working). But if you have 3.5Gb of memory to play with, you probably won't notice a difference. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From sean.farrell@sydney.XXX Wed Jul 24 07:36:16 2013 Date: Wed, 24 Jul 2013 07:35:38 +0100 From: sean.farrell@sydney.XXX To: topcat-user@sympa.XXX Subject: Replace functions don't like + symbols Hi Mark, i'm trying to replace all instances of + in a column of strings with %2B so I can read it into a URL string, but the replace functions don't seem to like this. Below is the syntax that I'm using: replaceAll(Name2,"+","%2B") where Name2 is the name of the column that contains all my strings. Instead of replacing all the +'s with %2B it spits out a blank string (i.e. an empty column). Any tips on how to get around this issue? Cheers, Sean From m.b.taylor@bristol.XXX Wed Jul 24 09:48:15 2013 Date: Wed, 24 Jul 2013 09:48:14 +0100 (BST) From: Mark Taylor To: sean.farrell@sydney.XXX Cc: topcat-user@sympa.XXX Subject: Re: Replace functions don't like + symbols On Wed, 24 Jul 2013, sean.farrell@sydney.XXX wrote: > Hi Mark, > > i'm trying to replace all instances of + in a column of strings with %2B so I > can read it into a URL string, but the replace functions don't seem to like > this. Below is the syntax that I'm using: > > replaceAll(Name2,"+","%2B") > > where Name2 is the name of the column that contains all my strings. Instead of > replacing all the +'s with %2B it spits out a blank string (i.e. an empty > column). > > Any tips on how to get around this issue? yes, "+" is a magic character for regular expressions so it has to be escaped. The easiest way to do that is: replaceAll(Name2,"\\+","%2B") (you need two backslashes because the first one escapes the second one, and the second one then escapes the plus sign). Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From Francois.Ochsenbein@astro.XXX Thu Jul 25 14:05:19 2013 Date: Thu, 25 Jul 2013 15:05:14 +0200 From: Francois Ochsenbein To: Mark Taylor Cc: topcat-user@mlist2.XXX Subject: Re: Replace functions don't like + symbols Hi, Just a brief suggestion: rather than the 2 backslashes, the brackets also works -- easier to remember and works for any character, being "magic" or not: replaceAll(Name2,"[+]","%2B") Cheers, françois Le 24/07/2013 10:48, Mark Taylor a écrit : > On Wed, 24 Jul 2013, sean.farrell@sydney.XXX wrote: > >> Hi Mark, >> >> i'm trying to replace all instances of + in a column of strings with %2B so I >> can read it into a URL string, but the replace functions don't seem to like >> this. Below is the syntax that I'm using: >> >> replaceAll(Name2,"+","%2B") >> >> where Name2 is the name of the column that contains all my strings. Instead of >> replacing all the +'s with %2B it spits out a blank string (i.e. an empty >> column). >> >> Any tips on how to get around this issue? > > yes, "+" is a magic character for regular expressions so it has to > be escaped. The easiest way to do that is: > > replaceAll(Name2,"\\+","%2B") > > (you need two backslashes because the first one escapes the second one, > and the second one then escapes the plus sign). > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > -- ====================================================================== Francois Ochsenbein --- Observatoire Astronomique de Strasbourg ochsenbein@evc.XXX --- francois.ochsenbein@astro.XXX +33 (0)3 88 77 81 17 --- +33 (0)3 68 85 24 29 ====================================================================== From jascenso@eso.XXX Fri Jul 26 14:50:46 2013 Date: Fri, 26 Jul 2013 15:47:52 +0200 From: Joana Ascenso To: topcat-user@bristol.XXX Subject: update subsets in Plane Layer Plot Dear Mark, The new plot options are very useful, thanks for implementing this. It would be great to have them in the toolbar at some point. I am having problems updating the subsets in the plane layer plot. If I change the subset conditions in the subset window, the subsets are updated in the "traditional" plots (if I unclick and then click again on the subset), but not in the plane layer plot. Is there any way to force the update? Thanks and regards, Joana From m.b.taylor@bristol.XXX Fri Jul 26 14:59:59 2013 Date: Fri, 26 Jul 2013 14:59:59 +0100 (BST) From: Mark Taylor To: Joana Ascenso Cc: topcat-user@bristol.XXX Subject: Re: update subsets in Plane Layer Plot On Fri, 26 Jul 2013, Joana Ascenso wrote: > Dear Mark, > > The new plot options are very useful, thanks for implementing this. It thanks Joana, I'm very pleased that you're finding them useful, especially with multiple subsets, which is where I think they are a bit more fiddly to use than the classic plots. > would be great to have them in the toolbar at some point. My feeling is that there are still some issues with usability and rough edges that I'm slowly getting round to improving; I plan to deprecate the classic plots (i.e. add the new ones in the toolbar) when I've got it a bit more satisfactory, but if I get more positive feedback than negative maybe I'll make it sooner rather than later. > I am having problems updating the subsets in the plane layer plot. If I > change the subset conditions in the subset window, the subsets are updated > in the "traditional" plots (if I unclick and then click again on the > subset), but not in the plane layer plot. Is there any way to force the > update? Yes, you can hit the Replot button (in the Plot menu, or toolbar equivalent). I'll look into whether it should get updated automatically. Thanks for the query/report Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From dtucker@fnal.XXX Mon Jul 29 15:47:05 2013 Date: Mon, 29 Jul 2013 09:45:01 -0500 From: Douglas Tucker To: topcat-user@bristol.XXX Subject: multi-cone-search and the vo registries... Greetings, all, Anyone else been having problems with searching the registries this past week under the multi-cone-search function? Searching the registries for keywords for even popular surveys (2mass, sdss, gsc, ...) has been finding no results (see attached png) for any of the registries -- http://registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 http://alt.registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 http://nvo.stsci.edu/vor10/ristandardservice.asmx http://registry.euro-vo.org/services/RegistrySearch I don't think this is a topcat problem, since I've been getting the same problem using different versions of topcat and on different platforms, and just in the past several days. I also hear that at least some users at STScI have been experiencing similar problems the past week (so the problem does not appear to be localized). Thanks! Best regards, Douglas PS: Thanks, Mark, for creating TOPCAT and STILTS -- they are life-savers! PPS: Update: it seems that the European VO registry -- http://registry.euro-vo.org/services/RegistrySearch -- seems to be back up and working within the TOPCAT cone search function. -- ======================================================================= Douglas L. Tucker Fermilab Tel: +1-630-840-2267 MS 127 FAX: +1-630-840-8274 PO Box 500 E-mail: dtucker@fnal.XXX Batavia, IL 60510 USA http://home.fnal.gov/~dtucker/ ======================================================================= [ Part 2, Image/PNG (Name: "topcat-conesearch.png") 1.2 MB. ] [ Unable to print this part. ] From jbrinchmann@gmail.XXX Tue Aug 6 11:47:27 2013 Date: Tue, 6 Aug 2013 12:46:44 +0200 From: Jarle Brinchmann To: topcat-user Subject: Specifying type (Class) of columns in STILTS Hi, I'm probably overlooking something very fundamental but I am trying to simply specify excplicitly the type of each column in a set of csv files explicitly using stilts (I need to glue them later and the automatic guessing of type doesn't work for all columns because they look different in some files.). This seems like it ought to be easy to do but with colmeta I do not seem to be able to set the type of each column. Is there an easy way to do this? Cheers, Jarle. From m.b.taylor@bristol.XXX Tue Aug 6 12:01:04 2013 Date: Tue, 6 Aug 2013 12:01:03 +0100 (BST) From: Mark Taylor To: Jarle Brinchmann Cc: topcat-user Subject: Re: Specifying type (Class) of columns in STILTS Hi Jarle, On Tue, 6 Aug 2013, Jarle Brinchmann wrote: > I'm probably overlooking something very fundamental but I am trying to > simply specify excplicitly the type of each column in a set of csv files > explicitly using stilts (I need to glue them later and the automatic > guessing of type doesn't work for all columns because they look different > in some files.). > > This seems like it ought to be easy to do but with colmeta I do not seem to > be able to set the type of each column. Is there an easy way to do this? Yes, colmeta sounds like a good bet, but (at least the way stilts thinks about it) this is about changing the values rather than the metadata of the columns. So use replacecol with a type-changing function, e.g.: cmd='replacecol mag toDouble(mag)' Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From jbrinchmann@gmail.XXX Tue Aug 6 12:17:49 2013 Date: Tue, 6 Aug 2013 13:17:46 +0200 From: Jarle Brinchmann To: Mark Taylor Cc: topcat-user Subject: Re: Specifying type (Class) of columns in STILTS Ah, That works very nicely, except for columns that are actually string columns but which stilts think are Boolean in some cases... It complains that toString(boolean) can not be accepted (this is using STILTS v 2.5-1). I was a bit surprised by that because the documentation states *toString( objVal )* Turns any value into a string. As applied to existing string values this isn't really useful, but it means that you can apply toString() to any value without knowing its type and get a useful return from it. so I thought it would work. Is there a work-around for this? Cheers, Jarle. On Tue, Aug 6, 2013 at 1:01 PM, Mark Taylor wrote: > Hi Jarle, > > On Tue, 6 Aug 2013, Jarle Brinchmann wrote: > > > I'm probably overlooking something very fundamental but I am trying to > > simply specify excplicitly the type of each column in a set of csv files > > explicitly using stilts (I need to glue them later and the automatic > > guessing of type doesn't work for all columns because they look different > > in some files.). > > > > This seems like it ought to be easy to do but with colmeta I do not seem > to > > be able to set the type of each column. Is there an easy way to do this? > > Yes, colmeta sounds like a good bet, but (at least the way stilts thinks > about it) this is about changing the values rather than the metadata > of the columns. So use replacecol with a type-changing function, > e.g.: > > cmd='replacecol mag toDouble(mag)' > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > From m.b.taylor@bristol.XXX Tue Aug 6 12:50:14 2013 Date: Tue, 6 Aug 2013 12:50:14 +0100 (BST) From: Mark Taylor To: Jarle Brinchmann Cc: topcat-user Subject: Re: Specifying type (Class) of columns in STILTS On Tue, 6 Aug 2013, Jarle Brinchmann wrote: > Ah, That works very nicely, except for columns that are actually string > columns but which stilts think are Boolean in some cases... > > It complains that toString(boolean) can not be accepted (this is using > STILTS v 2.5-1). I was a bit surprised by that because the documentation > states > *toString( objVal )* Turns any value into a string. As applied to existing > string values this isn't really useful, but it means that you can apply > toString() to any value without knowing its type and get a useful return > from it. The fantastically unhelpful analysis of this is that a boolean isn't an object so you're not using the method toString(objVal) for which that documentation is written. Sorry about that, I'll add a variant of toString that works with booleans to prevent that happening. > so I thought it would work. Is there a work-around for this? Can you try the somewhat less intuitive form ""+xxx rather than toString(xxx) Quoting issues may make it challenging to get that right on the command line, if you have trouble I can try to help. Alternatively I can get you a pre-release version with the boolean-handling fix I mention above. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From jbrinchmann@gmail.XXX Tue Aug 6 13:05:29 2013 Date: Tue, 6 Aug 2013 14:05:27 +0200 From: Jarle Brinchmann To: Mark Taylor Cc: topcat-user Subject: Re: Specifying type (Class) of columns in STILTS The ""+xxx works nicely, thanks! However then I stumbled on a couple of columns that I try to convert to Floats (and doubles etc...) but which stilts think are Boolean again. Doing toDouble(0.0)+xxx doesn't work in this case so I needed to use parseFloat(""+xx) which works but seems a tad awkward - specifying column types for csv/ascii files in a future release would really be nice, but I have a work-around for now. Thanks a bunch! Cheers, Jarle. On Tue, Aug 6, 2013 at 1:50 PM, Mark Taylor wrote: > On Tue, 6 Aug 2013, Jarle Brinchmann wrote: > > > Ah, That works very nicely, except for columns that are actually string > > columns but which stilts think are Boolean in some cases... > > > > It complains that toString(boolean) can not be accepted (this is using > > STILTS v 2.5-1). I was a bit surprised by that because the documentation > > states > > *toString( objVal )* Turns any value into a string. As applied to > existing > > string values this isn't really useful, but it means that you can apply > > toString() to any value without knowing its type and get a useful return > > from it. > > The fantastically unhelpful analysis of this is that a boolean isn't > an object so you're not using the method toString(objVal) for which > that documentation is written. Sorry about that, I'll add a variant > of toString that works with booleans to prevent that happening. > > > so I thought it would work. Is there a work-around for this? > > Can you try the somewhat less intuitive form > > ""+xxx > > rather than > > toString(xxx) > > Quoting issues may make it challenging to get that right on the command > line, if you have trouble I can try to help. Alternatively I can get > you a pre-release version with the boolean-handling fix I mention above. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > From m.b.taylor@bristol.XXX Tue Aug 6 17:17:04 2013 Date: Tue, 6 Aug 2013 17:17:03 +0100 (BST) From: Mark Taylor To: Douglas Tucker Cc: topcat-user@bristol.XXX Subject: Re: multi-cone-search and the vo registries... Douglas, sorry for the delay, I was on vacation. Registry provision has long been in a fairly sorry state, which I will refrain from banging on about here. As you suspect, what you are seeing is a server-side issue and nothing to do with topcat as such. I'm glad that the euro-vo registry picked itself up again, and I've just mailed the (ex-)AstroGrid people so at least the main AG registry is now running, though the alt one seems to be still having problems. For the future: the way registry services talk to clients is currently being overhauled within the Virtual Observatory. I hope to update topcat accordingly in the medium term (next few months), and following that this should get considerably more reliable. While I'm there I may tweak the user interface a bit too, so if anyone has particular gripes or suggestions about that, by all means pass them on (though no guarantees I'll implement them). Mark On Mon, 29 Jul 2013, Douglas Tucker wrote: > Greetings, all, > > Anyone else been having problems with searching the registries > this past week under the multi-cone-search function? > > Searching the registries for keywords for even popular surveys > (2mass, sdss, gsc, ...) has been finding no results (see attached png) > for any of the registries -- > http://registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 > http://alt.registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 > http://nvo.stsci.edu/vor10/ristandardservice.asmx > http://registry.euro-vo.org/services/RegistrySearch > > I don't think this is a topcat problem, since I've been getting the same > problem using different versions of topcat and on different platforms, > and just in the past several days. I also hear that at least some users > at STScI have been experiencing similar problems the past week (so the > problem does not appear to be localized). > > Thanks! > > Best regards, > Douglas > > PS: Thanks, Mark, for creating TOPCAT and STILTS -- they are life-savers! > > PPS: Update: it seems that the European VO registry -- > http://registry.euro-vo.org/services/RegistrySearch > -- seems to be back up and working within the TOPCAT > cone search function. > > > -- > > ======================================================================= > Douglas L. Tucker > Fermilab Tel: +1-630-840-2267 > MS 127 FAX: +1-630-840-8274 > PO Box 500 E-mail: dtucker@fnal.XXX > Batavia, IL 60510 USA http://home.fnal.gov/~dtucker/ > ======================================================================= > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From dtucker@fnal.XXX Wed Aug 7 18:04:59 2013 Date: Wed, 7 Aug 2013 12:04:56 -0500 From: Douglas Tucker To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: multi-cone-search and the vo registries... Thanks, Mark! Yep, the registries seem to be doing a lot better this week. Looking forward to future releases of TOPCAT and STILTS! Best regards, Douglas On 8/6/13 11:17 AM, Mark Taylor wrote: > Douglas, > > sorry for the delay, I was on vacation. > > Registry provision has long been in a fairly sorry state, which I > will refrain from banging on about here. As you suspect, what you > are seeing is a server-side issue and nothing to do with topcat as such. > I'm glad that the euro-vo registry picked itself up again, > and I've just mailed the (ex-)AstroGrid people so at least the > main AG registry is now running, though the alt one seems to be > still having problems. > > For the future: the way registry services talk to clients is > currently being overhauled within the Virtual Observatory. > I hope to update topcat accordingly in the medium term > (next few months), and following that this should get considerably > more reliable. While I'm there I may tweak the user interface a > bit too, so if anyone has particular gripes or suggestions about that, > by all means pass them on (though no guarantees I'll implement them). > > Mark > > On Mon, 29 Jul 2013, Douglas Tucker wrote: > >> Greetings, all, >> >> Anyone else been having problems with searching the registries >> this past week under the multi-cone-search function? >> >> Searching the registries for keywords for even popular surveys >> (2mass, sdss, gsc, ...) has been finding no results (see attached png) >> for any of the registries -- >> http://registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 >> http://alt.registry.astrogrid.org/astrogrid-registry/services/RegistryQueryv1_0 >> http://nvo.stsci.edu/vor10/ristandardservice.asmx >> http://registry.euro-vo.org/services/RegistrySearch >> >> I don't think this is a topcat problem, since I've been getting the same >> problem using different versions of topcat and on different platforms, >> and just in the past several days. I also hear that at least some users >> at STScI have been experiencing similar problems the past week (so the >> problem does not appear to be localized). >> >> Thanks! >> >> Best regards, >> Douglas >> >> PS: Thanks, Mark, for creating TOPCAT and STILTS -- they are life-savers! >> >> PPS: Update: it seems that the European VO registry -- >> http://registry.euro-vo.org/services/RegistrySearch >> -- seems to be back up and working within the TOPCAT >> cone search function. >> >> >> -- >> >> ======================================================================= >> Douglas L. Tucker >> Fermilab Tel: +1-630-840-2267 >> MS 127 FAX: +1-630-840-8274 >> PO Box 500 E-mail: dtucker@fnal.XXX >> Batavia, IL 60510 USA http://home.fnal.gov/~dtucker/ >> ======================================================================= >> >> > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > -- ======================================================================= Douglas L. Tucker Fermilab Tel: +1-630-840-2267 MS 127 FAX: +1-630-840-8274 PO Box 500 E-mail: dtucker@fnal.XXX Batavia, IL 60510 USA http://home.fnal.gov/~dtucker/ ======================================================================= From m.b.taylor@bristol.XXX Thu Aug 29 11:20:03 2013 Date: Thu, 29 Aug 2013 11:20:02 +0100 (BST) From: Mark Taylor To: Sean Lake Cc: topcat-user@bristol.XXX Subject: Re: Suggested New Feature Sean, thanks for these interesting suggestions. Non-tophat variants of the convolution kernels (I'd probably truncate them fairly close to the centre for performance reasons) would require some infrastructure changes but shouldn't take a vast amount of effort. Viewing the density plots in terms of a joint probability distribution is something I hadn't thought about before. I think normalising the rows and columns should be a fairly easy option to introduce. However it's not obvious to me what the practical application is. Can you give me an example of when you'd want to see plots like this? +1s for these features from other readers either on list or direct to me are welcome. Mark On Wed, 28 Aug 2013, Sean Lake wrote: > Hello all, > > Might I suggest a new feature for the TOPCAT plots? Well, two new features, really. I really like the new density mode in the plotting window and would appreciate two extensions to it. > > First, the shapes available amount to convolution of the data with the symbol and for that the three most useful symbols are filled circle (euclidian distance tophat), filled diamond (Manhattan distance tophat), and filled square (max( abs(x - x0), abs(y-y0) ) tophat). It would be nice be to have some sort of "smoothed" versions of these symbols. For instance, small graphics with alpha channels for the points could be generated by, say, plotting exp( - distance * distance ) with distance = the relevant type of distance (euclidean, Manhattan, or max perpendicular offset). > > Second, the 2-dimensional density this plots is essentially proportional to the joint probability distribution function for x and y. The ability to plot a conditional probability is also useful. This is done by producing the joint image as normal then normalizing the rows (for x|y) or columns (for y|x). > > Thanks, > Sean -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From rbarba@dfuls.XXX Thu Aug 29 21:14:33 2013 Date: Thu, 29 Aug 2013 16:13:51 -0400 From: Rodolfo Barbá To: Mark Taylor Cc: Sean Lake , topcat-user@bristol.XXX Subject: Re: Suggested New Feature Hi all, The use of Hess diagrams is strongly needed to visualise plots with millions of dots. I'd like this feature in Topcat. All the best. Rodolfo 2013/8/29 Mark Taylor > Sean, > > thanks for these interesting suggestions. Non-tophat variants of the > convolution kernels (I'd probably truncate them fairly close to the > centre for performance reasons) would require some infrastructure > changes but shouldn't take a vast amount of effort. > > Viewing the density plots in terms of a joint probability > distribution is something I hadn't thought about before. > I think normalising the rows and columns should be a fairly easy > option to introduce. However it's not obvious to me what the > practical application is. Can you give me an example of when > you'd want to see plots like this? > > +1s for these features from other readers either on list or direct > to me are welcome. > > Mark > > On Wed, 28 Aug 2013, Sean Lake wrote: > > > Hello all, > > > > Might I suggest a new feature for the TOPCAT plots? Well, two new > features, really. I really like the new density mode in the plotting window > and would appreciate two extensions to it. > > > > First, the shapes available amount to convolution of the data with the > symbol and for that the three most useful symbols are filled circle > (euclidian distance tophat), filled diamond (Manhattan distance tophat), > and filled square (max( abs(x - x0), abs(y-y0) ) tophat). It would be nice > be to have some sort of "smoothed" versions of these symbols. For instance, > small graphics with alpha channels for the points could be generated by, > say, plotting exp( - distance * distance ) with distance = the relevant > type of distance (euclidean, Manhattan, or max perpendicular offset). > > > > Second, the 2-dimensional density this plots is essentially proportional > to the joint probability distribution function for x and y. The ability to > plot a conditional probability is also useful. This is done by producing > the joint image as normal then normalizing the rows (for x|y) or columns > (for y|x). > > > > Thanks, > > Sean > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > > -- La utopía está en el horizonte. Me acerco dos pasos, ella se aleja dos pasos. Camino diez pasos y el horizonte se desplaza diez pasos más allá. Por mucho que camine, nunca la alcanzaré. ¿Para qué sirve la utopia? Para eso: sirve para caminar. Eduardo Galeano From M.B.Taylor@bristol.XXX Thu Aug 29 23:41:01 2013 Date: Thu, 29 Aug 2013 23:40:55 +0100 (BST) From: Mark Taylor To: Rodolfo Barbá Cc: topcat-user@bristol.XXX Subject: Re: Suggested New Feature Rodolfo, Density/scatter plots available in the new-style Layer plots from the Graphics menu provide a generalisation of a density map, which I think can do what you're asking for. See the introduction to the new visualisation features in topcat v4 (especially the first couple of examples) here: http://www.star.bris.ac.uk/~mbt/topcat/v4_graphics.html However since I didn't know the term Hess diagram before I looked it up on Wikipedia ten minutes ago, I might be misinterpreting the requirements - let me know if so. Mark On Thu, 29 Aug 2013, Rodolfo Barbá wrote: > Hi all, > > The use of Hess diagrams is strongly needed to visualise plots with > millions of dots. I'd like this feature in Topcat. > All the best. > > Rodolfo > > > > 2013/8/29 Mark Taylor > > > Sean, > > > > thanks for these interesting suggestions. Non-tophat variants of the > > convolution kernels (I'd probably truncate them fairly close to the > > centre for performance reasons) would require some infrastructure > > changes but shouldn't take a vast amount of effort. > > > > Viewing the density plots in terms of a joint probability > > distribution is something I hadn't thought about before. > > I think normalising the rows and columns should be a fairly easy > > option to introduce. However it's not obvious to me what the > > practical application is. Can you give me an example of when > > you'd want to see plots like this? > > > > +1s for these features from other readers either on list or direct > > to me are welcome. > > > > Mark > > > > On Wed, 28 Aug 2013, Sean Lake wrote: > > > > > Hello all, > > > > > > Might I suggest a new feature for the TOPCAT plots? Well, two new > > features, really. I really like the new density mode in the plotting window > > and would appreciate two extensions to it. > > > > > > First, the shapes available amount to convolution of the data with the > > symbol and for that the three most useful symbols are filled circle > > (euclidian distance tophat), filled diamond (Manhattan distance tophat), > > and filled square (max( abs(x - x0), abs(y-y0) ) tophat). It would be nice > > be to have some sort of "smoothed" versions of these symbols. For instance, > > small graphics with alpha channels for the points could be generated by, > > say, plotting exp( - distance * distance ) with distance = the relevant > > type of distance (euclidean, Manhattan, or max perpendicular offset). > > > > > > Second, the 2-dimensional density this plots is essentially proportional > > to the joint probability distribution function for x and y. The ability to > > plot a conditional probability is also useful. This is done by producing > > the joint image as normal then normalizing the rows (for x|y) or columns > > (for y|x). > > > > > > Thanks, > > > Sean > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > > > > > > > -- > La utopía está en el horizonte. Me acerco dos pasos, ella se aleja dos > pasos. Camino diez pasos y el horizonte se desplaza diez pasos más allá. > Por mucho que camine, nunca la alcanzaré. ¿Para qué sirve la utopia? Para > eso: sirve para caminar. Eduardo Galeano > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From ivan.zolotukhin@gmail.XXX Wed Oct 9 15:36:25 2013 Date: Wed, 9 Oct 2013 16:35:38 +0200 From: Ivan Zolotukhin To: topcat-user Subject: enforce datatype when reading CSV Hi Mark, I'm reading a csv table with TOPCAT/STILTS and can't make it read second column col2 as a string without casting to integer and hence removing leading zeros. "col1","col2","col3","col4" "1042140","0551020901","17","510857" "1000127","0206320101","46","468817" Any clues how to enforce string datatype for col2 here? -- With best regards, Ivan From m.b.taylor@bristol.XXX Wed Oct 9 15:53:43 2013 Date: Wed, 9 Oct 2013 15:53:42 +0100 (BST) From: Mark Taylor To: Ivan Zolotukhin Cc: topcat-user Subject: Re: enforce datatype when reading CSV On Wed, 9 Oct 2013, Ivan Zolotukhin wrote: > Hi Mark, > > I'm reading a csv table with TOPCAT/STILTS and can't make it read > second column col2 as a string without casting to integer and hence > removing leading zeros. > > "col1","col2","col3","col4" > "1042140","0551020901","17","510857" > "1000127","0206320101","46","468817" > > Any clues how to enforce string datatype for col2 here? The quotes in CSV format are just a quoting mechanism to allow you to insert linebreaks or commas - they don't affect the type as which cell values are interpreted. Unless anyone can tell me different, I don't think what you want can be done with the file as it stands from TOPCAT/STILTS. The only thing you could do to prevent it from turning it into an integer before further processing would be to insert a non-integer value into the column, e.g. edit the file so the first entry is "0551020901x". However, as a special case if those values just have a fixed number of digits, you could recover the strings you want with the padWithZeros function: padWithZeros(col2,10). -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From zmzff@mail.XXX Tue Oct 22 04:40:53 2013 Date: Tue, 22 Oct 2013 04:40:07 +0100 From: zmzff@mail.XXX To: topcat-user@sympa.XXX Subject: problem of tcube task when working with jython Hi, I want to create a python script, which will include tcube task. However, I cannot make the task work. Here is the code: import stilts as st cat_fn = 'catalog.fits' cat = st.tread(cat_fn, fmt='fits') cat_fluxerr_histo = st.tcube(cat, cols='fluxerr', binsizes=0.1) # if it works as said in the help message, it should return a 1-d array to cat_fluxerr_histo This code raises the following exception: Traceback (most recent call last): File "./stat_phot.py", line 44, in phot_errcor cat_fluxerr_histo = st.tcube(in_=cat_fn, cols='fluxerr', binsizes=0.1) File "/home/ma/Codes/stilts/stilts.jar/stilts.py", line 4528, in tcube File "/home/ma/Codes/stilts/stilts.jar/stilts.py", line 1365, in _check_unused_args File "", line None SyntaxError: Unused STILTS parameters ('cols', 'binsizes') This is strange because >>help(st.tcube) gives clear document of the function, which takes kwargs of 'cols' and 'binsizes'. If anyone have an idea about this problem, I'd appreciate it if you share. Thanks. From m.b.taylor@bristol.XXX Tue Oct 22 11:45:29 2013 Date: Tue, 22 Oct 2013 11:45:25 +0100 (BST) From: Mark Taylor To: zmzff@mail.XXX Cc: topcat-user@sympa.XXX Subject: Re: problem of tcube task when working with jython Dear Zmzff, I'm afraid this is a bug, or rather two bugs. The first is a documentation bug. Where tcube's help says: "The return value is the resulting table." it lies. There is no return value, instead the output is written to a FITS file as N-dimensional image data, as determined by the "out" parameter (make sure you set it, or you'll get FITS to stdout). The second is an implementation bug - it wasn't even doing that. I've fixed the bugs, and you can find a version that works at ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre. However, it probably doesn't do what you want it to, because the output is only available as a FITS file and not as an object within the jython enviroment. Although you could imagine sensible 1-d output from this command as a table, in the N-dimensional case it's not really obvious how to make this information available within jython - in any case, tcube doesn't do it. Thanks for the report, and apologies for the bug, and for the fact that it may not do what you need it to. Mark On Tue, 22 Oct 2013, zmzff@mail.XXX wrote: > Hi, > > I want to create a python script, which will include tcube task. However, I > cannot make the task work. > Here is the code: > > import stilts as st > > cat_fn = 'catalog.fits' > cat = st.tread(cat_fn, fmt='fits') > cat_fluxerr_histo = st.tcube(cat, cols='fluxerr', binsizes=0.1) > # if it works as said in the help message, it should return a 1-d array to > cat_fluxerr_histo > > This code raises the following exception: > > Traceback (most recent call last): > File "./stat_phot.py", line 44, in phot_errcor > cat_fluxerr_histo = st.tcube(in_=cat_fn, cols='fluxerr', binsizes=0.1) > File "/home/ma/Codes/stilts/stilts.jar/stilts.py", line 4528, in tcube > File "/home/ma/Codes/stilts/stilts.jar/stilts.py", line 1365, in > _check_unused_args > File "", line None > SyntaxError: Unused STILTS parameters ('cols', 'binsizes') > > This is strange because > > >>help(st.tcube) > > gives clear document of the function, which takes kwargs of 'cols' and > 'binsizes'. > > If anyone have an idea about this problem, I'd appreciate it if you share. > Thanks. > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From john.morgan@curtin.XXX Wed Oct 23 09:25:36 2013 Date: Wed, 23 Oct 2013 11:12:13 +0800 From: John Morgan To: topcat-user@sympa.XXX Subject: Bug reports / feature requests Hi all, I have been using topcat and stilts very extensively over the last few months and have found them extremely useful to say the least (there will certainly be a glowing acknowledgement in my next paper!). I've also noted a couple of bugs (or at least counter-intuitive behaviours), and I also have a couple of feature requests. Giving a quick talk to collaborators on using topcat motivated me to write these up. *Bug** *In exact matches, "short" integers will not match with "int" integers even if they have the same value, they have to be coerced into the same data type via toInteger(). This is in stilts 2.4. I haven't tried to reproduce it in the latest version, but it isn't listed explicitly in the bugfixes in the version history. *Feature Requests* The NULL_COLNAME convention for checking for null values works fine, but it took me a while to find and isn't obvious to me. Is there any reason why there isn't a function called something like isNull(colname)? It's great to see hypot() in the latest version! Is there a way to extend topcat/stilts with little convenience functions (e.g. square(x) return(pow(x, 2))) *Documentation** *The documentation is generally excellent. Here's a couple of things that I only figured out recently which I wish I had known from the start (maybe they deserve to go in the FAQs, or be mentioned explicitly as an example in an appropriate place?) - combining two tables side by side via an exact match of $0 and $0 (can be dangerous if the catalogue could have been reordered, but very useful if care is taken!) - getting a nice numbered list of columns via "stilts tpipe in=my_table omode=meta" -- Dr John Morgan Research Fellow VLBI Ph: +61 8 9266 9105 ICRAR: Discovering the hidden Universe through radio astronomy Web: http://www.icrar.org ICRAR's eNewsletter: http://www.icrar.org#subscribe ICRAR on Twitter: http://twitter.com/icrar ICRAR on Facebook: http://www.icrar.org/facebook Curtin Astronomy Club: http://www.facebook.com/curtinastronomyclub From m.b.taylor@bristol.XXX Wed Oct 23 14:44:52 2013 Date: Wed, 23 Oct 2013 14:44:51 +0100 (BST) From: Mark Taylor To: John Morgan Cc: topcat-user@sympa.XXX Subject: Re: Bug reports / feature requests John, thanks for the positive comments and the feedback, and the intention to ackowledge the software in publications. Citations to the topcat paper (2005ASPC..347...29T) especially are much appreciated. On Wed, 23 Oct 2013, John Morgan wrote: > I have been using topcat and stilts very extensively over the last few months > and have found them extremely useful to say the least (there will certainly be > a glowing acknowledgement in my next paper!). I've also noted a couple of bugs > (or at least counter-intuitive behaviours), and I also have a couple of > feature requests. Giving a quick talk to collaborators on using topcat > motivated me to write these up. > > *Bug** > *In exact matches, "short" integers will not match with "int" integers even if > they have the same value, they have to be coerced into the same data type via > toInteger(). > > This is in stilts 2.4. I haven't tried to reproduce it in the latest version, > but it isn't listed explicitly in the bugfixes in the version history. I would consider arguing that this is a feature rather than a bug ... but I've fixed it anyway. This behaviour will change in future releases, you can find a pre-release that matches numeric values by value not type here: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/ I'd be grateful if you could try it out and check that it behaves in what you consider a non-counter-intuitive (and still correct) way. > *Feature Requests* > > The NULL_COLNAME convention for checking for null values works fine, but it > took me a while to find and isn't obvious to me. Is there any reason why there > isn't a function called something like isNull(colname)? Yes. The technical details are somewhat intricate and available on request, but the bottom line is that it would be very difficult to add a function like this to the expression language with the normal function syntax. > It's great to see hypot() in the latest version! Is there a way to extend > topcat/stilts with little convenience functions (e.g. square(x) return(pow(x, > 2))) This kind of thing is easy to add, as long as a straightforward analytic expression is available. If you want to suggest other functions you'd like to see alongside square(x) I can add them. I'll note though that I'm wary of adding a huge number of trivial functions, just because it makes the documentation bigger, and makes it harder to find those functions that you actually want to use. > *Documentation** > *The documentation is generally excellent. Here's a couple of things that I thanks! > only figured out recently which I wish I had known from the start (maybe they > deserve to go in the FAQs, or be mentioned explicitly as an example in an > appropriate place?) > > - combining two tables side by side via an exact match of $0 and $0 (can be > dangerous if the catalogue could have been reordered, but very useful if care > is taken!) Ha! this is in the FAQ already: http://www.star.bristol.ac.uk/~mbt/topcat/faq.html#Q3.11 > - getting a nice numbered list of columns via "stilts tpipe in=my_table > omode=meta" and this one does exist in an example, if you look hard enough (http://www.star.bristol.ac.uk/~mbt/stilts/sun256/tpipeExamples.html) But I think the basic problem here is that there's an awful lot of functionality, so presenting it in a way that means people will end up knowing what they need to without expending a fair amount of effort to find it is hard. No doubt I could do it better though - concrete suggestions for how are welcome. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From S.Oliver@sussex.XXX Thu Oct 31 12:35:59 2013 Date: Thu, 31 Oct 2013 12:35:02 +0000 From: Seb Oliver To: topcat-user@sympa.XXX Subject: Two questions Dear All, Couple of questions.... 1. Does anyone know if there is a simple way within VO friendly tools to read a map at the positions in a catalogue? E.g. I have a map which I want to load into (say) Aladin and a catalogue in Topcat and I want to create a column in the catalogue which is the flux in that map at the position of the catalogue. 2. Does anyone know if there is a way of doing a region file query of catalogue servers. I.e. something like a cone search but for an arbitrary shapped region (defined e.g. in a .reg file), specifically not a circular region or rectangular region in RA,DEC. Many thanks, Seb. From thomas.boch@astro.XXX Thu Oct 31 14:24:38 2013 Date: Thu, 31 Oct 2013 15:23:55 +0100 From: Thomas Boch To: Seb Oliver Cc: topcat-user@mlist2.XXX Subject: Re: Two questions Dear Seb, Le 31/10/13 13:35, Seb Oliver a écrit : > Dear All, > > Couple of questions.... > > > 1. Does anyone know if there is a simple way within VO friendly tools to read a map at the positions in a catalogue? E.g. I have a map which I want to load into (say) Aladin and a catalogue in Topcat and I want to create a column in the catalogue which is the flux in that map at the position of the catalogue. If the map is not too large, you could do the following : - load the map (FITS image) in Aladin - extract the pixel values (menu Image-->Pixel extraction). This will create a new catalogue plane with one row per pixel. - send the catalogue to TOPCAT - use TOPCAT cross-match tool to join your catalogue with the catalogue exported from the map > > 2. Does anyone know if there is a way of doing a region file query of catalogue servers. I.e. something like a cone search but for an arbitrary shapped region (defined e.g. in a .reg file), specifically not a circular region or rectangular region in RA,DEC. TAP (Table Access Protocol) allows for this kind of query, but I'm not sure how (if) this is supported by services implementing the protocol. Cheers, -- Thomas From odysseus9672@gmail.XXX Thu Nov 14 06:16:44 2013 Date: Wed, 13 Nov 2013 22:15:50 -0800 From: Sean Lake To: "topcat-user@sympa.XXX" Subject: Layer Plot Density Mode Label Hello all, Is it possible to get a labeled color bar on a density mode plot? Kind of like the coverage plot here: http://www.aavso.org/apass Thanks, Sean From m.b.taylor@bristol.XXX Thu Nov 14 09:51:01 2013 Date: Thu, 14 Nov 2013 09:51:00 +0000 (GMT) From: Mark Taylor To: Sean Lake Cc: "topcat-user@sympa.XXX" Subject: Re: Layer Plot Density Mode Label Hi Sean, On Wed, 13 Nov 2013, Sean Lake wrote: > Is it possible to get a labeled color bar on a density mode plot? Kind of like the coverage plot here: http://www.aavso.org/apass It's not implemented in the code as it stands. One reason is that (unlike aux mode colouring) the density scale is calculated separately for each dataset plotted, so for multi-set plots you'd need multiple colour bars. Another thing to be aware of is the density colours are per screen pixel, so as it stands you couldn't label an all-sky plot like the one you reference above, since the per-pixel density is a function of the solid-angle-per-pixel value which varies with screen position as well as the points-per-solid-angle value which is of more physical interest. So, I guess at the moment it's more of a visual cue about the structure of the data than a quantitative analysis tool. I don't have plans to improve this for now, but I'll bear it in mind for future developments. If other people are particularly keen to see this functionality, feel free to lobby me. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From nathansecrest@msn.XXX Thu Nov 21 21:31:37 2013 Date: Thu, 21 Nov 2013 16:13:47 -0500 From: Nathan Secrest To: "topcat-user@bristol.XXX" Cc: Shobita Satyapal Subject: TOPCAT WCS matching issue Hello, I've noticed that when I try to match the NASA-Sloan Atlas, which has over 145,000 objects, to either the 3XMM-DR4 or Chandra Point Source Catalogs, which have around 500,000 and 120,000 sources, respectively, I get only a few hundred per catalog. I am matching these catalogs using the "Match Tables" window with the "Sky with Errors" algorithm for XMM, since XMM gives the 1-sigma positional errors on their sources, and the "Sky Ellipses" algorithm for Chandra, since Chandra gives the r0, r1, and PA values for their sources. I am matching these sources to a positional error of 95%, so I am using columns with the 95% positional error, and setting the "Scale" to the mean value of the 95% positional errors. What could be going wrong? Has anyone else reported a similar issue? Thank you, Nathan SecrestGraduate Research AssistantGeorge Mason University From m.b.taylor@bristol.XXX Fri Nov 22 10:14:04 2013 Date: Fri, 22 Nov 2013 10:14:04 +0000 (GMT) From: Mark Taylor To: Nathan Secrest Cc: "topcat-user@bristol.XXX" , Shobita Satyapal Subject: Re: TOPCAT WCS matching issue Nathan, thanks for sending me the data. I think the answer is ... there just are not many sources that appear in both catalogues. I see about a thousand in common between NSA/3XMM and about 200 between NSA/CSC, and from the data that looks to me like the right answer. Have you got a reason for believing this is not the correct result? The best way to work out why a match is not doing what you're expecting is to overplot both catalogues, and the results of the match, and see how it looks. For now that means setting up the plot yourself. I'm working on some changes that mean in a future release this kind of xmatch result visualisation should become absurdly easy to do. If you still think there's a problem, get back to me, probably best to do it off-list. Mark On Thu, 21 Nov 2013, Nathan Secrest wrote: > Hello, > I've noticed that when I try to match the NASA-Sloan Atlas, which has over 145,000 objects, to either the 3XMM-DR4 or Chandra Point Source Catalogs, which have around 500,000 and 120,000 sources, respectively, I get only a few hundred per catalog. > I am matching these catalogs using the "Match Tables" window with the "Sky with Errors" algorithm for XMM, since XMM gives the 1-sigma positional errors on their sources, and the "Sky Ellipses" algorithm for Chandra, since Chandra gives the r0, r1, and PA values for their sources. I am matching these sources to a positional error of 95%, so I am using columns with the 95% positional error, and setting the "Scale" to the mean value of the 95% positional errors. > What could be going wrong? Has anyone else reported a similar issue? > Thank you, > Nathan SecrestGraduate Research AssistantGeorge Mason University -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From jgarrido@iaa.XXX Wed Dec 11 11:52:37 2013 Date: Wed, 11 Dec 2013 12:51:17 +0100 From: Julián Garrido To: topcat-user@bristol.XXX Subject: exceptions querying for tap services Dear Mark and topcat users, TAP searches in topcat (4.0-1) were failing and throwing exceptions. But it looks to me that it doesn't always happen. Do you know if there is any problem in the registries? We have run the code and these were the exceptions (in case it helps). Registry: http://registry.astrogrid.org/astrogrid-registry/services/ RegistryQueryv1_0v Error: Search failed: java.lang.ExceptionInInitializerError Registry: http://registry.astrogrid.org/astrogrid-registry/services/ RegistryQueryv1_0 Error: Search failed: java.lang.NoClassDefFoundError: Could not initialize class com.sun.xml.internal.ws.fault.SOAPFaultBuilder Registry: http://registry.euro-vo.org/services/RegistrySearch Error: Search failed: com.sun.xml.internal.ws.client.ClientTransportException: The server sent HTTP status code 502: Proxy Error Best regards, Julián From m.b.taylor@bristol.XXX Wed Dec 11 12:14:09 2013 Date: Wed, 11 Dec 2013 12:14:08 +0000 (GMT) From: Mark Taylor To: Julián Garrido Cc: topcat-user@bristol.XXX Subject: Re: exceptions querying for tap services Julián, business as usual I fear. The registries are not very robust. I tried it just now and at least the main astrogrid registry and the euro-vo one appear to be OK; certainly I'm not getting those exceptions. If you think they're not working now, can you tell me the details of the search you're doing and I'll have another look. I will be overhauling the registry search in topcat sometime next year (the VO registry interface standard is in the throes of redesign). Mark On Wed, 11 Dec 2013, Julián Garrido wrote: > Dear Mark and topcat users, > > TAP searches in topcat (4.0-1) were failing and throwing exceptions. But it > looks to me that it doesn't always happen. > > Do you know if there is any problem in the registries? > > > We have run the code and these were the exceptions (in case it helps). > > > Registry: http://registry.astrogrid.org/astrogrid-registry/services/ > RegistryQueryv1_0v > Error: Search failed: java.lang.ExceptionInInitializerError > > Registry: http://registry.astrogrid.org/astrogrid-registry/services/ > RegistryQueryv1_0 > Error: Search failed: java.lang.NoClassDefFoundError: Could not initialize > class com.sun.xml.internal.ws.fault.SOAPFaultBuilder > > Registry: http://registry.euro-vo.org/services/RegistrySearch > Error: Search failed: com.sun.xml.internal.ws.client.ClientTransportException: > The server sent HTTP status code 502: Proxy Error > > Best regards, Julián > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From jgarrido@iaa.XXX Fri Dec 13 07:26:20 2013 Date: Fri, 13 Dec 2013 08:25:58 +0100 From: Julián Garrido To: Mark Taylor Cc: Julián Garrido , topcat-user@bristol.XXX Subject: Re: exceptions querying for tap services Hi Mark, you are right, it seems they are working now. If we can reproduce the exceptions at some point, I will contact you again. Thank you, Julián 2013/12/11 Mark Taylor > Julián, > > business as usual I fear. The registries are not very robust. > I tried it just now and at least the main astrogrid registry and the > euro-vo one appear to be OK; certainly I'm not getting those exceptions. > If you think they're not working now, can you tell me the details > of the search you're doing and I'll have another look. > > I will be overhauling the registry search in topcat sometime next year > (the VO registry interface standard is in the throes of redesign). > > Mark > > On Wed, 11 Dec 2013, Julián Garrido wrote: > > > Dear Mark and topcat users, > > > > TAP searches in topcat (4.0-1) were failing and throwing exceptions. But > it > > looks to me that it doesn't always happen. > > > > Do you know if there is any problem in the registries? > > > > > > We have run the code and these were the exceptions (in case it helps). > > > > > > Registry: http://registry.astrogrid.org/astrogrid-registry/services/ > > RegistryQueryv1_0v > > Error: Search failed: java.lang.ExceptionInInitializerError > > > > Registry: http://registry.astrogrid.org/astrogrid-registry/services/ > > RegistryQueryv1_0 > > Error: Search failed: java.lang.NoClassDefFoundError: Could not > initialize > > class com.sun.xml.internal.ws.fault.SOAPFaultBuilder > > > > Registry: http://registry.euro-vo.org/services/RegistrySearch > > Error: Search failed: > com.sun.xml.internal.ws.client.ClientTransportException: > > The server sent HTTP status code 502: Proxy Error > > > > Best regards, Julián > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From M.B.Taylor@bristol.XXX Fri Dec 13 10:00:57 2013 Date: Fri, 13 Dec 2013 10:00:41 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: exceptions querying for tap services thanks. Probably the exceptions originated at the service end, so it may be more use to contact the registry operators if you have the same problem again. Mark On Fri, 13 Dec 2013, Julián Garrido wrote: > Hi Mark, > > you are right, it seems they are working now. If we can reproduce the > exceptions at some point, I will contact you again. > > > Thank you, Julián > > > 2013/12/11 Mark Taylor > > > Julián, > > > > business as usual I fear. The registries are not very robust. > > I tried it just now and at least the main astrogrid registry and the > > euro-vo one appear to be OK; certainly I'm not getting those exceptions. > > If you think they're not working now, can you tell me the details > > of the search you're doing and I'll have another look. > > > > I will be overhauling the registry search in topcat sometime next year > > (the VO registry interface standard is in the throes of redesign). > > > > Mark > > > > On Wed, 11 Dec 2013, Julián Garrido wrote: > > > > > Dear Mark and topcat users, > > > > > > TAP searches in topcat (4.0-1) were failing and throwing exceptions. But > > it > > > looks to me that it doesn't always happen. > > > > > > Do you know if there is any problem in the registries? > > > > > > > > > We have run the code and these were the exceptions (in case it helps). > > > > > > > > > Registry: http://registry.astrogrid.org/astrogrid-registry/services/ > > > RegistryQueryv1_0v > > > Error: Search failed: java.lang.ExceptionInInitializerError > > > > > > Registry: http://registry.astrogrid.org/astrogrid-registry/services/ > > > RegistryQueryv1_0 > > > Error: Search failed: java.lang.NoClassDefFoundError: Could not > > initialize > > > class com.sun.xml.internal.ws.fault.SOAPFaultBuilder > > > > > > Registry: http://registry.euro-vo.org/services/RegistrySearch > > > Error: Search failed: > > com.sun.xml.internal.ws.client.ClientTransportException: > > > The server sent HTTP status code 502: Proxy Error > > > > > > Best regards, Julián > > > > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From gtr@physics.XXX Fri Jan 17 19:18:27 2014 Date: Fri, 17 Jan 2014 14:17:42 -0500 From: Gordon Richards To: topcat-user@sympa.XXX Subject: Reading WISE files with TOPCAT Is there a way to get TOPCAT (or STILTS) to read the bar-delimited bulk catalogs from WISE? They look like this: > head wise-allwise-cat-part18 J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014||||||8.2317599| I can convert them to CSV first, but that is cumbersome. And since the files don't have any header information, is there a way to specify that at the time of reading or after the fact so that the columns are labeled? The problem is with the WISE data file format and not with TOPCAT per se, but if I could solve it with TOPCAT alone, that would be great. Thanks, Gordon Richards From ipf@iac.XXX Fri Jan 17 19:22:32 2014 Date: Fri, 17 Jan 2014 19:20:59 +0000 From: Ismael Perez Fournon To: Gordon Richards , topcat-user@sympa.XXX Ismael Perez Fournon Subject: Re: Reading WISE files with TOPCAT this looks like the IPAC table format, which is one of the input formats in Topcat, but not available as output format. Ismael On 17/01/14 19:17, Gordon Richards wrote: > Is there a way to get TOPCAT (or STILTS) to > read the bar-delimited bulk catalogs from WISE? > > They look like this: > > head wise-allwise-cat-part18 > J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014||||||8.2317599| > > > I can convert them to CSV first, but that is cumbersome. > > And since the files don't have any header information, is there a way > to specify that > at the time of reading or after the fact so that the columns are labeled? > > The problem is with the WISE data file format and not with TOPCAT per > se, but if I > could solve it with TOPCAT alone, that would be great. > > Thanks, > Gordon Richards > > > > --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protección de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From ipf@iac.XXX Fri Jan 17 19:34:23 2014 Date: Fri, 17 Jan 2014 19:33:10 +0000 From: Ismael Perez Fournon To: Gordon Richards Cc: topcat-user@sympa.XXX Ismael Perez Fournon Subject: Re: Reading WISE files with TOPCAT operating with the table in stilts would be something like: stilts tpipe in=table ifmt=ipac ... etc stilts tpipe in=table ifmt=ipac omode=topcat will send the table to topcat if samp has been activated adding the header information should be possible, I think with "colmeta": Look at: http://www.star.bristol.ac.uk/~mbt/stilts/sun256/colmeta.html Ismael On 17/01/14 19:20, Ismael Perez Fournon wrote: > this looks like the IPAC table format, which is one of the input > formats in Topcat, but > not available as output format. > > Ismael > > On 17/01/14 19:17, Gordon Richards wrote: >> Is there a way to get TOPCAT (or STILTS) to >> read the bar-delimited bulk catalogs from WISE? >> >> They look like this: >> > head wise-allwise-cat-part18 >> J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014||||||8.2317599| >> >> >> I can convert them to CSV first, but that is cumbersome. >> >> And since the files don't have any header information, is there a way >> to specify that >> at the time of reading or after the fact so that the columns are >> labeled? >> >> The problem is with the WISE data file format and not with TOPCAT per >> se, but if I >> could solve it with TOPCAT alone, that would be great. >> >> Thanks, >> Gordon Richards >> >> >> >> > > --------------------------------------------------------------------------------------------- > > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de > Protección de Datos, acceda a http://www.iac.es/disclaimer.php > WARNING: For more information on privacy and fulfilment of the Law > concerning the Protection of Data, consult > http://www.iac.es/disclaimer.php?lang=en > > --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protección de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From ipf@iac.XXX Fri Jan 17 19:52:28 2014 Date: Fri, 17 Jan 2014 19:51:13 +0000 From: Ismael Perez Fournon To: Gordon Richards Cc: topcat-user@sympa.XXX Ismael Perez Fournon Subject: Re: Reading WISE files with TOPCAT sorry, the IPAC table format is different from your WISE bar-delimited table: http://www.star.bris.ac.uk/~mbt/topcat/sun253/sun253.html#inIpac Ismael On 17/01/14 19:33, Ismael Perez Fournon wrote: > > operating with the table in stilts would be something like: > > stilts tpipe in=table ifmt=ipac ... etc > > stilts tpipe in=table ifmt=ipac omode=topcat > will send the table to topcat if samp has been activated > > adding the header information should be possible, I think with "colmeta": > > Look at: > > http://www.star.bristol.ac.uk/~mbt/stilts/sun256/colmeta.html > > > Ismael > > > On 17/01/14 19:20, Ismael Perez Fournon wrote: >> this looks like the IPAC table format, which is one of the input >> formats in Topcat, but >> not available as output format. >> >> Ismael >> >> On 17/01/14 19:17, Gordon Richards wrote: >>> Is there a way to get TOPCAT (or STILTS) to >>> read the bar-delimited bulk catalogs from WISE? >>> >>> They look like this: >>> > head wise-allwise-cat-part18 >>> J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014||||||8.2317599| >>> >>> >>> I can convert them to CSV first, but that is cumbersome. >>> >>> And since the files don't have any header information, is there a >>> way to specify that >>> at the time of reading or after the fact so that the columns are >>> labeled? >>> >>> The problem is with the WISE data file format and not with TOPCAT >>> per se, but if I >>> could solve it with TOPCAT alone, that would be great. >>> >>> Thanks, >>> Gordon Richards >>> >>> >>> >>> >> >> --------------------------------------------------------------------------------------------- >> >> ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de >> Protección de Datos, acceda a http://www.iac.es/disclaimer.php >> WARNING: For more information on privacy and fulfilment of the Law >> concerning the Protection of Data, consult >> http://www.iac.es/disclaimer.php?lang=en >> >> > > --------------------------------------------------------------------------------------------- > > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de > Protección de Datos, acceda a http://www.iac.es/disclaimer.php > WARNING: For more information on privacy and fulfilment of the Law > concerning the Protection of Data, consult > http://www.iac.es/disclaimer.php?lang=en > > --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protección de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en From bilicki@camk.XXX Fri Jan 17 20:24:44 2014 Date: Fri, 17 Jan 2014 22:23:25 +0200 From: Maciej Bilicki To: Gordon Richards Cc: topcat-user@sympa.XXX Subject: Re: Reading WISE files with TOPCAT Dear Gordon, What do you mean by "catalogs from WISE"? If you download WISE data from IPAC (Gator), normally you should obtain IPAC tables with appropriate headers, easily opened by TOPCAT once you chose "IPAC" data format. TOPCAT can also save tables in this format. http://www.star.bris.ac.uk/~mbt/topcat/sun253/sun253.html#outIpac Best wishes Maciek 2014/1/17 Gordon Richards > Is there a way to get TOPCAT (or STILTS) to > read the bar-delimited bulk catalogs from WISE? > > They look like this: > > head wise-allwise-cat-part18 > J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0. > 0689|-0.014||||||8.2317599| > > I can convert them to CSV first, but that is cumbersome. > > And since the files don't have any header information, is there a way to > specify that > at the time of reading or after the fact so that the columns are labeled? > > The problem is with the WISE data file format and not with TOPCAT per se, > but if I > could solve it with TOPCAT alone, that would be great. > > Thanks, > Gordon Richards > > > > > From gtr@physics.XXX Fri Jan 17 20:36:18 2014 Date: Fri, 17 Jan 2014 15:35:40 -0500 From: Gordon Richards To: Maciej Bilicki Cc: topcat-user@sympa.XXX Subject: Re: Reading WISE files with TOPCAT I mean the bulk catalogs from http://irsadist.ipac.caltech.edu/wise-allwise/ For some reason these aren't in IPAC format. However, other than that, they are preferable to using Gator if you want every single WISE source (and want to do the matching yourself, say with TOPCAT...) Gator only gives you 2 Gb at a time and the full catalogs are 1.1 Tb. Gordon On 1/17/14 3:23 PM, Maciej Bilicki wrote: > Dear Gordon, > > What do you mean by "catalogs from WISE"? If you download WISE data > from IPAC (Gator), normally you should obtain IPAC tables with > appropriate headers, easily opened by TOPCAT once you chose "IPAC" > data format. TOPCAT can also save tables in this format. > http://www.star.bris.ac.uk/~mbt/topcat/sun253/sun253.html#outIpac > > > Best wishes > Maciek > > > 2014/1/17 Gordon Richards > > > Is there a way to get TOPCAT (or STILTS) to > read the bar-delimited bulk catalogs from WISE? > > They look like this: > > head wise-allwise-cat-part18 > J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014||||||8.2317599| > > I can convert them to CSV first, but that is cumbersome. > > And since the files don't have any header information, is there a > way to specify that > at the time of reading or after the fact so that the columns are > labeled? > > The problem is with the WISE data file format and not with TOPCAT > per se, but if I > could solve it with TOPCAT alone, that would be great. > > Thanks, > Gordon Richards > > > > > From odysseus9672@gmail.XXX Fri Jan 17 21:42:50 2014 Date: Fri, 17 Jan 2014 13:42:12 -0800 From: Sean Lake To: Gordon Richards Cc: Maciej Bilicki , topcat-user@sympa.XXX Subject: Re: Reading WISE files with TOPCAT Interesting. It looks like the made the decision to pre-compress the tables. A short python script will change the files to csv format, though. And example script follows. Sean #!/usr/bin/env python import sys import bz2 import gzip header for fname in sys.argv[1:]: if infl[-2:] == "gz": infl = gzip.GzipFile( fname, "r" ) elif infl[-3:] == "bz2": infl = bz2.BZ2File( fname, "r" ) else: infl = open( fname, "r" ) #Use this for uncompressed input files # outfl = bz2.BZ2File( fname.split(".")[0] + ".csv.bz2", "w" ) outfl = gzip.GzipFile( fname.split(".")[0] + ".csv.gz", "w" ) # outfl = open( fname.split(".")[0] + ".csv", "w" ) for line in infl: parts = line.split("|")[:-1] #Drop the newline outfl.write( ", ".join( parts ) + "\n" ) infl.close() outfl.close() On Jan 17, 2014, at 12:35, Gordon Richards wrote: > I mean the bulk catalogs from > http://irsadist.ipac.caltech.edu/wise-allwise/ > > For some reason these aren't in IPAC format. However, other than that, > they are preferable to using Gator if you want every single WISE source > (and want to do the matching yourself, say with TOPCAT...) > Gator only gives you 2 Gb at a time and the full catalogs are 1.1 Tb. > > Gordon > > > On 1/17/14 3:23 PM, Maciej Bilicki wrote: >> Dear Gordon, >> >> What do you mean by "catalogs from WISE"? If you download WISE data from IPAC (Gator), normally you should obtain IPAC tables with appropriate headers, easily opened by TOPCAT once you chose "IPAC" data format. TOPCAT can also save tables in this format. >> http://www.star.bris.ac.uk/~mbt/topcat/sun253/sun253.html#outIpac >> >> Best wishes >> Maciek >> >> >> 2014/1/17 Gordon Richards > >> >> Is there a way to get TOPCAT (or STILTS) to >> read the bar-delimited bulk catalogs from WISE? >> >> They look like this: >> > head wise-allwise-cat-part18 >> J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014||||||8.2317599| >> >> I can convert them to CSV first, but that is cumbersome. >> >> And since the files don't have any header information, is there a >> way to specify that >> at the time of reading or after the fact so that the columns are >> labeled? >> >> The problem is with the WISE data file format and not with TOPCAT >> per se, but if I >> could solve it with TOPCAT alone, that would be great. >> >> Thanks, >> Gordon Richards >> >> >> >> >> > From croman@astro.XXX Fri Jan 17 22:19:07 2014 Date: Fri, 17 Jan 2014 14:18:31 -0800 From: Carlos Roman To: topcat-user@sympa.XXX Subject: STILTS 3-table match Hi, I am trying to run this STILTS command stilts tmatchn in1=table1.cat in2=table2.cat in3=table3.cat \ ifmt1=ascii ifmt2=ascii ifmt3=ascii omode=out ofmt=ascii out= table123.cat \ matcher=sky params=1.5 nin=3 values1=' RAJ2000 DECJ2000 ' \ values2=' RAJ2000 DECJ2000 ' \ values3=' RAJ2000 DECJ2000 ' join1=always join2=always join3=always \ multimode=pairs so I can use it instead of TOPCAT->joins->triple match with sky matching and "All Rows" in all output rows options. However, the run from TOPCAT appears to do what I want, that is, a table that contains matches of tables 1+2+3, 1+2, 1+3 and 2+3. But it is not doing it so, I think it does not include 1+3. I can tell by looking at the output tables from both TOPCAT and STILTS and by counting output rows: topcat yields more rows than stilts. I am sure I am missing something obvious but I cannot tell what... could you help? Thank you (for this and in general for TOPCAT+STILTS, there was a life before and after them) C. Roman From bilicki@camk.XXX Sat Jan 18 06:31:41 2014 Date: Sat, 18 Jan 2014 08:30:12 +0200 From: Maciej Bilicki To: Sean Lake Cc: Gordon Richards , topcat-user@sympa.XXX Subject: Re: Reading WISE files with TOPCAT I see. Now it they indeed don't have column headers, one probably needs to write some extra script to add them, based on http://irsadist.ipac.caltech.edu/wise-allwise/wise-allwise-cat-schema.txt However, do you really want to work in TOPCAT with a combined table of ~1.1 TB? I can't imagine handling such tables. Already loading such a csv, or saving it into fits, will take ages . Also, x-matching two catalogs with "just" a couple dozen million sources each (order of magnitude smaller than full WISE) is really painful, even if you have loads of RAM and CPUs. Same must be for any other operations with such huge tables. M 2014/1/17 Sean Lake > Interesting. It looks like the made the decision to pre-compress the > tables. A short python script will change the files to csv format, though. > And example script follows. > > Sean > > #!/usr/bin/env python > > import sys > import bz2 > import gzip > > header > > for fname in sys.argv[1:]: > if infl[-2:] == "gz": > infl = gzip.GzipFile( fname, "r" ) > > elif infl[-3:] == "bz2": > infl = bz2.BZ2File( fname, "r" ) > > else: > infl = open( fname, "r" ) #Use this for uncompressed input > files > > # outfl = bz2.BZ2File( fname.split(".")[0] + ".csv.bz2", "w" ) > outfl = gzip.GzipFile( fname.split(".")[0] + ".csv.gz", "w" ) > # outfl = open( fname.split(".")[0] + ".csv", "w" ) > > for line in infl: > parts = line.split("|")[:-1] #Drop the newline > outfl.write( ", ".join( parts ) + "\n" ) > > infl.close() > outfl.close() > > > On Jan 17, 2014, at 12:35, Gordon Richards wrote: > > > I mean the bulk catalogs from > > http://irsadist.ipac.caltech.edu/wise-allwise/ > > > > For some reason these aren't in IPAC format. However, other than that, > > they are preferable to using Gator if you want every single WISE source > > (and want to do the matching yourself, say with TOPCAT...) > > Gator only gives you 2 Gb at a time and the full catalogs are 1.1 Tb. > > > > Gordon > > > > > > On 1/17/14 3:23 PM, Maciej Bilicki wrote: > >> Dear Gordon, > >> > >> What do you mean by "catalogs from WISE"? If you download WISE data > from IPAC (Gator), normally you should obtain IPAC tables with appropriate > headers, easily opened by TOPCAT once you chose "IPAC" data format. TOPCAT > can also save tables in this format. > >> http://www.star.bris.ac.uk/~mbt/topcat/sun253/sun253.html#outIpac < > http://www.star.bris.ac.uk/%7Embt/topcat/sun253/sun253.html#outIpac> > >> > >> Best wishes > >> Maciek > >> > >> > >> 2014/1/17 Gordon Richards gtr@physics.XXX>> > >> > >> Is there a way to get TOPCAT (or STILTS) to > >> read the bar-delimited bulk catalogs from WISE? > >> > >> They look like this: > >> > head wise-allwise-cat-part18 > >> > J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014||||||8.2317599| > >> > >> I can convert them to CSV first, but that is cumbersome. > >> > >> And since the files don't have any header information, is there a > >> way to specify that > >> at the time of reading or after the fact so that the columns are > >> labeled? > >> > >> The problem is with the WISE data file format and not with TOPCAT > >> per se, but if I > >> could solve it with TOPCAT alone, that would be great. > >> > >> Thanks, > >> Gordon Richards > >> > >> > >> > >> > >> > > > > From ivan.zolotukhin@gmail.XXX Sat Jan 18 08:17:14 2014 Date: Sat, 18 Jan 2014 09:16:34 +0100 From: Ivan Zolotukhin To: Gordon Richards Cc: Maciej Bilicki , topcat-user@sympa.XXX Subject: Re: Reading WISE files with TOPCAT Hi, In case it's helpful. That depends on the exact problem of course, but in most of cases you can do match from TOPCAT with every single WISE source without having the local copy of it. First, there's Conesearch service with WISE data at CDS (web page is http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=II/311), then there's Table Access Protocol server at WFAU (http://wfaudata.roe.ac.uk/wise-dsa/TAP endpoint) and at CDS too, though both without data upload feature. Then, there's WISE catalog available for crossmatch with user tables or other large catalogs at CDS X-match service http://cdsxmatch.u-strasbg.fr/xmatch. You may want to consider these options for your problem too. -- With best regards, Ivan On Fri, Jan 17, 2014 at 9:35 PM, Gordon Richards wrote: > I mean the bulk catalogs from > http://irsadist.ipac.caltech.edu/wise-allwise/ > > For some reason these aren't in IPAC format. However, other than that, > they are preferable to using Gator if you want every single WISE source > (and want to do the matching yourself, say with TOPCAT...) > Gator only gives you 2 Gb at a time and the full catalogs are 1.1 Tb. > > Gordon > > > > On 1/17/14 3:23 PM, Maciej Bilicki wrote: > > Dear Gordon, > > What do you mean by "catalogs from WISE"? If you download WISE data from > IPAC (Gator), normally you should obtain IPAC tables with appropriate > headers, easily opened by TOPCAT once you chose "IPAC" data format. TOPCAT > can also save tables in this format. > http://www.star.bris.ac.uk/~mbt/topcat/sun253/sun253.html#outIpac > > Best wishes > Maciek > > > 2014/1/17 Gordon Richards >> >> Is there a way to get TOPCAT (or STILTS) to >> read the bar-delimited bulk catalogs from WISE? >> >> They look like this: >> > head wise-allwise-cat-part18 >> >> J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014||||||8.2317599| >> >> I can convert them to CSV first, but that is cumbersome. >> >> And since the files don't have any header information, is there a way to >> specify that >> at the time of reading or after the fact so that the columns are labeled? >> >> The problem is with the WISE data file format and not with TOPCAT per se, >> but if I >> could solve it with TOPCAT alone, that would be great. >> >> Thanks, >> Gordon Richards >> >> >> >> > > From croman@astro.XXX Sat Jan 18 23:39:30 2014 Date: Sat, 18 Jan 2014 15:38:32 -0800 From: Carlos Roman To: topcat-user@sympa.XXX Subject: STILTS 3-table match Hi, I am trying to run this STILTS command stilts tmatchn in1=table1.cat in2=table2.cat in3=table3.cat \ ifmt1=ascii ifmt2=ascii ifmt3=ascii omode=out ofmt=ascii out= table123.cat \ matcher=sky params=1.5 nin=3 values1=' RAJ2000 DECJ2000 ' \ values2=' RAJ2000 DECJ2000 ' \ values3=' RAJ2000 DECJ2000 ' join1=always join2=always join3=always \ multimode=pairs so I can use it instead of TOPCAT->joins->triple match with sky matching and "All Rows" in all output rows options. However, the run from TOPCAT appears to do what I want, that is, a table that contains matches of tables 1+2+3, 1+2, 1+3 and 2+3. But it is not doing it so, I think it does not include 1+3. I can tell by looking at the output tables from both TOPCAT and STILTS and by counting output rows: topcat yields more rows than stilts. I am sure I am missing something obvious but I cannot tell what... could you help? Thank you (for this and in general for TOPCAT+STILTS, there was a life before and after them) C. Roman From M.B.Taylor@bristol.XXX Sun Jan 19 22:24:42 2014 Date: Sun, 19 Jan 2014 22:24:40 +0000 (GMT) From: Mark Taylor To: Maciej Bilicki Cc: Sean Lake , Gordon Richards , topcat-user@sympa.XXX Subject: Re: Reading WISE files with TOPCAT On Sat, 18 Jan 2014, Maciej Bilicki wrote: > I see. > > Now it they indeed don't have column headers, one probably needs to write > some extra script to add them, based on > http://irsadist.ipac.caltech.edu/wise-allwise/wise-allwise-cat-schema.txt > > However, do you really want to work in TOPCAT with a combined table of ~1.1 > TB? I can't imagine handling such tables. Already loading such a csv, or > saving it into fits, will take ages . I don't know how many rows 1.1 Tb of WISE data corresponds to, but it's probably not realistic to work with that amount of data in topcat. If you could download it then doing something with it in STILTS might just be feasible though. In that case you'd almost certainly want to turn it into fits, probably colfits, before doing any processing. > Also, x-matching two catalogs with "just" a couple dozen million sources > each (order of magnitude smaller than full WISE) is really painful, even if > you have loads of RAM and CPUs. Same must be for any other operations with > such huge tables. Crossmatching two very large tables won't work well in topcat, but crossmatching one pretty large one and one smaller one could work in stilts. However, Ivan's suggestions are more likely to be useful here, especially the fearsomely fast CDS xmatch service. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From sajjad_aftabi@yahoo.XXX Mon Jan 20 11:42:53 2014 Date: Mon, 20 Jan 2014 03:11:47 -0800 (PST) From: sajjad aftabi To: "topcat-user@bristol.XXX" Subject: dat or txt file as the input Hi I have a question. How can I read a ".dat" or ".txt" file? Thank you so much. From Elisabetta.Valiante@astro.XXX Mon Jan 20 12:16:11 2014 Date: Mon, 20 Jan 2014 11:51:01 +0000 From: Elisabetta Valiante To: sajjad aftabi Cc: "topcat-user@bristol.XXX" Subject: Re: dat or txt file as the input Have you tried to select 'ASCII' as table format when you open the file? Elisabetta On 20 Jan 2014, at 11:11, sajjad aftabi wrote: > Hi > I have a question. How can I read a ".dat" or ".txt" file? > Thank you so much. ------------------------------------------------------------------------------------------------------------ Elisabetta Valiante Phone +44 - (0)29 - 2087 - 5106 School of Physics and Astronomy Fax +44 - (0)29 - 2087 - 4056 Cardiff University Room N/3.13A Queen's Buildings 5 The Parade Cardiff CF24 3AA Wales, UK ------------------------------------------------------------------------------------------------------------ From m.b.taylor@bristol.XXX Mon Jan 20 11:55:09 2014 Date: Mon, 20 Jan 2014 11:55:08 +0000 (GMT) From: Mark Taylor To: sajjad aftabi Cc: "topcat-user@bristol.XXX" Subject: Re: dat or txt file as the input Sajjad, On Mon, 20 Jan 2014, sajjad aftabi wrote: > Hi > > I have a question. How can I read a ".dat" or ".txt" file? > Thank you so much. Probably the answer is to use the ASCII input format. You have to select "ASCII" from the "Format" selector. If you're using the Filestore Browser, use the Format selector in that window. If you're using the System Browser or typing in the location directly, use the Format selector in the Load New Table dialogue itself: http://www.star.bristol.ac.uk/~mbt/topcat/sun253/LoadWindow.html Note this doesn't guarantee it will work with your .dat or .txt files - these file endings do not entail any well-defined format. If that doesn't work, you can try to tweak the file content in an editor or with a script to match the format that topcat is expecting for ASCII files: http://www.star.bristol.ac.uk/~mbt/topcat/sun253/inAscii.html Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Jan 20 12:08:41 2014 Date: Mon, 20 Jan 2014 12:08:40 +0000 (GMT) From: Mark Taylor To: Carlos Roman Cc: topcat-user@sympa.XXX Subject: Re: STILTS 3-table match Hi Carlos, the answer (I think) is that TOPCAT matches always work like multimode=group, never like multimode=pairs. STILTS is more flexible, you can choose either. It's been on the to-do list for a long time to add both options to TOPCAT. So if you use multimode=group in your STILTS command the results should be the same as from TOPCAT. Let me know if you don't think this is accounting for the difference. Mark On Sat, 18 Jan 2014, Carlos Roman wrote: > Hi, > > I am trying to run this STILTS command > > stilts tmatchn in1=table1.cat in2=table2.cat in3=table3.cat \ > ifmt1=ascii ifmt2=ascii ifmt3=ascii omode=out ofmt=ascii out= > table123.cat \ > matcher=sky params=1.5 nin=3 values1=' RAJ2000 DECJ2000 ' \ > values2=' RAJ2000 DECJ2000 ' \ > values3=' RAJ2000 DECJ2000 ' join1=always join2=always join3=always \ > multimode=pairs > > so I can use it instead of TOPCAT->joins->triple match > with sky matching and "All Rows" in all output rows options. > > However, the run from TOPCAT appears to do what I want, > that is, a table that contains matches of tables 1+2+3, 1+2, 1+3 and 2+3. > But it is not doing it so, I think it does not include 1+3. I can tell > by looking at the output tables from both TOPCAT and STILTS > and by counting output rows: topcat yields more rows than stilts. > > I am sure I am missing something obvious but I cannot tell what... > > could you help? > Thank you (for this and in general for TOPCAT+STILTS, there > was a life before and after them) > > C. Roman > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Mon Jan 20 12:27:34 2014 Date: Mon, 20 Jan 2014 12:27:33 +0000 (GMT) From: Mark Taylor To: sajjad aftabi Cc: topcat-user@bristol.XXX Subject: Re: dat or txt file as the input Sajjad, that file is going to be fairly painful to use in topcat, since the column metadata format is not understood and the coordinates are in multiple columns, which will require some explicit fiddling around with to turn into plottable values. However, you can get the Union2 catalogue in other ways. One way is to use the menu item: VO | VizieR Catalogue Service (http://www.star.bristol.ac.uk/~mbt/topcat/sun253/VizierTableLoadDialog.html), select the "All Rows" checkbox, select the "By Keyword" tab, and enter "Union2" in the "Keywords" field. Then hitting OK will load the Union2 catalogue into TOPCAT. Mark On Mon, 20 Jan 2014, sajjad aftabi wrote: > I want to plot SNe Ia and their positions in the sky which are in "Union2wpos.dat" > > you can find the file attached. > Thank you so much for your consideration. > > > > > On Monday, January 20, 2014 3:25 PM, Mark Taylor wrote: > > Sajjad, > > > On Mon, 20 Jan 2014, sajjad aftabi wrote: > > > Hi > > > > I have a question. How can I read a ".dat" or ".txt" file? > > Thank you so much. > > Probably the answer is to use the ASCII input format. > > You have to select "ASCII" from the "Format" selector. > If you're using the Filestore Browser, use the Format selector > in that window.  If you're using the System Browser or typing > in the location directly, use the Format selector in the > Load New Table dialogue itself: > >   http://www.star.bristol.ac.uk/~mbt/topcat/sun253/LoadWindow.html > > Note this doesn't guarantee it will work with your .dat or .txt > files - these file endings do not entail any well-defined format. > If that doesn't work, you can try to tweak the file content in an > editor or with a script to match the format that topcat is > expecting for ASCII files: > >   http://www.star.bristol.ac.uk/~mbt/topcat/sun253/inAscii.html > > Mark > > -- > Mark Taylor  Astronomical Programmer  Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776  http://www.star.bris.ac.uk/~mbt/ -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +43-117-9288776 http://www.star.bris.ac.uk/~mbt/ From Hauser@ari.XXX Mon Jan 20 14:30:05 2014 Date: Mon, 20 Jan 2014 15:29:25 +0100 From: Marcus Hauser To: topcat-user@bristol.XXX Subject: Two y-axes in Topcat Hi Mark, hi list! Is there a way to use 2 y-axes in a plot, so that I can plot 2 quantities with different dynamical ranges over a shared x-axis? E.g. a time plot (same x-axis), showing quantity A from 0.2 to 5.8 in linear, and quantity B from 0.1 to 1e6 in log scale? Cheers, Marcus [ Part 2, "OpenPGP digital signature" Application/PGP-SIGNATURE ] [ (Name: "signature.asc") 501 bytes. ] [ Unable to print this part. ] From m.b.taylor@bristol.XXX Mon Jan 20 14:34:18 2014 Date: Mon, 20 Jan 2014 14:34:18 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Two y-axes in Topcat On Mon, 19 Jan 2014, Marcus Hauser wrote: > Hi Mark, > hi list! > > Is there a way to use 2 y-axes in a plot, so that I can plot 2 > quantities with different dynamical ranges over a shared x-axis? > > E.g. a time plot (same x-axis), showing quantity A from 0.2 to 5.8 in > linear, and quantity B from 0.1 to 1e6 in log scale? Does the stacked line plot do what you want? http://www.star.bristol.ac.uk/~mbt/topcat/sun253/LinesWindow.html -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From Hauser@ari.XXX Mon Jan 20 14:38:44 2014 Date: Mon, 20 Jan 2014 15:38:41 +0100 From: Marcus Hauser To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Two y-axes in Topcat On 20.01.2014 15:34, Mark Taylor wrote: > Does the stacked line plot do what you want? > > http://www.star.bristol.ac.uk/~mbt/topcat/sun253/LinesWindow.html Not really, I am looking for something like this example: http://matplotlib.org/mpl_examples/api/fahrenheit_celsius_scales.png But stacked lines will help for the moment. Thanks! Marcus [ Part 2, "OpenPGP digital signature" Application/PGP-SIGNATURE ] [ (Name: "signature.asc") 501 bytes. ] [ Unable to print this part. ] From m.b.taylor@bristol.XXX Mon Jan 20 14:43:20 2014 Date: Mon, 20 Jan 2014 14:43:19 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Two y-axes in Topcat On Mon, 20 Jan 2014, Marcus Hauser wrote: > On 20.01.2014 15:34, Mark Taylor wrote: > > Does the stacked line plot do what you want? > > > > http://www.star.bristol.ac.uk/~mbt/topcat/sun253/LinesWindow.html > > Not really, I am looking for something like this example: > http://matplotlib.org/mpl_examples/api/fahrenheit_celsius_scales.png OK, that's not currently possible. I'll stick it on the to-do list. If others are keen to see this as well, feel free to send "me too" emails. -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Mon Jan 20 14:54:42 2014 Date: Mon, 20 Jan 2014 16:53:29 +0200 From: Maciej Bilicki To: sajjad aftabi Cc: "topcat-user@bristol.XXX" Subject: Re: dat or txt file as the input Dear Sajjad, I believe such issues are well described in TOPCAT documentation, so please make sure to read through relevant sections, as well as try playing with your data a bit more. Regards Maciej Bilicki 2014/1/20 sajjad aftabi > Hi > I have a question. How can I read a ".dat" or ".txt" file? > Thank you so much. > From croman@astro.XXX Mon Jan 20 15:20:55 2014 Date: Mon, 20 Jan 2014 07:20:54 -0800 From: Carlos Roman To: Mark Taylor Cc: topcat-user@bristol.XXX Subject: Re: Two y-axes in Topcat Me too. Useful for angular vs. Parsec scales, for instance. El ene 20, 2014 6:44 AM, "Mark Taylor" escribió: > On Mon, 20 Jan 2014, Marcus Hauser wrote: > > > On 20.01.2014 15:34, Mark Taylor wrote: > > > Does the stacked line plot do what you want? > > > > > > http://www.star.bristol.ac.uk/~mbt/topcat/sun253/LinesWindow.html > > > > Not really, I am looking for something like this example: > > http://matplotlib.org/mpl_examples/api/fahrenheit_celsius_scales.png > > OK, that's not currently possible. I'll stick it on the to-do list. > If others are keen to see this as well, feel free to send "me too" emails. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > > From topcat-user-request@sympa.XXX Mon Jan 20 15:48:47 2014 Date: Mon, 20 Jan 2014 15:48:45 +0000 From: topcat-user-request@sympa.XXX Reply-To: sympa@sympa.XXX To: topcat-user-editor@sympa.XXX Subject: Article to be approved for list topcat-user from cbaorion2008@gmail.XXX You have 2 options to distribute the attached message in list topcat-user: 1) By visting: 2) Or send a message to sympa@sympa.XXX with the following subject : DISTRIBUTE topcat-user 20a725a4f09d345c879aa3d8d2285e1c You have 2 options to reject it (it will be removed): 1) By visiting: 2) Or send a message to sympa@sympa.XXX with the following subject : REJECT topcat-user 20a725a4f09d345c879aa3d8d2285e1c Number of messages awaiting moderation for this list : 1 [ Part 2: "Included Message" ] Date: Mon, 20 Jan 2014 12:48:31 -0300 From: Cesar Briceño To: topcat-user@sympa.XXX Subject: Re: Two y-axes in Topcat I also agree this feature would be very useful to have. Cesar Briceno On 01/20/2014 12:20 PM, Carlos Roman wrote: > > Me too. Useful for angular vs. Parsec scales, for instance. > > El ene 20, 2014 6:44 AM, "Mark Taylor" > escribió: > > On Mon, 20 Jan 2014, Marcus Hauser wrote: > > > On 20.01.2014 15:34, Mark Taylor wrote: > > > Does the stacked line plot do what you want? > > > > > > > http://www.star.bristol.ac.uk/~mbt/topcat/sun253/LinesWindow.html > > > > > Not really, I am looking for something like this example: > > http://matplotlib.org/mpl_examples/api/fahrenheit_celsius_scales.png > > OK, that's not currently possible. I'll stick it on the to-do list. > If others are keen to see this as well, feel free to send "me too" > emails. > > -- > Mark Taylor Astronomical Programmer Physics, Bristol > University, UK > m.b.taylor@bris.XXX > +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > > -- *********************************************** * César Briceño * * ********************************************* * AURA/CTIO * * Observatorio Interamericano de Cerro Tololo * * Casilla 603 * * La Serena, Chile * * Email: cbriceno@ctio.XXX * * cbaorion2008@gmail.XXX * *********************************************** From croman@astro.XXX Mon Jan 20 20:47:32 2014 Date: Mon, 20 Jan 2014 12:47:31 -0800 From: Carlos Roman To: Mark Taylor Cc: topcat-user@sympa.XXX Subject: Re: STILTS 3-table match Hi Mark Nope. The result from TopCat shows over 21000 rows, while the one from STILTS shows about 17000. I plotted positions for both and I only see coincidences, which makes me suspect that TOPCAT may be generating some duplicated rows.... Could that be the case? Carlos On Mon, Jan 20, 2014 at 4:08 AM, Mark Taylor wrote: > Hi Carlos, > > the answer (I think) is that TOPCAT matches always work like > multimode=group, never like multimode=pairs. STILTS is more flexible, > you can choose either. It's been on the to-do list for a long time > to add both options to TOPCAT. So if you use multimode=group in > your STILTS command the results should be the same as from TOPCAT. > > Let me know if you don't think this is accounting for the difference. > > Mark > > On Sat, 18 Jan 2014, Carlos Roman wrote: > > > Hi, > > > > I am trying to run this STILTS command > > > > stilts tmatchn in1=table1.cat in2=table2.cat in3=table3.cat \ > > ifmt1=ascii ifmt2=ascii ifmt3=ascii omode=out ofmt=ascii out= > > table123.cat \ > > matcher=sky params=1.5 nin=3 values1=' RAJ2000 DECJ2000 ' \ > > values2=' RAJ2000 DECJ2000 ' \ > > values3=' RAJ2000 DECJ2000 ' join1=always join2=always > join3=always \ > > multimode=pairs > > > > so I can use it instead of TOPCAT->joins->triple match > > with sky matching and "All Rows" in all output rows options. > > > > However, the run from TOPCAT appears to do what I want, > > that is, a table that contains matches of tables 1+2+3, 1+2, 1+3 and 2+3. > > But it is not doing it so, I think it does not include 1+3. I can tell > > by looking at the output tables from both TOPCAT and STILTS > > and by counting output rows: topcat yields more rows than stilts. > > > > I am sure I am missing something obvious but I cannot tell what... > > > > could you help? > > Thank you (for this and in general for TOPCAT+STILTS, there > > was a life before and after them) > > > > C. Roman > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > From croman@astro.XXX Mon Jan 20 20:48:28 2014 Date: Mon, 20 Jan 2014 12:48:27 -0800 From: Carlos Roman To: Mark Taylor Cc: topcat-user@sympa.XXX Subject: Re: STILTS 3-table match forgot to clarify, this is after using multimode=group in STILTS On Mon, Jan 20, 2014 at 12:47 PM, Carlos Roman wrote: > Hi Mark > > Nope. The result from TopCat shows over 21000 rows, while the one > from STILTS shows about 17000. I plotted positions for both and > I only see coincidences, which makes me suspect that TOPCAT may be > generating > some duplicated rows.... Could that be the case? > > Carlos > > > On Mon, Jan 20, 2014 at 4:08 AM, Mark Taylor wrote: > >> Hi Carlos, >> >> the answer (I think) is that TOPCAT matches always work like >> multimode=group, never like multimode=pairs. STILTS is more flexible, >> you can choose either. It's been on the to-do list for a long time >> to add both options to TOPCAT. So if you use multimode=group in >> your STILTS command the results should be the same as from TOPCAT. >> >> Let me know if you don't think this is accounting for the difference. >> >> Mark >> >> On Sat, 18 Jan 2014, Carlos Roman wrote: >> >> > Hi, >> > >> > I am trying to run this STILTS command >> > >> > stilts tmatchn in1=table1.cat in2=table2.cat in3=table3.cat \ >> > ifmt1=ascii ifmt2=ascii ifmt3=ascii omode=out ofmt=ascii out= >> > table123.cat \ >> > matcher=sky params=1.5 nin=3 values1=' RAJ2000 DECJ2000 ' \ >> > values2=' RAJ2000 DECJ2000 ' \ >> > values3=' RAJ2000 DECJ2000 ' join1=always join2=always >> join3=always \ >> > multimode=pairs >> > >> > so I can use it instead of TOPCAT->joins->triple match >> > with sky matching and "All Rows" in all output rows options. >> > >> > However, the run from TOPCAT appears to do what I want, >> > that is, a table that contains matches of tables 1+2+3, 1+2, 1+3 and >> 2+3. >> > But it is not doing it so, I think it does not include 1+3. I can tell >> > by looking at the output tables from both TOPCAT and STILTS >> > and by counting output rows: topcat yields more rows than stilts. >> > >> > I am sure I am missing something obvious but I cannot tell what... >> > >> > could you help? >> > Thank you (for this and in general for TOPCAT+STILTS, there >> > was a life before and after them) >> > >> > C. Roman >> > >> >> -- >> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >> m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ >> > > From croman@astro.XXX Mon Jan 20 21:20:30 2014 Date: Mon, 20 Jan 2014 13:20:29 -0800 From: Carlos Roman To: Mark Taylor Cc: topcat-user@sympa.XXX Subject: Re: STILTS 3-table match Wait. I take it back! sorry! had a typo in the script. Now it is 21761 vs 21863 rows. On Mon, Jan 20, 2014 at 12:48 PM, Carlos Roman wrote: > forgot to clarify, this is after using multimode=group in STILTS > > > On Mon, Jan 20, 2014 at 12:47 PM, Carlos Roman wrote: > >> Hi Mark >> >> Nope. The result from TopCat shows over 21000 rows, while the one >> from STILTS shows about 17000. I plotted positions for both and >> I only see coincidences, which makes me suspect that TOPCAT may be >> generating >> some duplicated rows.... Could that be the case? >> >> Carlos >> >> >> On Mon, Jan 20, 2014 at 4:08 AM, Mark Taylor wrote: >> >>> Hi Carlos, >>> >>> the answer (I think) is that TOPCAT matches always work like >>> multimode=group, never like multimode=pairs. STILTS is more flexible, >>> you can choose either. It's been on the to-do list for a long time >>> to add both options to TOPCAT. So if you use multimode=group in >>> your STILTS command the results should be the same as from TOPCAT. >>> >>> Let me know if you don't think this is accounting for the difference. >>> >>> Mark >>> >>> On Sat, 18 Jan 2014, Carlos Roman wrote: >>> >>> > Hi, >>> > >>> > I am trying to run this STILTS command >>> > >>> > stilts tmatchn in1=table1.cat in2=table2.cat in3=table3.cat \ >>> > ifmt1=ascii ifmt2=ascii ifmt3=ascii omode=out ofmt=ascii out= >>> > table123.cat \ >>> > matcher=sky params=1.5 nin=3 values1=' RAJ2000 DECJ2000 ' \ >>> > values2=' RAJ2000 DECJ2000 ' \ >>> > values3=' RAJ2000 DECJ2000 ' join1=always join2=always >>> join3=always \ >>> > multimode=pairs >>> > >>> > so I can use it instead of TOPCAT->joins->triple match >>> > with sky matching and "All Rows" in all output rows options. >>> > >>> > However, the run from TOPCAT appears to do what I want, >>> > that is, a table that contains matches of tables 1+2+3, 1+2, 1+3 and >>> 2+3. >>> > But it is not doing it so, I think it does not include 1+3. I can tell >>> > by looking at the output tables from both TOPCAT and STILTS >>> > and by counting output rows: topcat yields more rows than stilts. >>> > >>> > I am sure I am missing something obvious but I cannot tell what... >>> > >>> > could you help? >>> > Thank you (for this and in general for TOPCAT+STILTS, there >>> > was a life before and after them) >>> > >>> > C. Roman >>> > >>> >>> -- >>> Mark Taylor Astronomical Programmer Physics, Bristol University, UK >>> m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ >>> >> >> > From jbrinchmann@gmail.XXX Tue Jan 21 08:38:30 2014 Date: Tue, 21 Jan 2014 09:37:53 +0100 From: Jarle Brinchmann To: Gordon Richards Cc: topcat-user@sympa.XXX Subject: Re: Reading WISE files with TOPCAT When I converted the WISE tables to match to SDSS DR7 data I first converted them to CSV using a trivial shell script: for file in *part??; do newname="tmp.file" cat $file | tr "|" "," > $newname mv $newname $file done I then used STILTS to do a prefiltering on the S/N in W1 (using tpipe) to get rid of chaff - that was appropriate for my needs but might not be suitable for random use. Then using a match along the lines of for file in wise-cleaned*part??.fits; do newname=${file/wise-cleaned-subset-cat/wise-sdss} echo "Matching $file to SDSS, outputting to $newname" ~/Source/Stilts/stilts tskymatch2 in1=$file in2="sdss-mpajhu-pos.fits" ra1=ra dec1=dec ra2=RA dec2=DEC error=3 out=$newname done worked fine. I saw precious little reason to convert it all to one big file so didn't bother with that - I found it easier in general to work with the individual sub-files. Of course if you need to do many matches multiple times you probably want to do something a bit more efficient outside of TOPCAT, like creating optimised HEALPIX tables or somesuch - or indeed you might be better off using an existing interface as Ivan suggested. Cheers, Jarle. On Fri, Jan 17, 2014 at 8:17 PM, Gordon Richards wrote: > Is there a way to get TOPCAT (or STILTS) to > read the bar-delimited bulk catalogs from WISE? > > They look like this: > > head wise-allwise-cat-part18 > J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014|| > ||||8.2317599| > > I can convert them to CSV first, but that is cumbersome. > > And since the files don't have any header information, is there a way to > specify that > at the time of reading or after the fact so that the columns are labeled? > > The problem is with the WISE data file format and not with TOPCAT per se, > but if I > could solve it with TOPCAT alone, that would be great. > > Thanks, > Gordon Richards > > > > > From vanhouttewouter@gmail.XXX Tue Jan 21 08:55:30 2014 Date: Tue, 21 Jan 2014 09:54:57 +0100 From: Wouter Vanhoutte To: Jarle Brinchmann Cc: Gordon Richards , topcat-user@sympa.XXX Subject: Re: Reading WISE files with TOPCAT Dear TOPCAT list, The best way for matching sdss and wise. Is using a .fits file from the sdssserver and select in TOPCAT "Joins", then "Multicone" then using the link "registry.astrogrid.org/ ...." Then select the catalog II/311 WISE All-Sky Data Release (Cutri+ 2012). Then choose a small search radius (<5"), because of the performance. This is the most easy way for sdss wise matching, you haven't to download the gigantic WISE catalog, and there will not be compatibility problems, first try it with small sdss selection. Best regards, Wouter Vanhoutte 2014/1/21 Jarle Brinchmann > When I converted the WISE tables to match to SDSS DR7 data I first > converted them to CSV using a trivial shell script: > > for file in *part??; do > newname="tmp.file" > cat $file | tr "|" "," > $newname > mv $newname $file > done > > I then used STILTS to do a prefiltering on the S/N in W1 (using tpipe) to > get rid of chaff - that was appropriate for my needs but might not be > suitable for random use. Then using a match along the lines of > > for file in wise-cleaned*part??.fits; do > newname=${file/wise-cleaned-subset-cat/wise-sdss} > echo "Matching $file to SDSS, outputting to $newname" > ~/Source/Stilts/stilts tskymatch2 in1=$file in2="sdss-mpajhu-pos.fits" > ra1=ra dec1=dec ra2=RA dec2=DEC error=3 out=$newname > done > > worked fine. I saw precious little reason to convert it all to one big > file so didn't bother with that - I found it easier in general to work with > the individual sub-files. Of course if you need to do many matches multiple > times you probably want to do something a bit more efficient outside of > TOPCAT, like creating optimised HEALPIX tables or somesuch - or indeed you > might be better off using an existing interface as Ivan suggested. > > Cheers, > Jarle. > > > > On Fri, Jan 17, 2014 at 8:17 PM, Gordon Richards wrote: > >> Is there a way to get TOPCAT (or STILTS) to >> read the bar-delimited bulk catalogs from WISE? >> >> They look like this: >> > head wise-allwise-cat-part18 >> J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014|| >> ||||8.2317599| >> >> I can convert them to CSV first, but that is cumbersome. >> >> And since the files don't have any header information, is there a way to >> specify that >> at the time of reading or after the fact so that the columns are labeled? >> >> The problem is with the WISE data file format and not with TOPCAT per se, >> but if I >> could solve it with TOPCAT alone, that would be great. >> >> Thanks, >> Gordon Richards >> >> >> >> >> > From m.b.taylor@bristol.XXX Tue Jan 21 10:01:36 2014 Date: Tue, 21 Jan 2014 10:01:35 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Reading WISE files with TOPCAT Hi all, I don't know what Gordon was trying to do with the WISE data, but I'll make a few comments on general question of the "best" way to match two tables: Multicone works well if you have a small input catalogue to match against a large external catalogue. But it makes one query for each input catalogue so if it's more than a few hundred or thousand rows it does not scale well. A match can be done much more efficiently if the two catalogues are in the same place. If they are both available and small enough (few million), then doing the match in topcat is sensible. If not, the best thing is to do it on a remote server which has at least one of the catalogues. One way to do this is TAP with uploads. The GAVO TAP server has the wise.main catalogue, so if you have a medium-sized local catalogue to match against WISE: - open the VO|TAP window - search for the GAVO TAP service - hit "Enter Query" - select "wise.main" from the Table selector - select "Upload Join" from the Examples button - adapt the ADQL to match your table against WISE (your local table is TAP_UPLOAD.t, where is the number in the topcat table list) - if it's a moderately large query, uncheck the "Synchronous" box it should be fairly fast. Even faster is the CDS Xmatch service (http://cdsxmatch.u-strasbg.fr/xmatch). This is currently not available from within topcat, but it will be at some point in the future. Mark On Tue, 21 Jan 2014, Wouter Vanhoutte wrote: > Dear TOPCAT list, > > The best way for matching sdss and wise. Is using a .fits file from > the sdssserver and select in > TOPCAT "Joins", then "Multicone" then using the link "registry.astrogrid.org/ > ...." > Then select the catalog II/311 WISE All-Sky Data Release (Cutri+ 2012). > Then choose a small search radius (<5"), because of the performance. > > This is the most easy way for sdss wise matching, you haven't to download > the gigantic WISE catalog, and there will not be compatibility problems, > first try it with small sdss selection. > > Best regards, > > Wouter Vanhoutte > > > 2014/1/21 Jarle Brinchmann > > > When I converted the WISE tables to match to SDSS DR7 data I first > > converted them to CSV using a trivial shell script: > > > > for file in *part??; do > > newname="tmp.file" > > cat $file | tr "|" "," > $newname > > mv $newname $file > > done > > > > I then used STILTS to do a prefiltering on the S/N in W1 (using tpipe) to > > get rid of chaff - that was appropriate for my needs but might not be > > suitable for random use. Then using a match along the lines of > > > > for file in wise-cleaned*part??.fits; do > > newname=${file/wise-cleaned-subset-cat/wise-sdss} > > echo "Matching $file to SDSS, outputting to $newname" > > ~/Source/Stilts/stilts tskymatch2 in1=$file in2="sdss-mpajhu-pos.fits" > > ra1=ra dec1=dec ra2=RA dec2=DEC error=3 out=$newname > > done > > > > worked fine. I saw precious little reason to convert it all to one big > > file so didn't bother with that - I found it easier in general to work with > > the individual sub-files. Of course if you need to do many matches multiple > > times you probably want to do something a bit more efficient outside of > > TOPCAT, like creating optimised HEALPIX tables or somesuch - or indeed you > > might be better off using an existing interface as Ivan suggested. > > > > Cheers, > > Jarle. > > > > > > > > On Fri, Jan 17, 2014 at 8:17 PM, Gordon Richards wrote: > > > >> Is there a way to get TOPCAT (or STILTS) to > >> read the bar-delimited bulk catalogs from WISE? > >> > >> They look like this: > >> > head wise-allwise-cat-part18 > >> J175047.62-200235.1|267.6984422|-20.0430917|0.0663|0.0689|-0.014|| > >> ||||8.2317599| > >> > >> I can convert them to CSV first, but that is cumbersome. > >> > >> And since the files don't have any header information, is there a way to > >> specify that > >> at the time of reading or after the fact so that the columns are labeled? > >> > >> The problem is with the WISE data file format and not with TOPCAT per se, > >> but if I > >> could solve it with TOPCAT alone, that would be great. > >> > >> Thanks, > >> Gordon Richards > >> > >> > >> > >> > >> > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From P.W.Lucas@herts.XXX Tue Jan 21 13:56:27 2014 Date: Tue, 21 Jan 2014 13:54:59 +0000 From: Philip Lucas To: Gordon Richards Cc: "topcat-user@sympa.XXX" Subject: Re: Reading WISE files with TOPCAT Sadly this doesn't suprise me, since funding for the WISE team was cut in spring 2012, just as soon as they had made the all sky data public. A poor reward for all their hard work I feel. Phil Lucas U Hertfordshire On Tue, 21 Jan 2014, Gordon Richards wrote: > > On 1/21/14 5:01 AM, Mark Taylor wrote: >> Hi all, >> >> I don't know what Gordon was trying to do with the WISE data, but >> I'll make a few comments on general question of the "best" way to >> match two tables: > Without boring everyone with details, I know that I can do what > I want to do b/c I've already done it with the WISE allsky data. > I just want to do the same with ALLWISE. Basically, as Jarle says, > I converted to CSV first and then used subsamples of "good" data > from the individual bulk files. > > It is just that that would be much easier with FITS. However, the > WISE folks claim not to have the resources to make FITS files. > > Gordon > > > > From bilicki@camk.XXX Tue Jan 21 14:15:13 2014 Date: Tue, 21 Jan 2014 16:13:58 +0200 From: Maciej Bilicki To: P.W.Lucas@herts.XXX Cc: Gordon Richards , "topcat-user@sympa.XXX" Subject: Re: Reading WISE files with TOPCAT Anyhow, CDS is currently not a solution for Gordon, as it doesn't offer AllWISE yet, but only the older version, "WISE All-Sky". And it doesn't give as much flexibility in x-matching as STILTS/TOPCAT do. Maciek 2014/1/21 Philip Lucas > Sadly this doesn't suprise me, since funding for the WISE > team was cut in spring 2012, just as soon as they had made the > all sky data public. A poor reward for all their hard work I feel. > > Phil Lucas > U Hertfordshire > > > > On Tue, 21 Jan 2014, Gordon Richards wrote: > > >> On 1/21/14 5:01 AM, Mark Taylor wrote: >> >>> Hi all, >>> >>> I don't know what Gordon was trying to do with the WISE data, but >>> I'll make a few comments on general question of the "best" way to >>> match two tables: >>> >> Without boring everyone with details, I know that I can do what >> I want to do b/c I've already done it with the WISE allsky data. >> I just want to do the same with ALLWISE. Basically, as Jarle says, >> I converted to CSV first and then used subsamples of "good" data >> from the individual bulk files. >> >> It is just that that would be much easier with FITS. However, the >> WISE folks claim not to have the resources to make FITS files. >> >> Gordon >> >> >> >> >> > From francois-xavier.pineau@astro.XXX Tue Jan 21 16:07:15 2014 Date: Tue, 21 Jan 2014 17:06:24 +0100 From: François-Xavier Pineau To: bilicki@camk.XXX Gordon Richards Cc: "topcat-user@sympa.XXX" Subject: Re: Reading WISE files with TOPCAT Dear all, The AllWISE catalogue is being ingested at CDS and will be available in VizieR and the XMatch service in a few weeks (at most). fx Le 21/01/2014 15:13, Maciej Bilicki a écrit : > Anyhow, CDS is currently not a solution for Gordon, as it doesn't > offer AllWISE yet, but only the older version, "WISE All-Sky". And it > doesn't give as much flexibility in x-matching as STILTS/TOPCAT do. > > Maciek > > > 2014/1/21 Philip Lucas > > > Sadly this doesn't suprise me, since funding for the WISE > team was cut in spring 2012, just as soon as they had made the > all sky data public. A poor reward for all their hard work I feel. > > Phil Lucas > U Hertfordshire > > > > On Tue, 21 Jan 2014, Gordon Richards wrote: > > > On 1/21/14 5:01 AM, Mark Taylor wrote: > > Hi all, > > I don't know what Gordon was trying to do with the WISE > data, but > I'll make a few comments on general question of the "best" > way to > match two tables: > > Without boring everyone with details, I know that I can do what > I want to do b/c I've already done it with the WISE allsky data. > I just want to do the same with ALLWISE. Basically, as Jarle > says, > I converted to CSV first and then used subsamples of "good" data > from the individual bulk files. > > It is just that that would be much easier with FITS. However, the > WISE folks claim not to have the resources to make FITS files. > > Gordon > > > > > > -- François-Xavier Pineau Université de Strasbourg CDS, Observatoire de Strasbourg francois-xavier.pineau@astro.XXX 11, rue de l'Université Phone +33 (0)3 68 85 24 14 F - 67000 Strasbourg http://snob.u-strasbg.fr/~pineau http://astro.u-strasbg.fr [ Part 2.1, Text/HTML (charset: ISO-8859-1 "Latin 1 (Western Europe)") ] [ 130 lines. ] [ Unable to print this part. ] [ Part 2.2, Image/PNG (Name: "logoObs_h60l63_Grey.png") 4.3 KB. ] [ Unable to print this part. ] From m.b.taylor@bristol.XXX Tue Jan 21 17:43:22 2014 Date: Tue, 21 Jan 2014 17:43:21 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: STILTS 3-table match for interested readers ... we sorted this out off-list, it was not a problem with the topcat/stilts matching functions. On Sat, 18 Jan 2014, Carlos Roman wrote: > Hi, > > I am trying to run this STILTS command > > stilts tmatchn in1=table1.cat in2=table2.cat in3=table3.cat \ > ifmt1=ascii ifmt2=ascii ifmt3=ascii omode=out ofmt=ascii out= > table123.cat \ > matcher=sky params=1.5 nin=3 values1=' RAJ2000 DECJ2000 ' \ > values2=' RAJ2000 DECJ2000 ' \ > values3=' RAJ2000 DECJ2000 ' join1=always join2=always join3=always \ > multimode=pairs > > so I can use it instead of TOPCAT->joins->triple match > with sky matching and "All Rows" in all output rows options. > > However, the run from TOPCAT appears to do what I want, > that is, a table that contains matches of tables 1+2+3, 1+2, 1+3 and 2+3. > But it is not doing it so, I think it does not include 1+3. I can tell > by looking at the output tables from both TOPCAT and STILTS > and by counting output rows: topcat yields more rows than stilts. > > I am sure I am missing something obvious but I cannot tell what... > > could you help? > Thank you (for this and in general for TOPCAT+STILTS, there > was a life before and after them) > > C. Roman > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From gilles.duvert@obs.XXX Thu Jan 23 21:35:16 2014 Date: Thu, 23 Jan 2014 21:30:52 +0100 From: Gilles DUVERT To: topcat-user@bristol.XXX Subject: Joining tables: automatic data conversion? Hi, Thanks for TOPCAT, it's a wonderful tool. I do not know if this topic has been adressed before, since the archive are out of reach now: TOPCAT would be handier if, when concatenating two tables, it would silently perform a minimum set of automatic conversions between columns that match but have different types. Namely, promote real to double integer to real and (possibly) double to text (with a warning in this case). Best regards, Gilles [ Part 2, Text/X-VCARD (Name: "gilles_duvert.vcf") 18 lines. ] [ Unable to print this part. ] From gilles.duvert@obs.XXX Thu Jan 23 21:35:15 2014 Date: Thu, 23 Jan 2014 21:37:00 +0100 From: Gilles DUVERT To: topcat-user@bristol.XXX Subject: Reading TST files: One Column is Not Enough Hi, Thanks for making TOPCAT available! I would like to mention that TOPCAT fails to read a TST format file which has less than 3 columns, such as the attached file. (???) Best Regards Gilles [ Part 2, Text/PLAIN (Name: "solar.tst") 5 lines. ] [ Unable to print this part. ] [ Part 3, Text/X-VCARD (Name: "gilles_duvert.vcf") 18 lines. ] [ Unable to print this part. ] From M.B.Taylor@bristol.XXX Fri Jan 24 23:00:35 2014 Date: Fri, 24 Jan 2014 23:00:33 +0000 (GMT) From: Mark Taylor To: topcat-user@bristol.XXX Subject: Re: Reading WISE files with TOPCAT On Tue, 21 Jan 2014, Gordon Richards wrote: > > On 1/21/14 5:01 AM, Mark Taylor wrote: > > Hi all, > > > > I don't know what Gordon was trying to do with the WISE data, but > > I'll make a few comments on general question of the "best" way to > > match two tables: > Without boring everyone with details, I know that I can do what > I want to do b/c I've already done it with the WISE allsky data. > I just want to do the same with ALLWISE. Basically, as Jarle says, > I converted to CSV first and then used subsamples of "good" data > from the individual bulk files. > > It is just that that would be much easier with FITS. However, the > WISE folks claim not to have the resources to make FITS files. To tie this one up - with some assistance from Gordon I've written an ALLWISE build ASCII file input handler for STIL, so it's now straightforward to convert those 10Gb zipped text files to 30Gb FITS or colfits files with a reasonable amount of column metadata. I believe that IPAC will do those conversions and make the FITS files available from the archive at some point, but if you want to have a go yourself, see the README and other files at ftp://andromeda.star.bris.ac.uk/pub/star/allwise/ Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From cgp@leicester.XXX Tue Jan 28 11:06:37 2014 Date: Tue, 28 Jan 2014 11:03:20 +0000 From: Clive Page To: topcat-user@bristol.XXX Subject: More statistical info on boolean columns Mark, Here's a very minor enhancement suggestion: many of my tables have a few columns of flags which are boolean/logical in type i.e. true/false. It would be useful when using the statistics function in Topcat to be able to see the proportion of flags in the column that are set true. Would it be possible for the mean value, say, to display a mean computed using the C convention that false=0 and true=1? Regards -- Clive Page From m.b.taylor@bristol.XXX Tue Jan 28 12:27:06 2014 Date: Tue, 28 Jan 2014 12:27:05 +0000 (GMT) From: Mark Taylor To: Gilles DUVERT Cc: topcat-user@bristol.XXX Subject: Re: Reading TST files: One Column is Not Enough Gilles, sorry for the delay answering this one (and your other message). On Thu, 23 Jan 2014, Gilles DUVERT wrote: > I would like to mention that TOPCAT fails to read a TST format file which has > less than 3 columns, such as the attached file. (???) You're quite right, thanks for spotting it. This bug will be fixed in the next version. You can find a pre-release version with the fix here if you like: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Tue Jan 28 12:31:25 2014 Date: Tue, 28 Jan 2014 12:31:24 +0000 (GMT) From: Mark Taylor To: Clive Page Cc: topcat-user@bristol.XXX Subject: Re: More statistical info on boolean columns Clive, On Tue, 28 Jan 2014, Clive Page wrote: > Here's a very minor enhancement suggestion: many of my tables have a few > columns of flags which are boolean/logical in type i.e. true/false. It would > be useful when using the statistics function in Topcat to be able to see the > proportion of flags in the column that are set true. Would it be possible for > the mean value, say, to display a mean computed using the C convention that > false=0 and true=1? In fact that's not an enhancement suggestion, it's a bug report - I thought it did that already but a lurking bug means it always reports zero for the mean values of boolean columns. It will do what you suggest in the next release. A pre-release version with this fix is at: ftp://andromeda.star.bris.ac.uk/pub/star/topcat/pre/ Note it just stops at the mean, it doesn't e.g. report standard deviation/skew/kurtosis for boolean values. I think that would be a bit over the top, but if people feel it's useful functionality I could add it fairly easily. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From m.b.taylor@bristol.XXX Tue Jan 28 13:09:16 2014 Date: Tue, 28 Jan 2014 13:09:16 +0000 (GMT) From: Mark Taylor To: Gilles DUVERT Cc: topcat-user@bristol.XXX Subject: Re: Joining tables: automatic data conversion? Gilles, the suggestion to auto-widen columns as required for table concatenation has come up before, but so far I haven't fixed it. It's one of those things that turns out to be harder than you'd think to do. For now it's on the to-do list. Thanks for the suggestion in any case. Mark On Thu, 23 Jan 2014, Gilles DUVERT wrote: > Hi, > > Thanks for TOPCAT, it's a wonderful tool. > I do not know if this topic has been adressed before, since the archive are > out of reach now: > TOPCAT would be handier if, when concatenating two tables, it would silently > perform a minimum set of automatic conversions between columns that match but > have different types. > Namely, > promote real to double > integer to real > and (possibly) double to text (with a warning in this case). > > Best regards, > Gilles > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From bsversut@gmail.XXX Thu Jan 30 10:09:35 2014 Date: Thu, 30 Jan 2014 11:06:52 +0100 From: Bruno Sversut Arsioli To: topcat-user@bristol.XXX Subject: doubt (File with more than 2 million rows) Dears Learning how to use TopCat, I start facing some issues that I couldn't solve in the last few days. Came out to work with the WISE catalog, which is a source list based on Infra Red measurements, with about 500 Million entries. I have found the database using the tool on the TopCat tab: VO / Table Acess Protocol Query there I search for "Wise" Build my query using the protocol: SELECT TOP 500000000 designation, ra, dec, w1mpro, w2mpro, w3mpro, w4mpro FROM wise_allskysc Also, I have selected Max number of Rows: unlimited In the end, no mater what I change, I always get a sample with 2 Million objects. Is there any idea what I should do to get more than that? Thanks for your attention, Bruno Sversut Arsioli From m.b.taylor@bristol.XXX Thu Jan 30 10:26:10 2014 Date: Thu, 30 Jan 2014 10:26:09 +0000 (GMT) From: Mark Taylor To: Bruno Sversut Arsioli Cc: topcat-user@bristol.XXX Subject: Re: doubt (File with more than 2 million rows) Bruno, any given TAP service may impose a hard limit on the number of rows it is prepared to give you. I think you're using the WFAU service, and it does seem to impose a limit of 2,000,000 rows, so no matter how much you ask for that's the most you'll get. WISE is available from other places. The GAVO TAP service (search for "gavo" rather than "wise" in the TAP Select Service tab) has the catalog wise.main, though I'm not sure if that's the same data set you're requesting from WFAU. This service reports a hard limit of 20,000,000 rows, so you can get a larger result from there. There are other ways of getting larger amounts of WISE data, for instance VizieR or possibly bulk download of the catalogues. *However*: 20 Mrow is already quite a lot. 500 Mrow is really a lot. If you have a few Gbyte on your machine topcat may cope with 20Mrow, but not half a billion. If you're a data center or have serious compute resources available to you it's possible that you might have some use for the entire WISE catalogue, but if you're just a scientist trying to explore the data, grabbing the whole lot to look at is not likely to be a sensible way to go about it. Instead you should think about some subset that you're actually interested in, and refine your query to request a more manageable number of rows (or wait a few years for Moore's law to catch up). Mark On Thu, 30 Jan 2014, Bruno Sversut Arsioli wrote: > Dears > > Learning how to use TopCat, I start facing some issues that I couldn't > solve in the last few days. Came out to work with the WISE catalog, which > is a source list based on Infra Red measurements, with about 500 Million > entries. > > I have found the database using the tool on the TopCat tab: > > VO / Table Acess Protocol Query > > there I search for "Wise" > > Build my query using the protocol: > > SELECT > TOP 500000000 > designation, ra, dec, w1mpro, w2mpro, w3mpro, w4mpro > FROM wise_allskysc > > > Also, I have selected Max number of Rows: unlimited > > > In the end, no mater what I change, I always get a sample with 2 Million > objects. > > > Is there any idea what I should do to get more than that? > > Thanks for your attention, > Bruno Sversut Arsioli > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Thu Jan 30 13:37:52 2014 Date: Thu, 30 Jan 2014 15:36:34 +0200 From: Maciej Bilicki To: Bruno Sversut Arsioli Cc: "topcat-user@bristol.XXX" Subject: Re: doubt (File with more than 2 million rows) Dear Bruno, The most recent WISE release, the AllWISE catalogue, has 750 million rows and 334 columns. http://wise2.ipac.caltech.edu/docs/release/allwise/ This would be easily a terabyte if not more (probably much more). I don't believe any service would allow you to download it in bulk. However, you can either download selected columns and objects within some criteria directly from the WISE database ( http://irsa.ipac.caltech.edu/cgi-bin/Gator/nph-scan?mission=irsa&submit=Select&projshort=WISE) or gzipped chunks of the full catalogue. I'll forward to you a related discussion we had here. Still,I can't imagine handling this in TOPCAT. Perhaps STILTS would allow you to do some selected operations on such a huge catalogue, but even that to a very limited extent and with loads of patience. But normally you won't need all WISE columns and sources at once, unless you just want to archive it. Best wishes Maciek 2014-01-30 Bruno Sversut Arsioli : > Dears > > Learning how to use TopCat, I start facing some issues that I couldn't > solve in the last few days. Came out to work with the WISE catalog, which > is a source list based on Infra Red measurements, with about 500 Million > entries. > > I have found the database using the tool on the TopCat tab: > > VO / Table Acess Protocol Query > > there I search for "Wise" > > Build my query using the protocol: > > SELECT > TOP 500000000 > designation, ra, dec, w1mpro, w2mpro, w3mpro, w4mpro > FROM wise_allskysc > > > Also, I have selected Max number of Rows: unlimited > > > In the end, no mater what I change, I always get a sample with 2 Million > objects. > > > Is there any idea what I should do to get more than that? > > Thanks for your attention, > Bruno Sversut Arsioli > From bsversut@gmail.XXX Fri Jan 31 13:08:23 2014 Date: Fri, 31 Jan 2014 11:08:20 -0200 From: Bruno Sversut Arsioli To: Mark Taylor , Maciejbilicki@camk.XXX topcat-user@bristol.XXX Subject: Re: doubt (File with more than 2 million rows) Dear Mark and Maciej, Thanks for your quick reply! I am now aware of the ADTP and TAP protocols , and I manage to do impose many selection rules to the ALLWISE catalog. I got some output tables with 5 to 8 Million objects, which I guess are still possible to handle with TOPCAT. I am now using Vi, trying to optimize even more the tables, taking out some columns . Thanks for your attention, Bruno S. Arsioli 2014-01-30 Mark Taylor : > Bruno, > > any given TAP service may impose a hard limit on the number of > rows it is prepared to give you. I think you're using the WFAU > service, and it does seem to impose a limit of 2,000,000 rows, > so no matter how much you ask for that's the most you'll get. > > WISE is available from other places. The GAVO TAP service > (search for "gavo" rather than "wise" in the TAP Select Service tab) > has the catalog wise.main, though I'm not sure if that's the same > data set you're requesting from WFAU. This service reports a hard > limit of 20,000,000 rows, so you can get a larger result from there. > > There are other ways of getting larger amounts of WISE data, > for instance VizieR or possibly bulk download of the catalogues. > > *However*: 20 Mrow is already quite a lot. 500 Mrow is really a lot. > If you have a few Gbyte on your machine topcat may cope with 20Mrow, > but not half a billion. If you're a data center or have serious > compute resources available to you it's possible that you might > have some use for the entire WISE catalogue, but if you're just > a scientist trying to explore the data, grabbing the whole lot to > look at is not likely to be a sensible way to go about it. > Instead you should think about some subset that you're actually > interested in, and refine your query to request a more manageable > number of rows (or wait a few years for Moore's law to catch up). > > Mark > > On Thu, 30 Jan 2014, Bruno Sversut Arsioli wrote: > > > Dears > > > > Learning how to use TopCat, I start facing some issues that I couldn't > > solve in the last few days. Came out to work with the WISE catalog, which > > is a source list based on Infra Red measurements, with about 500 Million > > entries. > > > > I have found the database using the tool on the TopCat tab: > > > > VO / Table Acess Protocol Query > > > > there I search for "Wise" > > > > Build my query using the protocol: > > > > SELECT > > TOP 500000000 > > designation, ra, dec, w1mpro, w2mpro, w3mpro, w4mpro > > FROM wise_allskysc > > > > > > Also, I have selected Max number of Rows: unlimited > > > > > > In the end, no mater what I change, I always get a sample with 2 Million > > objects. > > > > > > Is there any idea what I should do to get more than that? > > > > Thanks for your attention, > > Bruno Sversut Arsioli > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > From m.b.taylor@bristol.XXX Fri Jan 31 13:17:26 2014 Date: Fri, 31 Jan 2014 13:17:25 +0000 (GMT) From: Mark Taylor To: Bruno Sversut Arsioli Cc: topcat-user@bristol.XXX Subject: Re: doubt (File with more than 2 million rows) Good! 5-8 Mrow should be OK, but you may need to boost java's heap memory a bit - see here for some tips on using large tables: http://www.star.bristol.ac.uk/~mbt/topcat/sun253/largeTables.html Using vi to remove columns sounds unnecessarily painful. You can remove columns in topcat by hiding them in the columns window: http://www.star.bristol.ac.uk/~mbt/topcat/sun253/ColumnInfoWindow.html if you then save the table the hidden columns will not be included. However, having lots of columns doesn't usually bother topcat much, especially if you use colfits format (again, see the tips above). Mark On Fri, 31 Jan 2014, Bruno Sversut Arsioli wrote: > Dear Mark and Maciej, > > Thanks for your quick reply! I am now aware of the ADTP and TAP protocols , > and I manage to do impose many selection rules to the ALLWISE catalog. I > got some output tables with 5 to 8 Million objects, which I guess are still > possible to handle with TOPCAT. > > I am now using Vi, trying to optimize even more the tables, taking out some > columns . > > Thanks for your attention, > > Bruno S. Arsioli > > > 2014-01-30 Mark Taylor : > > > Bruno, > > > > any given TAP service may impose a hard limit on the number of > > rows it is prepared to give you. I think you're using the WFAU > > service, and it does seem to impose a limit of 2,000,000 rows, > > so no matter how much you ask for that's the most you'll get. > > > > WISE is available from other places. The GAVO TAP service > > (search for "gavo" rather than "wise" in the TAP Select Service tab) > > has the catalog wise.main, though I'm not sure if that's the same > > data set you're requesting from WFAU. This service reports a hard > > limit of 20,000,000 rows, so you can get a larger result from there. > > > > There are other ways of getting larger amounts of WISE data, > > for instance VizieR or possibly bulk download of the catalogues. > > > > *However*: 20 Mrow is already quite a lot. 500 Mrow is really a lot. > > If you have a few Gbyte on your machine topcat may cope with 20Mrow, > > but not half a billion. If you're a data center or have serious > > compute resources available to you it's possible that you might > > have some use for the entire WISE catalogue, but if you're just > > a scientist trying to explore the data, grabbing the whole lot to > > look at is not likely to be a sensible way to go about it. > > Instead you should think about some subset that you're actually > > interested in, and refine your query to request a more manageable > > number of rows (or wait a few years for Moore's law to catch up). > > > > Mark > > > > On Thu, 30 Jan 2014, Bruno Sversut Arsioli wrote: > > > > > Dears > > > > > > Learning how to use TopCat, I start facing some issues that I couldn't > > > solve in the last few days. Came out to work with the WISE catalog, which > > > is a source list based on Infra Red measurements, with about 500 Million > > > entries. > > > > > > I have found the database using the tool on the TopCat tab: > > > > > > VO / Table Acess Protocol Query > > > > > > there I search for "Wise" > > > > > > Build my query using the protocol: > > > > > > SELECT > > > TOP 500000000 > > > designation, ra, dec, w1mpro, w2mpro, w3mpro, w4mpro > > > FROM wise_allskysc > > > > > > > > > Also, I have selected Max number of Rows: unlimited > > > > > > > > > In the end, no mater what I change, I always get a sample with 2 Million > > > objects. > > > > > > > > > Is there any idea what I should do to get more than that? > > > > > > Thanks for your attention, > > > Bruno Sversut Arsioli > > > > > > > -- > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > > > -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From bilicki@camk.XXX Fri Jan 31 15:56:37 2014 Date: Fri, 31 Jan 2014 17:55:12 +0200 From: Maciej Bilicki To: Mark Taylor Cc: Bruno Sversut Arsioli , "topcat-user@bristol.XXX" Subject: Re: doubt (File with more than 2 million rows) Dear Mark, You're being too pessimistic about the capabilities of the great tool that you've developed. TOPCAT handles huge tables very well, of many more rows than 5 million. All is limited by the amount of memory, disk space and CPUs one has (of course TC needs to be started with relevant Java parameters as you mention - for instance -Xmx30G if you have this amount of RAM :) ). What gets painful with too large tables is plotting or defining subsets - but is still doable if one is patient [see Bilicki et al. 2014 for TC-generated Aitoff plots of hundreds of millions sources]. Creating subsets of huge tables, as well as cross-matching, is then much easier with STILTS :) All the best Maciek 2014-01-31 Mark Taylor : > Good! 5-8 Mrow should be OK, but you may need to boost java's heap > memory a bit - see here for some tips on using large tables: > > http://www.star.bristol.ac.uk/~mbt/topcat/sun253/largeTables.html > > Using vi to remove columns sounds unnecessarily painful. > You can remove columns in topcat by hiding them in the columns window: > > http://www.star.bristol.ac.uk/~mbt/topcat/sun253/ColumnInfoWindow.html > > if you then save the table the hidden columns will not be included. > However, having lots of columns doesn't usually bother topcat much, > especially if you use colfits format (again, see the tips above). > > Mark > > On Fri, 31 Jan 2014, Bruno Sversut Arsioli wrote: > > > Dear Mark and Maciej, > > > > Thanks for your quick reply! I am now aware of the ADTP and TAP > protocols , > > and I manage to do impose many selection rules to the ALLWISE catalog. I > > got some output tables with 5 to 8 Million objects, which I guess are > still > > possible to handle with TOPCAT. > > > > I am now using Vi, trying to optimize even more the tables, taking out > some > > columns . > > > > Thanks for your attention, > > > > Bruno S. Arsioli > > > > > > 2014-01-30 Mark Taylor : > > > > > Bruno, > > > > > > any given TAP service may impose a hard limit on the number of > > > rows it is prepared to give you. I think you're using the WFAU > > > service, and it does seem to impose a limit of 2,000,000 rows, > > > so no matter how much you ask for that's the most you'll get. > > > > > > WISE is available from other places. The GAVO TAP service > > > (search for "gavo" rather than "wise" in the TAP Select Service tab) > > > has the catalog wise.main, though I'm not sure if that's the same > > > data set you're requesting from WFAU. This service reports a hard > > > limit of 20,000,000 rows, so you can get a larger result from there. > > > > > > There are other ways of getting larger amounts of WISE data, > > > for instance VizieR or possibly bulk download of the catalogues. > > > > > > *However*: 20 Mrow is already quite a lot. 500 Mrow is really a lot. > > > If you have a few Gbyte on your machine topcat may cope with 20Mrow, > > > but not half a billion. If you're a data center or have serious > > > compute resources available to you it's possible that you might > > > have some use for the entire WISE catalogue, but if you're just > > > a scientist trying to explore the data, grabbing the whole lot to > > > look at is not likely to be a sensible way to go about it. > > > Instead you should think about some subset that you're actually > > > interested in, and refine your query to request a more manageable > > > number of rows (or wait a few years for Moore's law to catch up). > > > > > > Mark > > > > > > On Thu, 30 Jan 2014, Bruno Sversut Arsioli wrote: > > > > > > > Dears > > > > > > > > Learning how to use TopCat, I start facing some issues that I > couldn't > > > > solve in the last few days. Came out to work with the WISE catalog, > which > > > > is a source list based on Infra Red measurements, with about 500 > Million > > > > entries. > > > > > > > > I have found the database using the tool on the TopCat tab: > > > > > > > > VO / Table Acess Protocol Query > > > > > > > > there I search for "Wise" > > > > > > > > Build my query using the protocol: > > > > > > > > SELECT > > > > TOP 500000000 > > > > designation, ra, dec, w1mpro, w2mpro, w3mpro, w4mpro > > > > FROM wise_allskysc > > > > > > > > > > > > Also, I have selected Max number of Rows: unlimited > > > > > > > > > > > > In the end, no mater what I change, I always get a sample with 2 > Million > > > > objects. > > > > > > > > > > > > Is there any idea what I should do to get more than that? > > > > > > > > Thanks for your attention, > > > > Bruno Sversut Arsioli > > > > > > > > > > -- > > > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > > > m.b.taylor@bris.XXX +44-117-9288776 > http://www.star.bris.ac.uk/~mbt/ > > > > > > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > > From bilicki@camk.XXX Fri Jan 31 16:20:09 2014 Date: Fri, 31 Jan 2014 18:18:36 +0200 From: Maciej Bilicki To: "topcat-user@bristol.XXX" Subject: Accessing pixel count values Dear Mark, This time I have a question. I have used the Plane Layer Plot, with Form=density, to generate a 2D density plot of "X" vs. "Y". I understand that TOPCAT has pixelised the X--Y data to do that, i.e. each pixel ta a given X,Y has some count. Now I'd like to access the value of each of these pixels, i.e. to be able to produce an array of X,Y vs. count. Is this anyhow possible from TOPCAT, or do I need to use some other tools? Best wishes, Maciek From m.b.taylor@bristol.XXX Fri Jan 31 16:29:00 2014 Date: Fri, 31 Jan 2014 16:29:00 +0000 (GMT) From: Mark Taylor To: Maciej Bilicki Cc: "topcat-user@bristol.XXX" Subject: Re: Accessing pixel count values Hi Maciej. On Fri, 31 Jan 2014, Maciej Bilicki wrote: > I have used the Plane Layer Plot, with Form=density, to generate a 2D > density plot of "X" vs. "Y". I understand that TOPCAT has pixelised the > X--Y data to do that, i.e. each pixel ta a given X,Y has some count. Now > I'd like to access the value of each of these pixels, i.e. to be able to > produce an array of X,Y vs. count. Is this anyhow possible from TOPCAT, or > do I need to use some other tools? You can't do it from those plot windows. There are two ways you can get this information: - the STILTS command tcube is designed for making N-dimensional histograms. If N=2, you've got a density map. http://www.star.bristol.ac.uk/~mbt/stilts/sun256/tcube.html - if you use the old-style Density Plot (Graphics|Density menu item) you can save the plotted grid to FITS (or send it via SAMP if you prefer). Either way, what you get is a FITS file containing an image array because the result is really an array not a table. You can load it into e.g. ds9 or Aladin to look at it, but you can't pull it back into topcat. If you have some kind of library that understands FITS image arrays, it's easy enough to turn that into a mapping of (X,Y) to pixel values. Mark -- Mark Taylor Astronomical Programmer Physics, Bristol University, UK m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ From croman@astro.XXX Fri Jan 31 16:55:43 2014 Date: Fri, 31 Jan 2014 08:55:41 -0800 From: Carlos Roman To: Mark Taylor Cc: Maciej Bilicki , "topcat-user@bristol.XXX" Subject: Re: Accessing pixel count values this sounds very useful, thanks!! On Fri, Jan 31, 2014 at 8:29 AM, Mark Taylor wrote: > Hi Maciej. > > On Fri, 31 Jan 2014, Maciej Bilicki wrote: > > > I have used the Plane Layer Plot, with Form=density, to generate a 2D > > density plot of "X" vs. "Y". I understand that TOPCAT has pixelised the > > X--Y data to do that, i.e. each pixel ta a given X,Y has some count. Now > > I'd like to access the value of each of these pixels, i.e. to be able to > > produce an array of X,Y vs. count. Is this anyhow possible from TOPCAT, > or > > do I need to use some other tools? > > You can't do it from those plot windows. There are two ways you can > get this information: > > - the STILTS command tcube is designed for making N-dimensional > histograms. If N=2, you've got a density map. > > http://www.star.bristol.ac.uk/~mbt/stilts/sun256/tcube.html > > - if you use the old-style Density Plot (Graphics|Density menu item) > you can save the plotted grid to FITS (or send it via SAMP if you > prefer). > > Either way, what you get is a FITS file containing an image array > because the result is really an array not a table. You can load it > into e.g. ds9 or Aladin to look at it, but you can't pull it back > into topcat. If you have some kind of library that understands > FITS image arrays, it's easy enough to turn that into a mapping > of (X,Y) to pixel values. > > Mark > > -- > Mark Taylor Astronomical Programmer Physics, Bristol University, UK > m.b.taylor@bris.XXX +44-117-9288776 http://www.star.bris.ac.uk/~mbt/ > >