uk.ac.starlink.ttools.cone

Class BlockUploader

java.lang.Object

uk.ac.starlink.ttools.cone.BlockUploader

public class BlockUploader
extends Object

Works with an UploadMatcher dividing the input table into chunks and uploading them separately to produce an arbitrarily large result while each upload/match operation is of a limited size.

Since:

15 May 2014

Author:

Mark Taylor

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	BlockUploader.BlockStats Provides information about a match.

Constructor Summary

Constructors

Constructor and Description
BlockUploader(UploadMatcher umatcher, int blocksize, long maxrec, String outName, JoinFixAction uploadFixAct, JoinFixAction remoteFixAct, ServiceFindMode serviceMode, boolean oneToOne, boolean uploadEmpty) Constructor.

Method Summary

Modifier and Type	Method and Description
Bi<StarTable,BlockUploader.BlockStats>	runMatch(StarTable inTable, QuerySequenceFactory qsFact, StoragePolicy storage) Performs an upload join in blocks.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BlockUploader

public BlockUploader(UploadMatcher umatcher,
                     int blocksize,
                     long maxrec,
                     String outName,
                     JoinFixAction uploadFixAct,
                     JoinFixAction remoteFixAct,
                     ServiceFindMode serviceMode,
                     boolean oneToOne,
                     boolean uploadEmpty)

Constructor.

Parameters:

umatcher - upload matcher

blocksize - maximum number of rows per uploaded block

maxrec - maximum number of output rows accepted in total

outName - name of output table

uploadFixAct - name deduplication policy for upload table

remoteFixAct - name deduplication policy for remote table

serviceMode - upload match mode

oneToOne - true iff output rows match 1:1 with input rows

uploadEmpty - determines behaviour if there are no input rows: true means attempt the match anyway, false means throw an IOException

Method Detail

runMatch

public Bi<StarTable,BlockUploader.BlockStats> runMatch(StarTable inTable,
                                                       QuerySequenceFactory qsFact,
                                                       StoragePolicy storage)
                                                throws IOException

Performs an upload join in blocks.

As currently implemented, the input table needs to have random access, and the output table has random access. It would be possible to stream on input and output to some extent, though each input chunk will still have to be random access.

The row indices given by ConeQueryRowSequence.getIndex calls associated with the supplied qsFact object must correspond to the row indices in the supplied inTable.

The service may truncate results for some of the blocks. Warnings will be written through the logging system if this happens. Summary information about such truncation will also be returned in the BlockStats object.

Parameters:

inTable - input table, must have random access

qsFact - object to generate positional queries when applied to a table

storage - storage policy for storing raw result table

Returns:

the result table and additional information about the match

Throws:

IOException