Access Keys:
Skip to content (Access Key - 0)

Bulk Data Transfer

Bulk Data Transfer Overview

As of caGrid 1.2 caGrid provides 2 types of 3rd party transfer mechanisms. The original GridFTP based Bulk Data Transfer and the new http based caGrid Transfer. The original BDT might still be used for very performance intensive transfer requirements such as parallel or striped data transfer, however, the new caGrid Transfer mechanism should satisfy the vast majority of the transfer use cases, and is much simpler to use, and is cross platform and secure.

As of caGrid 1.3, BDT has been deprecated and it's use is discouraged as it will be removed in a subsequent release.

The BulkDataHandler service aims to gather useful standard transfer mechanisms together into one standard service interface. The service will advertise metadata describing the possible transfer mechanisms which the BulkDataHandler service will support. The BulkDataHandler architecture utilizes the WSRF (Web Services Resource Framework) in order to enable the data provider to create resources which can then later be used to transmit the results to the user via the BulkDataHandler service. The data provider will be required to implement a resource interface on the server to enable the BulkDataHandler to be able to transmit their data. The current version of the BulkDataHandler supports 3 different transfer mechanisms: WS-Transfer, WS-Enumeration, and an accessor for obtaining GridFTP URLs. We believe that this current set of exposed functionality will enable users to effectively move data of larger sizes from service to client. The WS-Transfer and WS-Enumeration are standard data transportation standards for grid based computing and will fully adhere to security and data standards requirements of caBIG. GridFTP, however, is a transfer technology but not a standard protocol for doing so, that is, using GridFTP does not ensure interoperability from the server and the client. In this version we will be assuming that that transfer of the large data over the GridFTP channel is not guaranteed to be interoperable with the client. In our next version we plan to address this issue by providing specification of the "wire contents" of the GridFTP channel (such as requiring it to be XML representation, adopting a binary format such as BINX or DFDL, etc).

Last edited by
Sarah Honacki (1814 days ago) , ...
Adaptavist Theme Builder Powered by Atlassian Confluence