Access Keys:
Skip to content (Access Key - 0)

caGrid Transfer

caGrid Transfer 1.3 Developers Guide

Contents

Before You Begin

Proficiency in developing, deploying, and invoking grid services with Introduce is critical to successful use of the caGrid Transfer Tools; they are simple extensions to the normal service generation process.
Information and tutorials are available on the Introduce web site. We recommend you take the Introduce 1.2 tutorial and familiarize yourself with the Transfer Service Architecture. An example service, which is detailed in the guide below, can be found here.

Utilizing the Transfer Service

Introduce Configuration and Usage

Add the Transfer Extension

In Introduce, add the caGrid_Transfer extension to your service to utilize Grid Transfer. Prior to service generation, this is found in the Advanced tab of the creation panel. After the service has been generated, select caGrid_Transfer in the Extensions tab as shown in the image below:

Adding the Transfer Extension
The TransferServiceHelper API allows either the staging of data (in the service) for later retrieval by a client, or creation of a resource that allows a user to upload data for the service to retrieve. The first three methods can be used to stage data and return a TransferServiceContextReference, which contains an EndPointReference that can be used to retrieve the data on the client. The last method creates a resource where the client can upload data, and the service can locally retrieve.

Create an Operation that Uses Transfer

Using Introduce, you create a new Operation on your service in the usual manner (either on the main service or on a service context). The screenshot below shows the user creating a new operation on the main service context named "getSomeData". This operation returns a TransferServiceContextReference.

Transfer Service Data Types
Please refer to the next section for example code showing the service method implementation.

Using Transfer to Enable Clients to Download Data

Code Needed in the Client

On the client side, if a method from your service utilized the TransferServiceHelper to locally stage data, and returned the type TransferServiceContextReference, the TransferServiceContextClient and TransferClientHelper can be used to retrieve the data on the client. Below is a client example of how this can be executed in an insecure environment:

HelloWorldClient client = new HelloWorldClient(args[1]);
//create transfer is a method that staged some data and returned the Reference
TransferServiceContextReference ref = client.getSomeData();
//create a client that enables me to talk to my transfer resource
TransferServiceContextClient tclient = new TransferServiceContextClient(ref.getEndpointReference());
//use the TransferClientHelper to get an InputStream to the data
InputStream stream = TransferClientHelper.getData(tclient.getDataTransferDescriptor());

Again, in a secure environment:

HelloWorldClient client = new HelloWorldClient(args[1],GlobusCredential.getDefaultCredential());
//create transfer is a method that staged some data and returned the Reference
TransferServiceContextReference ref = client.getSomeData();
//create a client that enables me to talk to my transfer resource
TransferServiceContextClient tclient = new TransferServiceContextClient(ref.getEndpointReference(),GlobusCredential.getDefaultCredential());
//use the TransferClientHelper to get an InputStream to the data
InputStream stream = TransferClientHelper.getData(tclient.getDataTransferDescriptor(),GlobusCredential.getDefaultCredential());

You can always use the secure method of retrieving data, but if the connection URL is http then the credentials will not be used, and regular sockets will be used to retrieve the data.

Code Needed in the Service

Code inside the service is straightforward: when the user calls a method you created (i.e. getSomeData()) that you provided them, your service must find their data, create the TransferContextResourceReference, and return the reference. Once the user has received the data, they can call destroy() on the resource. An example:

public org.cagrid.transfer.context.stubs.types.TransferServiceContextReference getSomeData() throws RemoteException {
    // create some data to sent
    byte[] data = new String("This is my bulk data").getBytes();
    // create a descriptor for that data
    DataDescriptor dd = new DataDescriptor(null, "My Data");
    // create the transfer resource that will handle delivering the data and
    // return the reference to the user
    return TransferServiceHelper.createTransferContext(data, dd);
}

Using Transfer to Enable Clients to Upload Data

Code Needed in the Client

The caGrid Transfer service can also be used to upload data from client to service. The service, as in the download case, must provide a method which returns an TransferServiceContextReference. Thay can do this by using the TransferServiceHelper API, except, in this case they will call the createTransferContext(DataDescriptor dd) operation. This operation does not take any data in, as the intent is that the data is staged in by something else, however, it does generate the resource and reserve a url where that data can be written and then eventually stored on the local machine. Below is an example of how the client is to use the TransferServiceContextClient and TransferClientHelper in order to upload data to grid service so that it can be used:

HelloWorldClient client = new HelloWorldClient(args[1]);
// upload example
TransferServiceContextReference ref1 = client.putSomeData();
// use the EPR from the reference to create a client to talk
// to my resource
TransferServiceContextClient tclient1 = new TransferServiceContextClient(ref1.getEndpointReference());
// use the helper to get the output stream that I can upload
// my data with
DataDescriptor dd = new DataDescriptor(null, "My Data");
String myDataToSend = new String("New upload test");
InputStream is = new ByteArrayInputStream(myDataToSend.getBytes());
TransferClientHelper.putData(is, myDataToSend.length(), tclient1.getDataTransferDescriptor());
// tell the resource that the data has been uploaded.
tclient1.setStatus(Status.Staged);

Again, in a secure environment:

HelloWorldClient client = new HelloWorldClient(args[1],GlobusCredential.getDefaultCredential());
// upload example
TransferServiceContextReference ref1 = client.putSomeData();
// use the EPR from the reference to create a client to talk
// to my resource
TransferServiceContextClient tclient1 = new TransferServiceContextClient(ref1.getEndpointReference(),GlobusCredential.getDefaultCredential());
// use the helper to get the output stream that I can upload
// my data with
DataDescriptor dd = new DataDescriptor(null, "My Data");
String myDataToSend = new String("New upload test");
InputStream is = new ByteArrayInputStream(myDataToSend.getBytes());
TransferClientHelper.putData(is, myDataToSend.length(), tclient1.getDataTransferDescriptor(),GlobusCredential.getDefaultCredential());
// tell the resource that the data has been uploaded.
tclient1.setStatus(Status.Staged);

Again, as described before, you can always use the secure method of retrieving data but if the connection URL is http, the credentials will not be use and regular sockets will be used to upload the data.

Code Needed in the Service

When a user calls the createTransfer method that you provided, the TransferContextResourceReference is created, a DataStagedCallback is registered, and the reference is returned. Once the the data is uploaded and the setStatus operation is called on the resource (to let the resource know that the data is staged), the DataStagedCallback that was registered with the resource will be executed. This allows your implementation to process the uploaded data. An example of this code is as follows:

public org.cagrid.transfer.context.stubs.types.TransferServiceContextReference putSomeData() throws RemoteException {
    // create a data descriptor for the upload for thed ata to be uploaded
    DataDescriptor dd = new DataDescriptor(null, "Users Data");
    // create a callback that will handle the data once it is uploaded
    DataStagedCallback callback = new DataStagedCallback() {

        public void dataStaged(TransferServiceContextResource resource) {
            File dataFileUserSentMe = new File(resource.getDataStorageDescriptor().getLocation());
            // do something with this data
        }

    };

    // create the transfer resource that will handle receiving the data and
    // return the reference to the user
    return TransferServiceHelper.createTransferContext(dd, callback);
}

Note that the invocation of the DataStagedCallback is an asynchronous operation. That is, when the client calls tclient1.setStatus(Status.Staged), the call will return without waiting for the DataStagedCall function to complete its operation. On one hand, this implementation allows a client to proceed and overlap data staging operations with other client-side operations. On the other hand, if a client calls a service method that needs the data being uploaded, there is no guarantee that the DataStagedCall function will have completed before the client calls said service method. To address this issue, implement a checkUpload() service method, which will return false if the DataStagedCall function has not finished and true otherwise, and/or a waitUpload() service method, which will block until the DataStagedCall function finishes its process. To implement these functions, the service developer can use a service resource to monitor the state of the DataStagedCall function. An example is provided below:

A service resource class called MonitorUpload is created:

class MonitorUpload {
	boolean uploadCompleted;
}

This class (represented in a XML schema) is registered as service resource while creating the service using the Introduce toolkit. The example putSomeData method should be modified as follows:

 public org.cagrid.transfer.context.stubs.types.TransferServiceContextReference putSomeData() throws RemoteException {
    // create a data descriptor for the upload for thed ata to be uploaded
    DataDescriptor dd = new DataDescriptor(null, "Users Data");

    // get the MonitorUpload resource
    // Here, resource is of type your service context resource
    resource = getResourceHome().getAddressedResource();

    // Get the MonitorUpload object.
    // Note the object should be declared "final"
    final MonitorUpload myMonitor = resource.getMonitorUpload();

    // set the upload completed flag to false
    myMonitor.setUploadCompleted(false);

    // create a callback that will handle the data once it is uploaded
    DataStagedCallback callback = new DataStagedCallback() {

        public void dataStaged(TransferServiceContextResource resource) {
            File dataFileUserSentMe = new
File(resource.getDataStorageDescriptor().getLocation());
                // do something with this data

		// right before return, set the upload completed flag to true
		myMonitor.setUploadCompleted(true);
		return;
        }

    };

    // create the transfer resource that will handle receiving the data and
    // return the reference to the user
    return TransferServiceHelper.createTransferContext(dd, callback);
}

The two additional service methods could be implemented as follows:

public boolean checkUpload()
{
        resource = getResourceHome().getAddressedResource();
	MonitorUpload myMonitor = resource.getMonitorUpload();
        return myMonitor.getUploadCompleted();
}

public void waitUpload()
{
        resource = getResourceHome().getAddressedResource();
	MonitorUpload myMonitor = resource.getMonitorUpload();
	While (myMonitor.getUploadCompleted()==false);
}

The client would call these service methods if it wanted to check the upload or wait until the upload is completed.

Secure Uploads

When uploading data to a secure transfer service, it has been discovered that the DataStagedCallback object used to process the completed updoad might be destroyed by the Java garbage collector. Details about this issue can be located on the in the caGrid Issue Tracker on Gforge in Bug 22321.

A fix has been implemented for this bug and applied to the caGrid 1.4 source code, but has not been back-ported to caGrid 1.3. We recommend that you upgrade to Transfer 1.4 if you encounter this bug or need to use secure data uploads.

About the DataDescriptor Constructor (Optional Metadata on the Transfer Context)

As of caGrid 1.2, the DataDescriptor constructor can take in an Object as the first parameter. Technically, anything that the service (and the transfer context specifically) knows how to serialize (as specified in both .wsdd files) can be handed to this constructor to pass some metadata about the transfer to the client. In practice, this Object is a placeholder for a future iteration of the transfer service that will add in standardized support for describing the Transfer streams. At this time, it is not recommended to put anything other than null in the constructor, because a new client of your service would not have the beans and type descriptor files to be able to retrieve your metadata without customizing their client.
However, if you would like to customize the metadata on a Transfer context using a basic Axis type (such as a String or another basic type that is known to all service and doesn't require custom serialization), pass in the data to the server operation that creates the Transfer context. Then you can use this String in your DataDescriptor constructor to set a String descriptor (metadata) on the DataDescriptor object:

public org.cagrid.transfer.context.stubs.types.TransferServiceContextReference getContext(String metadata) throws RemoteException {

   DataDescriptor dd = new DataDescriptor(metadata, "Users Data");
 ...
}

The client can retrieve this metadata when it receives the transfer context:

String metadata = "JPEG";
TransferServiceContextReference ref = client.getContext(metadata);
TransferServiceContextClient tclient = new TransferServiceContextClient(ref.getEndpointReference());

DataTransferDescriptor d = tclient.getDataTransferDescriptor();
String metadataValue = (String)d.getDataDescriptor().getMetadata();
String dataDescriptorName = d.getDataDescriptor().getName();
Note that trying to set DataDescriptor fields client-side will NOT update the metadata value and data descriptor name for others. (i.e., the "set" methods on the DataDescriptor object, when used client-side, have no effect).

Deploying a Service that Uses caGrid Transfer

The caGrid Transfer service MUST be deployed to the same container that the invoking grid service in deployed. Like all caGrid services, only Tomcat and JBoss are currently supported. Prior to deployment of caGrid Transfer, Globus must be deployed to the container, and (if desired) security configuration must be completed. This can be accomplished by the caGrid Installer, or manually by following the instructions provided by Globus.

After you have acquired caGrid and setup your container, you can deploy the caGrod Transfer Service. This is accomplished by entering the caGrid Transfer project directory. From the top of the caGrid release:

 cd projects/transfer
Then deploy to either Tomcat or JBoss (tomcat example below):
 ant deployTomcat
Next, deploy your service by entering the services directory:
 cd <my services directory>
Then deploy to either Tomcat or JBoss (tomcat example below):

 ant deployTomcat
Last edited by
Joe George (575 days ago) , ...
Adaptavist Theme Builder Powered by Atlassian Confluence