caGrid Transfer 1.4 Developers Guide

	Contents Before You Begin Utilizing the Transfer Service Introduce Configuration and Usage Add the Transfer Extension Create an Operation that Uses Transfer Using Transfer to Enable Clients to Download Data Code Needed in the Client Code Needed in the Service Using Transfer to Enable Clients to Upload Data Code Needed in the Client Code Needed in the Service About the DataDescriptor Constructor (Optional Metadata on the Transfer Context) Deploying a Service that Uses caGrid Transfer

Before You Begin

Proficiency in developing, deploying, and invoking grid services with Introduce is critical to successful use of the caGrid Transfer Tools; they are simple extensions to the normal service generation process.
Information and tutorials are available on the Introduce web site. We recommend you take the Introduce 1.4 tutorial and familiarize yourself with the Transfer Service Architecture. An example service, which is detailed in the guide below, can be found here.

Utilizing the Transfer Service

Introduce Configuration and Usage

Add the Transfer Extension

In Introduce, add the caGrid_Transfer extension to your service to utilize Grid Transfer. Prior to service generation, this is found in the Advanced tab of the creation panel. After the service has been generated, select caGrid_Transfer in the Extensions tab as shown in the image below:

Adding the Transfer Extension

Create an Operation that Uses Transfer

Using Introduce, you create a new Operation on your service in the usual manner (i.e., either on the main service or on a service context). The screenshot below shows the user creating a new operation on the main service context named "getSomeData". This operation returns a TransferServiceContextReference.

Note: Please refer to the next section for sample code showing the service method implementation.

Transfer Service Data Types

Using Transfer to Enable Clients to Download Data

Code Needed in the Client

On the client side, if a method from your service utilized the TransferServiceHelper to locally stage data, and returned the type TransferServiceContextReference, the TransferServiceContextClient and TransferClientHelper can be used to retrieve the data on the client. Below is a client example of how this can be executed in an insecure environment:

HelloWorldClient client = new HelloWorldClient(args[1]);
//create transfer is a method that staged some data and returned the Reference
TransferServiceContextReference ref = client.getSomeData();
//create a client that enables me to talk to my transfer resource
TransferServiceContextClient tclient = new TransferServiceContextClient(ref.getEndpointReference());
//use the TransferClientHelper to get an InputStream to the data
InputStream stream = TransferClientHelper.getData(tclient.getDataTransferDescriptor());

Again, in a secure environment:

HelloWorldClient client = new HelloWorldClient(args[1],GlobusCredential.getDefaultCredential());
//create transfer is a method that staged some data and returned the Reference
TransferServiceContextReference ref = client.getSomeData();
//create a client that enables me to talk to my transfer resource
TransferServiceContextClient tclient = new TransferServiceContextClient(ref.getEndpointReference(),GlobusCredential.getDefaultCredential());
//use the TransferClientHelper to get an InputStream to the data
InputStream stream = TransferClientHelper.getData(tclient.getDataTransferDescriptor(),GlobusCredential.getDefaultCredential());

You can always use the secure method of retrieving data, but if the connection URL is http, then the credentials will not be used and regular sockets will be used instead to retrieve the data.

Code Needed in the Service

Code inside the service is straightforward. When the user calls a method you created (i.e., getSomeData()) that you provided to them, your service must find their data, create the TransferContextResourceReference, and return the reference. Once the user has received the data, they can call destroy() on the resource. For example:

public org.cagrid.transfer.context.stubs.types.TransferServiceContextReference getSomeData() throws RemoteException {
    // create some data to sent
    byte[] data = new String("This is my bulk data").getBytes();
    // create a descriptor for that data
    DataDescriptor dd = new DataDescriptor(null, "My Data");
    // create the transfer resource that will handle delivering the data and
    // return the reference to the user
    return TransferServiceHelper.createTransferContext(data, dd);
}

Using Transfer to Enable Clients to Upload Data

Code Needed in the Client

The caGrid Transfer service can also be used to upload data from client to service. The service, as in the download case, must provide a method which returns an TransferServiceContextReference. They can do this by using the TransferServiceHelper API, except in this case they will call the createTransferContext(DataDescriptor dd) operation. This operation does not take in any data, as the intent is that the data is staged in by something else. It does, however, generate the resource and reserve a url where that data can be written and then eventually stored on the local machine. Below is an example of how the client is to use the TransferServiceContextClient and TransferClientHelper in order to upload data to the grid service so that it can be used:

HelloWorldClient client = new HelloWorldClient(args[1]);
// upload example
TransferServiceContextReference ref1 = client.putSomeData();
// use the EPR from the reference to create a client to talk
// to my resource
TransferServiceContextClient tclient1 = new TransferServiceContextClient(ref1.getEndpointReference());
// use the helper to get the output stream that I can upload
// my data with
DataDescriptor dd = new DataDescriptor(null, "My Data");
String myDataToSend = new String("New upload test");
InputStream is = new ByteArrayInputStream(myDataToSend.getBytes());
TransferClientHelper.putData(is, myDataToSend.length(), tclient1.getDataTransferDescriptor());
// tell the resource that the data has been uploaded.
tclient1.setStatus(Status.Staged);

Again, in a secure environment:

HelloWorldClient client = new HelloWorldClient(args[1],GlobusCredential.getDefaultCredential());
// upload example
TransferServiceContextReference ref1 = client.putSomeData();
// use the EPR from the reference to create a client to talk
// to my resource
TransferServiceContextClient tclient1 = new TransferServiceContextClient(ref1.getEndpointReference(),GlobusCredential.getDefaultCredential());
// use the helper to get the output stream that I can upload
// my data with
DataDescriptor dd = new DataDescriptor(null, "My Data");
String myDataToSend = new String("New upload test");
InputStream is = new ByteArrayInputStream(myDataToSend.getBytes());
TransferClientHelper.putData(is, myDataToSend.length(), tclient1.getDataTransferDescriptor(),GlobusCredential.getDefaultCredential());
// tell the resource that the data has been uploaded.
tclient1.setStatus(Status.Staged);

Again, as described before, you can always use the secure method of retrieving data but if the connection URL is http, the credentials will not be used and regular sockets will be used instead to upload the data.

Code Needed in the Service

When a user calls the createTransfer method that you provided for them, the TransferContextResourceReference is created, a DataStagedCallback is registered, and the reference returned. Once the the data is uploaded, and the setStatus operation is called on the resource (to let the resource know that the data is staged), the DataStagedCallback that was registered with the resource is executed. This allows your implementation to process the uploaded data. An example of this code is as follows:

public org.cagrid.transfer.context.stubs.types.TransferServiceContextReference putSomeData() throws RemoteException {
    // create a data descriptor for the upload for thed ata to be uploaded
    DataDescriptor dd = new DataDescriptor(null, "Users Data");
    // create a callback that will handle the data once it is uploaded
    DataStagedCallback callback = new DataStagedCallback() {

        public void dataStaged(TransferServiceContextResource resource) {
            File dataFileUserSentMe = new File(resource.getDataStorageDescriptor().getLocation());
            // do something with this data
        }

    };

    // create the transfer resource that will handle receiving the data and
    // return the reference to the user
    TransferServiceContextReference context = TransferServiceHelper.createTransferContext(dd, callback);

    // Store the service's content within the service's resource to prevent it from
    // being randomly destroyed by the Java garbage collector
    try 
    {
        // Here, resource is of type your service context resource
        // e.g. <service-java-package-name>.service.globus.resource.<service-name>Resource resource
        resource = getResourceHome().getAddressedResource();
        
        resource.setCallback(TransferServiceHelper.getTransferServiceContextID(context), callback);
    } catch (Exception e) {
        throw new RemoteException("Unable to obtain Service Context ID: " + e.getMessage());
    }

    return context;
}

Due to how callbacks are normally stored by the service, it is possible for the Java garbage collector to randomly destroy the callback object. To prevent this, the user needs to implement the PersistentTransferCallback into the service's Resource class. An example of this code is as follows:

public class HelloWorldResource extends HelloWorldResourceBase implements PersistentTransferCallback {

	private HashMap<String, DataStagedCallback> callbacks = new HashMap<String, DataStagedCallback>();

	public DataStagedCallback getCallback(String transferContextID) throws Exception {
		return callbacks.get(transferContextID);
	}

	public void setCallback(String transferContextID, DataStagedCallback callback) {
		callbacks.put(transferContextID, callback);
	}

}

About the DataDescriptor Constructor (Optional Metadata on the Transfer Context)

As of caGrid 1.2, the DataDescriptor constructor can take in an Object as the first parameter. Technically, anything that the service (and the transfer context specifically) knows how to serialize (as specified in both .wsdd files) can be handed to this constructor to pass some metadata about the transfer to the client. In practice, this Object is a placeholder for a future iteration of the transfer service that will add in standardized support for describing the Transfer streams. At this time, it is not recommended to put anything other than null in the constructor because a new client of the service would not have the beans and type descriptor files to be able to retrieve your metadata without customizing their client.
However, if you would like to customize the metadata on a Transfer context using a basic Axis type (such as a String or another basic type that is known to all services and doesn't require custom serialization), pass in the data to the server operation that creates the Transfer context. Then you can use this String in your DataDescriptor constructor to set a String descriptor (metadata) on the DataDescriptor object:

public org.cagrid.transfer.context.stubs.types.TransferServiceContextReference getContext(String metadata) throws RemoteException {

   DataDescriptor dd = new DataDescriptor(metadata, "Users Data");
 ...
}

The client can retrieve this metadata when it receives the transfer context:

String metadata = "JPEG";
TransferServiceContextReference ref = client.getContext(metadata);
TransferServiceContextClient tclient = new TransferServiceContextClient(ref.getEndpointReference());

DataTransferDescriptor d = tclient.getDataTransferDescriptor();
String metadataValue = (String)d.getDataDescriptor().getMetadata();
String dataDescriptorName = d.getDataDescriptor().getName();

Note that trying to set DataDescriptor fields client-side will NOT update the metadata value and data descriptor name for others. I other words, the "set" methods on the DataDescriptor object, when used client-side, have no effect.

Deploying a Service that Uses caGrid Transfer

The caGrid Transfer service MUST be deployed to the same container that the invoking grid service is deployed. Like all caGrid services, only Tomcat is currently supported. Prior to deployment of caGrid Transfer, Globus must be deployed to the container, and (if desired) security configuration must be completed. This can be accomplished by the caGrid Installer or manually by following the instructions provided by Globus.

After you have acquired caGrid and setup your container, deploy the caGrod Transfer Service. This is accomplished by entering the caGrid Transfer project directory. From the top of the caGrid release:

 cd projects/transfer

 ant deployTomcat

 cd <my services directory>

 ant deployTomcat