Troubleshoot Index Service Registration


	Contents Steps Necessary for Success Registration Problems Aggregation Problems Diagnosing the Problem Step 1: Turn on debug logging and monitor the log file. Step 2: Verify that your Grid service is registering with the Index service Step 3: Using the caBIG Portal to diagnose your service 1. Index Make sure you are advertising to the correct Index service Make sure your container is publishing the correct host name Advertise the correct protocol and port 2. Status in Portal 3. Ping Service Changing your port when using a web proxy (Apache) in front of your service 4. Service Metadata Confirm that the service's metadata is accessible Step 4: Verify that your host is reachable from outside your institution Step 5: Check Container Logs Additional Steps Everything seems okay, but it's still not working Appendix Container File Locations <SERVICE NAME>_registration.xml Diagnostic Tools

Steps Necessary for Success

For successful registration, there are two processes that need to work:

Registration - The service must successfully "register" to the Index Service; the service must be able to connect to the Index Service. This simply means the service will show up in the list of services registered but does not mean it will be "discoverable."
Aggregation - The service's metadata must be able to be retrieved and aggregated by the Index Service; the Index Service must be able to connect to the service.

Registration Problems

The service in question is not actually initialized and running.
The hosting machine's system clock is significantly off.

Aggregation Problems

The container is not configured to run with a public IP or externally resolvable domain name.
The service is running behind a firewall.
The service or container is running with untrusted credentials.
The service must be providing ServiceMetadata and specify that it should be registered to the Index Service.

Diagnosing the Problem

Step 1: Turn on debug logging and monitor the log file.

Enable debug logging. Alternatively, you can use this example log4 configuration file.
Restart the container.
You should see information similar to that below, indicating what's going on:

2007-01-29 09:43:25,875 INFO resource.BaseResourceHome [gov.nih.nci.cagrid.fqp.service.globus.resource.BaseResourceHome.initialize(BaseResourceHome.java:47)] Attempting to initialize resource.
2007-01-29 09:43:25,875 INFO resource.BaseResourceHome [gov.nih.nci.cagrid.fqp.service.globus.resource.BaseResourceHome.findSingleton(BaseResourceHome.java:22)] Creating a single resource.
2007-01-29 09:43:26,015 INFO resource.BaseResource [gov.nih.nci.cagrid.fqp.service.globus.resource.BaseResource.refreshRegistration(BaseResource.java:126)] Attempting registration for the first time[container URL=https://localhost:8443/wsrf/services/].
2007-01-29 09:43:26,093 INFO resource.BaseResourceHome [gov.nih.nci.cagrid.fqp.service.globus.resource.BaseResourceHome.initialize(BaseResourceHome.java:52)] Successfully initialized resource.
2007-01-29 09:43:26,265 DEBUG client.ServiceGroupRegistrationClient [org.globus.wsrf.impl.servicegroup.client.ServiceGroupRegistrationClient.status(ServiceGroupRegistrationClient.java:474)] Renewing/Adding:
<ns1:ServiceGroupRegistrationParameters xsi:type="ns1:ServiceGroupRegistrationParameters" xmlns:ns1="http://mds.globus.org/servicegroup/client" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<ns1:ServiceGroupEPR xsi:type="ns2:EndpointReferenceType" xmlns:ns2="http://schemas.xmlsoap.org/ws/2004/03/addressing">
		<ns2:Address xsi:type="ns2:AttributedURI">http://cagrid01.bmi.ohio-state.edu:8080/wsrf/services/DefaultIndexService</ns2:Address>
		<ns2:ReferenceProperties xsi:type="ns2:ReferencePropertiesType"/>
		<ns2:ReferenceParameters xsi:type="ns2:ReferenceParametersType"/>
	</ns1:ServiceGroupEPR>
	<ns1:RegistrantEPR xsi:type="ns3:EndpointReferenceType" xmlns:ns3="http://schemas.xmlsoap.org/ws/2004/03/addressing">
		<ns3:Address xsi:type="ns3:AttributedURI">https://localhost:8443/wsrf/services/cagrid/FederatedQueryProcessor</ns3:Address>
		<ns3:ReferenceProperties xsi:type="ns3:ReferencePropertiesType"/>
		<ns3:ReferenceParameters xsi:type="ns3:ReferenceParametersType"/>
	</ns1:RegistrantEPR>
	<ns1:RefreshIntervalSecs xsi:type="xsd:int" xmlns:xsd="http://www.w3.org/2001/XMLSchema">600</ns1:RefreshIntervalSecs>
	<ns1:Content xsi:type="ns4:AggregatorContent" xmlns:ns4="http://mds.globus.org/aggregator/types">
		<ns4:AggregatorConfig xsi:type="ns4:AggregatorConfig">
			<ns9:GetMultipleResourcePropertiesPollType xmlns:ns0="http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceLifetime-1.2-draft-01.xsd" xmlns:ns1="http://caGrid.caBIG/1.0/gov.nih.nci.cagrid.dcqlresult" xmlns:ns2="http://CQL.caBIG/1/gov.nih.nci.cagrid.CQLResultSet" xmlns:ns3="http://CQL.caBIG/1/gov.nih.nci.cagrid.CQLQuery" xmlns:ns4="http://caGrid.caBIG/1.0/gov.nih.nci.cagrid.dcql" xmlns:ns5="http://fqp.cagrid.nci.nih.gov/FederatedQueryProcessor/types" xmlns:ns6="gme://caGrid.caBIG/1.0/gov.nih.nci.cagrid.metadata.security" xmlns:ns7="gme://caGrid.caBIG/1.0/gov.nih.nci.cagrid.metadata" xmlns:ns8="http://fqp.cagrid.nci.nih.gov/FederatedResults/types" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ns9="http://mds.globus.org/aggregator/types">
				<!-- Specifies that the index should refresh information every 300000 milliseconds (once every 5 minutes) -->
				<ns9:PollIntervalMillis>300000</ns9:PollIntervalMillis>
				<!-- specifies all Resource Properties that should be retrieved from the service -->
				<ns9:ResourcePropertyNames>ns7:ServiceMetadata</ns9:ResourcePropertyNames>
			</ns9:GetMultipleResourcePropertiesPollType>
		</ns4:AggregatorConfig>
		<ns4:AggregatorData xsi:type="ns4:AggregatorData"/>
	</ns1:Content>
</ns1:ServiceGroupRegistrationParameters>
2007-01-29 09:43:26,421 DEBUG client.ServiceGroupRegistrationClient [org.globus.wsrf.impl.servicegroup.client.ServiceGroupRegistrationClient.status(ServiceGroupRegistrationClient.java:474)] Attempting to get current time, preferring remote.
2007-01-29 09:43:26,421 DEBUG client.ServiceGroupRegistrationClient [org.globus.wsrf.impl.servicegroup.client.ServiceGroupRegistrationClient.status(ServiceGroupRegistrationClient.java:474)] No remote current time available. Getting from local clock instead.
2007-01-29 09:43:26,421 DEBUG client.ServiceGroupRegistrationClient [org.globus.wsrf.impl.servicegroup.client.ServiceGroupRegistrationClient.status(ServiceGroupRegistrationClient.java:474)] Attempting add
2007-01-29 09:43:29,671 DEBUG client.ServiceGroupRegistrationClient [org.globus.wsrf.impl.servicegroup.client.ServiceGroupRegistrationClient.status(ServiceGroupRegistrationClient.java:474)] Add response: null
2007-01-29 09:43:29,671 INFO client.ServiceGroupRegistrationClient [org.globus.wsrf.impl.servicegroup.client.ServiceGroupRegistrationClient.status(ServiceGroupRegistrationClient.java:478)] Successfully registered https://localhost:8443/wsrf/services/cagrid/FederatedQueryProcessor to servicegroup at http://cagrid01.bmi.ohio-state.edu:8080/wsrf/services/DefaultIndexService

In this example, the FederatedQueryProcessor service is registering to the Index Service running at cagrid01.bmi.ohio-state.edu. We can also see which metadata is supposed to be aggregated at the Index Service from this line:

<ns9:ResourcePropertyNames>ns7:ServiceMetadata</ns9:ResourcePropertyNames>

That line illustrates that the caGrid standard common metadata (ServiceMetadata) will be aggregated. By turning on this debugging information, we can determine:

What URL the service is trying to register
To which Index Service it is registering
What metadata it is registering
Whether or not the "registration" was successful

Verify the URL is correct by entering the URL in your web browser. You should see a result similar to the following:

If you do not see this, then the service may be experiencing a registration issue. Check that your container is started and that your system clock is accurate.

It is vital that your server have the correct system time to register with the Index service. Please double check your system time. We strongly recommend that you use NTP to keep your clock accurate.

In the example output above, we can see the service registration won't "work" (the Index Service won't have its metadata, and therefore the DiscoveryClient won't be able to discover it) because the service is running in a container that is configured to run as "localhost". The following section will correct this.

Step 2: Verify that your Grid service is registering with the Index service

Query the index service and check the output:

 $GLOBUS_LOCATION/bin/wsrf-query -a -z none -s http://cagrid-index.nci.nih.gov:8080/wsrf/services/DefaultIndexService | grep "<ns8:Address xmlns:ns8" | cut -d'>' -f2 | cut -d'<' -f1 | sort > all_grid_services.txt

Then inspect the file named "all_grid_services.txt". Check that your Grid service URL is listed.
This step verified that your Grid service is advertising to the Index service. However, you need to check the next step to make sure the Index service can retrieve your service's metadata.

Step 3: Using the caBIG Portal to diagnose your service

Go to the caBIG portal to check the status of your service: Production portal service diagnostics page Enter your Grid service URL in the "Service Diagnostics" pane and click "Diagnose."

If you are using the Training Grid, go to the training portal service diagnostics page.

The results will be similar to the following:

1. Index

If the result is not "Service found in the INDEX":

Make sure you are advertising to the correct Index service

Check your container for the <SERVICE NAME>_registration.xml file. Check the cagrid Appendix for the location of that file.

Here is a snippet from that file that indicates the service is registering to the Training Grid index service:

<ServiceGroupEPR>
<wsa:Address>http://index.training.cagrid.org:8080/wsrf/services/DefaultIndexService</wsa:Address></ServiceGroupEPR>

Make sure you are registering to the correct Index service. That is, make sure the URL here matches the URL for the Index service in your target Grid (caBIG Production, Training Grid, etc.).

If you need to update it, simply edit the file and restart the container. Verify that you are registering with the appropriate Index service by checking your container's log file. You should see something similar to the following:

Loading registration arguments from:/Users/jpermar/containers/jboss-4.0.5.GA-globus-secure-TestSecureTransfer/server/default/./deploy/wsrf.war/W EB-INF/etc/cagrid_TransferService/TransferServiceContext_registration.xml 2009-02-03 17:23:38,007 DEBUG [org.globus.wsrf.encoding.ObjectDeserializationContext]<ServiceGroupRegistrationParameters xmlns="http:/ /mds.globus.org/servicegroup/client"xmlns:agg="http://mds.globus.org/aggregator/types"xmlns:sgc="http://mds.globus.org/servicegroup/c lient"xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing"xmlns:xsd="http://www.w3.org/2001/XMLSchema"xmlns:xsi="http://www. w3.org/2001/XMLSchema-instance"><!-- The ServiceGroupEPR defines the servicegroup to which registrations will be made --><ServiceGroupEPR><wsa:Address>http://cagrid-index.nci.nih.gov:8080/wsrf/services/DefaultIndexService</wsa:Address></ServiceGroupEPR>

Additionally, update that URL in your service's deploy.properties file so that if you re-deploy it, it will point to the correct Index Service.

Then restart the container and check again with the portal diagnostics page. Please wait about 30 minutes to see your service register properly with the Index Service.

Make sure your container is publishing the correct host name

Your service must register with a publicly accessible address or DNS-resolvable host name, so the Index Service (and other clients) can connect to it. The caGrid Installer allows you to specify your container's host name and port, but if you used the incorrect settings or didn't use the installer, this section details how to change those settings.

The caGrid advertisement code uses Globus' ServiceHost API to determine the container's host name and port information. The host name of the container, unless otherwise specified, will be the hosting machine's IP address or primary name. This behavior is controllable by parameters in the global configuration settings for Globus.

Users of the Globus container should edit the file $GLOBUS_LOCATION/etc/globus_wsrf_core/server-config.wsdd to add a logicalHost parameter with a value of the IP or host name they wish to use, which should be publicly accessible. If you have already deployed your service then you need to modify this server-config.wsdd inside your container (file location is specified in the appendix).

For example:

...
<globalConfiguration>
...
<parameter name="sendXsiTypes"value="true"/>
<parameter name="logicalHost"value="somehost.cagrid.org"/> <parameter name="publishHostName"value="true"/>
...

NOTE: add the two lines for parameters named "logicalHost" and "publishHostName" just after the "sendXsiTyes" parameter line which is already in your configuration file.

The settings above will cause the container's services to advertise with the host somehost.cagrid.org (where somehost.cagrid.org is your externally visible DNS entry or IP value) regardless of the primary host name of the machine.

Tomcat users should also edit the deployed version of this file (which is what is used by Tomcat), in the same fashion, in the location $CATALINA_HOME/webapps/wsrf/WEB-INF/etc/globus_wsrf_core/server-config.wsdd.

JBoss users should edit the file at $JBOSS_HOME/server/default/deploy/wsrf.war/WEB-INF/etc/globus_wsrf_core/server-config.wsdd as specified above.

Advertise the correct protocol and port

Tomcat users may encounter this Globus bug, which causes the wrong port and/or protocol to be advertised if you did not use the caGrid Installer. You can work around this issue by setting parameters "defaultPort" and "defaultProtocol" in web.xml in tomcat to match the server.xml settings.

For example, if you are using Tomcat on port 8080 and using http, edit web.xml appropriately:

<servlet> <servlet-name>WSRFServlet</servlet-name> <display-name>WSRF Container Servlet</display-name> <servlet-class> org.globus.wsrf.container.AxisServlet? </servlet-class> <init-param> <param-name>defaultProtocol</param-name> <param-value>http</param-value> </init-param> <init-param> <param-name>defaultPort</param-name> <param-value>8080</param-value> </init-param> <load-on-startup>true</load-on-startup> </servlet>

NOTE: Depending upon the URL that was used to access the service for the first time, the container at times (mostly Tomcat & JBoss) caches the host name used in the URL, and this may cause issues when the container is trying to broadcast itself with a different name (via proxy settings in server.xml file). We recommend clearing the cache on the server.

For Tomcat, these folders are the work and temp folders in the $CATALINA_HOME.

For JBoss, the temporary folders are the tmp and work folders in $JBOSS_HOME/server/default: http://docs.jboss.org/jbossas/getting_started/v4/html/tour.html.

2. Status in Portal

Service is in ACTIVE status

This will show up ACTIVE if the previous checks are successful.

3. Ping Service

If the result does not say "Successfully pinged service", verify that:

you have the correct Grid Service URL.
there is no firewall or proxy preventing the Grid from contacting your service.
your DNS server is properly configured. You may need to contact your network administrator(s).

Changing your port when using a web proxy (Apache) in front of your service

You may have a deployment environment in which you cannot open a container port directly through your firewall. If you absolutely cannot open the container port directly, please contact the caGrid Knowledge Center for additional information. However, what follows are some guidelines for what needs to be done.

If you are using Apache or another web service to re-direct requests to your Grid service, you need to tell Tomcat about the external port to use. Edit Tomcat's connector settings in server.xml and set the proxyPort to your external port (e.g., 80).

The technical details are as follows (reference):

"Under some circumstances, the HttpConnector class reports the original port back to the client. Further requests will continue with that port, which is not the desired effect and might even be blocked by your firewall. You can circumvent that problem by adding a proxyPort to the HttpConnector declaration: <Connector className="org.apache.catalina.connector.http.HttpConnector"port="8080"proxyPort="80"> ...

Your connector will display something like <Connector ... port="<internal tomcat port>" proxyPort="<external proxy port>"...>
The "port" is used at startup time (outside of the context of any invocation) for the initial registration. However, when someone actually connects to your service, the registration code will attempt to use the port that was used to connect to it to renew the registration. If you are telling tomcat to connect to one port, Tomcat tells the client to use the "proxyPort" regardless of how the client connected.

JBoss configuration: Edit the file _$JBOSS_HOME/server/default/deploy/jbossweb-tomcat55.sar/server.xml.

More details on JBoss Tomcat connector options can be found in the jboss docs.

If you use a firewall, check that your firewall settings are configured properly.

To repeat this check yourself, enter the service URL in a web browser that is "on the Internet" (i.e., outside your firewall).

4. Service Metadata

If the result is not "Service Metadata retrieved successfully", please follow these instructions to verify that your service is properly advertising metadata:

Confirm that the service's metadata is accessible

If you've validated that the "registration" process is working, there may be an aggregation problem. Request the common service metadata from your service, preferably from an "external" machine (i.e., outside of your firewall).

On Unix-based systems, run this command (replacing your service's URL):

 $GLOBUS_LOCATION/bin/wsrf-get-property -a -z none -s <YOUR_SERVICE'S URL> {gme://caGrid.caBIG/1.0/gov.nih.nci.cagrid.metadata}ServiceMetadata

On Windows-based systems, run this command (replacing your service's URL):

 %GLOBUS_LOCATION%\bin\wsrf-get-property.bat -a -z none -s <YOUR_SERVICE'S URL> {gme://caGrid.caBIG/1.0/gov.nih.nci.cagrid.metadata}ServiceMetadata

Running this command should display your service's common service metadata. If you get an error, this is likely to be the error the Index Service is getting.

Domain Model

If your service is a Data Service and you want to check if the domain model for the service is being advertised correctly, run the following command:

On Unix-based systems, run this command (replacing your service's URL):

 $GLOBUS_LOCATION/bin/wsrf-get-property -a -z none -s <YOUR_SERVICE'S URL> {gme://caGrid.caBIG/1.0/gov.nih.nci.cagrid.metadata.dataservice}DomainModel

On Windows-based systems, run this command (replacing your service's URL):

 %GLOBUS_LOCATION%\bin\wsrf-get-property.bat -a -z none -s <YOUR_SERVICE'S URL> {gme://caGrid.caBIG/1.0/gov.nih.nci.cagrid.metadata.dataservice}DomainModel

Running this command will display your data service's domain model. If you get an error, this is likely the error the Index Service is getting.

Step 4: Verify that your host is reachable from outside your institution

As stated before, registration is a 2-phase process. Your service registers itself with the Index Service, then the Index Service connects to your service to aggregate the service metadata. This aggregation step requires that your service is available to connection from the Internet.

Please Test your service endpoint from a tool on the web that will verify that it is routable from outside your network.

External Web Site Test Tools:
http://validator.w3.org/checklink
http://www.websiteoptimization.com/services/analyze/

Step 5: Check Container Logs

Check the logs of your container to identify the specific problem. The locations of the log file are:

Tomcat: $CATALINA_HOME/logs/catalina.out
JBoss: $JBOSS_HOME/server/default/log

Additional Steps

Peruse the grid service deployment checklist: Grid Service Deployment Checklist.

Everything seems okay, but it's still not working

If you can't figure out the problem, contact the caBIG caGrid Knowledge Center.

Appendix

Container File Locations

server.xml

Notes: container startup information, including host and port to listen on

Tomcat: $CATALINA_HOME/conf/server.xml
JBoss: $JBOSS_HOME/server/default/deploy/jbossweb-tomcat55.sar/server.xml

web.xml

Notes: port that is advertised to the Index Service. Must match your port in server.xml

Tomcat: $CATALINA_HOME/webapps/wsrf/WEB-INF/web.xml
JBoss: $JBOSS_HOME/server/default/deploy/wsrf.war/WEB-INF/web.xml

server-config.wsdd

Notes: hostname that you publish to the Index Service

Tomcat: $CATALINA_HOME/webapps/wsrf/WEB-INF/etc/globus_wsrf_core/server-config.wsdd.
JBoss: $JBOSS_HOME/server/default/deploy/wsrf.war/WEB-INF/etc/globus_wsrf_core/server-config.wsdd as specified above.

jndi-config.xml

Notes: perform Registration parameter to determine whether to advertise to Index Service

Tomcat: $CATALINA_HOME/webapps/wsrf/WEB-INF/etc/cagrid_<SERVICE NAME>/jndi-config.xml
JBoss: $JBOSS_HOME/server/default/deploy/wsrf.war/WEB-INF/etc/cagrid_TestDataServiceQuery/jndi-config.xml

<SERVICE NAME>_registration.xml

Notes: the Index Service URL with which to register

Tomcat: $CATALINA_HOME/webapps/wsrf/WEB-INF/etc/cagrid_<SERVICE NAME>/<SERVICE NAME>_registration.xml
JBoss: $JBOSS_HOME/server/default/deploy/wsrf.war/WEB-INF/etc/cagrid_<SERVICE NAME>/<SERVICE NAME>_registration.xml

Other XML Files Related to Service Configuration

Tomcat: $CATALINA_HOME/webapps/wsrf/WEB-INF/etc/cagrid_<SERVICE NAME>
JBoss: $JBOSS_HOME/server/default/deploy/wsrf.war/WEB-INF/etc/cagrid_<SERVICE NAME>

Diagnostic Tools

If you are installing your service to a Unix/Linux OS, you can use the following Perl script to pull all of your service settings from the specified files for you.

This script is built to use the CATALINA_HOME and JBOSS_HOME environment variables. Please make sure they are defined on your system.

Download the zip file and extract a directory that is on your PATH, like $HOME/bin.
Set execute permissions on the script:
```
 chmod u+x chkUnixContainer.pl 
```
Run it using one of the following (Note: if your Perl is not in /usr/bin/perl use the second option):
```
 ./chkUnixContainer.pl (tomcat|jboss) 
```
```
 perl chkUnixContainer.pl (tomcat|jboss) 
```
You can easily capture the output and send it to the caBIG caGrid Knowledge Center.

Knowledgebase