Brian
One of the strengths of the Enterprise Pipeline is it's flexibility, so there are not many hard and fast rules. Below, I will identify each part of the Enterprise Pipeline and any installation/configuration guidance.


JMS Queue: The JMS Queue is used for communication between the Enterprise Pipeline pieces (ie status messages or task instructions). If the JMS queue is down the Enterprise Pipeline will not function. The JMS queue can be run either as a stand-alone service or in memory on the webserver. The main difference between running it as a stand-alone service and running it in memory is that if running in memory on the webserver, all message sitting on the queue waiting to be delivered will be lost during a restart of the webserver. We recommend running it as a stand-alone service.


Globus GRAM server: The server which runs the Globus GRAM server must have permissions to submit searches to the Cluster. Most of the current Enterprise Pipeline installations currently install this software on the same server that is acting as the Resource Manager for their cluster. The Globus GRAM software requires that use of SSL, you will need to plan carefully on how to deal with these certificates in EC2, as the instance names and IP address are not fixed (there are ways to get around this requirement, that might make an EC2 rollout easier. A quick phone call will be the easiest way to discuss them)


General Pipeline Guidance:
1) The Enterprise Pipeline currently only support Sun Grid Engine (SGE) and PBS-based clusters
2) The Enterprise Pipeline assumes that all pipeline files are located on a network storage device and can be accessed via NFS (by the web server, cluster nodes) and CIFS(by the Conversion Server).
3) Most of the current installs place the JMS Queue and Globus GRAM server on the same physical/virtual server


Conversion Server: As you noted, this must be a Windows Server.


LabKey Server and Database Server: No additional requirements.


-Brian

That's quite helpful, thanks. I'm not too worried about cert issues around shifting hostnames - with EC2 it's no bother to just rebuild the entire cluster should the head node go away for some reason, or one could use Elastic IP if a consistent FQDN is really required. Of course I'll probably hit some kink I haven't thought of yet...

So I'm looking at
Node A: public linux node, runs dB, webserver, NFS server
Node B: private node running CIFS
Node C: private node with JMS Queue, WS-GRAM, and PBS_server
Nodes D-xxx: private PBS_client nodes running X!Tandem etc

The goal of course is dynamic sizing of the client nodes, conceivably nodes B and C could be an on-demand thing as well.

I'm going to go ahead and implement with PBS, if only because our friends at ISB use it so we might get some nice reuse in TPP-world. For the most part I'm avoiding anything that's really AWS-specific, so we end up with something like a recipe for building clusters that aren't necessarily in the Amazon cloud.

- Brian

An important thing about AWS, as you point out, is the fact that IP addresses and FQDNs shift whenever you take an instance down and replace it with another (well, there are elastic IPs etc, but set that aside for now). Thus there are things like this: (from https://www.labkey.org/wiki/home/Documentation/page.view?name=configureFtp) Configuring LabKey Server to use the FTP Server Finally, you need to tell LabKey Server that there is a FTP server that it should tell users about. Go to the Site Settings (Manage Site->Admin Console->site settings). In the Configure File System Server section, fill in the server name and the port for the FTP server. If you have set up SSL on your FTP server, check the box. What's the proper way to do this programatically? I'd rather not hand users a list of manual steps for restarting a cluster (or for that matter for building an initial LabKey AMI). Thanks, Brian