That's quite helpful, thanks. I'm not too worried about cert issues around shifting hostnames - with EC2 it's no bother to just rebuild the entire cluster should the head node go away for some reason, or one could use Elastic IP if a consistent FQDN is really required. Of course I'll probably hit some kink I haven't thought of yet...

So I'm looking at
Node A: public linux node, runs dB, webserver, NFS server
Node B: private node running CIFS
Node C: private node with JMS Queue, WS-GRAM, and PBS_server
Nodes D-xxx: private PBS_client nodes running X!Tandem etc

The goal of course is dynamic sizing of the client nodes, conceivably nodes B and C could be an on-demand thing as well.

I'm going to go ahead and implement with PBS, if only because our friends at ISB use it so we might get some nice reuse in TPP-world. For the most part I'm avoiding anything that's really AWS-specific, so we end up with something like a recipe for building clusters that aren't necessarily in the Amazon cloud.

- Brian