Tuesday, April 22, 2008

Creating multiple Jetty Server instances

For my current project I was required to create multiple Jetty Server instances to run different applications and I found the documentation to be lacking.
It's possible that someone documented this somewhere but I couldn't find much info using the normal channels (google searches, etc).
I'm very hopeful that this will start some conversation and people will let me know if there are better ways to do this.

We're running Jetty on a Solaris 10 instance which I access using SSH. The requirements I had were:

  1. Jetty had to run in a shell-less mode. Meaning that when I logged out the server didn't shut down.

  2. I needed multiple instances of the Server to run different applications.
    It wasn't required for them to run in seperate JVMs which frankly I have yet to try.

Running Jetty shell-less is as simple as using the jetty.sh start script. This is probably obvious to old-time Jetty
users but it took me some researching to find this out. Starting Jetty this way requires you to be in the $JETTY_HOME directory and sexecute the script as follows:

./bin/jetty.sh start

This is a fairly useful script and you can view other useful commands by typing:

./bin/jetty.sh

...which will spit out all the usable commands.

Well that was easy enough, so how about starting multiple instances? Jetty uses the jetty.xml file (sometimes refered to generically as config.xml) found in the $JETTY_HOME/etc directory as the default configuration file for the server. You can override the default and pass in multiple files from the command line just by typing their path after the start command:

./bin/jetty.sh start $JETTY_HOME/env/foo.xml $JETTY_HOME/env/bar.xml

What will Jetty do with two configuration files? Well there is line very early in each config.xml file that identifies the server instance:
If the id for each file is the same then Jetty will combine both files. If they are different then Jetty creates multiple instances.

So I created a new file called jetty-cdr-instance.xml by copying the jetty.xml file. The first thing I did was change the line above:

After that all you have to do is look through the file and change ports or whatever other setting you need or want to change. I changed the port from 8080 to 8081 and changed all the other ports (not sure if I needed to):

<call name="addConnector">
<arg>
<new class="org.mortbay.jetty.nio.SelectChannelConnector">
<set name="port"><systemproperty name="jetty.port" default="8081"></systemproperty>
<set name="maxIdleTime">30000</set>
<set name="Acceptors">2</set>
<set name="statsOn">false</set>
<set name="confidentialPort">9453</set>
<set name="lowResourcesConnections">5000</set>
<set name="lowResourcesMaxIdleTime">5000</set>
</set>
</new>
</arg></call>

There are a number of other settings that can be configured but one that is important is where to find deployment archives (WAR files). By default Jetty looks into the $JETTY_HOME/webapps directory, which is fine for the default instance. But for my second instance I want to deploy different apps so I created a webapps2 directory and changed the configuration setting:

<call name="addLifeCycle">
<arg>
<new class="org.mortbay.jetty.deployer.WebAppDeployer">
<set name="contexts"><ref id="Contexts"></ref>
<set name="webAppDir"><systemproperty name="jetty.home" default=".">/webapps2</systemproperty>
<set name="parentLoaderPriority">false</set>
<set name="extract">true</set>
<set name="allowDuplicates">false</set>
<set name="defaultsDescriptor"><systemproperty name="jetty.home" default=".">/etc/webdefault.xml</systemproperty>
</set>
</set>
</set></new></arg></call>


And that's about it! So now if I feed both files at the command line I get two server instances, one running on the default port 8080 and the other 8081 and each running the applications in their respective webAppDir directory. But what if I don't want to pass the file names in the command line?

I'm sure there is a better way to do this, probably involving the jetty.conf file, but I was unable to find sufficient documentation. In the jetty.sh script you will find the lines:

#####################################################
# Run the standard server if there's nothing else to run
#####################################################
if [ -z "$CONFIGS" ]
then
CONFIGS="${JETTY_HOME}/etc/jetty.xml"
fi

I changed:

CONFIGS="${JETTY_HOME}/etc/jetty.xml"

to:

CONFIGS="${JETTY_HOME}/etc/jetty.xml ${JETTY_HOME}/etc/jetty-cdr-instance.xml"

and now by simply typing:

./bin/jetty.sh

I get both instances running.

And that's it! I'm still pretty new to administering Jetty so if you know better ways to handle these things please let me know.

5 comments:

Axe1 said...

Thank you, for that post. I'm faced with the similar problem, and being completely new to jetty, your post helped me a lot.

iamsteveholmes said...

I'm glad I could help!

EAI Engineer said...

Thanks for this post. Useful! I have made a reference to it in my blog.

Squint said...

There is a better way to do this rather than hacking the jetty.sh init script..

simply add your xml file into etc/jetty.conf

Jetty will load all xml files within that file. You can even specify a directory and Jetty will load all XML files within that directory into CONFIGS for you automatically..

You can also create a /etc/jetty.conf with the same thing.

Another way is to add these xml files into the start.ini..

Ideally you dont want to hack the init scripts unless really required.

Osher said...

I had another problem, where a delivery I got relayed on the jetty execution folder to get configuration files from.

Although it's a design problem from the first place - I had to find a way to run two versions of the same application with different configurations, and the jetty-root gave me a trouble with that.

Eventually I had to run two instances from two different installations of Jetty.

I crossed the following problems with using jetty.sh:
1 - the pid files were overrining each other. So I hacked the script to have every installation have it's own temp folder, instead of in /tmp - to have it in
/jetty/isnt1/run
/jetty/inst2/run
and so on.

I had to fix a bug in the script where it generates an argument for the path to the pid, but does not use it everywhere - there were places where this path was constructed again, so I changed it to be taken from the same argument that is built in the same place, and depended it on the temp folder that is now installation dependent.

2 - somewhere in the file, it initiates JAVA_ARGS with some socket address which was hardcoded to 4000. So I had to let every installation have it's own socket address.
I'm sure there are smarter ways, but I hacked the script for that.

All the rest - is just like described here:
/etc/jetty.xml and /contexts/app1.xml, app2.xml, app3

Monkey Search