Load balancing with web server failover (Apache)

April 9, 2009

This is the 7th article in the Getting Started with Grails tutorial series.  The entire series is as follows:

1. Introduction
2. Getting started with JRuby
3. Getting started with GlassFish
4. Restarting GlassFish
5. Getting started with load balancing (Apache)
6. Load balancing with web server redundancy (Apache)
7. Load balancing with web server failover (Apache)
8. Getting Started with Message Oriented Web Services (Java EE)
9. Setting up a local network using FireWire
10. Sending Email from Grails
11. Deploying Grails on GlassFish v3

In the previous sequence of articles, the ability to realize a High Availability (HA) cluster of GlassFish application servers was presented and then rudimentary load-balanced using multiple instances of the Apache web server. To note: the previously presented architecture does not support web-server failover, security zones or application maintainability. Within this article, the step-by-step technique of how we use an Apache web-server as an IP Sprayer thereby supporting multiple Apache web-servers with failover. Also within this developmental phase we’ll start to consider security issues via a multi-tiered security scheme and the ability to gracefully maintain your web-app using multiple clusters.

Obviously this architecture has a huge single-point failure opportunity called the IP Sprayer (and of course, your Mac laptop) – but we’ll get to those in another article.

As mentioned within the previous Adventures (but worth mentioning again) this Tutorial is based upon my current system configuration which is a MacBook Pro laptop running Mac OS X Version 10.5.6 (Leopard) and operating the Terminal as a bash shell – the default.

Disclaimer…

Within the above pictured system architecture, seven virtual machines are described to be running within your MacBook Pro laptop. Now, even though my laptop’s SDRAM has been maxed-out to 4 GB, I’m only able to realistically operate no more than five virtual machines at any one time. If no other applications are running and I’ve just done a clean reboot to ensure that the RAM is not fractured, I’m able to run six machines. At no time have I been able to run all seven virtual machines at the same time. This is OK though, the purpose of this article is to demonstrate a system that is robust enough to not only tolerate the loss of machines, but to do so gracefully (i.e.: automatically and transparently to the client). So, we will proceed to configure the system to contain seven virtual machines but it is up to you to decide which one(s) you wish to sacrifice to stay within the capacity of your laptop. This dynamic sacrificing and bringing on-line of virtual machines is a good opportunity to experience the day-to-day ebb-and-flow maintenance of an evolving web-site.

So here we go…

The above cluster-of-cluster architecture will be realized through the following steps:

1. Configure your Mac to support two new private networks
2. Generate and configure the three paranoid machines
3. Generate and configure the four private machines
4. Install OpenSolaris on each private machine
5. Download and install the JDK onto Private Machine 0
6. Download GlassFish into Private Machine 0
7. Install and configure GlassFish within Private Machine 0
8. Configure all machines with a static address
9. Rename the hostname of each paranoid machine
10. Configure the routing tables
11. Turn on the DAS
12. Generate node agents within paranoid machines 1,2 & 3
13. Setting up your GlassFish cluster
14. Deploy your web-app
15. Test your (non-load-balanced) Web-App
16. Install Apache within each private machine
17. Configure Apache for Load Balancing
18. Configure Apache as an IP Sprayer
19. Starting Apache on the Mac
20. Testing your load balanced cluster

Step 1 – Configure your Mac to support two new private networks

Within the previous articles, the separation of internal network traffic and the external network traffic was loosely addressed. Within this part we’ll set up two new private networks which will be internal to your mac. These private networks are segregated from the public (Internet) network and each other courtesy of various firewall techniques.

Within the System Preferences window:

Chose the Network icon to bring up…

Being that we wish to create a new network service, click on the (“Create a new service”) icon which is located on the bottom left of the Network window. This will bring up a data-entry sub-window as shown below. From the “Interface” selection box, select the “Ethernet Adaptor (en2)” item as shown:

Producing a new sub-window:

Now you can modify the “Service Name” entry field to be something more meaningful, such as:

Upon hitting the “Create” button, voila the new service is created (but not yet configured):

Establishing the static address…

… and upon clicking the “Apply” button, the new service will become enabled and float higher in the prioritized usage list:

And as we did to create the DMZ network on en2, let’s create the MZ network using en3:

And naming it:

Producing:

Which can be configured:

and made active (when the “Apply” button is clicked) to look like:

The above configuration of the DMZ and MZ networks will be lost whenever the Mac is rebooted. Therefore the above configuration will need to be redone on each power up.

When deployed, the DMZ and MZ networks will be isolated from the internet. Although this isolation is what we desire for these two protected nets, for now we have the luxury of temporarily allowing these networks to have access to the internet. This will assist in downloading/installing the software into the virtual machines. So to begin the configuration process to make these internal networks visible to the internet, we begin with the System Preferences window:

and select the “Sharing” icon to produce:

You’ll notice that the Internet Sharing is Off by default. Go ahead and select (check) both the Ethernet Adaptor en2 and en3 items from within the “To computers using:” selection box:

Now when you select the check-box to the left of the “Internet Sharing” Service item, you’ll get the following confirmation window:

Upon clicking the “Start” button, you’ll see that the Internet Sharing is now On. Don’t forget to turn this off prior to deploying your system. Forgetting to turn off this access path will expose your private and paranoid machines to intruders.

Cool. You have successfully created two new private networks wholly contained within your MacBook Pro. :-)

Step 2 – Generate and configure the three paranoid machines

Being that we established three virtual machines within the previous article:

Our easiest path of moving forwards is to just to rename and reconfigure our existing cluster. To rename Virtual Machine 1, highlight the machine name as shown above and then press the  icon to bring up the following Settings (General tab) window. As shown below, rename “Virtual Machine 1″ to “Paranoid Machine 1″.

and then click the “OK” button.

Redo the above steps to rename Virtual Machines 2 and 3 to Paranoid Machines 2 and 3 as shown below:

Now that the three machines have been renamed, we want to reconfigure them so that they attach to the MZ Network. Bring up the Settings window again for Paranoid Machine 1 and select “en3: MZ Ethernet Adaptor (en3)”:

And then click the “OK” button. You’ll see that Paranoid Machine 1 is now attached to the MZ Network:

Go ahead and redo the above attaching-to-the-MZ-Network steps for Paranoid Machines 2 & 3.

Step 3 – Generate and configure the four private machines

Since it may have been awhile since you created a virtual machine through the VirtualBox GUI, and being that the configuration of the private machines are slightly different than the paranoid machines, we’ll error on the verbose side an go through all of the windows again.

To start the process of creating a new private virtual machine, click on the  icon from the top of the VirtualBox home window. This will bring up the introduction window (the first window) of the Create New Virtual Machine wizard window sequence.

Clicking on the “Next >” button will bring up the window that will allow you to provide a name for your virtual machine. Go ahead and enter the name of your virtual machine within the “Name” text entry box and select “Solaris” and “OpenSolaris” for the “Operating System” and “Version” selection boxes as shown below.

Clicking the “Next >” button from the window above will step you to the next wizard window:

Accept the default by clicking the “Next >” button to produce the following screen. Now the virtual disk that is being presented to you is the last one that was created. We want a new one, therefore click the “New…” button from the screen below.

To send you down a little tangent wizard sequence to create a new virtual disk:

Clicking the “Next >” button will bring up:

Dynamically expanding storage is fine, therefore click the “Next >” button:

Now this is the tricky part. We are generating virtual hard disk that may grow to 16 GB. This virtual hard disk is part of a cluster of clusters of virtual machines (7 in total) each capable of growing their virtual hard disks to 16 GB. That’ll take upwards of 112 GB of disk space. This is probably not something that you wish to allocate within your laptop. When we transfer our final architecture to a computer located on a colocation site (many chapters from now), you’ll probably use the primary storage disk. For now though, I strongly urge you use an external disk. In my case I have a pair of external drives set up as a RAID drive supporting Time Machine and these virtual machines. OK, so much for the sales pitch. To identify the external drive location for your soon-to-be virtual disk, click the  icon to bring up the Mac Finder window:

Select the subdirectory that you’ve established to hold your VirtualBox virtual drives and supply a name for Private Machine 1 as shown above.

Upon clicking the “Save” button you’ll see the following window:

To which you click the “Next >” button to bring up:

As long as the information is as shown above, click the “Finish” button. This will create the virtual disk on your external drive and display it as shown below:

Clicking on the “Next >” button will bring up the last of the wizard windows:

To which clicking the “Finish” button will produce:

By repeating the above Step-3 entry sequence once for each of the remaining three private machines 2, 3 and 0 you’ll then see your system of virtual machines as shown below:

Now that each of your private machines have been generated, we need to configure each of the new private machines. After selecting Private Machine 0 from the list of virtual machines, click on the  icon to bring up the Settings’ General window. Select the “Advanced” tab as shown below:

Now click on the  icon, and then the  tab to bring up:

Selecting the “Mount CD/DVD Drive” and the “ISO Image File” will make the screen look like:

Luckily, the ISO Image File for OpenSolaris is still selected from an earlier chapter. If not use the  icon to locate the OpenSolaris image file that you downloaded (it’s probably still in your Download directory).

Once the “Enable IO APIC” checkbox is selected, click on the  icon to bring up:

But we wish to use a static address (as oppose to using NAT/DHCP) so we need to select “Host Interface” for the protocol and then attach Adapter 1 to the DMZ network as shown below:

Now although we will be eventually configuring the private machines with two adapters (Adapter 1 & Adapter 2) we cannot configure Adapter 2 yet. It turns out that by enabling Adapter 2 now, we will not be able to Install OpenSolaris onto the virtual machine (go figure). So for now, Adapter 2 will remain dormant.

Upon clicking the “OK” button, Private Machine 0 will be configured as we desire.

Go ahead and configure the remaining three private machines in the same fashion to produce:

Cool, all of your virtual machines are now in place and configured as shown within the below intermediate architecture:

Step 4 – Install OpenSolaris on each private machine

Now that all of your private machines have been created, select Private Machine 0 as shown below:

and either double-click on the Private Machine 0 Icon or the  icon to bring up the following boot disk selection screen:

The top one is fine. You can either let the 30 second timer count down to zero or simply press the Enter key to expedite the choosing of the top (default) boot disk. Once booting up, you will get the following Keyboard select query window:

In my case, just pressing the Enter key will select the default selection of 41 (US – English). This will bring up the following Language selection window:

In my case, pressing the Enter key will select the default selection 3 (English). This will proceed you to the following fake login window:

Just sit tight and DON’T DO ANYTHING. After about 15 seconds the fake login window will disappear and the following OpenSolaris desktop screen will be presented to you:

Although this has the look-and-feel of a functioning operating system, it is not. The OpenSolaris operating system is now operating out of the CD (not cool for anything other than continuing the install of the operating system onto the hard disk. To start the install-to-hard-disk wizard window cascade, double-clidk on the  icon which is blatantly presented to you on the desktop. This will bring up the following OpenSolaris introductory screen:

Clicking the “Next” button will produce:

Which you will probably want to use the entire (virtual) disk for this virtual machine. Clicking the Next button will produce:

Clicking on a blue dot representing a city which is local to you and within your timezone will autofill all of the selection boxes within this page. I personally like to use AM/PM as opposed to the default 24 hour time style. Clicking the “Next” button will produce:

Notice that each of the selection boxes within the above window have been filled in for you courtesy of the (blue dot) city that you identified in the previous window. Clicking the “Next” button will produce:

For consistency sake, you should use the same username and password that was set up for the Paranoid machines. Clicking the “Next” button will provide you with the following informational screen:

If everything is as you intend, click the “Install” button to start the install-OS-to-virtual-hard-disk process:

This install-OS-to-virtual-hard-disk process will take about 45 minutes.

While the install is progressing for Private Machine 0, you will probably want to go through the same start-the-virtual-machine and install-OS-to-virtual-hard-disk sequence for the remaining private machines (1, 2 & 3).

When the OpenSolaris installation process is complete you’ll see the following screen:

Cool, you will want to reboot your system, but this time when you come to the following boot-source screen, select the hard drive. OK, go ahead and click the above “Reboot” button to reboot Private Machine 0.

Within the above screen, make sure to scroll down using the down-arrow key so that you will be booting from the (virtual) hard disk as opposed to the CD. Once the “Boot from Hard Disk” is highlighted, press the “Enter” key to bring up the following window:

Pressing the “Enter” key to select the top default selection will bring you to the following screen:

After a couple of minutes you’ll see the login screen. This first boot-up is abnormally long. Successive boot-ups will not take this long:

Enter your username as established within the OpenSolaris install process and press the “Enter” key or click on the “OK” button to bring up the password screen:

Upon entering your password as established within the OpenSolaris install process, you’ll see the following booted-from-hard-disk desktop screen:

Ta-Daaa! Each of your four private virtual machines are now successfully configured to run the OpenSolaris operating system. Maybe it’s just me, but I find the fact that running five non-native operating systems simultaneously within the Mac OS somewhat humorous. Congratulations, this may be a new record for you as well.

But wait… there’s more.   ;-)

Before we can claim that our operating system install process is complete, we need to eject the CD.

To close Virtual Machine 0, click on the  (close window) icon belonging to Virtual Machine 0. This will bring up the following query window which you will want to enable as shown:

Upon clicking the “OK” button, Virtual Machine 0 will begin shutting down gracefully. This may take a couple of minutes so have patience. After a couple of minutes, the desktop will shrink in size to the following white-screen:

This white-screen will be displayed for another minute or three and then the following statement will be momentarily displayed:

and then the white-screen will disappear. Once all of the private machines have been shut down in this manner, Highlight Private Machine 0 from within the VirtualBox home screen:

and click on the  hyperlink to bring up:

which is showing you that the OpenSolaris installation CD is still mounted. Unclick the “Mount CD/DVD Drive” checkbox:

and then click the “OK” button:

And, of course, the CD from private machines 1, 2, & 3 should be ejected in a similar fashion.

Step 5 – Download and install JDK into Private Machine 0

Java is actually an amalgamation of a bunch of smaller packages, each with their own particular purpose. The OpenSolaris OS that we just installed onto your virtual machines in the previous section is bundled with only the Java Runtime Environment (JRE) package. GlassFish, that we will be installing later in this tutorial, will need another java package installed – namely:

SUNWj6dev

Built into OpenSolaris is a nifty application called the Package Manager which can manage the downloading/installing/updating of the multitudes of various components making up your particular configuration of the OpenSolaris (i.e.: UNIX) OS. Being that this is probably the first thing that you will be doing as soon as your desktop comes to life, an icon for this application has been conspicuously placed on the desktop. Double-click on the Package Manager’s desktop icon:

Which will bring up the Package Manager window:

Now you may be compelled to select the GlassFish package to install/update but don’t. Downloading GlassFish in this manner has two downsides:

1. This is a very slow technique in that you will be downloading this same (very large) package from a remote server once for each virtual machine within our cluster-of-clusters architecture.
2. The release of GlassFish that is downloaded via this Package Manager tool is kind of antiquated. We’ll download the more current version of GlassFish in the next step of this article.

Being that we are in need of the JDK packages SUNWj6cfg, SUNWj6dev, SUNWj6dmo, and SUNWj6man, highlight the “Java” selection from the left column and the stated packages from the right window as shown. Now you may click on the “Install/Update” icon which is located at the top of the Package Manager window:

You’ll then see the following install status message box:

Followed by the following query window:

When you click on the “Next” button, you’ll then begin seeing the following progress window:

culminating with:

Cool. All JDK packages necessary to run GlassFish within OpenSolaris are now installed within Private Machine 0.

We will NOT be running GlassFish within the remaining private machines (i.e.: 1, 2, & 3). Therefore we do not need to repeat this Step for the other private machines.

The following is the architecture so far:

Step 6 – Download GlassFish into Private Machine 0

If you remember, within Step 1 of this article we set up the DMZ Network to operate on en2 and specified that the Mac’s private network address is:

10.0.2.1

Also remember that we’ve already downloaded into our Mac laptop the most current GlassFish installer via Blog article 3 of this series.

We will now transfer the GlassFish installer file from the Mac into Private Machine 0. So here we go…

Execute the following sequence of commands from within Private Machine 0′s terminal window:

cd ~/Download

ftp 10.0.2.1

When the prompt asks you for your username and password, use your MacBook username and password.

Once inside of the FTP environment (you’ll be operating with an ftp> prompt on the command line). Execute the following commands from the VM terminal window:

ftp> cd ~/Downloads

ftp> get glassfish-installer-v2.1-b60e-sunos_x86.jar

Once you hit the carriage return you’ll see something like:

Opening BINARY mode data connection for ‘glassfish-installer-v2.1-b60e-sunos_x86.jar’ (74000843 bytes)

This intra-computer FTP file transfer is completed in a matter of just a couple of seconds.

Once you’ve ftp’ed the GlassFish installer file into your virtual machine, you can get out of the FTP environment by executing the following FTP command line statement:

ftp> quit

Once out of the FTP environment, you can verify that the GF installation file has been successfully copied by entering the following command into each of the virtual machine terminal windows:

ls -l

and you should see something listed out which looks like:

-rw-r–r– 1 fletch staff 74000843 2009-02-17 22:22 glassfish-installer-v2.1-b60e-sunos_x86.jar

Step 7 – Install and configure GlassFish within Private Machine 0

Side note: If perchance, the GlassFish environment becomes abnormal (i.e.: deviates from the functionality described within this tutorial), you may want to consider abandoning the installed set of GlassFish application and consider renaming the ~/glassfish directory something like ~/glassfish_attempt3, and then redo the following part of the tutorial from the following “GlassFish Restart Line”.

___________________ GlassFish Restart Line ___________________

To install GlassFish into Private Machine 0, enter the following commands into the terminal window of Virtual Machine 1:

cd

java -Xmx256m -jar ~/Download/glassfish-installer-v2.1-b60e-sunos_x86.jar

This will bring up a licensing window which will allow you to “Decline” or ”Accept” (once you have scrolled to the bottom of the EULA). The installation will take less than a minute following your acceptance of the licensing agreement.

Using the terminal window of Private Machine 0, cd into the newly unpacked ant/bin directory as follows:

cd ~/glassfish/lib/ant/bin

and then the permissions need to be relaxed so that the ant application can be executed. This changing of the access permissions is as follows:

chmod 755 ant

Then using the terminal window of Virtual Machine 1, cd back into the glassfish directory as follows:

cd ~/glassfish

Now the glassfish environment needs to be configured to operate within one of its possible configuration profiles. Being that I’m intending to operate my web-apps within a high-availability load-balanced environment, the ant script specific to setting up a clustering environment will need to be executed as follows (if however you are intending to run your JRuby-on-Rails on a single server (not recommended) then you would want to run the setup.xml script instead):

./lib/ant/bin/ant -f setup-cluster.xml

If you see the following error message:

Unable to locate tools.jar. Expected to find it in /usr/jdk/instances/jdk1.6.0/lib/tools.jar

That means that you did not successfully install the JDK package “SUNWj6dev” into this virtual machine as described within Part 7 above.

If/when the ant build is successful, the last two status lines should read something like:

BUILD SUCCESSFUL

Total time: 22 seconds

If your ant process fails due to having “port 4848 already in use”, then you have already installed Glassfish and are probably attempting to redo this step due to having your GlassFish environment hosed. Running this ant script a second time is not cool. If this is happening, you may want to consider deprecating your ~/glassfish directory (to something like ~/glassfish_attempt3, rebooting your computer (thereby releasing allocated ports (such as 4848)) and going back to this tutorial’s Restart Line.

You are just about ready to generate nodeagent2. Before we do though, I’d like to set up some system environment variables so that turning on and off and managing the server cluster is as easy as possible. To do this we need to edit the .bash_profile as follows:

From within Private Machine 0′s terminal window execute:

vi ~/.bashrc

and append the following two lines to the end of your .bashrc file:

export GLASSFISH_HOME=~/glassfish

PATH=$PATH:$GLASSFISH_HOME/bin

For the above system environments to take effect, (assuming that you’ve saved the changes to the .bashrc file you just made) you need to close and reopen Virtual Machine 1′s terminal window.

To verify that the extended paths have taken effect run the command:

env

and note that out of the listing produced there should be two lines that look somewhat like:

PATH=/usr/gnu/bin:/usr/bin:/usr/X11/bin:/usr/sbin:/sbin:/export/home/fletch/glassfish/bin

GLASSFISH_HOME=/export/home/fletch/glassfish

GlassFish is now installed and can be invoked from any directory from within Virtual Machine 1′s terminal window.

Step 8 – Configure all machines with a static address

Back in Step 1 we configured the Mac to operate two new internal networks — the DMZ and the MZ. In the steps since then, we’ve been using this network as a DHCP client just, well, for the simple reason that it’s the default setting that’s worked so well at getting us access to the internet. Before we turn on the DAS (in Step 9), we’ll need to convert each of the virtual machines from operating as a DHCP to using a static address. So here we go…

Within Private Machine 0, the status icons in the upper right of the screen have so far looked like this:

Upon selecting the following menu-item from within Private Machine 0:

System > Administration > Network

you’ll see the following window:

If the above Network Administration query window is not visible, it could be that it is hiding behind another window (such as your terminal window) as follows:

To bring the Network Administration window to the front, click on the  tab at the bottom of the window to make the Network Administration window appear:

When the above  button is clicked, the status icons will convert to looking like:

Although disconcerting, this is OK — and the following window will appear:

Select the above Ethernet connection item and then click on the “Properties” button to bring up:

Fill in the above Interface properties window to look as above. Then click the “OK” button to bring up:

Notice that the Ethernet connection is “not active”. To make it “active”, click on the  button.

and then the “OK” button.

Private Machine 0 is now configured to use a static IP address.

Please redo this Step 8 for Private Machines 1, 2 & 3 (using static IP addresses 10.0.2.3, 10.0.2.4 and 10.0.2.5 respectively).

All private machines are now configured to use a static IP address on the DMZ network. :-)

All of the private machines are now able to ping the other residents on the DMZ network, but are not able to see the Internet yet. :-(

The next Step will correct this inability to see/access the Internet.

Step 9 – Rename the hostname of each paranoid machine

Although the original hostnames hosting the GlassFish cluster are probably OK, lets never-the-less change them so that they are more descriptive.

To change the hostname from “opensolaris1″ to “opensolarisParanoid1″, we’ll need to modify only one file.

To modify the first file execute the following command from within the terminal window of Paranoid Machine 1:

pfexec vi /etc/nodename

and change its contents from “opensolaris1″ to “opensolarisParanoid1″. Then save and quit.

For this new hostname to take effect, reboot Paranoid Machine 1.

Redo this step again for Paranoid Machine 2 and Paranoid Machine 3 (using the new network names of “opensolarisParanoid2″ and “opensolarisParanoid3″ respectively).

Step 10 – Configure the routing tables

To verify the hostnames for the players within your private network, type the following statement into the MacBook’s terminal window and each of the virtual machine’s terminal window:

hostname

The presented ASCII host name for each machine will be used to configure the MacBook’s routing table. Now you are armed with enough information to edit the file:

/etc/hosts

Being that this is a file owned by root, we need to edit the file with root privileges. This can be done (assuming you can use the vi text editor) with the following command:

Within the Mac terminal window use: sudo vi /etc/hosts

Within the OpenSolaris terminal window use: pfexec vi /etc/hosts

Append the following 4 lines to the end of the hosts file WITHIN THE MAC AND EACH VIRTUAL MACHINE:

10.0.2.1 fletcher-mcbeths-macbook-pro.local

10.0.2.2 opensolarisPrivate0

10.0.2.3 opensolarisPrivate1

10.0.2.4 opensolarisPrivate2

10.0.2.5 opensolarisPrivate3

10.0.3.2 opensolarisParanoid1

10.0.3.3 opensolarisParanoid2

10.0.3.4 opensolarisParanoid3

Then save and quit.

At this point, the arp routing tables should be embellished with the ASCII hostnames as opposed to the numeric IP addresses. If you type the command into the MacBook terminal window:

arp -a

you should see a listing which includes something like the following lines:

opensolarisPrivate0 (10.0.2.2) at 8:0:27:de:11:14 on en2 [ethernet]

opensolarisPrivate1 (10.0.2.3) at (incomplete) on en2 [ethernet]

opensolarisPrivate2 (10.0.2.4) at 8:0:27:63:ba:90 on en2 [ethernet]

opensolarisPrivate3 (10.0.2.5) at (incomplete) on en2 [ethernet]

? (10.0.2.255) at (incomplete) on en2 [ethernet]

opensolarisParanoid1 (10.0.3.2) at 8:0:27:a1:72:12 on en3 [ethernet]

opensolarisParanoid2 (10.0.3.3) at 8:0:27:9:b:68 on en3 [ethernet]

opensolarisParanoid3 (10.0.3.4) at 8:0:27:6b:ad:63 on en3 [ethernet]

? (10.0.3.255) at (incomplete) on en3 [ethernet]

Those that are listed as “incomplete” are those machines that were queried but did not respond. In my case opensolarisPrivate1 and opensolarisPrivate3 are turned off so that I don’t exceed my system’s memory capacity.

Step 11 – Turn on the DAS

Before we start domain1 (i.e.: the DAS), we need to configure our MacBook’s routing table. You see, the DAS makes network references to itself and the other cluster members via their (ASCII) hostname, not their (numeric) IP address.

Let’s go ahead and turn on the DAS. This is done through the following Mac terminal command-line statement:

asadmin start-domain

The user name and password is still:

admin user name> admin

admin password> adminadmin

As the DAS starts up (about 20 to 30 seconds), there will be about 16 lines of status messages listed that concludes with the line:

Domain supports application server clusters and other standalone instances.

Your DAS for domain1 (the default) is now running on opensolarisPrivate0.

We can now open up a new browser window and enter the following URL:

http://opensolarisPrivate0:4848

Which will bring up the GlassFish login page:

and, with the entry of:

admin user name> admin

admin password> adminadmin

you’ll be presented with the DAS home screen for domain1:

We can look at the current status of our domain1 system by selecting the Node Agent item from the left column:

Congratulations, here is our current system architecture:

Step 12 – Generate node agents within paranoid machines 1, 2 & 3

Being that we are configuring GlassFish to operate as a distributed cluster, we need to create a node-agent within each of the paranoid machines. To generate the node-agent for Paranoid Machine 1, enter the following command into Virtual Machine 1′s terminal window (Caution: on some browsers the dash-dash-”host” and dash-dash-”port” syntax are incorrectly displayed as single dashes):

asadmin create-node-agent –host opensolarisPrivate0 –port 4848 nodeagent1

Use:

admin user name> admin

admin password> adminadmin

Within a couple of seconds, the following statement will be displayed:

Command create-node-agent executed successfully.

Notice that the command line statement used above to create nodeagent1 is referencing the location of the DAS (opensolarisPrivate0).

Redo the above Part-11 section two more times once for Paranoid Machine 2 and then again for Paranoid Machine 3 (using node-agent names of nodeagent3 and nodeagent4 respectively).

Upon refreshing your DAS, you should see that all three node agents are now recognized as shown below:

For security reasons, the act of turning on a nodeagent via the DAS GUI is not available. We can now turn on each node agent by entering the following command-line statements within its respective terminal window:

Within Paranoid Machine 1′s terminal window:

asadmin start-node-agent nodeagent1

Within Paranoid Machine 2′s terminal window:

asadmin start-node-agent nodeagent2

Within Paranoid Machine 3′s terminal window:

asadmin start-node-agent nodeagent3

Upon refreshing the DAS’ browser window you should see:

Pictorially our architecture has become:

Congratulations the node-agents within each of your virtual machine’s are up and running.   :-)

Step 13 – Setting up your GlassFish cluster

Within this section we will be interacting with the DAS (opensolarisPrivate0) and all of the paranoid machines. As such Private Machine 1, 2 & 3 are best left turned off. This will keep the limited 4 GB of DRAM on your Mac from being exceeded.

The first server management task is to create a new cluster. Select the “Clusters” item from the task tree on the left side of the window. The following screen will then be displayed:

Click the “New…” button to produce:

Within the Name text-box, enter the name “cluster1″.

Within the “Server Instances to be Created (0)” pane, hit the “New” button three times (and hesitate between hits to let the system stabilize) – each time introducing a new entry row into the instance name table.

Enter a name for each new instance (i.e.: instance1, instance2 and instance3) as shown below. Make sure that you select each of the instance’s respective nodeagent. The act of selecting a nodeagent is the act of identifying to which machine the GlassFish instance will be established:

When you click on the OK button, the newly specified cluster with the three named instances will be generated. In about 30 seconds the display window will look something like:

To turn on the cluster, first click on the check-box just to the left of the cluster name and then click the “Start Cluster” button. The following confirmation window will appear:

Once the confirmation window is acknowledged (and after about 45 seconds of processing), the screen will then show the new cluster status as follows:

If one or more of your instances won’t start, you may wish to double-check the /etc/hosts file of all your machines – in particular the machine that’s refusing to start.

Structurally we now have:

We will now repeat the above step but this time for Cluster2 as follows:

From within the “Clusters” window that we last looked at, click the  (cluster) button to bring up the following cluster creation window:

Within the Name text-box, enter the name “cluster2″.

Within the “Server Instances to be Created (0)” pane, hit the  (instance) button three times (and hesitate between hits to let the system stabilize) – each time introducing a new entry row into the instance name table.

Enter a name for each new instance (i.e.: instance4, instance5 and instance6) as shown below. Make sure that you select each of the instance’s respective nodeagent (nodeagent1, nodeagent2 and nodeagent3). The act of selecting a nodeagent is the act of identifying to which machine the GlassFish instance will be established:

When you click on the OK button, the newly specified cluster2 with the three named instances will be generated. In about 30 seconds the display window will look something like:

To turn on the cluster, first click on the check-box just to the left of the cluster2 name and then click the  button. The following confirmation window will appear:

Once the confirmation window is acknowledged (and after about 45 seconds of processing), the screen will then show the new cluster status as follows:

Before we get too excited about having multiple clusters up and running, we should check that the node agents are still alive and responding. It is at this point that my system usually goes a little wiggey. Check the node agents by selecting the “Node Agents” item in the Common Tasks selection list:

As shown above, the node agents have ceased being recognized by the DAS. This is not cool. If we continue delving into let’s say nodeagent1, we’ll see:

The above Node Agent Exception report which says:

The server encountered an internal error () that prevented it from fulfilling this request.

javax.servlet.ServletException: java.lang.reflect.InvocationTargetException while attempting to process a ‘beforeCreate’ event for ‘page1’.

java.lang.reflect.InvocationTargetException

javax.management.AttributeNotFoundException: Attribute not found: “RestartRequired” (Connection refused)

Is telling us that our node agents have gotten wiggy. This may have been caused by the OpenSolaris Network Monitor crashing:

By clicking on the “Reload” button, the Network Monitor did indeed restart. Hummm, even after the Network Monitor was restarted on each of my paranoid machines, the DAS still would not recognize the Node Agents. Within Paranoid Machines 1, I stopped and restarted the nodeagent1 via the CLI commands:

asadmin stop-node-agent nodeagent1

asadmin start-node-agent nodeagent1

Following this power cycling of nodeagent1, it became recognized by the DAS:

Now reinstate the Network Monitor and the Node Agent within Paranoid Machines 2 & 3.

On observing the status of the cluster we see that it is still a little wiggy:

As was done before, click on the check-box to the left of cluster1 and cluster2 and click the “Start Cluster” button producing:

and then, when the cluster screen comes back it should look like this:

Whoo-Hoo, now to verify that the nodeagents are registering properly, bring up the Node Agents window:

Cool, we have thus overcome the Node Agents getting wierd on us when starting the HA clusters.

Our intermediate architecture is now:

Congratulations! Your clustered GlassFish environment is now operating and ready to serve up (non-load-balanced) scalable and maintainable applications using multiple high-availability clusters.

Part 14 – Deploy your Web-App

Let’s now deploy a demo application that has already been distributed to you within the GlassFish installation files.

Select cluster1 from the Common Tasks item list (the left column of the screen) and then select the “Applications” tab to bring up the following window:

Upon clicking the  (web-app) button, you’ll get:

In my opinion, the singular purpose of a cluster of application servers is to provide the user with a high availability experience even if/when one or more servers go down.

That said, go ahead and check the check-box enabling the “(High) Availability” functionality for the application we are about to deploy.

Ensure that the radio-button is set to (the default setting) which chooses to use a “Packaged file to be uploaded to the server”.

Then click on the  button.

Using the Mac’s directory-browser window browse over and choose the supplied deployable web-app:

~/glassfish/samples/quickstart/clusterjsp/clusterjsp.ear

Your cluster deployment window should now look something like this:

Now we are wishing to install this application within multiple clusters. Therefore, at the bottom of the page select  from the left selection window and then click on the  button to transfer cluster2 over to the list of targets window. Your Application Deployment window should now look something like:

Upon clicking the  button, the DAS will begin to install the selected Enterprise Application Resource (EAR) file into each of the remote GlassFish application server instances within both cluster1 and cluster2. The installation process takes about 20 seconds. Your cluster1 DAS window should then look like:

And your cluster2 DAS window should look the same way:

Although your application is now installed within each of the instances of cluster1 and cluster2 and “enabled”, this status message is a little misleading in that your web-app is not yet turned on.

To turn on the newly installed web-app, enable the checkbox to the left of your web-app (clusterjsp) item and then using the “–MoreActions–” pull-down selection box, select “Disable” to produce:

And then re-enable the checkbox to the left of your web-app (clusterjsp) item and then use the “–MoreActions–” pull-down selection box again, this time selecting “Enable” to re-enable clusterjsp back to “true”:

Now your cluserjsp web-app is (supposed to be) fully deployed and functioning.

Sometimes though, the nodeagents, the DAS, the instances and the Web-apps will get a little wiggy. I found that working through the system and sequentially turning off and then back on the:

• Virtual Machines
• Node Agents
• Clusters
• Web Applications

you will be able to get back to a stable and functioning multi-cluster system. For now I’m contributing this wiggey-ness to a combination of my having too many virtual servers swedged within my Mac laptop and possibly the immaturity of the VirtualBox, multiple clusters and/or DAS software combination.

Through all this turmoil though, you get kudos for configuring your laptop to contain the following multi-tiered multi-cluster architecture:

Step 15 – Test your (non-load-balanced) Web-App

Open another Mac web browser window and type in the URL:

http://opensolarisParanoid1:38080/clusterjsp

As you could probably guess, port 38080 is the default port number for a clustered high-availability web-app. Your web-app will produce the following page:

Being that session replication across different machines is not yet fully supported in GlassFish release 2.1 (see discussion thread: http://forums.java.net/jive/thread.jspa?messageID=334651&tstart=0). This is a known bug that has been fixed. The fix is available to those savvy enough to grab the current GlassFish open-source code-base and do their own build. For those upstanding netizens that are under a maintenance contract with Sun, there’s a patch (Patch1) available that contains this fix. If you’re a troglodyte like me, you’ll have to wait for the next release (which occurs about every six months).

As mentioned above this is overcome by downloading GlassFish v2.1 patch1. For the purposes of this tutorial, I’ll let time (i.e. the next rev of GlassFish) be my ally and therefore only focus on the top part of the JSP response window.

By typing in the URL:

http://opensolarisParanoid2:38080/clusterjsp

You’ll get:

By typing the URL:

http://opensolarisParanoid3:38080/clusterjsp

You’ll get:

The URL:

http://opensolarisParanoid1:38081/clusterjsp

Will produce:

URL:

http://opensolarisParanoid2:38081/clusterjsp

Produces:

And

http://opensolarisParanoid3:38081/clusterjsp

Produces:

Cool, our multi-cluster web-app is functioning as desired. :-)

Step 16 – Install Apache within each private machine

Being that this section will be configuring Private Machine 1, 2 & 3, go ahead and shut down Private Machine 0 and all Paranoid Machines.

Sometimes when operating OpenSolaris with a static IP address, you may not be able to access the internet. If Virtual Machine 1 is not able to access the internet, the work-around is to temporarily switch the Network Administration back to Automatic mode. From Virtual Machine 1′s menu bar, select System > Administration > Network as shown below:

This will bring up the following Network Administration window. When this window arrives onto the desktop, it will show up under any existing windows. To bring the Network Administration window to the front, click on the Network Administration tab at the bottom of the screen.

…and click on the “Automatic” button. Your virtual machine is now operating with a DHCP-issued IP address and has a reliable connection to the internet.

Clicking on the “Add More Software” icon from the desktop of Virtual Machine 1:

will bring up the Package Manager console window:

Select the “Application and Web Servers” from the left pane and “SUNWapch22″ from the right pane as shown above. Clicking on the “Install/Update” icon from the top of this window will bring up a sequence of windows as shown below:

followed by:

Upon clicking the “Next” button, you’ll see the following cascade of progress windows:

followed by:

and concluding with:

Cool, Apache is now installed within Virtual Machine 1.

If you did the optional switchover to a non-static IP address in order to gain access to the internet, you should set Network Administration back to using a static IP.

Click on the “Manual” button.

Upon clicking the “OK” button, your Virtual machine 1 will be operating again with your originally established static IP address again.

The above installation process should be done within each of the remaining two virtual machines.

Step 17 – Configure Apache for load balancing

We need to configure the Apache 2.2 web server so that it’s aware of our cluster topography as well as our load-balancing desires. This is done via a configuration file called “httpd.conf” which is located at:

/etc/apache2/2.2/httpd.conf

Prior to editing this file, I’d recommend that you make a back-up copy just in case…

cd /etc/apache2/2.2

pfexec cp httpd.conf httpd.conf.original

“pfexec” is the OpenSolaris version of “sudo”. Once safely backed up, open this file via your favorite text editor. If you happen to use vi, the following command will work:

pfexec vi httpd.conf

Append the following text to the end of the httpd.conf file:

###############################

### Start of Fletch mods… ###

###############################

ProxyRequests Off

<Proxy *>

Order deny,allow

Allow from all

</Proxy>

<Proxy balancer://mycluster>

# instance1 is (pretending to be for this example) not very powerful,

# therefore send only one out of twelve requests to instance1…

BalancerMember http://10.0.3.2:38080 loadfactor=1

# instance2 is (pretending to be for this example) more powerful,

# therefore send two out of twelve requests to instance2…

BalancerMember http://10.0.3.3:38080 loadfactor=2

# instance3 is (pretending to be for this example) much more powerful,

# therefore send three out of twelve requests to instance2…

BalancerMember http://10.0.3.4:38080 loadfactor=2

# you get the idea…

# instance4, 5 & 6…

BalancerMember http://10.0.3.2:38081 loadfactor=1

BalancerMember http://10.0.3.3:38081 loadfactor=1

BalancerMember http://10.0.3.4:38081 loadfactor=1

# if we wanted instance3 to be the hot standby that is used only if all other instances in the cluster fail, we would comment out instance3 above and uncomment the statement below…

# BalancerMember http://10.0.3.4:38080 status=+H

ProxySet lbmethod=byrequests

Order deny,allow

Allow from all

</Proxy>

<Location /clusterjsp>

Order allow,deny

Allow from all

</Location>

ProxyPass /clusterjsp balancer://mycluster/clusterjsp stickysession=JSESSIONID

ProxyPassReverse / balancer://mycluster

##############################

### …End of Fletch mods. ###

##############################

Once the httpd.conf file has been modified with the above text and saved, (the) Apache (Web Server) can be turned on.

To turn on Apache 2.2 within OpenSolaris 2008.11 which is within VirtualBox’s Private Machine 1, execute the following command line from Private Machine 1′s terminal window:

pfexec /usr/apache2/2.2/bin/apachectl -f /etc/apache2/2.2/httpd.conf

The command-prompt should return in about a second. Just to maintain the balance of the universe, the following instruction can be used to stop the Apache Web Server:

pfexec /usr/apache2/2.2/bin/apachectl -k stop

Redo this part two more times, once for each of the remaining virtual machines 2 & 3.

Without turning on your paranoid machines, you can validate that your apache web servers are operating (serving up static pages) by opening a web browser window from within your Mac and enter the URL:

http://opensolarisPrivate1

and should see the following Default apache web page:

Likewise you can test out opensolarisPrivate2 and opensolarisPrivate3 in the same fashion.

Congratulations! Your redundant web-servers are now in place, configured and operating:

Disscussion: Notice that within the httpd.conf file we used the numerical IP addresses:

10.0.2.3

10.0.2.4

10.0.2.5

as opposed to the more human-readable hostnames:

opensolarisParanoid1

opensolarisParanoid2

opensolarisParanoid3

The first reason for using a numerical IP over a hostname is execution speed. By specifying the numerical IP addresses we avoid imposing a DNS lookup each time a server access is requested.

The second reason is laziness. If we used the more human readable ASCII names then we would become responsible to configure the DNS to be aware of the alphanumeric-to-numerical IP addresses. If perchance you did use the alphanumeric ASCII hostnames within the httpd.conf file and failed to do the extra step of informing DNS of your hostname-to-IP address mappings, you’d get one of the following error message within your browser when attempting to accesses your load balanced cluster:

Reason: DNS lookup failure for: opensolarisParanoid1

Reason: DNS lookup failure for: opensolarisParanoid2

Reason: DNS lookup failure for: opensolarisParanoid3

Which statement you get depends upon which back-end app server is chosen by the load balancer.

Being that you probably wish to test the as-yet-incomplete system you may want to power down Private Machine 2 and Private Machine 3 so that you can turn on Private Machine 0 and all three of the paranoid machines.

Once all three paranoid machines are powered up, you’ll want to start up the web-app in the following sequence:

1. Start nodeagent1, 2 and 3 within Paranoid Machine 1, 2 and 3 respectively from within each machines terminal window using the command: asadmin start-node-agent nodeagent1
2. Using the DAS, start up both clusters via the clusters window
3. Using the DAS, disable then enable each cluster via the cluster1 > Applications > pull-down menu then cluster2 > Applications > pull-down menu.

Access web-server within Private Machine 1 via the URL:

http://opensolarisPrivate1/clusterjsp

Each time you refresh this page, you’ll get served up a new load-balanced web-page.

Step 18 – Configure Apache as an IP Sprayer

Apache/2.2.9 (Unix) just happens to already be pre-installed and ready to go within Mac OS X Version 10.5.6 (Leopard). Before we turn it on though, we’ll need to configure Apache so that it’s aware of our web-server cluster topography as well as our load-balancing desires. This is done via a configuration file called “httpd.conf” which (using the Mac terminal window) is located at:

/etc/apache2/httpd.conf

Prior to editing this file, I’d recommend that you make a back-up copy just in case…

cd /etc/apache2

sudo cp httpd.conf httpd.conf.original

Once safely backed up, open this file via your favorite text editor. If you happen to use vi, the following command will work:

sudo vi httpd.conf

and just above the last line of this file which reads

Include /private/etc/apache2/other/*.conf

enter (i.e.: cut and paste) the following text:

###############################

### Start of Fletch mods… ###

###############################

ProxyRequests Off

<Proxy *>

Order deny,allow

Allow from all

</Proxy>

<Proxy balancer://mycluster>

BalancerMember http://10.0.2.3:80 loadfactor=1

BalancerMember http://10.0.2.4:80 loadfactor=1

BalancerMember http://10.0.2.5:80 loadfactor=1

ProxySet lbmethod=byrequests

Order deny,allow

Allow from all

</Proxy>

<Location /clusterjsp>

Order allow,deny

Allow from all

</Location>

ProxyPass /clusterjsp balancer://mycluster/clusterjsp stickysession=JSESSIONID

ProxyPassReverse / balancer://mycluster

##############################

### …End of Fletch mods. ###

##############################

Once the httpd.conf file has been modified with the above text and saved, (the) Apache (Web Server) can be turned on.

Step 19 – Starting Apache on the Mac

To turn on (the) Apache (Web Server), open the preferences window via the Apple>System Preferences… menu item to display the following:

…and select the “Sharing” icon from the “Internet & Network” row (third row, rightmost item). The Sharing window will then come up from which you can select the “Web Sharing” item from the selection box:

Specifically, you’ll want to turn on Web Sharing by clicking within the check-box just to the left of the “Web Sharing” selection. Doing so will cause the Apache Web Server to be turned on as follows:

That’s it. You’re ready to test!

(If your Web Sharing was already turned on, you’ll want to turn it off and then back on again such that the newest edits that you just did to the httpd.conf file will take effect)

Step 20 – Testing your load balanced cluster

Just for fun, you may want to refresh your DAS browser window to ensure that your GlassFish cluster is indeed still running:

instance3 and instance6 are as expected Stopped since I’ve sacrificed Private Machine 3 and Paranoid Machine 3 in order to operate within the 4 GB of my laptop’s RAM.

Then open a new browser window and enter the URL:

http://localhost/clusterjsp

which should bring up a JSP page generated from one of the GlassFish instances from your cluster like the one below:

Discussion: The URL that we used to get this JSP image specifically addressed the Apache Web Server residing on the Mac OS (i.e.: The “IP Sprayer”). Upon receipt of the http service request, the IP Sprayer selected an instance (instance1 or instance2) from our web-server cluster and forwarded the request to that web server. In turn, the selected web-server chose one of the available back-end Glassfish Servers (in this case instcnce5) and then dispatched a service request to Paranoid Machine 2/cluster3/instance5.

Clicking on the refresh button  within your Mac browser, the cascade chain of dispatches will be loadbalanced to a different web-server and culminate with another JSP page being served up such as:

In Conclusion…

We were able to configure and set up a cluster of three GlassFish instances and then load-balance them according to a user-specified algorithm to choose amongst the viable GlassFish instances of a cluster. The system design topography that we were able to exercise is as follows:

Within our current architecture we are using a single router (the Mac router) to service all three networks (i.e.: The Public, the DMZ, and the MZ networks). As such, we still have a couple single-point failure components.  Namely the:

• Router
• DMZ Network
• MZ Network
• IP Sprayer

With that we segue to our next chapter (Fault tolerance with Router/IPSprayer redundancy)…

Advertisement