Installing^[a] eXist on the Amazon EC2 Micro Instance^[b]

Note: I did this install on eXist 1.4.  There may be some changes needed for eXist 2.2, but the basic steps should be the same.  If you have updates let me know and I can grant you write access to the document.  I also want to thank Chris Wallace, Eric Palmer and Joe Wicentowski for their feedback.  Their notes and comments are on the right in the margin.

- Dan McCreary

Summary

Amazon’s EC2 Service is an attractive way to run eXist in the cloud.  Amazon currently has an offer for new customers allowing them to run a small (micro) instance of a Linux server for one year.  Although the free system is limited in its memory (613MB RAM), it lets you explore the EC2 service and even put a useful eXist-based service in the cloud.  This document describes how to get eXist 1.4 up and running with eXist on the free Amazon EC2 account.

Installing eXist on the Amazon EC2 Micro Instance

Summary

Background

Prerequisites

Key Concepts

Public/Private Keys

Images and Instances

Limitations of the Amazon Micro Instance

Public DNS

Elastic Static IP Address

Steps to get eXist running

Step 1: Create an EC2 Account at Amazon

Step 2 Create a security group for eXist

Step 3 Create a private key and download it to your local system

Step 4 Create a new instance of Linux using CentOS Image

Step 5 Login with Secure Shell (SSH) using your key

Step 6 Downloading the eXist binary

Step 7 Install eXist from the UNIX Command Line using the Java Command

In the rest of this document will refer to /usr/local/exist as $EXIST_HOME.

Step 9 Configuration of Startup Parameters

Option #1: Change the wrapper configuration jetty configuration file read by the wrapper

Change the startup shell script

Startup Method #1 (Preferred)

Startup Method #2 (Optional)

Change your Wrapper Setting

Step 10 Test your Amazon EC2 Installation of eXist

Step 11 Create passwords for the admin and guest account

Step 12 Assign a Static IP address to your instance

Step 13 Automatically Starting eXist when the server boots

Viewing the startup log file

sample listing of rc.local file

Next Steps

Creating a link to the eXist shell command

Links to Old Documents

Restricting Caching Space

Automatically Starting eXist with the chkconfig/service

Setting the Correct Wrapper Executable

Changing the Guest Password

Getting Security Updates

Alias for Wrapper Commands

Background

In October of 2010, Amazon Web Services division announced a “free” version of its hosted Linux servers through its Elastic Cloud Computing (EC2) system for their smallest UNIX instance (613MB RAM).  Amazon EC2 gives you a “virtual machine” to run.  You don’t really have a full computer, just a “virtual system” that looks like your own Linux box.  Here are the details about the free usage tier:

Amazon AWS Free Usage Tier (Per Month):

• 750 hours of Amazon EC2 Linux Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) – enough hours to run continuously each month*
• 750 hours of an Elastic Load Balancer plus 15 GB data processing
• 10 GB of Amazon Elastic Block Storage, plus 1 million I/Os, 1 GB of snapshot storage, 10,000 snapshot Get Requests and 1,000 snapshot Put Requests
• 5 GB of Amazon S3 storage, 20,000 Get Requests, and 2,000 Put Requests
• 30 GB per of internet data transfer (15 GB of data transfer “in” and 15 GB of data transfer “out” across all services except Amazon CloudFront)
• 25 Amazon SimpleDB Machine Hours and 1 GB of Storage
• 100,000 Requests of Amazon Simple Queue Service
• 100,000 Requests, 100,000 HTTP notifications and 1,000 email notifications for Amazon Simple Notification Service

Prerequisites

To use this demo you will need a local system that has an “Secure Shell” (SSH) client.  This program comes with all Mac systems and can be downloaded onto Windows systems using the cygwin program.  SSH is needed to be able to login to the remote CentOS version of Linux to download the eXist binaries.  All text between you and the remote system with SSH is fully encrypted using a public/private key pair.

It is also very helpful if you have some experience using a UNIX shell and a UNIX editor such as the “vi” or “emacs” editor.  This is not required if you are just following these instructions exactly, but some prior experience does help.

Key Concepts

There are several main concepts that you will need to understand before you begin.

Public/Private Keys

Just like a key to your car or house, Amazon creates a set of “keys” that you will use to get into one or more of your virtual machines.  Technically, there are two part to each “key”: a private and a public part.  The public part will be automatically stored on the Amazon Web Service (AWS) systems but the private part will be generated for you and you must download it to any local computer that you want to use to manage your servers.  This key should be protected and made “read only” to yourself and no other users of your local computer system.  The private key must then be protected just like you would protect your passwords.  If you need to allow a co-worker to administer your instance you will need to then send them your key.  If you have not used keys before there are good tutorial listed on Wikipedia.

Images and Instances

An “Amazon Machine Image” (AMI) is a stored image of an entire operating system that will be copied for you to use.  You do not have to install an operating system; you just have to “create a new instance” of an operating system.  Once a copy of that operating system is ready you just “start the instance” of that virtual machine.  After an image has be copied and then configured and ready to run it is then called an “instance of that virtual machine” or just an “instance” -- your instance.

There are many “actions” that you can perform on your instance.  For example you can stop your instance, restart your instance, etc.  Note that stopping an instance is not the same as shutting down your UNIX system.  It just moves the entire operating system image out or RAM onto the hard drive.  When you start an instance there is not long boot process that is needed.  It can be running in seconds.

Note that you cannot restart a terminated instance. However, you can launch additional instances of the same image.

Limitations of the Amazon Micro Instance

The “free” version on the Amazon EC2 service is only configured with 613MB of RAM.  This is fine if you just need a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available. They are well-suited for lower throughput applications and web sites that consume significant compute cycles periodically.^[c]  This is because you can use 100% of the CPU cycles in a Micro Instance without any additional charges.  But there is a limit on the amount of bandwidth that you can use without incurring additional fees.  The “burst” capacity is if you are doing a large product launches or announcements and expecting lots of web traffic in a short amount of time.

Here are the current specifications of the Micro Instance Type

• 613 MB memory
• Up to 2 EC2 Compute Units (for short periodic bursts)
• EBS storage only
• 32-bit or 64-bit platform
• I/O Performance: Low
• API name: t1.micro

Public DNS

Each instance will be assigned a “Public DNS” when it starts.  This Public DNS looks like this:^[d]

(image missing here)

Elastic Static IP Address

Since there are a very limited number of IPv4 addresses remaining, Amazon only uses static IP address when the system needs them.  Amazon will give you an static IP address, but note that it charges for them if they are not bound to a running server!  They do this to keep people from hoarding unused static IP addresses.  These static IP Addresses can be quickly assigned to different instances and they are started and stopped.  The poll of static IP addresses that Amazon uses is called your Elastic IP address.

If you establish a link between your Elastic IP Address and your instance and you reboot the Public DNS will stays the same.

If you stop and restart your instance before you do a bind to an Elastic IP then  AWS creates a new Public DNS for your instance.   Any references to the Public DNS must be updated.

If you then associate your Elastic IP with any newly started instance, you need to wait a few minutes and refresh and the public DNS servers will reflect the Elastic IP Address.

Other Terms

For a full list of terms see the EC2 Glossary here.

Steps to get eXist running

Note: Many of the basic steps here are covered in Amazon’s Getting Started Guide.

http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/^[e][f]

Step 1: Create an EC2 Account at Amazon

Sign up for the EC2 account here:  

http://aws-portal.amazon.com/gp/aws/developer/subscription/index.html?productCode=AmazonEC2

Step 2 Create a security group for eXist

Before you create an instance of Linux, you will need to configure the “firewall” to limit the types of Internet Protocols that will be accepted by the server.  This is called a “Security Group” on the EC2 system.

There are two main protocols you will need:

SSH (Secure Shell) - for logging in and downloading the eXist software

HTTP (Default web port) - to permit your eXist system to be a web server.  This is 8080 by default but we will show how this will be set to port 80 in this example.

Note that for the eXist security group, both SSH on port 22 and HTTP on port 80 must be selected.

Step 3 Create a private key and download it to your local system

Every time you login to your Amazon system you must do so using standard public key cryptography.  This is done by the use of two keys, a public key that everyone can see and a private key that only you will ever see.  Amazon makes the process easy by generating a private key for you.  You just download this file to your local system and then you can connect directly using a secure shell.

Sidebar: What is PEM?

PEM (Privacy Enhanced Mail) is a protocol originally developed to secure email. Although rarely deployed for its intended purpose, it’s encoding mechanism for generating certificates is used for quite a few web services including Amazon EC2, PayPal Web Payments Pro and SSH Key Pairs.

Step 4 Create a new instance of Linux using CentOS Image

Now that we have both a security group and a key set up we are ready to create a new instance of Linux to run eXist in.

Select an Instance Type.  In our example we are ONLY going to be showing you how to use a Micro image type.  Note that this is NOT the default so be careful if you don’t want to get charged by Amazon.

http://aws.amazon.com/amis

You have a variety of instance types http://aws.amazon.com/ec2/instance-types  The free one is the “Micro” instance:

Warning! If you pick any other option than the Micro option you will be charged a daily fee by Amazon!

Now you can select any of the thousands of pre-built Amazon operating system images (AMI) that Amazon and their partners have set up.  At this point you could select an image with eXist already installed  (try , or you can select version of Linux or Windows and install eXist in it.

In this example we will use a basic 64-bit Amazon Linux AMI 1.0 which is the second selection in the image above.

You are now ready to start your Linux operating system.  To do this just click the “Launch Instance”.

When you create a new “virtual server” you can pick from one of several pre-built and pre-tested “Amazon Virtual Machines”.

Once^[g] it is running your instance should look like the following:

Note that each instance has a Name, an ID (Instance), an AMI ID (the imaged used to start the VM), a root Device, a type (in this case t1.micro), a status (in this case the instance is running), a security group^[h], a key pair and a few other properties.

The most critical part of the login step is that you use the correct Key Pair Name that is associated with this “instance”.  The “public” key pair is always associated with the remote instance.

Step 5 Login with Secure Shell (SSH) using your key

http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/

You can run the Unix ssh shell command on a Mac by just opening the Terminal application.  You can also download the Cygwin unix shell application on Windows systems that contains a version of ssh.

Here are the “Actions” you can perform with your VM:

In this example you we want to able to easily stop our system when we are not using it (so we don’t get an hourly charge) and then restart it when we are using it again.

You can download the key in the form of a “pem” file from from the AWS management console using the download function.

From Jeff Barr’s Book on EC2
Our first important task is to set the correct file permissions, otherwise we'll be unable to use it to connect: we'll receive a big 'WARNING: UNPROTECTED PRIVATE KEY FILE!' error message. Use the chmod command to restrict read and write permissions to the file owner: chmod 600 your_key_file.pem."

You must now make the private key file readable by only your account:

My preference is to put this key into my home directory under a folder called “keys”.  You then need to make it so you are the only person that has read access to your private key:

$ cd keys

$ chmod 600 exist-1.4.pem^[i]

Here is what the UNIX “ls” command should show.  Note that only the “r” or “rw” in your own permissions should be set.  If r is set for group or other SSH will not work.

You are now ready to connect to your new server via SSH.

$ ssh -i keys/exist-1.4.pem ec2-user@ec2-nn-nnn-nnn-nn.compute-1.amazonaws.com^[j]

Note that on some newer UNIX images Amazon has disabled direct root access.  You MUST use the ec2-user to connect to these images but older images may still use the root account.  See the release notes for each image for which account is used.

$ ssh -i keys/exist-1.4.pem ec2-user@ec2-nn-nn-nn-nnn.compute-1.amazonaws.com

           __|  __|_  )  Amazon Linux AMI

           _|  (         /         Beta

          ___|\___|___|

See /etc/image-release-notes for latest release notes. :-)

Note that if the image does not permit the root account to be used you MUST prefix all commands with “sudo”.

Step 6 Downloading the eXist binary

We will now use the UNIX “Web Get” shell command to get the exist binary.  But to do this we need the exact URL to the right eXist binary.

First change your directory to a location that you know you have write access such as:

$ cd /tmp

On your local machine, browse to the eXist download page.  Copy the URL of the link of the release you would like to use; the image below shows us copying the location of the eXist 1.4.0 jar.

Now type in “wget ” and paste in the URL.  You should see the following line in your shell:

$ cd /tmp

$ wget http://sourceforge.net/projects/exist/files/Stable/1.4/eXist-setup-1.4.0-rev10440.jar/download

or for 1.4.1

wget http://sourceforge.net/projects/exist/files/Stable/1.4.1/eXist-setup-1.4.1dev-rev14275.jar/download

When you hit return, you should see a message similar to the following:

--2010-11-13 15:05:35--  http://sourceforge.net/projects/exist/files/Stable/1.4/eXist-setup-1.4.0-rev10440.jar/download

Resolving sourceforge.net... 216.34.181.60

Connecting to sourceforge.net|216.34.181.60|:80... connected.

HTTP request sent, awaiting response... 302 Found

Location: http://downloads.sourceforge.net/project/exist/Stable/1.4/eXist-setup-1.4.0-rev10440.jar?r=&ts=1289660736&use_mirror=surfnet [following]

--2010-11-13 15:05:36--  http://downloads.sourceforge.net/project/exist/Stable/1.4/eXist-setup-1.4.0-rev10440.jar?r=&ts=1289660736&use_mirror=surfnet

Resolving downloads.sourceforge.net... 216.34.181.59

Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected.

HTTP request sent, awaiting response... 302 Found

Location: http://surfnet.dl.sourceforge.net/project/exist/Stable/1.4/eXist-setup-1.4.0-rev10440.jar [following]

--2010-11-13 15:05:36--  http://surfnet.dl.sourceforge.net/project/exist/Stable/1.4/eXist-setup-1.4.0-rev10440.jar

Resolving surfnet.dl.sourceforge.net... 130.59.138.21, 2001:620:0:1b::21

Connecting to surfnet.dl.sourceforge.net|130.59.138.21|:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 41232922 (39M) [application/java-archive]

Saving to: “eXist-setup-1.4.0-rev10440.jar”

100%[======================================================================================================================>] 41,232,922  10.1M/s   in 6.7s    

2010-11-13 15:05:43 (5.84 MB/s) - “eXist-setup-1.4.0-rev10440.jar” saved [41232922/41232922]

You can confirm that the file downloaded correctly by listing the directory’s contents:

$ ls -l

total 40312

-rw-rw-r-- 1 ec2-user ec2-user 41232922 Nov 11  2009 eXist-setup-1.4.0-rev10440.jar

For version 2.2  type:

  $ wget http://sourceforge.net/projects/exist/files/Stable/2.2/eXist-db-setup-2.2RC1.jar/download

  $ mv download download.jar

  $ chmod +x download.jar

Step 7 Install eXist from the UNIX Command Line using the Java Command

This material is also covered in the eXist documentation’s “Quick Start” article about performing a “headless” installation of eXist.  In this context “headless” implies that there is no graphical monitor being used, only a shell console.  The nice GUI installer used on the Mac and Windows can not be used here.

Verify that the Java JDK (not just the JRE) is all set up and configured correctly.

Warning! Your Amazon image may not have a recent version of the Java JDK installed.  Note that a JDK is different from a JRE and note that the Oracle JDK is different from the OpenJDK.  You can run eXist binary with a JRE but to compile from source you will need the JDK.  Make sure whatever instance you have is compatible with the version of eXist you are using.

For more on the eXist Java see the following link to the eXist quickstart documentation:

http://exist-db.org/exist/apps/doc/quickstart.xml?q=amazon&field=all&id=D1.2.3#D1.2.3

Here is and example of what the OracleJDK looks like:

$ java -version

java version "1.6.0_65"

Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)

Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)

Note that the word "Oracle" does not actually appear in the version text!

Here is an example of what the OpenJDK looks like

$ java -version

java version "1.6.0_20"

OpenJDK Runtime Environment (IcedTea6 1.9.1) (amazon-44.1.9.1.15.amzn1-i386)

OpenJDK Client VM (build 19.0-b06, mixed mode)

Note that eXist requires Java JDK to compile from source and the JRE to run from binary.

You can also run the command “which java” to see what version will be run.

Note that getting the right version of Java correctly installed in your AWS system is a non-trivial exercise.  Amazon also does not make this easy and the issues around requiring the "Accept licensing" the Oracle Java has compounded the complexity of the problem.  There are key differences between the OpenJDK and the OracleJDK that may cause problems.  See the release notes of your current eXist installation for notes on this.

You are now ready to install eXist from the jar file you just downloaded.  In this example we will install eXist directly from our /tmp directly into /usr/local.

Question: can we specify a guest and admin password on the command line of this install?

Note that the “sudo” prefix is only needed if your image does not allow you to login as the root account.

[root@ip-10-28-173-32 local]# sudo java -jar download.jar

Select target path [/usr/local]

/usr/local/exist2.2

press 1 to continue, 2 to quit, 3 to redisplay

1

Set Data Directory

Please select a directory where eXist-db will keep its data files. On Windows, this should be outside the 'Program Files' directory. Please make sure eXist can write to the directory it is installed in.

Data dir:  [webapp/WEB-INF/data]

press 1 to continue, 2 to quit, 3 to redisplay

1

Set Admin Password and Configure Memory

Enter password:  []

your-password-here

Enter password:  [your-password-here]

------------------------------------------

Maximum memory in mb: [1024]

Cache memory in mb: [128]

press 1 to continue, 2 to quit, 3 to redisplay

1

[ Starting to unpack ]

[ Processing package: Sources (1/12) ]

[ Processing package: Core (2/12) ]

[ Processing package: Apps (3/12) ]

[ Processing package: shared (4/12) ]

[ Processing package: bf-XForms (5/12) ]

[ Processing package: dashboard (6/12) ]

[ Processing package: demo (7/12) ]

[ Processing package: doc (8/12) ]

[ Processing package: eXide (9/12) ]

[ Processing package: fundocs (10/12) ]

[ Processing package: xsltforms (11/12) ]

[ Processing package: xsltforms-demo (12/12) ]

[ Unpacking finished ]

[ Starting processing ]

Starting process Setting admin password ... (1/1)

--- Starting embedded database instance ---

Apr 01, 2014 7:11:36 PM org.expath.pkg.repo.util.Logger info

INFO: Create a new repository with storage: File system storage in /usr/local/exist2.2/webapp/WEB-INF/data/expathrepo

Setting admin user password...

--- Initialization complete. Shutdown embedded database instance ---

[ Console installation done ]

[root@ip-10-28-173-32 local]#

In the rest of this document will refer to /usr/local/exist as $^[k]EXIST_HOME.^[l]

Note that $EXIST_HOME is not automatically set up by the installation process.  This is just a convention that you can use.  Some people like to modify the environment variables to add this.  For example you can add the following line to your /user/ec2-user/.bash_profile file:

Step 9 Configuration of Startup Parameters

There are two separate methods of starting exist.  One is called the “wrapper” method and the other is the “bin/startup” method.  These startup methods are very independent and use different configuration processes.  This guide recommends always using the wrapper method for configuration of your headless servers since you can use shell commands to monitor the server activity, but the bin/startup.sh method can be used for testing.

For the wrapper method, all the changes will be made to the following two locations:

$EXIST_HOME/tools/jetty/etc/jetty.xml

$EXIST_HOME/tools/wrapper/wrapper.conf

($EXIST_HOME/tools/wrapper/conf/wrapper.conf in 1.4.1)^[m]

The method that uses the bin/startup.sh file will modify the following file:

$EXIST_HOME/bin/functions.d^[n]

Option #1: Change the wrapper configuration jetty configuration file read by the wrapper

The jetty.xml file is read by wrapper the server.sh shell program

In the file $EXIST_HOME/tools/jetty/etc/jetty.xml change the SystemProperty for jetty.port to be 80:

<SystemProperty name="jetty.port" default="80"/>

In the file $EXIST_HOME/tools/wrapper/conf/wrapper.conf change the default value of 128 to be 2048

wrapper.java.maxmemory=2048

Option #2: Change the functions.d used in the startup.sh process

There are two items to change that will impact in the startup.sh  script.

Change the startup shell script

In the file $EXIST_HOME/bin/functions.d/eXist-settings.sh

Change the JAVA_OPTIONS maximum memory setting (-Xms) to be 512 and and the jetty port option: -Djetty.port=80:

set_java_options() {

        if [ -z "${JAVA_OPTIONS}" ]; then

            JAVA_OPTIONS="-Xms512m -Xmx512m -Dfile.encoding=UTF-8 -Djetty.port=80";

        fi

        JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.endorsed.dirs=${JAVA_ENDORSED_DIRS}";

}^[o]

Now you can start the server:

sudo ln -s $EXIST_HOME/tools/wrapper/bin/exist.sh /usr/local/bin

Startup Method #1 (Preferred)

$ sudo $EXIST_HOME/tools/wrapper/bin/exist.sh start

$sudo $EXIST_HOME/bin/exist.sh status

$sudo $EXIST_HOME/bin/exist.sh start

Startup Method #2 (Optional)

$ cd $EXIST_HOME

$ sudo bin/startup.sh^[p]

Using locale: en_US.UTF-8

13 Nov 2010 17:32:22,769 [main] INFO  (JettyStart.java [run]:90) - Configuring eXist from /usr/local/exist/conf.xml

13 Nov 2010 17:32:22,769 [main] INFO  (JettyStart.java [run]:91) -  

13 Nov 2010 17:32:22,769 [main] INFO  (JettyStart.java [run]:92) - Running with Java 1.6.0_20 [Sun Microsystems Inc. (OpenJDK Client VM) in /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre]

13 Nov 2010 17:32:22,769 [main] INFO  (JettyStart.java [run]:97) -  

13 Nov 2010 17:32:22,770 [main] INFO  (JettyStart.java [run]:101) - [eXist Version : 1.4.0]

13 Nov 2010 17:32:22,770 [main] INFO  (JettyStart.java [run]:103) - [eXist Build : 20091111]

13 Nov 2010 17:32:22,770 [main] INFO  (JettyStart.java [run]:105) - [eXist Home : /usr/local/exist]

13 Nov 2010 17:32:22,770 [main] INFO  (JettyStart.java [run]:107) - [SVN Revision : 10440]

13 Nov 2010 17:32:22,770 [main] INFO  (JettyStart.java [run]:115) - [Operating System : Linux 2.6.34.7-56.40.amzn1.i686 i386]

13 Nov 2010 17:32:22,771 [main] INFO  (JettyStart.java [run]:118) - [jetty.home : /usr/local/exist/tools/jetty]

13 Nov 2010 17:32:22,771 [main] INFO  (JettyStart.java [run]:120) - [log4j.configuration : file:/usr/local/exist/log4j.xml]

13 Nov 2010 17:32:23,742 [main] INFO  (FileResource.java [<clinit>]:60) - Checking Resource aliases

13 Nov 2010 17:32:23,859 [main] INFO  (HttpServer.java [setStatsOn]:1130) - Statistics on = false for org.mortbay.jetty.Server@650892

13 Nov 2010 17:32:23,864 [main] INFO  (HttpServer.java [doStart]:684) - Version Jetty/5.1.12

13 Nov 2010 17:32:24,167 [main] INFO  (Container.java [start]:74) - Started org.mortbay.jetty.servlet.WebApplicationHandler@14b6b02

Logging already initialized. Skipping...

13 Nov 2010 17:32:24,871 [main] INFO  (Container.java [start]:74) - Started WebApplicationContext[/exist,eXist XML Database]

13 Nov 2010 17:32:24,880 [main] INFO  (SocketListener.java [start]:205) - Started SocketListener on 0.0.0.0:80

13 Nov 2010 17:32:24,880 [main] INFO  (Container.java [start]:74) - Started org.mortbay.jetty.Server@650892

13 Nov 2010 17:32:24,880 [main] INFO  (JettyStart.java [run]:174) - ----------------------------------------------------------------

13 Nov 2010 17:32:24,880 [main] INFO  (JettyStart.java [run]:175) - eXist-db has started on port 80. Configured contexts:

13 Nov 2010 17:32:24,880 [main] INFO  (JettyStart.java [run]:177) - http://localhost:80/exist

13 Nov 2010 17:32:24,880 [main] INFO  (JettyStart.java [run]:179) - ----------------------------------------------------------------

Change your Wrapper Setting

Maximum Java Memory Configuration

Step 10 Test your Amazon EC2 Installation of eXist

You are now ready to test your Amazon EC2 installation of eXist.  You should be able to go to the a URL similar to the following:

http://ec2-nn-nnn-nnn-nn.compute-1.amazonaws.com/exist

Note that EC2 URLs are dynamically changed each time you start and stop your instance!

Note that this address will ONLY work while the instance is running and has never been stopped^[q].  As^[r] soon as it is stopped and restarted (not terminated) the address originally used will no longer work.  If you stop and restart you will need to use the new ec2 subdomain (ec2-nn-nnn-nnn.compute-1).  Check your Amazon Web Services Management Console for the most up-to-date public domain name of your instance.

Step 11 Create passwords for the admin and guest account

Warning: When you do a headless install both the admin and guest accounts do not have any passwords set.  This means that your system is wide open and anyone can write to any collection.  You must secure you system right away.  Do not delay this step!

Note, the full description of how to secure eXist is here:

http://demo.exist-db.org/exist/security.xml#N10185

When you do a “headless” install of eXist, by default the “admin” account and the “guest” account will not have ANY password set.  This configuration should never be used when there is public access.  Furthermore when eXist starts up it checks to see if there is an admin password.  If there is no password set it will run without any security checks.

The preferred method to set the login and password is to use the Java Admin client.  To start up the Java Admin client you should go to the bottom left corner of the eXist startup page and click the “Launch” button.

Follow the instructions on the exist site to set the guest and admin passwords.

There is another possible way to configure the admin and guest password, but it is known to have problems.

As soon as an admin password is enabled then security checks will start running.  We strongly recommend you first add a “guest” password using the web console.

To setup the initial passwords go to the main web page and scroll down to the lower left till you see the admin menu.  The first time you will login as admin with no password and you will see that neither the admin or guest have a password set.  First set the guest password and then set the admin password.  As soon as the admin password is set security is now “on”.  Order is important!

Warning: If the guest account does not have a password set correctly the system will not run when security is enabled by adding an admin password.  Even the admin panels will not work.  You MUST set a guest password before you do anything else or you will have to start over from scratch! :-(

Here is a screen image from the Admin/User Management screen:

^[s]

You will have to click each one and rest the password.  Remember to do t^[t]he guest a^[u]ccount first.

Step 12 Assign a Static IP address to your instance

If the instance is stopped and restarted, the Public DNS such as (ec2-nnn-nn-nnn-nnn.compute-1.amazonaws.com) will have changed! (not the static IP address).   Amazon EC2 provides Elastic IP addresses which can be created and mapped to instances.  This allows a fixed IP address to be used in URLs.  If the instance is stopped and restarted, the Elastic IP address will need to be reassigned to the instance because the public DNS has changed.  (I think cw)

If you want to always have the same IP address point to different instances as they are started and stopped you will have to rebind them using the Elastic IP address.

You might want to have static IP addresses for your instances. Amazon EC2 provides elastic IP addresses that can be dynamically remapped to^[v]                 different instances. For more information, go to Elastic IP Addresses in the Amazon Elastic Compute Cloud User Guide.

If the instance is stopped and restarted, the public DNS will have changed  the Elastic IP address has to be reassigned to the new instance ID.

Step^[w] 13 Automatically Starting eXist when the server boots

The correct method to start eXist is to first create a link from the /etc/init.d folder to the exist.sh script:

$ sudo ln -s $EXIST_HOME/tools/wrapper/bin/exist.sh /etc/init.d/exist

or if you use the standard local location:

$ sudo ln -s /usr/local/exist/tools/wrapper/bin/exist.sh /etc/init.d/exist

Next, add the following lines to the /etc/rc.d/rc.local

EXIST_HOME=/usr/local/exist

echo 'Starting the eXist native XML database from within /etc/rc.d/rc.local ...'

grep 'jetty.port' $EXIST_HOME/tools/jetty/etc/jetty.xml

echo 'Starting with the following max memory (MB)'

grep 'wrapper.java.maxmemory' $EXIST_HOME/tools/wrapper/wrapper.conf

/etc/init.d/exist start

You should be able to restart the server and when you login after a reboot you should be able to see the following:

$ exist status

eXist Native XML Database is running (940).

Note, that the “exist start” process will read the Jetty configuration file in $EXIST_HOME/tools/jetty/etc/jetty.xml.  If this is not set to be port 80 it will come up on port 8080 and it will not work with the security configuration.  The incorrect method is to use the bin/startup.sh script and modify this with the Djetty.port-80 setting. That process will work but the exist shell command will not be able to connect to the eXist server.

Viewing the startup log file

When your instance starts, you can use the Amazon AWS Management console to view the log file during the boot process. To view the system log you must first select your instance, then click on the Instance Actions drop down and then select the “Get System Log” item:

After you select this you will see the following window.  Scroll down to the very end of the System log:

You will note that the “echo” message from the rc.local should be there as well as the “Starting the eXist native XML database from within /etc/rc.d/rc.local ...” from the exist shell command.  The only problem here is that the message does not tell you which port it is starting on which makes debugging much more challenging.  You can fix this by puting this command in the rc.local to :

sample listing of rc.local file

EXIST_HOME=/usr/local/exist

echo 'Starting the eXist native XML database from /etc/rc.d/rc.local'

# This will show the port that eXist will start on

# make sure that the jetty.xml file has the right port

grep 'jetty.port' $EXIST_HOME/tools/jetty/etc/jetty.xml

echo 'Starting with the following max memory in MB:'

grep 'wrapper.java.maxmemory' $EXIST_HOME/tools/wrapper/conf/wrapper.conf

/etc/init.d/exist start

This should also put the following line on the console:

                        <SystemProperty name="jetty.port" default="80"/>

The following is an example of what you might see at the end of the startup log file:

Starting the eXist native XML database from /etc/rc.d/rc.local

 <!-- The default port can be changed using: java -Djetty.port=80     -->

                   <SystemProperty name="jetty.port" default="80"/>

Starting with the following max memory in MB:

wrapper.java.maxmemory=512

Starting eXist Native XML Database...

Note that the root use my not have /home/ec2-user/bin it the path so you will need to use an absolute path to restart eXist:

$ sudo /home/ec2-user/bin/exist restart

In the same way you want to put the stop in the shutdown:

/etc/init.d/exist stop

On Ubuntu 10.04 this file would be in rc6.d:

Next Steps

You can connect to the eXist Java admin client using the following string:

You can also add the EXIST_HOME to your .bash_profile

There are several other next steps you can now take.  You can monitor your system (for a fee), you can upgrade your memory (for a fee), you can have multiple instances running with a load balancer between the systems to reduce the load on any one system.^[x]

You can also assign a real “host name” to this IP address.  Organizations such as GoDaddy will create a DNS entry so that your domain name can point to this IP address.

Creating a link to the eXist shell command

It is very convenient to be able to just type “exist status” anywhere from the UNIX command line.  To do this you can create a “symbolic” link as follows 

$ sudo ln -s /usr/local/exist/tools/wrapper/bin/exist.sh /home/ec2-user/bin/exist

$ exist status

Links to Old Documents

The following links are now out-of-date

http://exist-db.org/installing-exist-on-amazon-ec2.html

http://exist-db.org/readying-centos-for-exist.html

Restricting Caching Space

When eXist loads new documents and indexes them, it likes to grab a lot of RAM.  Because we have limited memory of 613 MB we will need to limit how much memory it uses in the process of indexing large document collections.  The default eXist install does not restrict this.  To restrict this value you must edit the following file:

$EXIST_HOME/conf.xml

Change line line 122 from:^[y]

   nodesBuffer=”-1”>

to be

   nodesBuffer=”1000”>

Automatically Starting eXist with the chkconfig/service

From Anton: I think, that in CentOS, the best way for launching application servers is chkconfig/service, instead of rc.local

One can add something like following:

#!/bin/sh

#

# exist eXist Native XML Database and Application server

#

# chkconfig: 345 97 3

# description: eXist Native XML Database

# processname: exist

# pidfile: /var/run/exist/eXist.pid

# config:  /opt/exist/eXist/conf.xml

#

in the head of tools/wrapper/bin/exist.sh

and "activate" the service (after ln -s ... /etc/init.d/exist) using

chkconfig exist on

(The start/stop commands are: service exist start, service exist stop)

Also I think that it would be a good idea for eXist service to use separate linux user (setting RUN_AS_USER in the wrapper script and of course setting proper permissions on data and log directories and pid file)

/usr/local is good location for non-packaged locals, but for big system with specific directory layout containing data, subproducts and configuration, you may consider /opt location too.

Kind Regards,

Anton

Andrzej Taramina Said:

The problem you run into, with trying to run eXist as non-root, is that on a linux platform (why

would you run a production server on anything else? LOL), you have to have root priviledges to open

a low numbered port, that is port 80 or 443...the standard HTTP/HTTPS ports for web access.

Using 8080 works in a dev environment, but typically is not practical in production, unless you get

into multi-layered network architectures with bastion servers, port redirection, DMZs and the like,

which for smaller deployments, is not usually pragmatic.

Thus the best way to deploy eXist in a production, secure environment (on Linux...see comment above

;-) ), is to run it in a chroot jail, where it's isolated from the rest of the OS, but still can

have enough privileges to open restricted ports.

Setting up a chroot jail is non-trivial...but probably a lot easier to do and maintain than changing

all the permissions.

My 2 cents worth.

Incidentally, I've been running our eXist-based healthcare software on Amazon EC2 (Ubuntu instances)

for a few years now. It's not that difficult, barring locking things down for production use.

Setting the Correct Wrapper Executable

Loren Cahlander said:

If you look in

/usr/local/exist/tools/wrapper/bin,

you will find several files with the names starting with wrapper.  These are the wrapper executables for each of the system environments.  e.g. wrapper-linux-86-32 is for running within a linux environment running on a 32bit x86 system.  You need to either rename one of these files to wrapper or more appropriately link the particular executable to the file named wrapper.

If you are running on a linux environment running on a 32bit x86 system, then run the following command in the /usr/local/exist/tools/wrapper/bin directory:

$ cd /usr/local/exist/tools/wrapper/bin

$ ln -s wrapper-linux-x86-32 wrapper

Changing the Guest Password

But default the guess password is set to be "guest".  If you change this you will no longer be able to access the system admin pages.  We only recommend that you change the guest password if you are an experienced user and you are in the process of "locking" down the system.

If you do change the guest password the first function that will fail is a module call the "URLRewriting" module.  This module is used by all the web admin pages.  You can re-enable these functions with the new guess password if you add the following to the $EXIST_HOME/webapp/WEB-INF/web.xml file:

<filter-name>XQueryURLRewrite</filter-name>

   ….snip....

   <init-param>

      <param-name>password</param-name>

      <param-value>your-new-password</param-value>

</init-param>

….snip...

Once this is done your URLRewriting functions will work again.

Note that you can still secure you system by removing the "read" access for guest for any collection in the system.  This method is frequently the preferred way of doing this.

Getting Security Updates

Amazon provides a service that allows you to get Linux security updates if you use the Amazon default Linux AMI.  You use the "yum" tool to install these updates.

$ sudo yum upgrade

In general these are small UNIX library changes and the eXist server does not need to be restarted during an upgrade.

Eric Palmer (Eric.DaddyOh) Says:  Thanks for putting this together.  I was able to get this running in about 1.5 hours. The major problem I had is that with 1.4 the guest password needs to be “guest” and that was not obvious.  Otherwise this is pretty straight forward.  I haven’t set exist to start up on a reboot but that should be easy.  Yes, just make sure there is a link in /etc/inet.d/exist.sh to the actual wrapper script  

off to lunch now!

let me know if you have any questions…

Alias for Wrapper Commands

When you login to the eXist it is nice to have your startup shell automatially create a few handy admin shortcuts for you so you can type "exist-status" right from the command line shell.  To do this you can put some "alias" commands in your shell startup.  By doing this you don't have to remember to put "sudo" in front of the command and you don't have to add /usr/local/bin to the sudo path, which can be a security issue.

Put the following in your .bash_profile in your home directory

# commands to start and stop exist

alias restart-exist='sudo /usr/local/bin/exist restart'

alias start-exist='sudo /usr/local/bin/exist start'

alias stop-exist='sudo /usr/local/bin/exist stop'

alias status='sudo /usr/local/bin/exist status'

Or if you prefer consistency (thanks Eric Palmer)

# commands to start and stop exist

alias exist-restart='sudo /usr/local/bin/exist restart'

alias exist-start='sudo /usr/local/bin/exist start'

alias exist-stop='sudo /usr/local/bin/exist stop'

alias exist-status='sudo /usr/local/bin/exist status'