<b>J2EE Hosting Standards</b>

<u>Purpose</u>
Intended for development groups, deployment groups, java administrators, server administrators, 3rd party application vendors and requestors of new java app servers

<u>References</u>
1) Linux Server Standards and Provisioning - http://&lt;sharepoint_host&gt;/dept/it/InfrastructureSupport/dcteam/Shared%20Documents/Linux/Linux%20Server%20Standards%20and%20Provisioning.doc
2) Linux Provisioning Instructions - http://&lt;sharepoint_host&gt;/dept/it/InfrastructureSupport/dcteam/Shared%20Documents/Linux/Linux%20Provisioning%20Instructions.docx
3) J2EE Application Server Standards: http://&lt;sharepoint_host&gt;/dept/it/InfrastructureSupport/mwteam/Shared%20Documents/J2EE/J2EE%20Application%20Server%20Standards%20and%20Provisioning.doc
4) JBoss 5 install script: https://docs.google.com/document/pub?id=1-dOWCFugK_ZkbCbsHMq3n7a3JVFrhvsK1B6178s8s1o
5) Apache Multi-Purpose Reverse Proxy: https://docs.google.com/document/view?id=1Q5iMm16sKkVl2LKu8ae1srCh-EEK8sq1j_WE1QO5hgc&authkey=CIqoneIH
6) Reverse Proxy Diagram - Exposed J2EE Apps: https://docs.google.com/drawings/d/1jJtkb8IQjZZ9BjZt5C3kTMmdMYxxoBk0X3CEoRxowEA/view
7) PasLinuxPermissions.xls - http://&lt;sharepoint_host&gt;/dept/it/InfrastructureSupport/mwteam/Shared%20Documents/J2EE/PasLinuxPermissions.xls
8) Documents to support and justify standards - http://&lt;sharepoint_host&gt;/dept/it/InfrastructureSupport/mwteam/Shared%20Documents/J2EE/RapidsureInfrastructureRisks.pdf, http://&lt;sharepoint_host&gt;/dept/it/InfrastructureSupport/mwteam/Shared%20Documents/J2EE/PIC%20Migration%20Assessment.doc, and http://&lt;sharepoint_host&gt;/dept/it/InfrastructureSupport/mwteam/Shared%20Documents/J2EE/Philadelphia%20Insurance%20JBoss%20Recommendations.pdf

The hosting department strongly recommends utilizing the JBoss EAP standards outlined below so that J2EE applications can be hosted to the highest standards. This includes usage in any and all code, scripts, configuration files, documentation, and/or anywhere relevant in the hosting environment. These standards have been gathered from multiple independent assessments by expert consulting bodies, professional training, research and, most of all, experience. Following these standards is the only way for the hosting department to provide the best and most efficient J2EE provisioning and hosting experience for the business.

<u>Collaborative Items</u>:

1)  <u>Slimming</u>
 a. The hosting department will slim all unnecessary components including hypersonic, hot deploy, clustering, JMS, quartz scheduler, etc. If the customer application relies on any of these components, it is up to the customer  to provide a case to keep them when the request for server provisioning is made. See below for more information regarding specific modules, what they do, and their dependencies.
 b.    Since the linux server provisioning process is push button and middleware installation and configuration is tightly coupled to this process, all configurations including slimming information are required before the server is built. This process creates snapshots of server build instructions serving as our documentation for the server runbook and enhancing our business continuity ability for linux servers.
2)    <u>Class loading and Shared Libraries</u>
 a.   It is required that all 3rd party applications utilize isolated class loading for portability, sustainability and supportability. This also prevents logging problems. It is mechanically enforced in $JBOSS_HOME/server/$JBOSS_CONF/deployers/ear-deployer-jboss-beans.xml: &lt;isolated&gt;true&lt;/isolated&gt;. As such, write access to shared libraries such as $JBOSS_HOME/lib/, $JBOSS_HOME/common/lib/, and $JBOSS_HOME/server/$JBOSS_CONF/lib/ for shared 3rd party libraries is disabled.
3) <u>Logs</u>
 a. Application logging should always be separated from jboss or system logging (logs/ instead of log/).
 b. In keeping with best practices, please use a consistent logging methodology for easier tuning and control. Please do not utilize STDERR or STDOUT for logging as it circumvents tuning and control.
 c. Please use rolling file appenders and jboss log directory and jboss log level variables at all times. Example:
<i>
 &lt;?xml version="1.0" encoding="UTF-8" ?&gt;
 &lt;!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"&gt;
 &lt;log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"&gt;
 &lt;appender name="FILE" class="org.apache.log4j.<b>RollingFileAppender</b>"&gt;
   &lt;param name="File" value="<b>${jboss.server.log.dir}s/acme.log</b>"/&gt;
   &lt;param name="Append" value="true"/&gt;
   &lt;param name="MaxFileSize" value="5000KB"/&gt;
   &lt;param name="MaxBackupIndex" value="5"/&gt;
   &lt;layout class="org.apache.log4j.PatternLayout"&gt;
    &lt;param name="ConversionPattern" value="%d %-5p [%c] %m%n"/&gt;
   &lt;/layout&gt;
 &lt;/appender&gt;
 &lt;root&gt;
   &lt;priority value="<b>${jboss.server.log.threshold}</b>" /&gt;
   &lt;appender-ref ref="FILE" /&gt;
 &lt;/root&gt;
 &lt;/log4j:configuration&gt;
</i>
4) <u>Path variables and aliases</u>
 a. <b>$JBOSS_HOME</b> (OS variable) and <b>${jboss.server.home}</b> (java variable) will always point to the current running jboss-as/ directory. Since this makes troubleshooting, support, patching, updates, and other tasks much easier for the hosting department, one must utilize this variable as opposed to hard coding the absolute path in any and all configuration files, scripts, or documentation. If one is unable to use this OS level variable, they must use the generic path of <b>/usr/local/jboss/jboss-as/</b>.
 b. <b>$JBOSS_CONF</b> will always point to the working node directory. The hosting department will always copy from the "default" node to a custom node. The hosting department does not plan on utilizing JBoss clustering for high availability unless absolutely needed, so the hardcoded name will probably be "&lt;client&gt;-nocluster”. However, use of the variable instead of the name is standard.
 c.    <b>cdj</b> is an alias for cd $JBOSS_HOME/server/$JBOSS_CONF/ and is available to all users for the user's convenience
5) <u>Datasources</u>
 a. All JBoss 5 servers will have generic and comprehensive datasources setup at <b>$JBOSS_HOME/server/$JBOSS_CONF/deploy/mssql-ds.xml</b>. Please utilize this file and only this file for all datasources. Please overwrite only the relevant settings within DefaultDS node: sql server name, sql server instance, username and password. Do not overwrite or paste your own xml node as it would overwrite tuned datasource settings. Similary, a comprehensive XA datasource is provided but commented.
 b. The hosting department will be using the latest Microsoft supported SQL JDBC driver for all SQL Server connections.
 c.    For the most stable datasources and to avoid problems with database connection pooling, it is highly recommended to utilize the included xml nodes called &lt;new-connection-sql&gt; and &lt;check-valid-connection-sql&gt;. To perform the most accurate health check for sql connections, application developers should supply lightweight sql queries specific to their application in lieu of the generic "select 1" statements.
6) <u>Redundancy</u>
 a. As a general rule, the hosting department plans on architecting high availability, active/active nodes in all MO and production environments. This will be achieved through hardware load balancing over port 8080*. Note that the utilization of JBoss application clustering will be avoided unless absolutely necessary for session replication reasons as it introduces an unnecessary level of complexity and load. *Note that if reverse proxy functionality is needed, then load balancing would require an additional software load balancing layer in the DMZ network using mod_jk over port 8009.
 b.    This requires that the applications deployed can handle multiple active nodes and are not coded in a singleton context.

<u>Informational Items</u>:

1) <u>Reverse Proxies</u>
 a. If the java applications deployed need to be exposed to the internet, classic apache/mod_jk/F5 architectures will be utilized as outlined in the above references.
2)    <u>Systems Architecture - Scalability</u>
 a.    Horizontal scalability will be advised in all environments, especially MO and Production (see Redundancy above). Appropriate server sizing will be advised (ex: for improved performance, it is better to build multiple small servers rather than fewer larger servers).
3) <u>Cache</u>
 a. The contents of the work/, data/, and tmp/ directories are automatically removed on jboss shutdowns as well as any files in /tmp/ owned by jboss.
4) <u>OS</u>
 a. Application server provisioning for JBoss has been merged into our Linux server provisioning. Therefore all JBoss servers will be hosted on the latest supported version of Red Hat Enterprise Linux.
 b. The hosting department will always use 64 bit OS platforms and JDKs unless there is a reason to avoid it.
 c.    We run a customized service control script to ensure process shutdown, avoid duplicates, and provide troubleshooting assistance. Hard kills will be used when proper shutdown methods are unsuccessful. For the convenience of the application developer, a "dump" command is built in to automatically create dump files for developer troubleshooting.
 d.   All JBoss installations are done at provision time. Therefore, the hosting department will not clone JBoss servers or allow the copying of JBoss bundles from one server to the next. This ensures that the hosting department meets the goal of building customized, lean and mean J2EE servers tailored to each application as well as preserving permissions and folder naming conventions.
5) <u>Customizations</u>
 a. The hosting department may utilize <b>soft links</b> wherever necessary to denote customizations at the file or directory level. For example, any third party add-ons, such as JDBC drivers will always be soft linked from custom subfolders. This ensures that the hosting department can achieve the highest level of application administration, troubleshooting and maintenance.
6) <u>Permissions</u>
 a. In order to pass audits, guarantee service levels, and support our environments the best way possible, the hosting department will only issue the amount of permissions that are needed. Requests are subject to review by change control, architecture review boards, and the security team. See PasLinuxPermissions.xls in the middleware sharepoint site. All changes require ITSR requests and are not a guarantee for approval.
 b.    Permissions are configured so that group ownership of jboss will persist but user ownership will show the actual user who created the file or folder for trace-ability and audit reasons.

 Deployment teams will be able to stop or start jboss using the following syntax unless otherwise noted:
 sudo -u jboss service jbossas (stop|start|restart)
 Additionally, value added functions have been created in the service control script to generate thread dumps and tattle tale reports (WIP):
 sudo -u jboss service jbossas (dump|tattle)

 Anyone with UNIX permissions will ALWAYS have read access to most JBoss files including but not limited to:
 $JBOSS_CONF/server/$JBOSS_CONF/log/
 $JBOSS_CONF/server/$JBOSS_CONF/logs/

Developers are typically granted the same permissions as Deployment technicians in all environments except model office and production. This is subject to change.
The standard location for deployments should be kept in /var/tmp/releases/ instead of any one user's home folder. This improves portability and supportability for deployment teams.


The hosting department reserves the right to update these standards at anytime as necessary.