Tuesday, October 27, 2015

Automated WAR file redeployment in tomcat using a BASH script with the Linux 'at' command

Wouldn't it be nice to schedule a redeployment of your updated web-app during off-peak hours AUTOMATICALLY?


I tried searching for some existing scripts or blogs on how others might have done this, but surprisingly nothing came up. So I'm sharing how I did this from scratch and I'm hoping others will find this useful.

The Bash script

The bash script is responsible for shutting down tomcat, undeploying your webapp, deploying your new web-app and starting up tomcat. The script takes in two arguments.

  • The name of the web-app without the '.war' extension
  • The location of the new version of the web-app
# deploy.sh
# Script to shutdown tomcat and redeploy webapp

#Check the number of arguments
if [ "$#" -ne 2 ]
then
        echo "Missing arguments: webapp name and WAR file location"
        exit 1
fi

if [ -z "$1" ]
then
        echo "First argument cannot be empty"
        exit 1
fi

if [ -z "$2" ]
then
        echo "Second argument cannot be empty"
        exit 1
fi

tomcatHome="/usr/local/apache-tomcat-7.0.55"
echo "Tomcat home: $tomcatHome"

# Get the process ID of tomcat
pid=$(ps h -C java -o "%p:%a" | grep catalina | cut -d: -f1)
if [ "$pid" > 0 ]
then
        echo "Shutting down tomcat PID $pid"

        # Shutdown tomcat
        #$tomcatHome/bin/shutdown.sh
        kill -9 $pid

        # Wait until tomcat is shutdown
        while kill -0 $pid > /dev/null; do sleep 1; done

fi

# remove the old webapp
echo "Removing webapp $1"
mv $tomcatHome/webapps/$1.war $tomcatHome
rm -rf $tomcatHome/webapps/$1

# Copy the new WAR file to the webapps folder
cp $2 $tomcatHome/webapps/$1.war

# Change the permissions
chown tomcat:tomcat $tomcatHome/webapps/$1.war

# Start up tomcat
/etc/init.d/tomcat start

# Finished
echo "redeployed successfully"


An example of how to run the script:
./deploy.sh TaskRequest /home/philip/TaskRequest.war

If you plan on reusing my script, one of the things you'll most likely want to change is the 'tomcatHome' variable and the way I move copies of the old WAR file out of the web-apps folder.

Make sure that you have permissions to execute the bash script:

chmod a+x deploy.sh

Scheduling

 Now that we have our script to execute the redeployment and restart of tomcat, we'll need to schedule a one-time job using linux's 'at' command. As an example, if you want to schedule your redeployment to occur at 9pm tonight, you could use the following command:

at 21:00

This will start a command prompt, where you can now specify what commands or scripts to be executed:

at> /root/deploy.sh TaskRequest /home/philip/TaskRequest.war
CTRL-D
CRTL-D will end the prompt

And that's pretty much it. If you want to view the list scheduled jobs you can use the 'atq' command:


[root@apntest1 webapps]# atq
2       2015-10-28 09:30 a root


To view the contents of the scheduled job use 'at -c id ', for example:


[root@apntest1 webapps]# at -c 2
#!/bin/sh
# atrun uid=0 gid=0
# mail   philip 0
umask 22
HOSTNAME=apntest1; export HOSTNAME
SHELL=/bin/bash; export SHELL
OLDPWD=/usr/local/apache-tomcat-7.0.55; export OLDPWD
USER=root; export USER
LS_COLORS=no=00:fi=00:di=00\;34:ln=00\;36:pi=40\;33:so=00\;35:bd=40\;33\;01:cd=40\;33\;01:or=01\;05\;37\;41:mi=01\;05\;37\;41:ex=00\;32:\*.cmd=00\;32:\*.exe=00\;32:\*.com=00\;32:\*.btm=00\;32:\*.bat=00\;32:\*.sh=00\;32:\*.csh=00\;32:\*.tar=00\;31:\*.tgz=00\;31:\*.arj=00\;31:\*.taz=00\;31:\*.lzh=00\;31:\*.zip=00\;31:\*.z=00\;31:\*.Z=00\;31:\*.gz=00\;31:\*.bz2=00\;31:\*.bz=00\;31:\*.tz=00\;31:\*.rpm=00\;31:\*.cpio=00\;31:\*.jpg=00\;35:\*.gif=00\;35:\*.bmp=00\;35:\*.xbm=00\;35:\*.xpm=00\;35:\*.png=00\;35:\*.tif=00\;35:; export LS_COLORS
SUDO_USER=philip; export SUDO_USER
SUDO_UID=501; export SUDO_UID
USERNAME=root; export USERNAME
PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/philip/bin; export PATH
MAIL=/var/spool/mail/philip; export MAIL
PWD=/usr/local/apache-tomcat-7.0.55/webapps; export PWD
LANG=en_AU; export LANG
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass; export SSH_ASKPASS
SHLVL=1; export SHLVL
SUDO_COMMAND=/bin/su; export SUDO_COMMAND
HOME=/root; export HOME
LOGNAME=root; export LOGNAME
CVS_RSH=ssh; export CVS_RSH
LESSOPEN=\|/usr/bin/lesspipe.sh\ %s; export LESSOPEN
SUDO_GID=501; export SUDO_GID
G_BROKEN_FILENAMES=1; export G_BROKEN_FILENAMES
cd /usr/local/apache\-tomcat\-7\.0\.55/webapps || {
         echo 'Execution directory inaccessible' >&2
         exit 1
}
${SHELL:-/bin/sh} << `(dd if=/dev/urandom count=200 bs=1 2>/dev/null|LC_ALL=C tr -d -c '[:alnum:]')`

/root/deploy.sh TaskRequest /home/philip/TaskRequest.war

Confirmation email

 Let's take it one step further. If we want to receive email notifications of the automated deployment, we can pipe the output of the script into the mail command as follows:


at 21:00
at> /root/deploy.sh TaskRequest /home/philip/TaskRequest.war | mail -s "Webapp deployment" jo.blo@gmail.com
CTRL-D

One liner

If you want to execute the command all in one line without a command prompt, here's an example


echo "/root/deploy.sh TaskRequest /home/philip/TaskRequest.war | mail -s \"Task request redeployed\" jo.blow@gmail.com" | at 19:45

Configuring RHEL to send emails

To install use the following
yum install postfix

If the mail command doesn't work then try installing mailx as well:

# mail
bash: mail: command not found
# yum install mailx

If you're not receiving emails, it's probably because you haven't configured it yet. If you're like me using Red Hat Linux, then here are some basic instructions on how I set it up. Edit the file /etc/postfix/main.cf and set the parameter called relayhost to your smtp server:


relayhost=some.ip.address

Then restart postfix: /etc/init.d/postfix restart
Try testing by sending a test email:

echo "please work" | mail -s "Test email" email@gmail.com

If you didn't receive an email you can debug it by adding the -v flag for verbose output as follows

echo "please work" | mail -v -s "Test email" email@gmail.com
Mail Delivery Status Report will be mailed to root. 

You will get the following response: Mail Delivery Status Report will be mailed to root, here's how to view the message sent to root by using the 'mail' command:

[root@apn-lsrv01 ~]# mail
Heirloom Mail version 12.4 7/29/08.  Type ? for help.
"/var/spool/mail/root": 13 messages 2 new 11 unread
   10 Mail Delivery System  Thu Oct 29 08:09  68/2348  "Mail Delivery Status Report"
   11 Mail Delivery System  Thu Oct 29 08:13  66/2276  "Mail Delivery Status Report"
>N 12 Mail Delivery System  Thu Oct 29 08:28  65/2239  "Mail Delivery Status Report"
 N 13 Mail Delivery System  Thu Oct 29 08:28  65/2239  "Mail Delivery Status Report"

A command prompt will appear asking you which message to open. Enter the ID of the message in this case '13' to see what went wrong with your email.

& 13
Message 13:
From MAILER-DAEMON  Thu Oct 29 08:09:19 2015
Return-Path: <>
X-Original-To: root@blah
Delivered-To: root@blah
Date: Thu, 29 Oct 2015 08:09:19 +1100 (AEDT)
From: MAILER-DAEMON@blah (Mail Delivery System)
Subject: Mail Delivery Status Report
To: root@blah
Auto-Submitted: auto-replied
Content-Type: multipart/report; report-type=delivery-status;
        boundary="745D91181D8A.1446066559/blah"
Status: R

Part 1:
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii

This is the mail system at host blah.

Enclosed is the mail delivery report that you requested.

                   The mail system

<someones@email>: delivery temporarily suspended: Host or domain name not
    found. Name service error for name=bad.smtp.server type=AAAA: Host not
    found



In this message I realized, that the relayhost I configured earlier was incorrect. Once I obtained the correct IP address, sending emails started to work.

Use CTRL-D to end the prompt.

Happy to get some feedback on how you might have improved your bash script! Cheers!

References:

Getting the tomcat process PID
How to use the 'at' command
Send emails from linux

Sunday, September 20, 2015

Adding more images to the Nexus theme for Drupal

By default the Nexus theme only allows 3 images in the front page slider. For many of us this is not enough, so I outline the steps I took to add additional images by modifying some of the underlying code.

Add a new image

Once you've unpacked the theme, go to the /images folder and add your image and change the name to 'slide-image-4.jpg'

Modify  /templates/page.tpl.php

Edit the template found in /templates/page.tpl.php as follows and add the lines highlighted in red:


  <?php  
   $slide1_head = check_plain(theme_get_setting('slide1_head','nexus'));  $slide1_desc = check_markup(theme_get_setting('slide1_desc','nexus'), 'full_html'); $slide1_url = check_plain(theme_get_setting('slide1_url','nexus'));  
   $slide2_head = check_plain(theme_get_setting('slide2_head','nexus'));  $slide2_desc = check_markup(theme_get_setting('slide2_desc','nexus'), 'full_html'); $slide2_url = check_plain(theme_get_setting('slide2_url','nexus'));  
   $slide3_head = check_plain(theme_get_setting('slide3_head','nexus'));  $slide3_desc = check_markup(theme_get_setting('slide3_desc','nexus'), 'full_html'); $slide3_url = check_plain(theme_get_setting('slide3_url','nexus'));  
   $slide4_head = check_plain(theme_get_setting('slide4_head','nexus'));  $slide4_desc = check_markup(theme_get_setting('slide4_desc','nexus'), 'full_html'); $slide4_url = check_plain(theme_get_setting('slide4_url','nexus'));  
  ?>  
  <div id="slidebox" class="flexslider">  
   <ul class="slides">  
    <li>  
     <img src="<?php print base_path() . drupal_get_path('theme', 'nexus') . '/images/slide-image-1.jpg'; ?>"/>  
     <?php if($slide1_head || $slide1_desc) : ?>  
      <div class="flex-caption">  
       <h2><?php print $slide1_head; ?></h2><?php print $slide1_desc; ?>  
       <a class="frmore" href="<?php print url($slide1_url); ?>"> <?php print t('READ MORE'); ?> </a>  
      </div>  
     <?php endif; ?>  
    </li>  
    <li>  
     <img src="<?php print base_path() . drupal_get_path('theme', 'nexus') . '/images/slide-image-2.jpg'; ?>"/>  
     <?php if($slide2_head || $slide2_desc) : ?>  
      <div class="flex-caption">  
       <h2><?php print $slide2_head; ?></h2><?php print $slide2_desc; ?>  
       <a class="frmore" href="<?php print url($slide2_url); ?>"> <?php print t('READ MORE'); ?> </a>  
      </div>  
     <?php endif; ?>  
    </li>  
    <li>  
     <img src="<?php print base_path() . drupal_get_path('theme', 'nexus') . '/images/slide-image-3.jpg'; ?>"/>  
     <?php if($slide3_head || $slide3_desc) : ?>  
      <div class="flex-caption">  
       <h2><?php print $slide3_head; ?></h2><?php print $slide3_desc; ?>  
       <a class="frmore" href="<?php print url($slide3_url); ?>"> <?php print t('READ MORE'); ?> </a>  
      </div>  
     <?php endif; ?>  
    </li>  
    <li>  
     <img src="<?php print base_path() . drupal_get_path('theme', 'nexus') . '/images/slide-image-4.jpg'; ?>"/>  
     <?php if($slide4_head || $slide4_desc) : ?>  
      <div class="flex-caption">  
       <h2><?php print $slide4_head; ?></h2><?php print $slide4_desc; ?>  
       <a class="frmore" href="<?php print url($slide4_url); ?>"> <?php print t('READ MORE'); ?> </a>  
      </div>  
     <?php endif; ?>  
    </li>  

Modify theme-settings.php

To configure the title, description and URL of the image through the user interface, add the following lines of code to the file /theme-settings.php


  $form['nexus_settings']['slideshow']['slide4'] = array(  
   '#type' => 'fieldset',  
   '#title' => t('Slide 4'),  
   '#collapsible' => TRUE,  
   '#collapsed' => TRUE,  
  );  
  $form['nexus_settings']['slideshow']['slide4']['slide4_head'] = array(  
   '#type' => 'textfield',  
   '#title' => t('Slide Headline'),  
   '#default_value' => theme_get_setting('slide4_head','nexus'),  
  );  
  $form['nexus_settings']['slideshow']['slide4']['slide4_desc'] = array(  
   '#type' => 'textarea',  
   '#title' => t('Slide Description'),  
   '#default_value' => theme_get_setting('slide4_desc','nexus'),  
  );  
  $form['nexus_settings']['slideshow']['slide4']['slide4_url'] = array(  
   '#type' => 'textfield',  
   '#title' => t('Slide URL'),  
   '#default_value' => theme_get_setting('slide4_url','nexus'),  
  );  

Then you should see the following when configuring the 'appearance':


 References: https://www.drupal.org/node/1672396

Sunday, August 9, 2015

Using a basic groovy library packaged as a JAR, for sharing domain class definitions across many projects

Let's say I have multiple web applications all developed in Grails that needed access to a shared database as shown in the diagram below.



What would be the best way to define domain classes without duplicating code?

Prior to Grails 3.x, I typically created Grails Plugins to define domain classes shared across multiple web applications. All you had to do was define a plugin dependency in your BuildConfig.groovy configuration file.


 plugins {  
           runtime "my-plugin:my-plugin:1.0-SNAPSHOT"  
 }  

This worked reasonably well, that is, until we decided to add a new web application using Grails 3.x to the mix of existing web applications as shown in the diagram below:





 Grails 3.x was a massive change, now using Gradle as opposed to Maven to construct the builds. This meant that the file structure and layout changed dramatically due to differences in convention. What this also meant, was that the previously defined plugins in Grails 2.x no longer worked in Grails 3.x projects.
We were not in a position to upgrade all of our applications to 3.x because the example I've provided is simplified. We actually have many more web applications accessing the same shared database and would require extensive regression testing.

The other alternative, would be to maintain 2 versions of the plugin, one in grails 2.x and the other in grails 3.x. For obvious reasons, this was not ideal.

What I wanted was a single library (JAR file) with my domain class definitions that could be used by any web application regardless of the underlying grails version.

I've manged to find a way and present my findings here with you.

No more plugins, just a groovy library

Rather than using a plugin, I've created a library containing all  my domain class definitions using a combination of

  • Gradle - as the build tool
  • Groovy - to define my domain classes
  • JPA - to define domain class relational mappings - Java Persistence API

 Gradle installation

  • Download Gradle from the following webiste: Download Gradle
  • Set the bin folder to be in your environment variable PATH. This will allow you to run the gradle from the command-line
  • To test run the following from the command-line, 'gradle -v'

Define build.gradle in your library

 // Apply plugin must come first, the order is very important!  
 apply plugin: 'groovy'  
 apply plugin: 'java'  
 apply plugin: 'eclipse'  
 apply plugin: 'maven'  
 group = 'my-plugin'  
 version = '1.0-SNAPSHOT'  
 repositories {  
      mavenCentral()  
      mavenLocal()  
 }  
 dependencies {  
      compile 'org.codehaus.groovy:groovy-all:2.3.10'  
      compile 'org.hibernate.javax.persistence:hibernate-jpa-2.1-api:1.0.0.Final'  
 }  

Define your domain classes using JPA

 package anu.individual  
 import java.util.Collection;  
 import java.util.List;  
 import java.util.Set;  
 import javax.persistence.ElementCollection  
 import javax.persistence.Entity  
 import javax.persistence.EnumType  
 import javax.persistence.Enumerated  
 import javax.persistence.GeneratedValue  
 import javax.persistence.Id  
 import javax.persistence.JoinColumn  
 import javax.persistence.JoinTable  
 import javax.persistence.ManyToOne  
 import javax.persistence.OneToMany  
 /**  
  * Individual domain class, defined using JPA   
  *   
  * @author Philip  
  *  
  */  
 // Using JPA annotations  
 @Entity  
 class Individual {  
      @Id  
      @GeneratedValue  
      Long id  
      String externalId  
      String externalIdUpperCase // variable not used  
      Boolean deleted= false  
      @ManyToOne  
      @JoinColumn(name="mother_id")  
      Individual mother  
      @ManyToOne  
      @JoinColumn(name="father_id")  
      Individual father  
      Long mustererBarcode  
      @ElementCollection  
      Set<String> phenotypes // snowmed CT IDs  
      @Enumerated(EnumType.STRING)  
      Species species  
      Boolean affected  
      Boolean proband  
      @Enumerated(EnumType.STRING)  
      Gender gender  
      @OneToMany       
      @JoinTable(  
           name="individual_team",  
           joinColumns= @JoinColumn(name="individual_id", referencedColumnName="id"),  
           inverseJoinColumns= @JoinColumn(name="team_id", referencedColumnName="id")   
      )  
      Set<Team> teams  

Package the library

To package the library using gradle, you can use the 'gradle install' from the command-line. This will automatically install the library in your local maven repository (.m2 folder) which can be used by your Grails application as a depedency.

Add the dependency to your Grails 3 application

If you're using Grails 3, then the way to configure your dependencies is in the build.gradle file. An example is shown below with the library highlighted in red:


 buildscript {  
   ext {  
     grailsVersion = project.grailsVersion  
   }  
   repositories {  
     mavenLocal()  
     maven { url "https://repo.grails.org/grails/core" }  
   }  
   dependencies {  
     classpath "org.grails:grails-gradle-plugin:$grailsVersion"  
     classpath 'com.bertramlabs.plugins:asset-pipeline-gradle:2.1.1'  
   }  
 }  
 plugins {  
   id "io.spring.dependency-management" version "0.5.2.RELEASE"  
 }  
 version "0.1"  
 group "individual"  
 apply plugin: "spring-boot"  
 apply plugin: "war"  
 apply plugin: "asset-pipeline"  
 apply plugin: 'eclipse'  
 apply plugin: 'idea'  
 apply plugin: "org.grails.grails-web"  
 apply plugin: "org.grails.grails-gsp"  
 ext {  
   grailsVersion = project.grailsVersion  
   gradleWrapperVersion = project.gradleWrapperVersion  
 }  
 assets {  
   minifyJs = true  
   minifyCss = true  
 }  
 repositories {  
   mavenLocal()  
   maven { url "https://repo.grails.org/grails/core" }  
 }  
 dependencyManagement {  
   imports {  
     mavenBom "org.grails:grails-bom:$grailsVersion"  
   }  
   applyMavenExclusions false  
 }  
 dependencies {  
   compile "org.springframework.boot:spring-boot-starter-logging"  
   compile "org.springframework.boot:spring-boot-starter-actuator"  
   compile "org.springframework.boot:spring-boot-autoconfigure"  
   compile "org.springframework.boot:spring-boot-starter-tomcat"  
   compile "org.grails:grails-dependencies"  
   compile "org.grails:grails-web-boot"  
   compile "org.grails.plugins:hibernate"  
   compile "org.grails.plugins:cache"  
   compile "org.hibernate:hibernate-ehcache"  
   compile "org.grails.plugins:scaffolding"  
   runtime "org.grails.plugins:asset-pipeline"  
   testCompile "org.grails:grails-plugin-testing"  
   testCompile "org.grails.plugins:geb"  
   // Note: It is recommended to update to a more robust driver (Chrome, Firefox etc.)  
   testRuntime 'org.seleniumhq.selenium:selenium-htmlunit-driver:2.44.0'  
   console "org.grails:grails-console"  
      // custom dependencies  
      compile 'anu.individual:individual-lib:1.0-SNAPSHOT'  
      runtime 'postgresql:postgresql:9.1-901.jdbc4'  
 }  
 task wrapper(type: Wrapper) {  
   gradleVersion = gradleWrapperVersion  
 }  

Tell Grails which domain classes should apply GORM

In the Grails app, create a file called hibernate.cfg.xml and place it in the /conf folder
 <!DOCTYPE hibernate-configuration SYSTEM  
  "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">  
 <hibernate-configuration>  
   <session-factory>  
     <mapping package="anu.individual" />  
     <mapping class="anu.individual.Individual" />  
     <mapping class="anu.individual.Team" />  
   </session-factory>  
 </hibernate-configuration>  

The file should list all the domain classes defined in the library

If you want to provide a different datasource for the domain classes defined in your external library, you can prefix the datasource name to the hibernate.cfg.xml file. Quote from the grails website says the following:

"To specify that an annotated class uses a non-default datasource, create a hibernate.cfg.xml file for that datasource with the file name prefixed with the datasource name."
For example, if I have a datasource named ds1, then i would change the name of the hibernate config file to ds1_hibernate.cfg.xml

Run the grails application

To test if your configuration is working correctly, run the application using 'grails run-app' command and your tables should automatically be generated.

Good luck!


References:
http://grails.github.io/grails-doc/latest/guide/plugins.html#creatingAndInstallingPlugins
http://stackoverflow.com/questions/23654116/importing-external-domain-classes-into-grails
https://grails.github.io/grails-doc/latest/guide/hibernate.html
https://spring.io/blog/2010/08/26/reuse-your-hibernate-jpa-domain-model-with-grails
https://jira.grails.org/browse/GRAILS-9389
https://jira.grails.org/browse/GRAILS-9410
http://blog.bripkens.de/2012/03/Install-artefacts-local-repository-gradle/
https://spring.io/guides/gs/accessing-data-jpa/
https://docs.gradle.org/current/userguide/tutorial_groovy_projects.html
https://grails.github.io/grails-doc/latest/guide/single.html#dependencyResolution

Sunday, August 2, 2015

Setting up Grails 3 in Spring Tool Suite (STS) or Eclipse

Steps to follow

1) Download grails 3.0.4
3) Add grails bin folder to your PATH environment variable
2) Reconfigure eclipse following the instructions here for Gradle support: https://tedvinke.wordpress.com/2015/04/10/grails-3-released-installing-gradle-and-groovy-2-4-support-in-eclipseggts/
3) Import from bitbucket
4) Learn how to run application following documentation notes: http://grails.github.io/grails-doc/3.0.4/guide/gettingStarted.html#aHelloWorldExample
5) Open command-line, then change directory to the project folder, type 'grails run-app'

The other alternative is to try intelliJ

To use intelliJ with Grails, the following link can be helpful:

IntelliJ + Grails

1) Download IntelliJ community edition
2) Download Git SCM
3)  Open IntelliJ and click 'Check out from Version Control' and select 'Git'
 

4)  Copy and paste the Git URL and click Clone



5) Enter your password to access bitbucket

 Even IntelliJ suffers from the same problems as STS. So you will still have to use the command-line to invoke grails specific commands as mentioned in the following link:

http://stackoverflow.com/questions/28261484/intellij-idea-grails-3-0-error