Development, Analysis And Research

Updated bash script 23/01/2010: The script was assumed to run from EC2 itself, however I have since modified this so its applicable to local environments and made a little more robust.

There are quite a number of new tools from EC2, each requiring some form of setup on the server. As a result I have created a bash script to install them automatically.

• AMI Tools
• API Tools
• Elastic Load Balancing Tools
• Cloud Watch Tools

The only prerequisite is to install the certificate and private key within ~/.ec2/pk.pem and ~/.ec2/cert.pem and Java if on Fedora.

I have tested this on Debian and Fedora, which uses either yum or apt to execute the install of some dependencies, although it does assume that you are running under root.

You can download it from here ec2.sh

#!/bin/bash

DEBUG=1;

if [ "$(id -u)" != "0" ]; then
  echo "This script must be run as root" 1>&2
  exit 1
fi

GREP="`which grep`";
if which apt-get >/dev/null; then
  PACKAGE_MANAGEMENT="`which apt-get` "
else
  PACKAGE_MANAGEMENT="`which yum`"
fi

if which dpkg >/dev/null; then
  PACKAGE_TEST="dpkg --get-selections | $GREP -q "
else
  PACKAGE_TEST="rpm -qa | $GREP -q "
fi

CURL_OPTS=" --silent --retry 1 --retry-delay 1 --retry-max-time 1 "

function log
{
  if [ "$DEBUG" -eq 1 ]; then
    echo $1;
  fi;
}

function bail
{
  echo -e $1;
  exit 1;
}

function check_env
{
  if [ -f ~/.bashrc ]; then
    . ~/.bashrc;
  fi

  # Tools already exist
  if [ -z `which ec2-describe-instances` ] || [ -z `which ec2-upload-bundle` ] || [ -z `which ec2-describe-instances` ] || [ -z `which elb-create-lb` ] || [ -z `which mon-get-stats` ] || [ -z `which as-create-auto-scaling-group` ]; then
    log "Amazon EC2 toolkit missing!"
    install_ec2;
  fi

  # EC2_HOME set
  if [ -z "$EC2_HOME" ]; then
    log "Amazon EC2 is not set-up correctly! EC2_HOME not set"
    if ! grep EC2_HOME ~/.bashrc; then
      echo "export EC2_HOME=/usr/local/ec2-api-tools/" >> ~/.bashrc
    fi;
    export EC2_HOME=/usr/local/ec2-api-tools/
    source ~/.bashrc
  fi

  # Java
  if [ -z "$JAVA_HOME" ]; then

    if grep -i yum "$PACKAGE_MANAGEMENT" > /dev/null; then
      bail "nPlease install java manually (do not use yum install java, it is incompatible)nsee JRE http://java.sun.com/javase/downloads/index.jspnDownload, run the bin file, place in /opt/ and update ~/.bashrc. Once complete run 'source ~/.bashrc;'";
    fi;

    $PACKAGE_MANAGEMENT install -y sun-java6-jdk
    JAVA_PATH=/usr/lib/jvm/java-6-sun/jre/;

    echo "export JAVA_HOME=$JAVA_PATH" >> ~/.bashrc
    export JAVA_HOME=$JAVA_PATH
    source ~/.bashrc
  fi

  # Keys
  EC2_HOME_DIR='.ec2';
  EC2_PRIVATE_KEY_FILE="$HOME/$EC2_HOME_DIR/pk.pem";
  EC2_CERT_FILE="$HOME/$EC2_HOME_DIR/cert.pem";

  if [ ! -d "$HOME/$EC2_HOME_DIR" ]; then
    mkdir -pv "$HOME/$EC2_HOME_DIR";
  fi

  install_ec2_env EC2_PRIVATE_KEY "$EC2_PRIVATE_KEY_FILE";
  install_ec2_env EC2_CERT        "$EC2_CERT_FILE";

  install_ec2_keys_files "$EC2_PRIVATE_KEY_FILE" "Private key";
  install_ec2_keys_files "$EC2_CERT_FILE" "Certificate";

  install_ec2_env AWS_AUTO_SCALING_HOME "/usr/local/ec2-as-tools/"
  install_ec2_env AWS_ELB_HOME          "/usr/local/ec2-elb-tools/"
  install_ec2_env AWS_CLOUDWATCH_HOME   "/usr/local/ec2-cw-tools/"

  get_region
  get_availability_zone

}

function install_ec2_env
{
  # Variable Variable for $1
  EC2_VARIABLE=${!1};
  EC2_VARIABLE_NAME=$1;
  EC2_FILE=$2;

  #log "VARIABLE: $EC2_VARIABLE_NAME=$EC2_VARIABLE";

  # Variable Variable
  if [ -z "$EC2_VARIABLE" ]; then
    log "Amazon $EC2_VARIABLE_NAME is not set-up correctly!";

    if ! grep -q "$EC2_VARIABLE_NAME" ~/.bashrc > /dev/null; then
      echo "export $EC2_VARIABLE_NAME=$EC2_FILE" >> ~/.bashrc;
    fi;
    export $EC2_VARIABLE_NAME=$EC2_FILE;
    source ~/.bashrc
  else
    if ! grep -q "$EC2_VARIABLE_NAME" ~/.bashrc > /dev/null; then
      echo "export $EC2_VARIABLE_NAME=$EC2_FILE" >> ~/.bashrc;
    else
      log "Amazon $EC2_VARIABLE_NAME var installed";
    fi;
  fi
}

function install_ec2_keys_files
{
  EC2_FILE=$1;
  EC2_DESCRIPTION=$2;
  EC2_CONTENTS='';

  if [ ! -f "$EC2_FILE" ]; then
    bail "Amazon $EC2_FILE does not exist, please copy your $EC2_DESCRIPTION to $EC2_FILE and re-run this script";
  else
    log "Amazon $EC2_FILE file installed";
  fi
}

function install_ec2
{

  for PACKAGE in curl wget tar bzip2 unzip zip symlinks unzip ruby; do
    if ! which "$PACKAGE" >/dev/null; then
      $PACKAGE_MANAGEMENT install -y $PACKAGE;
    fi
  done;

  # AMI Tools
  if [ -z "`which ec2-upload-bundle`" ]; then
    curl -o /tmp/ec2-ami-tools.zip $CURL_OPTS --max-time 30 http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
    rm -rf /usr/local/ec2-ami-tools;
    cd /usr/local && unzip /tmp/ec2-ami-tools.zip
    ln -svf `find . -type d -name ec2-ami-tools*` ec2-ami-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  # API Tools
  if [ -z "`which ec2-describe-instances`" ]; then
    log "Amazon EC2 API toolkit is not installed!"
    curl -o /tmp/ec2-api-tools.zip $CURL_OPTS --max-time 30 http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
    rm -rf /usr/local/ec2-api-tools;
    cd /usr/local && unzip /tmp/ec2-api-tools.zip
    ln -svf `find . -type d -name ec2-api-tools*` ec2-api-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  # ELB Tools
  if [ -z "`which elb-create-lb`" ]; then
    curl -o /tmp/ec2-elb-tools.zip $CURL_OPTS --max-time 30 http://ec2-downloads.s3.amazonaws.com/ElasticLoadBalancing-2009-05-15.zip
    rm -rf /usr/local/ec2-elb-tools;
    cd /usr/local && unzip /tmp/ec2-elb-tools.zip
    mv ElasticLoadBalancing-1.0.3.4 ec2-elb-tools-1.0.3.4;
    ln -svf `find . -type d -name ec2-elb-tools*` ec2-elb-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  # Cloud Watch Tools
  if [ -z "`which mon-get-stats`" ]; then
    curl -o /tmp/ec2-cw-tools.zip $CURL_OPTS --max-time 30 http://ec2-downloads.s3.amazonaws.com/CloudWatch-2009-05-15.zip
    rm -rf /usr/local/ec2-cw-tools;
    mv -v CloudWatch-1.0.2.3 ec2-cw-tools-1.0.2.3
    cd /usr/local && unzip /tmp/ec2-cw-tools.zip
    ln -svf `find . -type d -name ec2-cw-tools*` ec2-cw-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  if [ -z "`which as-create-auto-scaling-group`" ]; then
    curl -o /tmp/ec2-as-tools.zip $CURL_OPTS --max-time 30 http://ec2-downloads.s3.amazonaws.com/AutoScaling-2009-05-15.zip
    rm -rf /usr/local/ec2-as-tools;
    mv -v AutoScaling-1.0.9.0 ec2-as-tools-1.0.9.0
    cd /usr/local && unzip /tmp/ec2-as-tools.zip
    ln -svf `find . -type d -name ec2-as-tools*` ec2-as-tools
    chmod -R go-rwsx ec2* && rm -rvf /tmp/ec2*
  fi

  ln -sf /usr/local/ec2-api-tools/bin/* /usr/bin/;
  ln -sf /usr/local/ec2-ami-tools/bin/* /usr/bin/;
  ln -sf /usr/local/ec2-elb-tools/bin/* /usr/bin/;
  ln -sf /usr/local/ec2-cw-tools/bin/* /usr/bin/;
  ln -sf /usr/local/ec2-as-tools/bin/* /usr/bin/;

  rm -f /usr/bin/ec2-*.cmd;

}

function get_availability_zone
{
  # Not reliable between availability zones using meta-data
  # export EC2_AVAILABILITY_ZONE="`curl $CURL_OPTS --max-time 2 http://169.254.169.254/2009-04-04/meta-data/placement/availability-zone`"

  get_instance_id;

  if [ ! -z "$EC2_INSTANCE_ID" ]; then
    EC2_AVAILABILITY_ZONE="`ec2-describe-instances | grep -q $EC2_INSTANCE_ID | awk '{print $11}'`"
    if [ -z "$EC2_AVAILABILITY_ZONE" ] && [ ! "$EC2_AVAILABILITY_ZONE"="" ]; then
      export EC2_AVAILABILITY_ZONE=$EC2_AVAILABILITY_ZONE;
      install_ec2_env EC2_AVAILABILITY_ZONE $EC2_AVAILABILITY_ZONE;
    fi;
  fi;
}

function get_region
{
  get_instance_id;
  if [ ! -z "$EC2_INSTANCE_ID" ]; then
    EC2_REGION="`ec2-describe-instances | grep $EC2_INSTANCE_ID | awk '{print $11}'`"
    if [ -z "$EC2_REGION" ]; then
      export EC2_REGION=$EC2_REGION;
      install_ec2_env EC2_REGION $EC2_REGION;
      install_ec2_env EC2_URL "https://ec2.$EC2_REGION.amazonaws.com" | sed 's/a.amazonaws.com/.amazonaws.com/'
    fi;
  fi;
}

function get_instance_id
{
  instanceId="`curl $CURL_OPTS --max-time 2 http://169.254.169.254/1.0/meta-data/instance-id`"
  if [ ! -z "$instanceId" ]; then
    export EC2_INSTANCE_ID="$instanceId";
  fi;
}

check_env

I just read “How to use locks in PHP cron jobs to avoid cron overlaps” and I thought I would elaborate on this and provide some more examples. In order for a lock to work correctly it must handle, Atomicity / Race Conditions, and Signaling.

I use the following bash script to create locks for crontabs and ensure single execution of scripts.

“The clever bit is to get a lock file test and creation (if needed) to be atomic, that is done without interruption. The set -C stops a redirection from over writing a file. The : > touches a file. In combination, the effect is, when the lock file exists, the redirection fails and exits with an error. If it does not exist, the redirection creates the lock file and exits without an error.

The final part is to make sure that the lock file is cleaned up. To makes sure it is removed even if the script is terminated with a ctrl-c, a trap is used. Simply, when the script exits, the trap is run and the lock file is deleted.”, The Lab Book Pages

In addition it also checks the process list and tests whether the pid within the lock file is active.

#!/bin/bash

LOCK_FILE=/tmp/my.lock
CRON_CMD="php /var/www/..../fork.php -t17"

function check_lock {
    (set -C; : > $LOCK_FILE) 2> /dev/null
    if [ $? != "0" ]; then
        RUNNING_PID=$(cat $LOCK_FILE 2> /dev/null || echo "0");
        if [ "$RUNNING_PID" -gt 0 ]; then
            if [ `ps -p $RUNNING_PID -o comm= | wc -l` -eq 0 ]; then
                echo "`date +'%Y-%m-%d %H:%M:%S'` WARN [Cron wrapper] Lock File exists but no process running $RUNNING_PID, continuing";
            else
                echo "`date +'%Y-%m-%d %H:%M:%S'` INFO [Cron wrapper] Lock File exists and process running $RUNNING_PID - exiting";
                exit 1;
            fi
        else
            echo "`date +'%Y-%m-%d %H:%M:%S'` CRIT [Cron wrapper] Lock File exists with no PID, wtf?";
            exit 1;
        fi
    fi
    trap "rm $LOCK_FILE;" EXIT
}

check_lock;
echo "`date +'%Y-%m-%d %H:%M:%S'` INFO [Cron wrapper] Starting process";
$CRON_CMD &
CURRENT_PID=$!;
echo "$CURRENT_PID" > $LOCK_FILE;
trap "rm -f $LOCK_FILE 2> /dev/null ; kill -9 $CURRENT_PID 2> /dev/null;" EXIT;
echo "`date +'%Y-%m-%d %H:%M:%S'` INFO [Cron wrapper] Started ($CURRENT_PID)";
wait;
# remove the trap kill so it won't try to kill process which took place of the php one in mean time (paranoid)
trap "rm -f $LOCK_FILE 2> /dev/null" EXIT;
rm -f $LOCK_FILE 2> /dev/null;
echo "`date +'%Y-%m-%d %H:%M:%S'` INFO [Cron wrapper] Finished process";

With the implementation described in the post at abhinavsingh.com, it will fail if you put it as a background process as an example see below.

andrew@andrew-home:~/tmp.lock$ php x.php
==16169== Lock acquired, processing the job...
^C
andrew@andrew-home:~/tmp.lock$ php x.php
==16169== Previous job died abruptly...
==16170== Lock acquired, processing the job...
^C

andrew@andrew-home:~/tmp.lock$ php x.php
==16170== Previous job died abruptly...
==16187== Lock acquired, processing the job...
^Z
[1]+  Stopped                 php x.php
andrew@andrew-home:~/tmp.lock$ ps aux | grep php
andrew   16187  0.5  0.5  50148 10912 pts/2    T    09:53   0:00 php x.php
andrew   16192  0.0  0.0   3108   764 pts/2    R+   09:53   0:00 grep --color=auto php
andrew@andrew-home:~/tmp.lock$ php x.php
==16187== Already in progress...

You can use pcntl_signal to trap interruptions to the application and handle cleanup of the process. Here is a slightly modified implementation to handle cleanup. Just to highlight the register_shutdown_function will not help to cleanup on any signal/interruption.

<?php
class lockHelper {

	protected static $_pid;

	protected static $_lockDir = '/tmp/';

	protected static $_signals = array(
		// SIGKILL,
		SIGINT,
		SIGPIPE,
		SIGTSTP,
		SIGTERM,
		SIGHUP,
		SIGQUIT,
	);

	protected static $_signalHandlerSet = FALSE;

	const LOCK_SUFFIX = '.lock';

	protected static function isRunning() {
		$pids = explode(PHP_EOL, `ps -e | awk '{print $1}'`);
		return in_array(self::$_pid, $pids);
	}

	public static function lock() {
		self::setHandler();

		$lock_file = self::$_lockDir . $_SERVER['argv'][0] . self::LOCK_SUFFIX;
		if(file_exists($lock_file)) {
			self::$_pid = file_get_contents($lock_file);
			if(self::isrunning()) {
				error_log("==".self::$_pid."== Already in progress...");
				return FALSE;
			}
			else {
				error_log("==".self::$_pid."== Previous job died abruptly...");
			}
		}

		self::$_pid = getmypid();
		file_put_contents($lock_file, self::$_pid);
		error_log("==".self::$_pid."== Lock acquired, processing the job...");
		return self::$_pid;
	}

	public static function unlock() {
		$lock_file = self::$_lockDir . $_SERVER['argv'][0] . self::LOCK_SUFFIX;
		if(file_exists($lock_file)) {
			error_log("==".self::$_pid."== Releasing lock...");
			unlink($lock_file);
		}
		return TRUE;
	}

	protected static function setHandler() {
		if (!self::$_signalHandlerSet) {

			declare(ticks = 1);

			foreach(self::$_signals AS $signal) {
				if (!pcntl_signal($signal, array('lockHelper',"signal"))) {
					error_log("==".self::$_pid."== Failed assigning signal - '{$signal}'");
				}
			}
		}
		return TRUE;
	}

	protected static function signal($signo) {
		if (in_array($signo, self::$_signals)) {
			if(!self::isrunning()) {
				self::unlock();
			}
		}
		return FALSE;
	}
}

As an example:

andrew@andrew-home:~/tmp.lock$ php t.php
==16268== Lock acquired, processing the job...
^Z==16268== Releasing lock...

Whilst the implementation above simply uses files, it could be implemented with shared memory (SHM/APC), distributed caching (memcached), or a database. If over a network, factors such as packet loss, latency etc can cause race conditions and should be taken into account. Depending on the application it maybe better to implement as a daemon. If your looking to distribute tasks amongst servers, take a look at Gearman

• supervisord
• Create daemons in PHP

Last year I wrote an application to highlight media outlets and their reach (coverage of media outlets), selecting regions within the UK and highlighting aspects of a map. This had many issues where by hitting performance problems of rendering within browsers and also limitations of converting KML to tiles via google. A list of these limitations are:

1. Timeouts from google on large KML files.
2. Responsiveness of servers to deliver KML files to google.
3. Max KML size (Even when gzipped)
4. 500 Errors from google
5. Transparency within IE
6. ….

Some of these limits have since been increased by google and are documented.

Maximum fetched file size (raw KML, raw GeoRSS, or compressed KMZ)	3MB
Maximum uncompressed KML file size	10MB
Maximum number of Network Links	10
Maximum number of total document-wide features	1,000

In order to alleviate these issues I ended up with the following

• Caching KML files to avoid latency on a expensive database lookups/response.
• Chunking the response into 250 records and writing to individual static KML files. (Files would become very large and google would time out retrieving data sets).
• Proxying googles tiles after they had been converted from KML to images and caching them locally on our servers and then applying the overlays from our servers once merged

So depending on the depth (zoom) of the map and the area selected as well the volume of data, it would either use tiles or googles KML directly (Increased functionality).

In order to have greater control over the spatial data within our database we split this into areas, regions, and sub_regions, which held lookups to postcodes, towns and spatial data itself (There are a lot of discrepancies over outlines of maps).

Left hand menu:

<ul style="display: block;" class="ulTree jsTree">
	<li id="East"><a href="#" onclick="loadTilesFromGeoXML('|1|'); return false;">East</a>
		<ul style="display: none;" class="ulTree jsTree">
			<li><a href="#" onclick="loadTilesFromGeoXML('|1|6'); return false;">Bedfordshire</a></li>
			<li><a href="#" onclick="loadTilesFromGeoXML('|1|18'); return false;">Cambridgeshire</a></li>
			...
		</ul>
	</li>
</ul>

Javascript to locate tiles

  function loadTilesFromGeoXML(entity_id) {
    // Matches database record ids that are mapped to spatial data within MySQL
    mapTownsId = entity_id.toString().split('|')[0];
    mapRegionsId = entity_id.toString().split('|')[1];
    mapSubRegionsId = entity_id.toString().split('|')[2];
    locationUrl ='map_towns_id='+mapTownsId+'&map_regions_id='+mapRegionsId+'&map_sub_regions_id='+mapSubRegionsId;

    var cc = map.fromLatLngToDivPixel(map.getCenter());
    map.setZoom(1);

    // Request URL to cached titles links
    geoXMLUrl = '/ajax/mapping/get/overlays/region?'+locationUrl;
    geoXMLUrl+='&format=JSON&method=getLinks&x='+cc.x+'&y='+cc.y+'&zoom='+map.getZoom();

    // tileUrlTemplate: 'http://domain.com/maps/proxy/regions/?url=http%3A%2F%2Fdomain.com/ajax/mapping/get/cache/?filename=.1.6.0&x={X}&y={Y}&zoom={Z}',

    $.getJSON(geoXMLUrl, function(data) {
      $.each(data, function(i,link) {
        kmlLinks+=encodeURIComponent(link)+',';
      });

      // Builds the location for tiles to be mapped
      tileUrlTemplate = '/maps/proxy/regions/?url='+kmlLinks+'&x={X}&y={Y}&zoom={Z}';
      var tileLayerOverlay = new GTileLayerOverlay(
        new GTileLayer(null, null, null, {
          tileUrlTemplate: tileUrlTemplate,
          isPng:true,
          opacity:1.0
        })
      );
      if (debug) GLog.writeUrl('/maps/proxy/regions/?url='+kmlLinks+'&x={X}&y={Y}&zoom={Z}');
      map.addOverlay(tileLayerOverlay);
    });
  }

Response whilst retrieving links (if cached)

The code behind this simply caches the KML files, if it does not exist, otherwise attempts to create it and also outputs a json request with the files matching the sequence and globs for any files with a similar pattern, all files are suffixed with their page number.

["/ajax/mapping/get/cache/?filename=.1..0&x=250&y=225&zoom=5","/ajax/mapping/get/cache/?filename=.1..1&x=250&y=225&zoom=5"]

Proxying googles tiles and merging the layer ids

    $kmlUrls = urlencode($_GET['url']);
    $cachePath = dirname(__FILE__).'/cache.maps/tiles/';

    $cachedFiles = array_filter(explode(',',rawurldecode($kmlUrls)));
    $hash = sha1(rawurldecode($kmlUrls).".w{$_GET['w']}.h{$_GET['h']}.x{$_GET['x']}.y{$_GET['y']}.{$_GET['zoom']}");
    $cachePath.="{$_GET['x']}.{$_GET['y']}/{$_GET['zoom']}/";
    if (!is_dir($cachePath)) {
      @mkdir($cachePath, 0777, true);
    }

    // Returns image if cached already and aggregated.
    if (file_exists($path = $cachePath.$hash)) {
      header('Content-Type: image/png');
      $fp = fopen($path, 'rb');
      fpassthru($fp);
    }

    // Extract layer id's from KML files that are to be merged.
    $layerIds = array();
    foreach( $cachedFiles AS $kmlFile) {
      $kmlFile="http://{$_SERVER['HTTP_HOST']}{$kmlFile}";

      $url = "http://maps.google.com/maps/gx?q={$kmlFile}&callback=_xdc_._1fsue7g2w";
      @$c = file_get_contents($url);
      if (!$c)
        throw new Exception("Failed to request {$url} - {$c}");
      preg_match_all('/layer_id:"kml:(.*)"/i', $c, $matches);
      if (count($matches)>0 && isset($matches[1][0])) {
        $layerIds[] = "kml:{$matches[1][0]}";
      }
    }

    // Cache locally.
    if (count($layerIds)>0) {
      header('Content-Type: image/png');
      // Aggregate layers into a single image
      $link = "http://mlt0.google.com/mapslt?lyrs=" . implode(',',$layerIds);
      $link.="&x={$_GET['x']}&y={$_GET['y']}&z={$_GET['zoom']}&w={$_GET['w']}&h={$_GET['h']}&source=maps_api";
      echo $c = file_get_contents($link);
      @file_put_contents($path, $c);
    } else {
      // Output 1x1 png
      header('Content-Type: image/png');
      echo base64_decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAC0lEQVQIHWNgAAIAAAUAAY27m/MAAAAASUVORK5CYII=');
    }
  }

Paging GeoXML loading

    function loadGeoXMLPaged(geoXMLUrl) {
      var cc = map.fromLatLngToDivPixel(map.getCenter());
      geoXMLUrl+='&format=JSON&method=getLinks&x='+cc.x+'&y='+cc.y+'&zoom='+map.getZoom();

      if (debug) GLog.writeUrl(geoXMLUrl);

      $.getJSON(geoXMLUrl, function(data) {
	  geoXmlPager = data;
          loadGeoXmlPage();
        });
      }

      var timeoutPID = null;

      function loadGeoXmlPage(){
  	if (data = geoXmlPager.pop()){
	 if (debug)
            GLog.writeUrl(BASE_URL+data);

	 geoXmlStack.push(new GGeoXml(BASE_URL+data));
	 map.addOverlay(geoXmlStack[geoXmlStack.length - 1]);

         GEvent.addListener(geoXmlStack[geoXmlStack.length - 1],"load",function() {
	  timeoutPID = setTimeout("loadGeoXmlPage()", 500);
         });
	}else{
          clearTimeout(timeoutPID);
	  map.setZoom(map.getBoundsZoomLevel(bounds));
          map.setCenter(bounds.getCenter());
          try {
            geoXmlStack[geoXmlStack.length - 1].gotoDefaultViewport(map);
          } catch(e) {}
	}
      }

All the code above has been modified slightly to make it applicable to others, however don’t accept raw input as its simply an example.

I recently came across a peculiar issue that meant dates and times were causing issues with a product we had developed within Australia. The issue being that within “Red Hat Enterprise Linux Server release 5 (Tikanga)” the date within PHP was being read as EST instead of AEST/AEDT, however running “date” from the terminal or running “SELECT NOW()” from MySQL displayed the correct time.

[user@server ~]$ date
Wed Oct 14 22:24:20 EST 2009

[user@server ~]$ php -r'var_dump(date("r"));'
string(51) "Wed, 14 Oct 2009 21:25:07 +1000 Australia/Melbourne"

[user@server ~]$ php -r'var_dump(date("r e"));var_dump(getenv("TZ"));var_dump(ini_get("date.timezone"));var_dump(date_default_timezone_get());';
string(51) "Wed, 14 Oct 2009 21:25:07 +1000 Australia/Melbourne"
bool(false)
string(0) ""
string(19) "Australia/Melbourne"

[user@server ~]$ mysql -uuser -ppassword -e 'SELECT NOW();'
+---------------------+
| NOW()               |
+---------------------+
| 2009-10-14 22:26:12 |
+---------------------+

As you can see php incorrectly gets the time, being an hour off. Running the above on debian worked perfectly fine and comparing the zoneinfo matched my local machine.


[user@server ~]$ md5sum /etc/localtime && md5sum /usr/share/zoneinfo/Australia/Sydney && md5sum /usr/share/zoneinfo/Australia/Melbourne
85285c5495cd5b8834ab62446d9110a9 /etc/localtime
85285c5495cd5b8834ab62446d9110a9 /usr/share/zoneinfo/Australia/Sydney
8a7f0f78d5a146db4bf865ca91cc1c42 /usr/share/zoneinfo/Australia/Melbourne


After a fair amount of digging I ended up coming across the following ticket @478566. Amazingly the ticket is marked as “CLOSED WONTFIX”.

There were a few interesting points from some of the conversations I read.

” Alphabetic time zone abbreviations should not be used as unique identifiers for UTC offsets as they are ambiguous in practice. For example, “EST” denotes 5 hours behind UTC in English-speaking North America, but it denotes 10 or 11 hours ahead of UTC in Australia; and French-speaking North Americans prefer “HNE” to “EST”. twinsun”

Due to different locations in Australia having various interpretations of summer time with start/end dates and clock shifts. As well as the operating system not having zoneinfo data for DEST, AEDT etc (unless you create these yourself) it means you cannot rely on the correct time from php on redhat.

So far I have resorted to the following

[user@server ~]$ php -r 'date_default_timezone_set("Etc/GMT-11"); var_dump(date("r"));';
string(31) "Wed, 14 Oct 2009 22:24:29 +1100"


I have been migrating a large number of websites and consolidating servers to reduce costs.
As a result it is important to ensure that services are migrated smoothly, planned effectively,
after which I had a think about aspects to consider prior to migrating services.

Planning

• Make a preliminary checklist of services actively in use by each active domain, I.e. FTP, HTTP, SMTP, IMAP, POP3, MySQL etc.
• What maintenance periods do you have available, if at all?
  • What volume of traffic and when are your quietest periods?
• Do you have dedicated infrastructure, sharded, split by service/role?
  • Can parts of the infrastructure be migrated as an individual component
• List core functionality from the domain for testing purposes
  • Ideally this should be wrapped in unit tests as both functional
    • Examples are email, upload (permissions), adding/editing/removing users
• How many servers are you migrating?
  • Large quantities should be automated.
• How critical is the site/service
  • Does it stop 80 staff working?

Specific

• Services
  • Ensure services are initially installed on new server(s).
  • List all configuration files for a particular service (tree).
    • Ensure configuration between each service are identical or compromises are made.
  • List data directories for each service I.e. /var/lib/mysql
    • Can data be transferred automically.
    • Can services be replicated and brought into sync
    • Can data be back filled?
      • I.e Are large log tables required to make the site functional, what is the minimal effort required to bring the site functional?
• SSL
  • Ensure valid certificate exists for any CDN, sub-domain, domain.
• Email
  • Are there any special firewall, configuration requirements?
• DNS
  • Lower the TTL for a domain your preparing to transfer (if possible)
    • Cannot rely on low TTLs, these are cached amongst large corporates, ISPs etc.
  • Ensure the domain is bound to a unique VIP on new servers, if DNS resolution fails, you can put a header(‘Location 10.10.10.10′); in the old site to ensure the domain will resolve correctly.
    • Test this prior to transfer for both HTTP & HTTPS if applicable
• Permissions
  • Do you upload content to the servers, does your code write to the filesystem?
    • Is this writtable?
  • Under which user/group is this written?
• Cache
  • Does your site make use of distributed or local cache?
    • Could there be collisions between different sites, I.e. Do you prefix cache key names based on site?
• Networking
  • Can specific services be migrated prematurely?
    • Repoint via iptables, and keep an eye on bytes passing through the interface till redundant
• Security
  • Were there any firewall restrictions that need to be replicated, either hardware, iptables etc.
  • Chrooted, users copied, ssh keys copied.
• Optimizations
  • Were there any special optimizations, I.e. DnsMasq?, sysctl changes?
• Load balancing
  • Ensure each domain has its own VIP – HTTP_HOST fails in HTTP 1.0 clients
  • Ensure wild cards are not specified within virtual hosts – see above
  • Ensure sites with load balancing and over SSL use TCP requests correctly, in addition see first point.
  • ifdown each VIP in the webserver pool, does it failover with the correct site on all nodes?
• Monitoring
  • If previously had monitoring on servers (should do), has this been replicated to new servers?
• Database (Will vary depending on setup)
  • Is the database replicated?
    • Take LVM snapshots of the raw data on slave and rsync to new servers.
      • Ensure to change configuration such as server id’s, permissions on master, firewall, start service and start replication. Will be ready to start replicating with correct binlog positions etc.
• Other general changes
  • Are there customizations to /etc/hosts get sites working?

Let me know if there is anything you think I have missed.

Recently we had an issue with one of our hosting providers load balancing (LVS), which resulted in some very small outages. As a result we decided to setup our own load balancing that we had full control over, and could manage ourselves. In addition to choosing a better suited weighting algorithm.

Each webserver is setup using ucarp an implementation of Common Address Redundancy Protocol (CARP) allowing failover of a single Virtual IP (VIP) for high availability. We bound multiple VIPs for each host as we noticed some HTTP 1.0 clients incorrectly sending the host address to the server.

There are many ways you can then proxy the webservers and load balance, however we decided to use haproxy. This can also be acheived by pound, apache mod_proxy, mod_backhand etc.

In order to setup ucarp & haproxy:

apt-get install -y haproxy ucarp


Modify /etc/network/interfaces giving each interface a unique ucarp-vid and adjust ucarp-advskew for weighting on each server (increment by one for each server) and set ucarp-master to yes if it is to be the master. Modify the configuration below appropriately.

# The primary network interface
auto eth0
iface eth0 inet static
        address   10.10.10.2 # IP address of server
        netmask   255.255.255.255
        broadcast 10.10.10.10
        gateway   10.10.10.1
        ucarp-vid 3
        ucarp-vip 10.110.10.20 # VIP to listen to
        ucarp-password password
        ucarp-advskew 10
        ucarp-advbase 1
        ucarp-facility local1
        ucarp-master yes
iface eth0:ucarp inet static
        address 10.10.10.20# VIP to listen to
        netmask 255.255.255.255

To bring the interface up, simply run the following:

ifdown eth0; ifup etho0
ifdown eth0:ucarp; ifup eth0:ucarp


In order to configure haproxy:

sed -i -e 's/^ENABLED.*$/ENABLED=1/' /etc/default/haproxy


Reconfigure apache to listen only on local interfaces (/etc/apache2/ports.conf):
So replace “Listen 80″ with


Listen 10.10.10.20:80
Listen 10.10.10.2:80


edit /etc/haproxy/haproxy.cfg

listen web 10.10.10.20:80
        mode http
        balance leastconn
        stats enable
        stats realm Statistics
        stats auth stats:password
        stats scope .
        stats uri /stats?stats
        #persist
        server web1 10.10.10.2:80 check inter 2000 fall 3
        server web2 10.10.10.3:80 check inter 2000 fall 3
        server web3 10.10.10.4:80 check inter 2000 fall 3
        server web4 10.10.10.5:80 check inter 2000 fall 3
        server web5 10.10.10.6:80 check inter 2000 fall 3

Then restart haproxy with /etc/init.d/haproxy restart

After changing your DNS to point to 10.10.10.20 you will be able to see the traffic balanced between the servers by going to the URL http://10.10.10.20/stats?stats with the credentials assigned above and see the bytes balanced between the servers listed.

Some other alternatives are:

• Wackamole
• LVS

The Project

I was recently working on a project to expose our trading systems via XmlRpc, Rest and SOAP. It was quite an interesting project, which took two of us three weeks to develop (Amongst other things).

This involved creating a testbed, that would automatically generate the payload and response for each protocol. The parameters are introspected for each class method capturing each parameters data type, allowing for user input via standard html forms. This is probably best described with a picture or two.

Most of the documentation was generated via reflection and comments within the docblocks, parameters, notes were also generated making it quick and simple to update. In addition to parsing the start and end line of each method for any applicable error codes/faults that may be returned.

Zend Framework

Using the Zend Framework for the first time in a commercial product was not exactly hassle free, and still has quite a few issues with its webservices implementation. Currently there seems to be quite a bit of confusion regarding its Rest implementation and whether it is to be merged, would be great if someone clarify this.

The main issue I found with the Zend Frameworks implementation of XmlRpc and Rest is that it assumes that the payload it receives is valid. During my development, I tended to mix the payloads from SOAP, XmlRpc and Rest, yet it would assume that simple_xml can parse the input.

For example $this->_sxml is assumed to be a valid object, if not you will either get invalid method call or an undefined index, which doesn’t render well for an xmlrpc server.

    /**
     * Constructor
     *
     * @param string $data XML Result
     * @return void
     */
    public function __construct($data)
    {
        $this->_sxml = simplexml_load_string($data);
    }

    /**
     * toString overload
     *
     * Be sure to only call this when the result is a single value!
     *
     * @return string
     */
    public function __toString()
    {
        if (!$this->getStatus()) {
            $message = $this->_sxml->xpath('//message');
            return (string) $message[0];
        } else {
            $result = $this->_sxml->xpath('//response');
            if (sizeof($result) > 1) {
                return (string) "An error occured.";
            } else {
                return (string) $result[0];
            }
        }
    }

One of the main issues with Rest was that it needed ksort when using the Rest client as the arguments were not necessarily passed in order. This can be “rest.php?method=x&arg1=1&arg0=0″ and it would interpret each arg in the order it received them. This should be sorted in the next release of the ZF.

As the webservices we are exposing needs to have quite good performance with the number of transactions it will be handling and the amount of reflection that Zend Server Reflection (Only noticed after I started profiling) performs and I wanted to optimize any overhead, which got me looking at Zend_XmlRpc_Server_Cache. First thing I did was profile Zend_XmlRpc_Server_Cache, which added a considerable amount of overhead. Looking at its implementation, it uses serialize, which is a relatively slow process and should be avoided, unless there is a large overhead in initializing objects. So most likely Zend_XmlRpc_Server_Cache will not add any benefit. And var_dump’ing out the reflection in XmlRpc spews out a shocking amount of information on some fairly large classes.

  if (!Zend_XmlRpc_Server_Cache::get($cacheFile, $server)) {

  }

Generating WSDL

I tried a number of WSDL generators including the implementation in incubator for ZF, which I found to be the best, yet I still had to write a large chunk of the WSDL by hand and adapt it.

The best way to debug is to run the soap client with verbose mode on, and it will typically tell you the issue straight away.

• Zend_Soap_AutoDiscover: Duplicates an operation in WSDL for methods with parameters that are optional. (ZF-2642)
• Zend_Soap_AutoDiscover: If missing the @return in your docblock the message response in the WSDL is not generated. (ZF-2643)
• AutoDiscover duplicates response if using set class multiple times. (ZF-2641 )
• One of my colleagues typically writes their docblocks with “@return int, comment.”, which the comma caused return types to be dropped with AutoDiscover, more of an issue with Zend Server Reflection

Other odd issues

Raw input bug

Some other obscurities I found was capturing the raw request data. In our local development environment reading the raw request input, and then once again within the Zend Frameworks appears to work fine. However in our pre-production environment it fails to read the second request to read the raw request. (PHP 5.2.2)


if (!isset($HTTP_RAW_POST_DATA)){
$HTTP_RAW_POST_DATA = file_get_contents('php://input');
}


It does seem a little odd that the XmlRpc does not check whether $HTTP_RAW_POST_DATA isset before attempting to re-read raw input.

Internal error: Wrong return type

Whilst running PHPUnit I noticed a very weird quirk in our local dev environment, which essentially did the following… You would expect this to output the contents of an array right? Well between the method call to x and return the result back to method y returns NULL. This is very obscure and i’ve never seen anything like it especially considering it is explicitly set. I had a number of colleagues check this, which had us all scratching our heads. Has anyone else seen anything similar to this?

class test {

  public function x() {
    $ret = array();
    for(...) {
      $ret[] = $row;
    }
    return $ret;
  }

  public function y() {
    $response = $this->x();
    var_dump($response);
  }
}

$t = new test();
$t->y();

Conclusion

Overall the project went pretty well, I’m confident it is now stable especially with the number of tests we ran against it. It is adaptable to other projects that we may need to expose via an API, in total there is about 6000 lines of code alone just testing the 3 different protocols it supports. I would have rather avoided the Rest implementation with ZF as it still needs a lot of work, however XmlRpc is a lot more stable and I would quite happily use again. As there is a lot of overhead with reflection it is not the fastest implementation and was contrasted to some of the heavier web pages we have for some simple functionality. It would be ideal to replace the reflection with something lighter such as an array with the corresponding methods, parameters and types, however I would have to look into that if performance did become a major issue.

PS. Just to note I used PHP’s in built soap server.

I’ve been using Zend Studio for Eclipse (beta) for several weeks in a rewrite of a framework and numerous sites at work and overall I really like the IDE. Its got some great features and being based on the eclipse project makes it really extensible and customizable. With debugging, profiling, code completion, code formatting and more enabled can help with productivity.

A complete list of features can be found at Zend.

Zend Studio for Eclipse consumes quite a lot of memory and the recommended amount of RAM for eclipse based applications is 2GB, however you can control the amount of memory that eclipse will use by editing the zendStudio.ini file.

Whilst I do like the IDE I have found a number of issues with Zend Studio:

Bugs

• 1. There have been a number of issues revolving around the SVN implementation in Zend Studio for Eclipse which causes the application to hang (SVN support is provided to eclipse by a 3rd party plugin (a company called Polarion)):
  • a) When committing files it locks entire directories and often hangs making Zend Studio for Eclipse unusable, if you have files that are not saved and attempt to do so it queues it as a users pending tasks and because the commit has stalled you cannot save the file.

    To resolve this I have to kill the process for Zend Studio, shell into the server and cleanup the the projects src, and sometimes have to re-checkout the directories in a project

• 2. When developing via a samba share, it prompts with an incorrect error and does not attempt to re-authenticate when the samba share needs to re-connect and/or does recognise that it is talking via a remote device.
   
• 3. Auto format adds extra braces to statements, causing syntax errors, strips all comments out of files!
   
• 4. Importing an auto format does not seem to work correctly.
   
• 5. When working with multiple open files it can overwrite the contents with another, I believe this is the case with files of a similar name (I’ve only had this occur once, however a colleague experiences this quite frequently).
   
• 6. Modified file names are prefixed with “>” and when searching for files by pressing a character will not go to that file in PHP Explorer
   
• 7. Templates do not always get replaced e.g. If the system is slow or you type fnc real quick it doesn’t replace with the template for a function.
   
• 8.
  

  If you cause a syntax error, whether deliberate or not, a number of items such as auto formatting does not work, or you cannot open the php manual from the contexts menu for a php function. For example if I type “str_shuffle” then use a context menu to it does not open a manual.

• 9. introducing a syntax error and then removing the syntax error doesn’t clear until you save the document.

  

Resolved

• 1. There was a bug in automatically updating eclipse, which never seemed to work however in the latest release (beta 2) this has now been resolved.

In high performance web applications you will always have bottlenecks within your application. Identifying these bottlenecks and optimizing is a tedious task and typically show themselves underload. A single bad/unindexed query can bring a server to its knees. A large number of rows will also help to highlight any poor queries, and on very large datasets you may come to the point where you may have to make decisions whether to denormilize database schema.

Explain each page

Whilst I develope sites, I typically print out all queries, EXPLAIN each select statement at the bottom of each page, and highlight it red if its doing a full table scan, temp tables or a filesort. As well as displaying SHOWS INDEXES FROM TABLE…

Not only will it help you to optimize sites, you can also see bad logic and areas to optimize such as a query for each loop when looking through a users table for example.

How do you identify where bottlenecks occur?

One of my favourite linux commands lately is the watch command. For Mac users you can get this from macports via “sudo port install watch”. Also a few other handy applications are mysqlreport, mytop.

# Appends file with processlist
watch -n1 "mysqladmin -uroot processlist >>watch.processlist.txt"

# Count the number of locked processes
watch -n1 "mysqladmin -uroot processlist | grep -i 'lock' | wc -l ";

# Count the number of processes sleep
watch -n1 "mysqladmin -uroot processlist | grep -i 'sleep' | wc -l ";

# Run a specific query every second
watch -n1 "mysql -uadmin -p`cat /etc/psa/.psa.shadow` trade_engine --execute "SELECT NOW(),date_quote FROM sampleData WHERE 1=1 AND permission = '755' AND  symbol='IBZL' GROUP BY date_quote;" "

# Emails mysqlreport every 60 seconds
watch -n60 mysqlreport --all --email andrew@email.com

# Displays process list as well as appending the contents to a file
watch -n1 "mysqladmin -uadmin -p`cat /etc/psa/.psa.shadow` processlist | tee -a process.list.txt"

Watching the processlist is very handy in identifying locked, sleeping or sorting process states. If you have a large number of locked processes you typically should change the table type to INNODB, which supports row level locking. if you have a large number of sleeping connections, and you have persistent connections enabled, most likely indicates that connections are not being reused.

Running a specific query every second is exceptionally handy, the example I gave indicates whether one of our crons is correctly functioning and as each row is inserted you can watch something being either inserted or updated. mysqlreport gives numerous peices of information, extremely helpful in identifying issues, you can see more indepth at hackmysql.com/mysqlreportguide.

Look at the mysql slow query log and optimize each query starting with the most common, think whether you have to execute that query at all and use a cache such as memcached.

I also typically tend to look at the following:

• vmstat -S M
• ps axl | grep -i ‘mysql’
• pstree –G
• free –m

Reference:
http://dev.mysql.com/tech-resources/presentations/presentation-oscon2000-20000719/index.html

I’ve had alot of experience with other programming languages, however I had to learn C++ from scratch in a very short period of time, a number of weeks ago. This was to develop a real-time stock quote client, the goal was simply to push data from remote servers into our databases, filter what messages it would receive and get something up and running fast as deadlines lingured. This was simple enough, however with the rush the application had its inherent flaws, due to my lack of knowledge of C++, the API, and the goals it had to acomplish.

I’ve since had time to learn a little more C++ and limited time to design the application properly.

The Problems

The core problems with the application:

• refactor, refactor, refactor
• database connection pooling
• Query remote CSP servers^*1
• Query remote CSP servers^*1 from PHP
• Configuration management
• Monitoring
• Flexible Database schema
  • Add columns to database schema dependent on datatype.
  • Log messages in XML per trade message with date/time, columns and values.

Compatible GCC

The first issue was that I used an API from interactive-data, which was compatible with “gcc version 3.2.3″ and is not kept up to date. This meant compiling a compatible gcc from source for 32bit platforms only.


./configure --prefix=/usr/local/gcc/ --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --enable-languages=c,c++,objc,obj-c++

make bootstrap
cd gcc
make
sudo make install


Once having a compatible compiler, I then had to make modifications to the Makefile, move a number of lib/so files to get MySQL to compile and get things working. Unfortunately I did not have a local machine to attach a debugger, so everything was trial and error from the command line with g++32, which makes life difficult identifying runtime errors.

The Logic

Once everything was in place, the logic was fairly simple, foreach field retreived construct a query with the field name, checking the fields values datatype whether it be a datetime, varchar etc. Insert each trade message in a table, update another and if either failed, check if the fault was due to a missing column, if so add it and re-execute queries.

The problem soon arrises when you need to know when each column was actually last updated, with which field, value, datetime and the last insert id for the trade messages. Whilst looping through each trade message, I constructed an XML schema containing the above, however the tricky part is to ensure that it only updates the fragment matching the field in the schema. Not an ideal format to query from a database.

Storing Data

One of the fundemental issues is managing and storing data. For some exchanges you don’t want to store every trade message; simply storing the current data for a number of instruments is enough. Which servers or databases do you peg data to? If one database goes down, how do you handle fault tolerance? MySQL cluster is not a feasible solution, requiring multiple servers and large memory requirements per installation. The databases are highly susceptible to curruption or faults. Also particular sites may require data from multiple exchanges, so seperating trade messages per database is not also ideal.

All of this fundamentally comes down to configuration management.

Configuration

One of the fundamental aspects of the application is configuration management. This contains where data should be stored for a particular exchange, the type of data to store, whether it is per trade message, current data or both. Which servers to source data from, whether it is real time or delayed, whether to source data for bonds, equities, automated trades etc… All queries can be grouped, or to query remote servers. Some of the products for example just for the London Stock Exchange is:

• London Stock Exch – Covered Warrants L1
• London Stock Exch – International Equity Mkt Service L1
• London Stock Exch – International Equity Mkt Service Level 2
• London Stock Exch – UK Equity Mkt Service L1
• London Stock Exch – UK Equity Mkt Service Level 2 (Depth Refresh)
• London Stock Exchange: UK Equity Market Service Level 2

All of which is stored in several database tables and managed via a MySQL database and PHP frontend.