Development, Analysis And Research

Amazon ELB – monitoring packet count and byte size with Amazon Cloudwatch and VPC flow logs

16 Dec 2015

We recently exceeded the capacity for one of our Amazon ELBs in a 60 second period. One of the points of our debrief was to monitor the network from the ELB. Amazon doesn’t provide statistics from ELBs to Amazon cloudwatch. So I came up with the following:

Using amazons apis capture the network interfaces with attachment.instance-owner-id for “amazon-elb”.
For each network interface create a cloudwatch metric filter with the network interface.

Running the above in a lambda function with a scheduled event every 5 minutes will create the relevant filters and cloudwatch logs. This takes the overhead out of analysing the VPC flow logs.

Amazon ELBs rotate their servers and the attached ENIs will obviously rotate, so the second part is to identify whether the ENIs have been released and clean up.

Amazon ELBs will add/remove nodes to/from DNS, which should be faster than a 5 minute period (the current minimum for a scheduled task).
As such the time taken to start graphing new ENIs added to the ELBs will not be immediate. The above graphs should help us to infer an increase in network traffic through the ELB and to observe to set sensible alarms.

No Comments
Tags: Cloudwatch, ELB, ENI, Lambda, VPC, VPC Flow Logs

SSL – Amazon ELB Certificates

In: General|Linux

20 Jun 2015

I always forget the order to concatenate SSL certificates and to upload to Amazon ELBs as well as commands to verify SSL certificates.

Extract dates from certificate

echo | openssl s_client -connect co.photobox.com:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Feb 10 00:00:00 2015 GMT
notAfter=Jun  9 23:59:59 2016 GMT

cat 14619087repl_1.crt | openssl x509 -noout -dates

List DNS entries for SANs certificates

echo | openssl s_client -connect co.photobox.com:443 2>/dev/null | openssl x509 -noout -text | grep DNS | sed -r 's/[ ]+DNS://g;s/,/\n/g'

Extract certificate

openssl x509 -inform PEM -in xxx.crt

Concatenation of SSL Certificates for Comodo

cat COMODORSAOrganizationValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt >> gd_bundle.crt;

Upload SSL certificates

aws iam upload-server-certificate \
    --server-certificate-name sans-photobox-com-intermediate-expires-09-06-2016-v2 \
    --certificate-body file://14619087repl_1.crt \
    --private-key file://../co.key \
    --certificate-chain file://gd_bundle.crt

No Comments
Tags: Amazon, ELB, SSL, X509

Automatically update Amazon ELB SSL Negotiation Policies

In: General

31 May 2015

A quick script to update Amazon ELB SSL Negotiation policies for all Amazon ELBs.
The security policies are managed from Amazon and will only update the predefined security policies, choosing the latest one.

More information available on AWS predefined negotiation policies.

Sample output…

No Comments

Amazon Opsworks – Dependent on Monit

In: AWS|Linux

17 Oct 2014

I recently converted a number Amazon Opsworks instances to be time based, so that they shut down at the end of the working day.
After doing so, I kept seeing that the Opsworks life cycle events always being marked as in progress.

Every morning the instances would be marked booting and would be able to SSH into the instances and the applications would be running on them.
When I finally managed to investigate this morning I figured out why this was occurring and was a little surprised how amazon run the opsworks agent.

We have a number of custom monit configurations to ensure that our own applications are running. These do not interfere with the Amazon Opsworks monit configuration.
The development/qa team occasionally stop monit when doing development work or switching branches on these instances.

As can be seen from the following…

[UTC Oct 9 09:50:14] info : monit: generated unique Monit id 437edc13afbe8e51fde4501f9af8db8f and stored to ‘/var/lib/monit/id’
[UTC Oct 9 09:50:14] info : Starting monit daemon
[UTC Oct 9 09:50:14] info : ‘ip-10-75-15-96.eu-west-1.compute.internal’ Monit started
[UTC Oct 9 09:54:35] info : monit daemon with pid [6927] killed
[UTC Oct 9 09:54:35] info : ‘ip-10-75-15-96.eu-west-1.compute.internal’ Monit stopped

Amazon opsworks doesn’t use any form of upstart/init.d/systemd so when servers boot up its reliant on monit to be started. If the last state is monit being stopped this will fail to run opsworks and you will see the above state.

$ egrep -rin opsworks /etc/rc* /etc/init.d/

I’ve asked our development team to disable the correct processes within monit

E.g.

$ monit unmonitor node_checkout_process

It would be sensible not to rely on monit and have a relevant script to spawn the process on server restart.

root@nodejs-dev-andy:/opt/aws/opsworks/current/bin# ./opsworks-agent-cli agent_report
Pidfile /var/lib/aws/opsworks/pid/opsworks-agent.pid present but no matching process running - cleaning up

AWS OpsWorks Instance Agent State Report:

Last activity was a "configure" on 2014-09-26 18:03:22 UTC
Agent Status: No AWS OpsWorks agent running
Agent Version: 325, up to date

Hopefully Amazon will improve the way the opsworks agent is spawned.

(1) Comment
Tags: aws, Bash, monit, opsworks

Awk craziness: Processing log files

In: General|Linux

13 Apr 2013

A while back I wrote an awk script to process a custom log file to work out how an application was performing over a period of time.
The following script takes a timestamp in field one $1 and also an arbitrary field $10 in this case.

For every 100 lines of the log file processed it would calculate the min, max, avg, stddev for rolling stats.

zcat Summary.2012-12-09.gz | grep '2012-12-09 20:[234]' | awk '
  BEGIN{
    size=50;
    tmax = -999999999;
    tmin = 999999999;
  }
  {   # Accumulate basic data
      cnt[$10]++;
      item[++n] = $10;
      if ($10 > max) max = $10; if ($10 < min) min = $10;
      if ($10 > tmax) tmax = $10; if ($10 < tmin) tmin = $10;
  }
  {
    mod=NR%size;
    if(NR<=size) {
      count++;
    } else {
      sum-=array[mod];
      min=max=$10;
      
    };
    total+=$10;
    sum+=$10; sumsq+=$10*$10;
    array[mod]=$10;  
    print $1" "$2 "\tvalue: " $10 "\tmin: " min "\tmax: " max "\tavg: " sum/count "\tstddev: " sqrt(sumsq/NR - (sum/NR)**2)  "\t"
  }
  END {
    printf("\n\n# Lines processed = %d\n", NR);
    printf("# Total = %d\n", total);
    printf("# Sqrt = %d\n", sqrt(sumsq/NR - (sum/NR)**2));
    # Print Descriptive Statistics
    printf("# Count = %d\n", n);
    printf("# Min = %d\n", tmin);
    decile = 1;
    for (decile = 10; decile < 100; decile += 10) {
      idx = int((decile * n) / 100) + 1;
      printf("# %d%% decile = %d\n", decile, item[idx]);
      if (decile == 50)
        median = item[idx];
    }
    printf("# Max = %d\n", tmax);
 
    printf("# Median = %d\n", median);
    for (i in cnt) {
      if (cnt[i] > cnt[mode])
        mode = i;
    }
    printf("# Mode = %d\n", mode);
  }
' | less
 
...
2012-12-09 20:20:01     value: 18       min:    max: 18 avg: 5.8        stddev: 6.20967 
2012-12-09 20:20:01     value: 7        min:    max: 18 avg: 6  stddev: 5.68624 
2012-12-09 20:20:01     value: 7        min:    max: 18 avg: 6.14286    stddev: 5.27605 
2012-12-09 20:20:01     value: 4        min:    max: 18 avg: 5.875      stddev: 4.98592 
2012-12-09 20:20:01     value: 2        min:    max: 18 avg: 5.44444    stddev: 4.85595 
2012-12-09 20:20:02     value: 3        min:    max: 18 avg: 5.2        stddev: 4.66476 
2012-12-09 20:20:02     value: 4        min:    max: 18 avg: 5.09091    stddev: 4.46103 
2012-12-09 20:20:02     value: 104      min:    max: 104        avg: 13.3333    stddev: 27.6687 
2012-12-09 20:20:02     value: 2        min:    max: 104        avg: 12.4615    stddev: 26.7542 
2012-12-09 20:20:02     value: 25       min:    max: 104        avg: 13.3571    stddev: 25.9824 
2012-12-09 20:20:03     value: 3        min:    max: 104        avg: 12.6667    stddev: 25.234  
2012-12-09 20:20:03     value: 3        min:    max: 104        avg: 12.0625    stddev: 24.5445 
2012-12-09 20:20:03     value: 3        min:    max: 104        avg: 11.5294    stddev: 23.907  
2012-12-09 20:20:03     value: 2        min:    max: 104        avg: 11 stddev: 23.3357 
2012-12-09 20:20:03     value: 6        min:    max: 104        avg: 10.7368    stddev: 22.7407 
2012-12-09 20:20:04     value: 4        min:    max: 104        avg: 10.4       stddev: 22.2135 
2012-12-09 20:20:04     value: 3        min:    max: 104        avg: 10.0476    stddev: 21.7354 
2012-12-09 20:20:04     value: 2        min:    max: 104        avg: 9.68182    stddev: 21.3017 
2012-12-09 20:20:04     value: 70       min:    max: 104        avg: 12.3043    stddev: 24.1938 
2012-12-09 20:20:04     value: 4        min:    max: 104        avg: 11.9583    stddev: 23.7425 
2012-12-09 20:20:04     value: 12       min:    max: 104        avg: 11.96      stddev: 23.2628 
2012-12-09 20:20:04     value: 14       min:    max: 104        avg: 12.0385    stddev: 22.8144 
2012-12-09 20:20:04     value: 6        min:    max: 104        avg: 11.8148    stddev: 22.417  
2012-12-09 20:20:04     value: 5        min:    max: 104        avg: 11.5714    stddev: 22.0493 
2012-12-09 20:20:04     value: 2        min:    max: 104        avg: 11.2414    stddev: 21.7361 
2012-12-09 20:20:04     value: 30       min:    max: 104        avg: 11.8667    stddev: 21.6344 
2012-12-09 20:20:05     value: 5        min:    max: 104        avg: 11.6452    stddev: 21.3172 
2012-12-09 20:20:05     value: 22       min:    max: 104        avg: 11.9688    stddev: 21.0587 
2012-12-09 20:20:05     value: 4        min:    max: 104        avg: 11.7273    stddev: 20.7821 
2012-12-09 20:20:05     value: 2        min:    max: 104        avg: 11.4412    stddev: 20.54   
2012-12-09 20:20:05     value: 25       min:    max: 104        avg: 11.8286    stddev: 20.3701 
2012-12-09 20:20:05     value: 27       min:    max: 104        avg: 12.25      stddev: 20.2394 
2012-12-09 20:20:05     value: 3        min:    max: 104        avg: 12 stddev: 20.0203 
2012-12-09 20:20:06     value: 2        min:    max: 104        avg: 11.7368    stddev: 19.8198 
2012-12-09 20:20:06     value: 1        min:    max: 104        avg: 11.4615    stddev: 19.6375 
2012-12-09 20:20:06     value: 2        min:    max: 104        avg: 11.225     stddev: 19.4467 
2012-12-09 20:20:06     value: 36       min:    max: 104        avg: 11.8293    stddev: 19.5846 
2012-12-09 20:20:06     value: 3        min:    max: 104        avg: 11.619     stddev: 19.3968 
2012-12-09 20:20:06     value: 27       min:    max: 104        avg: 11.9767    stddev: 19.3096 
2012-12-09 20:20:06     value: 2        min:    max: 104        avg: 11.75      stddev: 19.1467 
2012-12-09 20:20:07     value: 5        min:    max: 104        avg: 11.6       stddev: 18.9589 
2012-12-09 20:20:07     value: 2        min:    max: 104        avg: 11.3913    stddev: 18.8039 
2012-12-09 20:20:07     value: 5        min:    max: 104        avg: 11.2553    stddev: 18.6256 
2012-12-09 20:20:07     value: 4        min:    max: 104        avg: 11.1042    stddev: 18.4597 
2012-12-09 20:20:07     value: 2        min:    max: 104        avg: 10.9184    stddev: 18.3156 
2012-12-09 20:20:07     value: 4        min:    max: 104        avg: 10.78      stddev: 18.1574 
...
 
# Lines processed = 9863
# Total = 80137
# Sqrt = 15
# Count = 9863
# Min = 1
# 10% decile = 8
# 20% decile = 1
# 30% decile = 3
# 40% decile = 3
# 50% decile = 17
# 60% decile = 7
# 70% decile = 5
# 80% decile = 1
# 90% decile = 4
# Max = 140
# Median = 17
# Mode = 2

No Comments
Tags: awk, Linux, log processing

Elastic Search Presentation

In: General|PHP

29 Oct 2012

I’ve implemented a number of sites with Elastic Search and recently had a two-day spike to implement Elastic Search for Photobox’s backoffice systems. As a result I have since wrote a presentation and presented at Photobox on Friday. I have posted the slides for the presentation here…

I wrote this presentation using rvl.io – reveal.js, however unfortunately does not allow you to nicely export the presentation to pdf or other non html formats. Best viewed full screen…

A few other interesting presentations I came across whilst writing up these slides.

Scaling massive elastic search clusters – Rafał Kuć – Sematext from Rafał Kuć

You know, for search. Querying 24 Billion Documents in 900ms from Jodok Batlogg

No Comments
Tags: Elastic Search, Elastica, PHP

Requests per second from Apache logs

In: General|Linux

22 Sep 2012

Whilst recently looking into usage of SSL, I needed to analyse some logs from apache to determine requests per second.

Initially I wrote a quick bash script to do this, however figured it would be handy to elaborate on this, as a result I have posted a perl script on github at apache-log-stats

    total_requests=$(zcat b2f01_ApacheAccess.2011-12-05.gz | \
    sort -b -k4.9,4.12 -k4.5b,4.7Mb -k4.2,4.3 -k4.14,4 | \
    awk -F '[ [/:]' '{print $7"-"$6"-"$5":"$8":"$9":"$10}' | \
    sort | uniq -c  | sort -g | tail -n1);

    echo $total_requests;

And the perl script.

andrew@andrew-OptiPlex-7010:/var/www/logs/bin$ ./apache-log-stat $(ls ../logs/b2f*_ApacheAccess.2011-12-05.gz)
Processing
====================

../logs/b2f01_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.92    variance_requests_per_second: 25.93 total_requests:  413603 stddev_requests_per_second: 5.09    max_requests_per_second:   57   
../logs/b2f02_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 6.05    variance_requests_per_second: 26.55 total_requests:  417711 stddev_requests_per_second: 5.15    max_requests_per_second:   45   
../logs/b2f03_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 6.09    variance_requests_per_second: 26.64 total_requests:  419073 stddev_requests_per_second: 5.16    max_requests_per_second:   48   
../logs/b2f04_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.07    variance_requests_per_second: 12.82 total_requests:  325374 stddev_requests_per_second: 3.58    max_requests_per_second:   29   
../logs/b2f05_ApacheAccess.2011-12-05.gz - avg_requests_per_second:    0    variance_requests_per_second:    0  total_requests:    0    stddev_requests_per_second:    0    max_requests_per_second:    0   
../logs/b2f06_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.20    variance_requests_per_second: 13.29 total_requests:  338946 stddev_requests_per_second: 3.65    max_requests_per_second:   30   
../logs/b2f07_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.09    variance_requests_per_second: 12.92 total_requests:  330180 stddev_requests_per_second: 3.59    max_requests_per_second:   28   
../logs/b2f08_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.08    variance_requests_per_second: 12.78 total_requests:  334373 stddev_requests_per_second: 3.57    max_requests_per_second:   33   
../logs/b2f09_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.05    variance_requests_per_second: 12.66 total_requests:  327368 stddev_requests_per_second: 3.56    max_requests_per_second:   28   
../logs/b2f10_ApacheAccess.2011-12-05.gz - avg_requests_per_second:    0    variance_requests_per_second:    0  total_requests:    0    stddev_requests_per_second:    0    max_requests_per_second:    0   
../logs/b2f11_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.51    variance_requests_per_second: 15.00 total_requests:  370115 stddev_requests_per_second: 3.87    max_requests_per_second:   35   
../logs/b2f12_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.43    variance_requests_per_second: 14.41 total_requests:  357580 stddev_requests_per_second: 3.80    max_requests_per_second:   37   
../logs/b2f13_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.47    variance_requests_per_second: 15.09 total_requests:  366275 stddev_requests_per_second: 3.88    max_requests_per_second:   36   
../logs/b2f14_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.39    variance_requests_per_second: 14.51 total_requests:  363630 stddev_requests_per_second: 3.81    max_requests_per_second:   34   
../logs/b2f15_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.40    variance_requests_per_second: 14.40 total_requests:  361463 stddev_requests_per_second: 3.80    max_requests_per_second:   41   
../logs/b2f16_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.87    variance_requests_per_second: 17.03 total_requests:  403126 stddev_requests_per_second: 4.13    max_requests_per_second:   48   
../logs/b2f17_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.90    variance_requests_per_second: 16.81 total_requests:  404777 stddev_requests_per_second: 4.10    max_requests_per_second:   37   
../logs/b2f18_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.89    variance_requests_per_second: 16.89 total_requests:  407635 stddev_requests_per_second: 4.11    max_requests_per_second:   30   
../logs/b2f19_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.82    variance_requests_per_second: 16.13 total_requests:  402808 stddev_requests_per_second: 4.02    max_requests_per_second:   36   
../logs/b2f20_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.84    variance_requests_per_second: 16.44 total_requests:  404247 stddev_requests_per_second: 4.05    max_requests_per_second:   35   
../logs/b2f21_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.91    variance_requests_per_second: 29.21 total_requests:  583211 stddev_requests_per_second: 5.40    max_requests_per_second:   41   
../logs/b2f22_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.86    variance_requests_per_second: 28.31 total_requests:  575122 stddev_requests_per_second: 5.32    max_requests_per_second:   38   
../logs/b2f23_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.68    variance_requests_per_second: 27.37 total_requests:  571349 stddev_requests_per_second: 5.23    max_requests_per_second:   39   
../logs/b2f24_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.76    variance_requests_per_second: 27.80 total_requests:  571728 stddev_requests_per_second: 5.27    max_requests_per_second:   37   
../logs/b2f25_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.90    variance_requests_per_second: 28.59 total_requests:  583242 stddev_requests_per_second: 5.35    max_requests_per_second:   51   
../logs/b2f26_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.87    variance_requests_per_second: 28.02 total_requests:  576603 stddev_requests_per_second: 5.29    max_requests_per_second:   36   
../logs/b2f27_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.91    variance_requests_per_second: 29.45 total_requests:  584972 stddev_requests_per_second: 5.43    max_requests_per_second:   42   
../logs/b2f28_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.66    variance_requests_per_second: 27.15 total_requests:  561073 stddev_requests_per_second: 5.21    max_requests_per_second:   38   
../logs/b2f29_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.83    variance_requests_per_second: 28.15 total_requests:  578100 stddev_requests_per_second: 5.31    max_requests_per_second:   47   
../logs/b2f30_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.74    variance_requests_per_second: 27.62 total_requests:  574167 stddev_requests_per_second: 5.26    max_requests_per_second:   39   
Total:  avg_requests_per_second: 178.19 total_requests: 12507851    variance_requests_per_second: 581.99    stddev_requests_per_second: 125.99  max_requests_per_second: 1075   


Requests per hour
====================

   00 #                                                                                                           199683 / 12507851 (  1.60% )  var: 12308168   rqs:  55    rqm: 3328
   01                                                                                                             121028 / 12507851 (  0.97% )  var: 12386823   rqs:  33    rqm: 2017
   02                                                                                                              92839 / 12507851 (  0.74% )  var: 12415012   rqs:  25    rqm: 1547
   03                                                                                                              79303 / 12507851 (  0.63% )  var: 12428548   rqs:  22    rqm: 1321
   04                                                                                                              76682 / 12507851 (  0.61% )  var: 12431169   rqs:  21    rqm: 1278
   05                                                                                                              96342 / 12507851 (  0.77% )  var: 12411509   rqs:  26    rqm: 1605
   06 #                                                                                                           141415 / 12507851 (  1.13% )  var: 12366436   rqs:  39    rqm: 2356
   07 #                                                                                                           245247 / 12507851 (  1.96% )  var: 12262604   rqs:  68    rqm: 4087
   08 ###                                                                                                         431089 / 12507851 (  3.45% )  var: 12076762   rqs: 119    rqm: 7184
   09 ####                                                                                                        589031 / 12507851 (  4.71% )  var: 11918820   rqs: 163    rqm: 9817
   10 #####                                                                                                       662950 / 12507851 (  5.30% )  var: 11844901   rqs: 184    rqm: 11049
   11 #####                                                                                                       637200 / 12507851 (  5.09% )  var: 11870651   rqs: 177    rqm: 10620
   12 #####                                                                                                       739012 / 12507851 (  5.91% )  var: 11768839   rqs: 205    rqm: 12316
   13 ######                                                                                                      780787 / 12507851 (  6.24% )  var: 11727064   rqs: 216    rqm: 13013
   14 ######                                                                                                      779051 / 12507851 (  6.23% )  var: 11728800   rqs: 216    rqm: 12984
   15 #####                                                                                                       667872 / 12507851 (  5.34% )  var: 11839979   rqs: 185    rqm: 11131
   16 ######                                                                                                      750655 / 12507851 (  6.00% )  var: 11757196   rqs: 208    rqm: 12510
   17 ######                                                                                                      804304 / 12507851 (  6.43% )  var: 11703547   rqs: 223    rqm: 13405
   18 #####                                                                                                       740101 / 12507851 (  5.92% )  var: 11767750   rqs: 205    rqm: 12335
   19 ######                                                                                                      795684 / 12507851 (  6.36% )  var: 11712167   rqs: 221    rqm: 13261
   20 #######                                                                                                     973406 / 12507851 (  7.78% )  var: 11534445   rqs: 270    rqm: 16223
   21 #######                                                                                                     979602 / 12507851 (  7.83% )  var: 11528249   rqs: 272    rqm: 16326
   22 #####                                                                                                       724108 / 12507851 (  5.79% )  var: 11783743   rqs: 201    rqm: 12068
   23 ###                                                                                                         400460 / 12507851 (  3.20% )  var: 12107391   rqs: 111    rqm: 6674


Comparison by distribution over an hour
====================

   00 ####################                                                                                      199683 / 979602 ( 20.38% )  var: 779919 rqs:  55    rqm: 3328
   01 ############                                                                                              121028 / 979602 ( 12.35% )  var: 858574 rqs:  33    rqm: 2017
   02 #########                                                                                                  92839 / 979602 (  9.48% )  var: 886763 rqs:  25    rqm: 1547
   03 ########                                                                                                   79303 / 979602 (  8.10% )  var: 900299 rqs:  22    rqm: 1321
   04 #######                                                                                                    76682 / 979602 (  7.83% )  var: 902920 rqs:  21    rqm: 1278
   05 #########                                                                                                  96342 / 979602 (  9.83% )  var: 883260 rqs:  26    rqm: 1605
   06 ##############                                                                                            141415 / 979602 ( 14.44% )  var: 838187 rqs:  39    rqm: 2356
   07 #########################                                                                                 245247 / 979602 ( 25.04% )  var: 734355 rqs:  68    rqm: 4087
   08 ############################################                                                              431089 / 979602 ( 44.01% )  var: 548513 rqs: 119    rqm: 7184
   09 ############################################################                                              589031 / 979602 ( 60.13% )  var: 390571 rqs: 163    rqm: 9817
   10 ###################################################################                                       662950 / 979602 ( 67.68% )  var: 316652 rqs: 184    rqm: 11049
   11 #################################################################                                         637200 / 979602 ( 65.05% )  var: 342402 rqs: 177    rqm: 10620
   12 ###########################################################################                               739012 / 979602 ( 75.44% )  var: 240590 rqs: 205    rqm: 12316
   13 ###############################################################################                           780787 / 979602 ( 79.70% )  var: 198815 rqs: 216    rqm: 13013
   14 ###############################################################################                           779051 / 979602 ( 79.53% )  var: 200551 rqs: 216    rqm: 12984
   15 ####################################################################                                      667872 / 979602 ( 68.18% )  var: 311730 rqs: 185    rqm: 11131
   16 ############################################################################                              750655 / 979602 ( 76.63% )  var: 228947 rqs: 208    rqm: 12510
   17 ##################################################################################                        804304 / 979602 ( 82.11% )  var: 175298 rqs: 223    rqm: 13405
   18 ###########################################################################                               740101 / 979602 ( 75.55% )  var: 239501 rqs: 205    rqm: 12335
   19 #################################################################################                         795684 / 979602 ( 81.23% )  var: 183918 rqs: 221    rqm: 13261
   20 ###################################################################################################       973406 / 979602 ( 99.37% )  var:   6196 rqs: 270    rqm: 16223
   21 ####################################################################################################      979602 / 979602 (100.00% )  var:      0 rqs: 272    rqm: 16326
   22 #########################################################################                                 724108 / 979602 ( 73.92% )  var: 255494 rqs: 201    rqm: 12068
   23 ########################################

(1) Comment
Tags: Bash, perl

Hackday @ Everlution

In: General|Linux|MySQL|PHP

25 Jul 2012

At Everlution we did a hack day to help motivate developers and to get the team working together. We started at 8.30am and finished at 5pm. It was pretty impressive what all the teams managed to achieve, with four teams of two developers and a designer.

It was set out in a competition style event with all teams competing against each other with a defined set of points for each task achieved. The tasks were very ambitious considering we had a day to complete the task.

The key task was to analyse a webcam and determine whether a meeting room was in use or not by analysing the web cams jpeg stream.

The motivation for which is that many staff members book meeting rooms and don’t use them or randomly hijack meeting rooms.

Task sheet/Points

#	Task	Points*
1	To build a webpage that tells the user if the boardroom is in use or not by parsing the JPEG stream emitted from the webcam. Scoring: Web page displays in real time whether Boardroom is in use or not. Using own parsing algorithm for detection and not an off-the-shelf library Detection system to not trigger with someone going in/out of Studio 2	+35 points +15 points +10 points
2	To use a Face detection and recognition library to say whether one member of your team is in the Boardroom or not.	+20 points
3	To build an MSN or Skype Bot that will answer a natural language question on whether the Boardroom is in use or not. Must be able to handle at least 3 variations of how question could be asked. Scoring: MSN bot or Skype bot: Using own parsing algorithm for detection and not an off-the-shelf library Also answers whether one of your team is in the room or not	+15 points +10 points
4	To build a native mobile app that will display same information. Scoring: Android or iPhone	+10 points
5	To come up with a name, brand/logo and design for the service. Scoring: Design Build/Implementation	+15 points +10 points
6	Bonus: Good team work	+5 points
7	Bonus: Quality, reliability and elegance of solution	+5 points
8	Sabotaging another team’s effort, being disruptive, not participating in spirit of competition, etc.	-150 points

With regards to “Detection system to not trigger with someone going in/out of Studio 2″. There is a door going into the boardroom and a door just outside going into another office of Everlution. So any body outside the glass should not be detected going into that office.

The implementations varied drastically between teams, some based on motion.

Team 1 – My Team:

Used a C# client application to detect motion in real time from the remote image and/or directly attached camera using AForge.

The sample client was more than enough and simply ran a timed event to push the stats across over http to a PHP based API and store within a MySQL database. The web page then polled periodically to check the availability from the MySQL database, which produces the following data set. If the motion_level was above 0.02 it would indicate the room is occupied. This histogram is indicated at the bottom of the image stream.

The main issue was that it took me till 11am to install Visual Studio. Additionally the implementation was improved by adjusting the motion detection method depending on whether the stream was sourced from the camera itself or by the image, which was updated every two seconds.

{

   "boardroom" : {

      "data" : {

         "date_created" : "2012-07-24 00:14:19",

         "name" : "boardroom",

         "signal_time" : "24/07/2012 00:14:20",

         "fqdn" : "HP28458242993",

         "motion_level" : "0.000406901",

         "detected_objects_count" : "-1",

         "has_motion_been_detected" : "",

         "id" : "19945",

         "motion_alarm_level" : "0.015",

         "flash" : "0"

      },

      "last_used" : {

         "date_created" : "2012-07-24 00:14:19",

         "name" : "boardroom",

         "signal_time" : "24/07/2012 00:14:20",

         "fqdn" : "HP28458242993",

         "motion_level" : "0.000406901",

         "detected_objects_count" : "-1",

         "has_motion_been_detected" : "",

         "id" : "19945",

         "motion_alarm_level" : "0.015",

         "flash" : "0"

      },

      "is_free" : true

   }

}

AForge is an excellent library and provides the following…

AForge.Imaging – library with image processing routines and filters
AForge.Vision – computer vision library
AForge.Neuro – neural networks computation library
AForge.Genetic – evolution programming library
AForge.Fuzzy – fuzzy computations library
AForge.MachineLearning – machine learning library
AForge.Robotics – library providing support of some robotics kits
AForge.Video – set of libraries for video processing

In order to achieve face detection, I was going to use OpenCV, however ran out of time to fully implement. As such we implemented face.com to attempt face recognition. Unfortunately for all implementations this API failed to recognise anyone.

Additionally an MSN bot was built on put on top although did not implement any real natural language implementation.

The results in action

Team 2:

The implementation was based on PHP Imagick using edge detection by enhancing the images with edgeImage and creating composite images.

Two sets of metrics were extracted for statistical analysis from the image. For all channels the mean, stddev from each image used (The raw image, the composite). A histogram was also generated for each channel with the std dev, variance and skew.

Team 2 were to use two important metrics that they did not manage to implement, however their intention was too…

Edge density (from http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1517&context=infopapers) , and
Image energy ( I would calculate then total amount, standard deviation etc.
All that after pre filtering with Sobel operator – http://www.win.tue.nl/~wstahw/2IV05/seamcarving.pdf)

These stats were then passed to Quinlan’s C4.5 and provided with some training pictures with people in the room or not.

The idea behind applying edge density & energy analysis was for the algorithm to learn about the room’s layout… i.e. moving a chair would not change density nor energy a lot, but adding another one would do, so theoretically this approach should learn about objects inside the room. Filtering by edge density to detect features would remove unimportant (not dense) objects such as chairs or people being too far away.

If C4.5 used that algorithm with enough data with people and other objects, it would learn itself about how to distinguish between a face and a chair.

Team 3:

Wrote their implementation using python with bindings for OpenCV. Initially loading the image creating 3 grayscale images.

Image 1: Last board room image
Image 2: The current board room image
Image 3: The AbsDiff between Image 1 and Image 2

Using ImageChops.difference the stats for the difference are stored in sqlite and divided by an average. If the condition is met it will mark the room as being occupied.

Snapshots from OpenCV can be seen below.

Team 4:

Used PHPs Imagick extension and the following function http://robert-lerner.com/imagecompare.php. They ran a series of slices whether someone was in the room and determined that if there was a 5% variance the meeting room was occupied.

The results in action

Other references

C Sharp / C / C++

Realtime Webcam Sudoku Solver

Motion Detection Algorithms

Face Detection in C#

Multiple face detection and recognition in real time

Emgu CV

Hands Gesture Recognition

Playing Card Recognition Using AForge.Net Framework

OpenCV

Feature detection

Motion Analysis and Object Tracking

Canny Edge Detector

Laplace Operator

Sobel Derivatives

No Comments
Tags: C Sharp, C#, C4.5, Image Analysis, Imagick, PHP

apt-get unattended / non-interactive installation

In: Linux|PHP

19 Jan 2012

I previously wrote about “Installing Java & MySQL – unattended/non-interactive installation”. Whilst using slack, I have been running apt-get during postinstall rather than preinstall.
Files are transferred from slack after preinstall as such transferring any files that conflict with the packages configuration files, apt-get will prompt regardless of DEBIAN_FRONTEND=noninteractive. Not ideal if your trying to run non-interactive. You can use the -o option to circumvent this…

apt-get -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'  -f -q -y php-cli

Previous example of non-interactive installations.

INSTALLER_LOG=/var/log/non-interactive-installer.log

installnoninteractive(){
    sudo bash -c "DEBIAN_FRONTEND=noninteractive aptitude install -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold'  -f -q -y $* >> $INSTALLER_LOG"
}

assert_package() {
    is_installed=$(apt-cache policy "$1" | grep 'Installed: (none)' | wc -l);
    if [ "$is_installed" -eq "1" ] ; then 
        #apt-get install -q -y $1;
        installnoninteractive $1;
    fi; 
}

(1) Comment
Tags: apt-get, Bash, Debian, non-interactive, PHP, Ubuntu

PHP apache_note, Graphite & Logster for logging

In: General|Linux|PHP

9 Jan 2012

A nice tip for logging data against apache logs within requests. Apache_note is a wrapper for Apache’s table_get and table_set. It edits the table of notes that exists during a request. The table’s purpose is to allow Apache modules to communicate.


if (function_exists('apache_note')) {

    if (isset($_SESSION) && !empty($_SESSION)) {
        apache_note('session_id',     session_id());
        apache_note('users_id',       isset($_SESSION['users_id'])? $_SESSION['users_id'] : null);

    }

    apache_note('start_time',     isset($_SERVER['REQUEST_TIME'])? $_SERVER['REQUEST_TIME'] : null);
    apache_note('end_time',       time());
    apache_note('peak_memory',    memory_get_peak_usage());

}

This will allow the variable to be exposed to apache logs in the form of the following.


LogFormat "%{session_id}n %{users_id}n " combined

You can easily add this globally by using the auto_append_file directive in the php ini.


auto_append_file = /var/www/footer.php

Additionally it is quite handy to include the following in a custom log format for debugging purposes. For example if you have hung apache processes.

Format String	Description
%…D	The time taken to serve the request, in microseconds.
%…B	Size of response in bytes, excluding HTTP headers.
%…P	The process ID of the child that serviced the request.
%…{format}P	The process ID or thread id of the child that serviced the request. Valid formats are pid and tid. (Apache 2.0.46 and later)
%…X	Connection status when response is completed: X = connection aborted before the response completed. + = connection may be kept alive after the response is sent. – = connection will be closed after the response is sent. (This directive was %…c in late versions of Apache 1.3, but this conflicted with the historical ssl %…{var}c syntax.)

Common Log Format with Virtual Host and above params


LogFormat "%v %h %l %u %t \"%r\" %>s %b %B %P %X %{session_id}n %{users_id}n %{peak_memory}n" custom_log

You can take this one step further and log these in realtime using Graphite, Logtailer and logster. This holds the offset from the last time you scanned your logs. So this will give you accurate graphs. I have also added a custom parser for PHP, which can be found here.

An example of this, although does not have any errors shown below (hidden the server names).

(1) Comment
Tags: Apache, LogFormat, Logging, PHP

Older Entries

Eugene: I've just hit exactly the same issue. Would have been be nice if Amazon mentioned it in documentatio [...]

Ozzy: Thanks Andy. This was useful (: [...]

shahzaib: Nice post . I have a question regarding BGP. As you said, we'll need to advertise our route from mul [...]

Steve: Hi there… sorry that this is old, but I’m trying to use your script to check for my u [...]

John Cannon: Hey that was really needful. Thanks for sharing. I’ll surely be looking for more. [...]

Development, Analysis And Research

Amazon ELB – monitoring packet count and byte size with Amazon Cloudwatch and VPC flow logs

SSL – Amazon ELB Certificates

Automatically update Amazon ELB SSL Negotiation Policies

Amazon Opsworks – Dependent on Monit

Awk craziness: Processing log files

Elastic Search Presentation

Requests per second from Apache logs

Hackday @ Everlution

Task sheet/Points

Team 1 – My Team:

The results in action

Team 2:

Team 3:

Team 4:

The results in action

Other references

C Sharp / C / C++

OpenCV

apt-get unattended / non-interactive installation

PHP apache_note, Graphite & Logster for logging

About this blog

Categories

Archives

Task sheet/Points

Team 1 – My Team:

The results in action

Team 2:

Team 3:

Team 4:

The results in action

Other references

C Sharp / C / C++

OpenCV

About this blog

Categories

Archives

Tags