<?xml version="1.0" encoding="UTF-8"?>
<!-- generator="wordpress/2.3.3" -->
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	>

<channel>
	<title>Development, Analysis And Research</title>
	<link>http://www.ajohnstone.com</link>
	<description>by Andrew Johnstone</description>
	<pubDate>Thu, 21 Feb 2008 02:30:20 +0000</pubDate>
	<generator>http://wordpress.org/?v=2.3.3</generator>
	<language>en</language>
			<item>
		<title>Soap, XmlRpc and Rest with the Zend Framework</title>
		<link>http://www.ajohnstone.com/archives/soap-xmlrpc-and-rest-with-the-zend-framework/</link>
		<comments>http://www.ajohnstone.com/archives/soap-xmlrpc-and-rest-with-the-zend-framework/#comments</comments>
		<pubDate>Thu, 21 Feb 2008 02:30:20 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[Db]]></category>

		<category><![CDATA[General]]></category>

		<category><![CDATA[PHP]]></category>

		<category><![CDATA[XML]]></category>

		<category><![CDATA[Rest API]]></category>

		<category><![CDATA[Soap]]></category>

		<category><![CDATA[Trading]]></category>

		<category><![CDATA[XmlRpc]]></category>

		<category><![CDATA[Zend Framework]]></category>

		<category><![CDATA[ZF]]></category>
<category>General</category><category>Perfomance</category><category>PHP</category><category>Rest API</category><category>Soap</category><category>Trading</category><category>XmlRpc</category><category>Zend Framework</category><category>ZF</category>
		<guid isPermaLink="false">http://www.ajohnstone.com/archives/soap-xmlrpc-and-rest-with-the-zend-framework/</guid>
		<description><![CDATA[The Project
I was recently working on a project to expose our trading systems via XmlRpc, Rest and SOAP. It was quite an interesting project, which took two of us three weeks to develop (Amongst other things).
This involved creating a testbed, that would automatically generate the payload and response for each protocol. The parameters are introspected [...]]]></description>
			<content:encoded><![CDATA[<h3>The Project</h3>
<p>I was recently working on a project to expose <a href="http://corporate.bullbearings.co.uk">our trading</a> systems via <a href="http://framework.zend.com/manual/en/zend.xmlrpc.html">XmlRpc</a>, <a href="http://framework.zend.com/manual/en/zend.rest.html">Rest</a> and <a href="http://uk3.php.net/soap">SOAP</a>. It was quite an interesting project, which took two of us three weeks to develop <em>(Amongst other things)</em>.</p>
<p>This involved creating a testbed, that would automatically generate the payload and response for each protocol. The parameters are introspected for each class method capturing each parameters data type, allowing for user input via standard html forms. This is probably best described with a picture or two.</p>
<p>Most of the documentation was generated via reflection and comments within the docblocks, parameters, notes  were also generated making it quick and simple to update. In addition to parsing the start and end line of each method for any applicable error codes/faults that may be returned.</p>
<p><a href='http://www.ajohnstone.com/wp-content/uploads/2008/02/restapiinterface.png' title='Rest API interface' rel="lightbox"><img src='http://www.ajohnstone.com/wp-content/uploads/2008/02/restapiinterface.thumbnail.png' alt='Rest API interface' /></a></p>
<p><a href='http://www.ajohnstone.com/wp-content/uploads/2008/02/xmlrpcapiinterfaceexecuted.png' rel="lightbox" title='XmlRpc API Interface - executed API method'><img src='http://www.ajohnstone.com/wp-content/uploads/2008/02/xmlrpcapiinterfaceexecuted.thumbnail.png' alt='XmlRpc API Interface - executed API method' /></a></p>
<h3><a href="http://framework.zend.com/">Zend Framework</a></h3>
<p>Using the <a href="http://framework.zend.com/">Zend Framework</a> for the first time in a commercial product was not exactly hassle free, and still has quite a few issues with its <a href="http://en.wikipedia.org/wiki/Web_service">webservices</a> implementation. Currently there seems to be quite a bit of confusion regarding its <a href="http://framework.zend.com/manual/en/zend.rest.html">Rest</a> implementation and whether it is to be merged, would be great if someone clarify this.</p>
<p>The main issue I found with the <a href="http://framework.zend.com/">Zend Frameworks</a> implementation of <a href="http://framework.zend.com/manual/en/zend.xmlrpc.html">XmlRpc</a> and <a href="http://framework.zend.com/manual/en/zend.rest.html">Rest</a> is that it assumes that the payload it receives is valid. During my development, I tended to mix the payloads from <a href="http://uk3.php.net/soap">SOAP</a>, <a href="http://framework.zend.com/manual/en/zend.xmlrpc.html">XmlRpc</a> and <a href="http://framework.zend.com/manual/en/zend.rest.html">Rest</a>, yet it would assume that <a href="http://uk2.php.net/simplexml">simple_xml</a> can parse the input.</p>
<p>For example $this->_sxml is assumed to be a valid object, if not you will either get invalid method call or an undefined index, which doesn&#8217;t render well for an <a href="http://framework.zend.com/manual/en/zend.xmlrpc.server.html">xmlrpc server</a>.</p>
<p><code>
<pre>
    /**
     * Constructor
     *
     * @param string $data XML Result
     * @return void
     */
    public function __construct($data)
    {
        $this->_sxml = simplexml_load_string($data);
    }

    /**
     * toString overload
     *
     * Be sure to only call this when the result is a single value!
     *
     * @return string
     */
    public function __toString()
    {
        if (!$this->getStatus()) {
            $message = $this->_sxml->xpath('//message');
            return (string) $message[0];
        } else {
            $result = $this->_sxml->xpath('//response');
            if (sizeof($result) > 1) {
                return (string) "An error occured.";
            } else {
                return (string) $result[0];
            }
        }
    }
</pre>
<p></code></p>
<p>One of the main issues with <a href="http://framework.zend.com/manual/en/zend.rest.html">Rest</a> was that it needed <a href="http://www.php.net/ksort">ksort</a> when using the <a href="http://framework.zend.com/manual/en/zend.rest.client.html">Rest client</a> as the arguments were not necessarily passed in order. This can be &#8220;rest.php?method=x&#038;arg1=1&#038;arg0=0&#8243; and it would interpret each arg in the order it received them. This should be sorted in the next release of the <a href="http://framework.zend.com/">ZF</a>.</p>
<p>As the <a href="http://en.wikipedia.org/wiki/Web_service">webservices</a> we are exposing needs to have quite good performance with the number of transactions it will be handling and the amount of <a href="http://framework.zend.com/manual/en/zend.server.reflection.html">reflection</a> that <a href="http://framework.zend.com/manual/en/zend.server.reflection.html">Zend Server Reflection</a> <em>(Only noticed after I started profiling)</em> performs and I wanted to optimize any overhead, which got me looking at <a href="http://framework.zend.com/apidoc/core/Zend_XmlRpc/Server/Zend_XmlRpc_Server_Cache.html">Zend_XmlRpc_Server_Cache</a>. First thing I did was profile <a href="http://framework.zend.com/apidoc/core/Zend_XmlRpc/Server/Zend_XmlRpc_Server_Cache.html">Zend_XmlRpc_Server_Cache</a>, which added a considerable amount of overhead. Looking at its implementation, it uses serialize, which is a relatively slow process and should be avoided, unless there is a large overhead in initializing objects. So most likely <a href="http://framework.zend.com/apidoc/core/Zend_XmlRpc/Server/Zend_XmlRpc_Server_Cache.html">Zend_XmlRpc_Server_Cache</a> will not add any benefit. And var_dump&#8217;ing out the reflection in XmlRpc spews out a shocking amount of information on some fairly large classes.</p>
<p><code>
<pre>
  if (!Zend_XmlRpc_Server_Cache::get($cacheFile, $server)) {

  }
</pre>
<p></code></p>
<h3>Generating WSDL</h3>
<p>I tried a number of <a href="http://en.wikipedia.org/wiki/Web_Services_Description_Language">WSDL</a> generators including the implementation in incubator for <a href="http://framework.zend.com/">ZF</a>, which I found to be the best, yet I still had to write a large chunk of the <a href="http://en.wikipedia.org/wiki/Web_Services_Description_Language">WSDL</a> by hand and adapt it. </p>
<p>The best way to debug is to run the soap client with verbose mode on, and it will typically tell you the issue straight away.</p>
<ul>
<li>Zend_Soap_AutoDiscover: Duplicates an operation in <a href="http://en.wikipedia.org/wiki/Web_Services_Description_Language">WSDL</a> for methods with parameters that are optional. (<a href="http://framework.zend.com/issues/browse/ZF-2642">ZF-2642</a>)</li>
<li>Zend_Soap_AutoDiscover: If missing the @return in your docblock the message response in the <a href="http://en.wikipedia.org/wiki/Web_Services_Description_Language">WSDL</a> is not generated. (<a href="http://framework.zend.com/issues/browse/ZF-2643">ZF-2643</a>)</li>
<li>AutoDiscover duplicates response if using set class multiple times. (<a href="http://framework.zend.com/issues/browse/ZF-2641">ZF-2641 </a>)</li>
<li>One of my colleagues typically writes their docblocks with &#8220;@return int, comment.&#8221;, which the comma caused return types to be dropped with AutoDiscover, more of an issue with <a href="http://framework.zend.com/manual/en/zend.server.reflection.html">Zend Server Reflection</a></li>
</ul>
<h3>Other odd issues</h3>
<h4>Raw input bug</h4>
<p>Some other obscurities I found was capturing the raw request data. In our local development environment reading the raw request input, and then once again within the <a href="http://framework.zend.com/">Zend Frameworks</a> appears to work fine. However in our pre-production environment it fails to read the second request to read the raw request. <em>(<a href="http://www.php.net/">PHP 5.2.2</a>)</em></p>
<p><code><br />
  if (!isset($HTTP_RAW_POST_DATA)){<br />
    $HTTP_RAW_POST_DATA = file_get_contents('php://input');<br />
  }<br />
</code></p>
<p>It does seem a little odd that the <a href="http://framework.zend.com/manual/en/zend.xmlrpc.html">XmlRpc</a> does not check whether $HTTP_RAW_POST_DATA isset before attempting to re-read raw input.</p>
<h4>Internal error: Wrong return type</h4>
<p>Whilst running <a href="http://www.phpunit.de/">PHPUnit</a> I noticed a very weird quirk in our local dev environment, which essentially did the following&#8230; You would expect this to output the contents of an array right? Well between the method call to x and return the result back to method y returns NULL. This is very obscure and i&#8217;ve never seen anything like it especially considering it is explicitly set. I had a number of colleagues check this, which had us all scratching our heads. Has anyone else seen anything similar to this?</p>
<p><code>
<pre>
class test {

  public function x() {
    $ret = array();
    for(...) {
      $ret[] = $row;
    }
    return $ret;
  }

  public function y() {
    $response = $this->x();
    var_dump($response);
  }
}

$t = new test();
$t->y();
</pre>
<p></code></p>
<h3>Conclusion</h3>
<p>Overall the project went pretty well, I&#8217;m confident it is now stable especially with the number of <a href="http://www.phpunit.de/">tests</a> we ran against it. It is adaptable to other projects that we may need to expose via an <a href="http://en.wikipedia.org/wiki/Application_programming_interface">API</a>, in total there is about 6000 lines of code alone just testing the 3 different protocols it supports. I would have rather avoided the <a href="http://framework.zend.com/manual/en/zend.rest.html">Rest</a> implementation with ZF as it still needs a lot of work, however <a href="http://framework.zend.com/manual/en/zend.xmlrpc.html">XmlRpc</a> is a lot more stable and I would quite happily use again. As there is a lot of overhead with reflection it is not the fastest implementation and was contrasted to some of the heavier web pages we have for some simple functionality. It would be ideal to replace the reflection with something lighter such as an array with the corresponding methods, parameters and types, however I would have to look into that if performance did become a major issue.</p>
<p>PS. Just to note I used PHP&#8217;s in built soap server.</p>
<a href="http://www.ajohnstone.com/tag/general" rel="tag">General</a><div class="awmp_tags"><a href="http://www.ajohnstone.com/search/Soap/" rel="tag">Soap</a> <a href="http://www.ajohnstone.com/search/XmlRpc/" rel="tag">XmlRpc</a> <a href="http://www.ajohnstone.com/search/PHP/" rel="tag">PHP</a> <a href="http://www.ajohnstone.com/search/Rest API/" rel="tag">Rest API</a> <a href="http://www.ajohnstone.com/search/Trading/" rel="tag">Trading</a> <a href="http://www.ajohnstone.com/search/ZF/" rel="tag">ZF</a> <a href="http://www.ajohnstone.com/search/Zend Framework/" rel="tag">Zend Framework</a> <a href="http://www.ajohnstone.com/search/Perfomance/" rel="tag">Perfomance</a></div>]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/soap-xmlrpc-and-rest-with-the-zend-framework/feed/</wfw:commentRss>
		</item>
		<item>
		<title>Zend Studio for Eclipse: Neon</title>
		<link>http://www.ajohnstone.com/archives/zend-studio-for-eclipse-neon/</link>
		<comments>http://www.ajohnstone.com/archives/zend-studio-for-eclipse-neon/#comments</comments>
		<pubDate>Sat, 12 Jan 2008 13:15:58 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[General]]></category>

		<category><![CDATA[PHP]]></category>
<category>General</category><category>PHP</category><category>eclipse-project</category><category>shell</category><category>zend-studio</category>
		<guid isPermaLink="false">http://www.ajohnstone.com/archives/zend-studio-for-eclipse-neon/</guid>
		<description><![CDATA[I&#8217;ve been using Zend Studio for Eclipse (beta) for several weeks in a rewrite of a framework and numerous sites at work and overall I really like the IDE. Its got some great features and being based on the eclipse project makes it really extensible and customizable. With debugging, profiling, code completion, code formatting and [...]]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve been using Zend Studio for Eclipse (beta) for several weeks in a rewrite of a framework and numerous sites at work and overall I really like the IDE. Its got some great features and being based on the eclipse project makes it really extensible and customizable. With debugging, profiling, code completion, code formatting and more enabled can help with productivity.</p>
<p>A complete list of features can be found at <a href="http://www.zend.com/en/products/studio/eclipse/compare">Zend</a>.</p>
<p>Zend Studio for Eclipse consumes quite a lot of memory and the recommended amount of RAM for eclipse based applications is 2GB, however you can control the amount of memory that eclipse will use by editing the zendStudio.ini file.</p>
<p>Whilst I do like the IDE I have found a number of issues with Zend Studio:</p>
<p>
<h3>Bugs</h3>
</p>
<ul>
<li>1. There have been a number of issues revolving around the SVN implementation in Zend Studio for Eclipse which causes the application to hang (SVN support is provided to eclipse by a 3rd party plugin (a company called Polarion)):
<ul>
<li>a) When committing files it locks entire directories and often hangs making Zend Studio for Eclipse unusable, if you have files that are not saved and attempt to do so it queues it as a users pending tasks and because the commit has stalled you cannot save the file.
<p>To resolve this I have to kill the process for Zend Studio, shell into the server and cleanup the the projects src, and sometimes have to re-checkout the directories in a project
      </li>
</ul>
</li>
<li>2. When developing via a samba share, it prompts with an incorrect error and does not attempt to re-authenticate when the samba share needs to re-connect and/or does recognise that it is talking via a remote device.<br />&nbsp;</li>
<li>3. Auto format adds extra braces to statements, causing syntax errors, strips all comments out of files!<br />&nbsp;</li>
<li>4. Importing an auto format does not seem to work correctly. <br />&nbsp;</li>
<li>5. When working with multiple open files it can overwrite the contents with another, I believe this is the case with files of a similar name (I&#8217;ve only had this occur once, however a colleague experiences this quite frequently).<br />&nbsp;</li>
<li>6. Modified file names are prefixed with &#8220;>&#8221; and when searching for files by pressing a character will not go to that file in PHP Explorer<br />&nbsp;</li>
<li>7. Templates do not always get replaced e.g. If the system is slow or you type fnc real quick it doesn&#8217;t replace with the template for a function.<br />&nbsp;</li>
<li>8.<br />
<table>
<tr>
<td valign="top">If you cause a syntax error, whether deliberate or not, a number of items such as auto formatting does not work, or you cannot open the php manual from the contexts menu for a php function. For example if I type &#8220;str_shuffle&#8221; then use a context menu to it does not open a manual.</td>
<td>
<a href='http://www.ajohnstone.com/wp-content/uploads/2008/01/contextmenuwithsyntaxerror.png' title='Context Menu With Syntax Error and Attempting to open PHP Manual' target="_blank"><img src='http://www.ajohnstone.com/wp-content/uploads/2008/01/contextmenuwithsyntaxerror.png' alt='Context Menu With Syntax Error and Attempting to open PHP Manual' /></a></td>
</tr>
</table>
</li>
<li>9. introducing a syntax error and then removing the syntax error doesn&#8217;t clear until you save the document.
<p><a href='http://www.ajohnstone.com/wp-content/uploads/2008/01/invalidsyntaxerror.png' title='Invalid Syntax Error' target="_blank"><img src='http://www.ajohnstone.com/wp-content/uploads/2008/01/invalidsyntaxerror.png' alt='Invalid Syntax Error' /></a></li>
</ul>
<p>
<h3> Resolved </h3>
</p>
<ul>
<li>1. There was a bug in automatically updating eclipse, which never seemed to work however in the latest release (beta 2) this has now been resolved.</li>
</ul>
<p><a rel="lightbox" href='http://www.ajohnstone.com/wp-content/uploads/2008/01/zendstudiofull.png' title='Zend Studio For Eclipse'><img src='http://www.ajohnstone.com/wp-content/uploads/2008/01/zendstudiofull.thumbnail.png' alt='Zend Studio For Eclipse' /></a></p>
<a href="http://www.ajohnstone.com/tag/general" rel="tag">General</a>, <a href="http://www.ajohnstone.com/tag/php" rel="tag">PHP</a>]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/zend-studio-for-eclipse-neon/feed/</wfw:commentRss>
		</item>
		<item>
		<title>MySQL &#038; PHP Performance Optimization Tips</title>
		<link>http://www.ajohnstone.com/archives/mysql-php-performance-optimization-tips/</link>
		<comments>http://www.ajohnstone.com/archives/mysql-php-performance-optimization-tips/#comments</comments>
		<pubDate>Wed, 25 Jul 2007 09:32:46 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[Db]]></category>

		<category><![CDATA[General]]></category>
<category>Db</category><category>General</category><category>MySQL</category><category>Optimization</category><category>performance</category><category>high-performance-web-applications</category><category>MySQL</category><category>optimization</category>
		<guid isPermaLink="false">http://www.ajohnstone.com/archives/mysql-php-performance-optimization-tips/</guid>
		<description><![CDATA[In high performance web applications you will always have bottlenecks within your application. Identifying these bottlenecks and optimizing is a tedious task and typically show themselves underload. A single bad/unindexed query can bring a server to its knees. A large number of rows will also help to highlight any poor queries, and on very large [...]]]></description>
			<content:encoded><![CDATA[<p>In high performance web applications you will always have bottlenecks within your application. Identifying these bottlenecks and optimizing is a tedious task and typically show themselves underload. A single bad/unindexed query can bring a server to its knees. A large number of rows will also help to highlight any poor queries, and on very large datasets you may come to the point where you may have to make decisions whether to denormilize database schema. </p>
<h3>Explain each page</h3>
<p>Whilst I develope sites, I typically print out all queries, EXPLAIN each select statement at the bottom of each page, and highlight it red if its doing a full table scan, temp tables or a filesort. As well as displaying SHOWS INDEXES FROM TABLE&#8230;</p>
<p>Not only will it help you to optimize sites, you can also see bad logic and areas to optimize such as a query for each loop when looking through a users table for example. </p>
<p><a href="http://www.ajohnstone.com/wp-content/uploads/2007/07/badindexing.png" rel="lightbox" title="MySQL indexing optimization"><img src="http://www.ajohnstone.com/wp-content/uploads/2007/07/badindexing.thumbnail.png" alt="MySQL indexing optimization" /></a></p>
<h3>How do you identify where bottlenecks occur?</h3>
<p>One of my favourite linux commands lately is the watch command. For Mac users you can get this from macports via &#8220;sudo port install watch&#8221;. Also a few other handy applications are <a href="http://hackmysql.com/mysqlreport">mysqlreport</a>, <a href="http://jeremy.zawodny.com/mysql/mytop/">mytop</a>.<br />
<code>
<pre>
# <b>Appends file with processlist</b>
watch -n1 &#8220;mysqladmin -uroot processlist >>watch.processlist.txt&#8221;

# <b>Count the number of locked processes</b>
watch -n1 &#8220;mysqladmin -uroot processlist | grep -i &#8216;lock&#8217; | wc -l &#8220;;

# <b>Count the number of processes sleep</b>
watch -n1 &#8220;mysqladmin -uroot processlist | grep -i &#8217;sleep&#8217; | wc -l &#8220;;

# <b>Run a specific query every second</b>
watch -n1 &#8220;mysql -uadmin -p`cat /etc/psa/.psa.shadow` trade_engine &#8211;execute &#8220;SELECT NOW(),date_quote FROM sampleData WHERE 1=1 AND permission = &#8216;755&#8242; AND  symbol=&#8217;IBZL&#8217; GROUP BY date_quote;&#8221; &#8221;

# <b>Emails mysqlreport every 60 seconds</b>
watch -n60 mysqlreport &#8211;all &#8211;email andrew@email.com

# <b>Displays process list as well as appending the contents to a file</b>
watch -n1 &#8220;mysqladmin -uadmin -p`cat /etc/psa/.psa.shadow` processlist | tee -a process.list.txt&#8221;
</pre>
<p></code></p>
<p>Watching the processlist is very handy in identifying locked, sleeping or sorting process states. If you have a large number of locked processes you typically should change the table type to INNODB, which supports row level locking. if you have a large number of sleeping connections, and you have persistent connections enabled, most likely indicates that connections are not being reused.</p>
<p>Running a specific query every second is exceptionally handy, the example I gave indicates whether one of our crons is correctly functioning and as each row is inserted you can watch something being either inserted or updated. mysqlreport gives numerous peices of information, extremely helpful in identifying issues, you can see more indepth at <a href="http://hackmysql.com/mysqlreportguide">hackmysql.com/mysqlreportguide</a>.</p>
<p>Look at the mysql slow query log and optimize each query starting with the most common, think whether you have to execute that query at all and use a cache such as memcached. </p>
<p>I also typically tend to look at the following:
<ul>
<li>vmstat -S M</li>
<li>ps axl | grep -i &#8216;mysql&#8217;</li>
<li>pstree –G</li>
<li>free –m</li>
</ul>
<p>Reference:<br />
http://dev.mysql.com/tech-resources/presentations/presentation-oscon2000-20000719/index.html</p>
<a href="http://www.ajohnstone.com/tag/db" rel="tag">Db</a>, <a href="http://www.ajohnstone.com/tag/general" rel="tag">General</a>, <a href="http://www.ajohnstone.com/tag/mysql" rel="tag">MySQL</a>, <a href="http://www.ajohnstone.com/tag/optimization" rel="tag">Optimization</a>, <a href="http://www.ajohnstone.com/tag/performance" rel="tag">performance</a>]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/mysql-php-performance-optimization-tips/feed/</wfw:commentRss>
		</item>
		<item>
		<title>C++</title>
		<link>http://www.ajohnstone.com/archives/c/</link>
		<comments>http://www.ajohnstone.com/archives/c/#comments</comments>
		<pubDate>Sun, 08 Jul 2007 13:20:31 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[C++]]></category>

		<category><![CDATA[General]]></category>

		<category><![CDATA[PHP]]></category>
<category>C++</category><category>General</category><category>MySQL</category><category>PHP</category><category>refactor</category><category>compatible-compiler</category><category>connection-pooling</category><category>database-connection</category><category>database-schema</category><category>gcc-version</category><category>log-messages</category><category>mysql</category><category>refactor</category><category>runtime-errors</category><category>trade-message</category>
		<guid isPermaLink="false">http://www.ajohnstone.com/archives/c/</guid>
		<description><![CDATA[I've had alot of experience with other programming languages, however I had to learn C++ from scratch in a very short period of time, a number of weeks ago. This was to develop a real-time stock quote client, the goal was simply to push data from remote servers into our databases, filter what messages it would receive and get something up and running fast as deadlines lingured. This was simple enough, however with the rush the application had its inherent flaws, due to my lack of knowledge of C++, the API, and the goals it had to acomplish.]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve had alot of experience with other programming languages, however I had to learn C++ from scratch in a very short period of time, a number of weeks ago. This was to develop a real-time stock quote client, the goal was simply to push data from remote servers into our databases, filter what messages it would receive and get something up and running fast as deadlines lingured. This was simple enough, however with the rush the application had its inherent flaws, due to my lack of knowledge of C++, the API, and the goals it had to acomplish.</p>
<p>I&#8217;ve since had time to learn a little more C++ and limited time to design the application properly. </p>
<h3>The Problems</h3>
<p>The core problems with the application:</p>
<ul>
<li>refactor, refactor, refactor</li>
<li>database connection pooling</li>
<li>Query remote CSP servers<sup>*1</sup></li>
<li>Query remote CSP servers<sup>*1</sup> from PHP</li>
<li>Configuration management</li>
<li>Monitoring</li>
<li>Flexible Database schema
<ul>
<li>Add columns to database schema dependent on datatype.</li>
<li>Log messages in XML per trade message with date/time, columns and values.</li>
</ul>
</li>
</ul>
<h3>Compatible GCC</h3>
<p>The first issue was that I used an API from <a href="http://www.interactivedata-rts.com/">interactive-data</a>, which was compatible with &#8220;gcc version 3.2.3&#8243; and is not kept up to date. This meant compiling a compatible gcc from source for 32bit platforms only.</p>
<p><code><br />
./configure --prefix=/usr/local/gcc/ --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --enable-languages=c,c++,objc,obj-c++</p>
<p>make bootstrap<br />
cd gcc<br />
make<br />
sudo make install<br />
</code></p>
<p>Once having a compatible compiler, I then had to make modifications to the Makefile, move a number of lib/so files to get MySQL to compile and get things working. Unfortunately I did not have a local machine to attach a debugger, so everything was trial and error from the command line with g++32, which makes life difficult identifying runtime errors.</p>
<h3>The Logic</h3>
<p>Once everything was in place, the logic was fairly simple, foreach field retreived construct a query with the field name, checking the fields values datatype whether it be a datetime, varchar etc. Insert each trade message in a table, update another and if either failed, check if the fault was due to a missing column, if so add it and re-execute queries.</p>
<p>The problem soon arrises when you need to know when each column was actually last updated, with which field, value, datetime and the last insert id for the trade messages. Whilst looping through each trade message, I constructed an XML schema containing the above, however the tricky part is to ensure that it only updates the fragment matching the field in the schema. Not an ideal format to query from a database. </p>
<h3>Storing Data</h3>
<p>One of the fundemental issues is managing and storing data. For some exchanges you don&#8217;t want to store every trade message; simply storing the current data for a number of instruments is enough. Which servers or databases do you peg data to? If one database goes down, how do you handle fault tolerance? MySQL cluster is not a feasible solution, requiring multiple servers and large memory requirements per installation. The databases are highly susceptible to curruption or faults. Also particular sites may require data from multiple exchanges, so seperating trade messages per database is not also ideal.</p>
<p>All of this fundamentally comes down to configuration management. </p>
<h3>Configuration</h3>
<p>One of the fundamental aspects of the application is configuration management. This contains where data should be stored for a particular exchange, the type of data to store, whether it is per trade message, current data or both. Which servers to source data from, whether it is real time or delayed, whether to source data for bonds, equities, automated trades etc&#8230; All queries can be grouped, or to query remote servers. Some of the products for example just for the London Stock Exchange is:
<ul>
<li>London Stock Exch - Covered Warrants L1</li>
<li>London Stock Exch - International Equity Mkt Service L1</li>
<li>London Stock Exch - International Equity Mkt Service Level 2</li>
<li>London Stock Exch - UK Equity Mkt Service L1</li>
<li>London Stock Exch - UK Equity Mkt Service Level 2 (Depth Refresh)</li>
<li>London Stock Exchange: UK Equity Market Service Level 2</li>
</ul>
<p>All of which is stored in several database tables and managed via a MySQL database and PHP frontend.</p>
<a href="http://www.ajohnstone.com/tag/c%2B%2B" rel="tag">C++</a>, <a href="http://www.ajohnstone.com/tag/general" rel="tag">General</a>, <a href="http://www.ajohnstone.com/tag/mysql" rel="tag">MySQL</a>, <a href="http://www.ajohnstone.com/tag/php" rel="tag">PHP</a>, <a href="http://www.ajohnstone.com/tag/refactor" rel="tag">refactor</a>]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/c/feed/</wfw:commentRss>
		</item>
		<item>
		<title>Installing memcached</title>
		<link>http://www.ajohnstone.com/archives/installing-memcached/</link>
		<comments>http://www.ajohnstone.com/archives/installing-memcached/#comments</comments>
		<pubDate>Sat, 07 Apr 2007 16:20:05 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[General]]></category>
<category>memcache optimization</category><category>performance</category><category>PHP</category>
		<guid isPermaLink="false">http://blog.ajohnstone.com/archives/installing-memcached/</guid>
		<description><![CDATA[Recently I had to install memcache on a number of servers, and I would always tend to end up with errors whilst memcache tries to locate libevent. I always seem to forgett LD_DEBUG, so I figured I would write up the process for installing memcache.
One of the dependencies of memcache is libevent, so firstly download [...]]]></description>
			<content:encoded><![CDATA[<p>Recently I had to install memcache on a number of servers, and I would always tend to end up with errors whilst memcache tries to locate libevent. I always seem to forgett LD_DEBUG, so I figured I would write up the process for installing memcache.</p>
<p>One of the dependencies of memcache is libevent, so firstly download the source files for <a href="http://www.monkey.org/~provos/libevent/">Libevent</a>.</p>
<p><code><br />
tar -xvf libevent-1.3b.tar.gz<br />
cd libevent-1.3b<br />
./configure;make;make install;<br />
</code></p>
<p>Download the latest Memcached source code from <a href="http://www.danga.com/memcached/">danga.com</a></p>
<p><code><br />
gunzip memcached-1.2.1.tar.gz<br />
tar -xvf memcached-1.2.1.tar<br />
cd memcached-1.2.1<br />
./configure;make;make install;<br />
</code></p>
<p>Often libevent.so cannot be found when executing memcache. A useful command LD_DEBUG, is very helpful to determine where libraries are being loaded from.</p>
<p><code><br />
LD_DEBUG=help memcached -v<br />
&#160;<br />
LD_DEBUG=libs memcached -v 2>&#038;1 > /dev/null | less<br />
18990: find library=libevent-1.3b.so.1 [0]; searching<br />
...<br />
18990: trying file=/usr/lib/libevent-1.3b.so.1<br />
18990:<br />
memcached: error while loading shared libraries: libevent-1.3b.so.1: cannot open shared object file: No such file or directory<br />
</code></p>
<p>Simply place the library where memcached will find it and execute memcached. </p>
<p><code><br />
ln -s /usr/local/lib/libevent-1.3b.so.1 /lib/libevent-1.3b.so.1<br />
memcached -d -u nobody -m 512 127.0.0.1 -p 11211<br />
&#160;<br />
</code></p>
<p>The options for memcached are:</p>
<blockquote><p>
-l &lt;ip_addr&gt;  <br />
Listen on &lt;ip_addr&gt;; default to INDRR_ANY. This is an important option to consider as there is no other way to secure the installation. Binding to an internal or firewalled network interface is suggested.<br />
-d<br />
Run memcached as a daemon.<br />
-u &lt;username&gt;<br />
Assume the identity of &lt;username&gt; (only when run as root).<br />
-m &lt;num&gt;<br />
Use &lt;num&gt; MB memory max to use for object storage; the default is 64 megabytes.<br />
-M<br />
Instead of throwing items from the cache when max memory is reached, throw an error<br />
-c &lt;num&gt;<br />
Use &lt;num&gt; max simultaneous connections; the default is 1024.<br />
-k<br />
Lock down all paged memory. This is a somewhat dangerous option with large caches, so consult the README and memcached homepage for configuration suggestions.<br />
-p &lt;num&gt;<br />
Listen on port &lt;num&gt;, the default is port 11211.<br />
-r<br />
Maximize core file limit<br />
-M<br />
Disable automatic removal of items from the cache when out of memory. Additions will not be possible until adequate space is freed up.<br />
-r<br />
Raise the core file size limit to the maximum allowable.<br />
-h<br />
Show the version of memcached and a summary of options.<br />
-v<br />
Be verbose during the event loop; print out errors and warnings.<br />
-vv<br />
Be even more verbose; same as -v but also print client commands and responses.<br />
-i<br />
Print memcached and libevent licenses.<br />
-P &lt;filename&gt;<br />
Print pidfile to &lt;filename&gt;, only used under -d option.
</p></blockquote>
<p>To install the pecl package for PHP<br />
<code><br />
wget http://pecl.php.net/get/memcache-2.1.2.tgz<br />
gzip -df memcache-2.1.2.tgz<br />
tar -xvf memcache-2.1.2.tar<br />
cd memcache-2.1.2<br />
phpize<br />
./configure;make;make install;<br />
</code></p>
<p>Add memcache.so to the php.ini file<br />
<code><br />
extension=memcache.so<br />
</code></p>
<p>Then run<br />
<code><br />
php -i | grep -i 'memcache'<br />
</code><br />
memcache should be listed and then restart the web server.</p>
<p>For further information:<br />
<a href="http://www.linuxjournal.com/article/7451">Distributed Caching with Memcached</a></p>
<a href="http://www.ajohnstone.com/tag/memcache-optimization" rel="tag">memcache optimization</a>, <a href="http://www.ajohnstone.com/tag/performance" rel="tag">performance</a>, <a href="http://www.ajohnstone.com/tag/php" rel="tag">PHP</a>]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/installing-memcached/feed/</wfw:commentRss>
		</item>
		<item>
		<title>PHP/MySQL &#038; the Real Time dilemma</title>
		<link>http://www.ajohnstone.com/archives/phpmysql-the-real-time-dilemma/</link>
		<comments>http://www.ajohnstone.com/archives/phpmysql-the-real-time-dilemma/#comments</comments>
		<pubDate>Tue, 26 Dec 2006 21:06:08 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[General]]></category>
<category>AJAX</category><category>JavaScript</category><category>MySQL</category><category>PHP</category>
		<guid isPermaLink="false">http://blog.ajohnstone.com/archives/phpmysql-the-real-time-dilemma/</guid>
		<description><![CDATA[Introduction
Currently I&#8217;m working with stock market data, and its quite an interesting topic when we are getting to the point of real time data as it brings a number of new concepts into the mix. The first challenge is to import information from the feeds into our databases (MySQL), whilst this should be a relatively [...]]]></description>
			<content:encoded><![CDATA[<h3>Introduction</h3>
<p>Currently I&#8217;m working with stock market data, and its quite an interesting topic when we are getting to the point of real time data as it brings a number of new concepts into the mix. The first challenge is to import information from the feeds into our databases (MySQL), whilst this should be a relatively straight forward task, I&#8217;m sure we are going to hit issues in terms of writes to the database (INSERTS/UPDATES). The information from these feeds will be used for various tasks, that will require alot of processing. The information displayed to the user will be via the web, therefore we have to maintain updated stock market information dynamically to the user via the use of AJAX.</p>
<p>The concept of real time computing should ideally be under 1 millisecond, however I have previously worked for companies where their distinction of real time meant a 15 minute delay. Whilst delays over the web are inevitable I believe a one to three second delay would be acceptable for users to view current information via AJAX. </p>
<h3>Replication</h3>
<p>As we will be using the data from the stock market for multiple applications, we will need to replicate the data from MySQL, this will only add a further bottleneck in the application. Most notably performance with replication will become an issue because every slave still needs to execute the same write queries as the master. Whilst the majority of queries, will be writes over reads, this becomes a fundamental problem in itself, making replication questionable. So we will have to look at multi-master MySQL server setup, or MySQL cluster, which holds databases in memory. The fundamental problem with replication is ensuring the consistency of the data between writes once replicated. Ideally if a slave falls behind we want to ignore Updates, that have previously been issued and just use the current values to ensure we do not have stale data.</p>
<h3>Heartbeat</h3>
<p>We will ideally have to create a heartbeat monitor and validate the latency of data between nodes. As mentioned previously we would want to ensure that all slaves do not fall behind, however any slave that did fall behind we would want to ensure that updates for stocks were only applied with the latest and the rest of the binary log is ignored. Additionally we would need to seperate inserts for historical data to be inserted based on a sample time (&#8217;1 Min&#8217;,'15 Min&#8217;,'Hour&#8217;,'Midday&#8217;,'End Of Day&#8217;,'End Of Week&#8217;,'End Of Month&#8217;), ideally this would most benefically be horizontally scaled.<br />
This could be extended to monitor the latency of the end user and notify them that the data is out of date between the last sync with a little javascript.</p>
<h3>Website Data</h3>
<p>The website itself will have to use AJAX to dynamically update all stock prices and activity in the market that are applicable on that page. The fundamental issue is that the prices are updating in real time, how often do we create a http request that is with in reason on server resources? Looking at this further, we will have the bottleneck of TCP/IP connections, the clients bandwidth, ideally testing users bandwidth, and whether the client accepts gzip or compressed content to reduce bandwidth costs.</p>
<p>AJAX request every second, servers typically handles 200 requests per second<br />
say 25 users online, 25*60 =1500 requests per minute or 2,160,000 p/d<br />
say 100 users online, 100*60 =6000 requests per minute or 8,640,000 p/d</p>
<p>We could optionally increase the clients connection limit in internet explorer, with a registry key to increase the<br />
2 connection limit standards from rfc 2216 for persistent connections to http 1.1 agents.<br />
IE7 release does not increase this limit by default however this is more notable when a user downloads 2 files and IE waits for the connection to release before starting a 3rd download for example.</p>
<p><code>
<pre>Windows Registry Editor Version 5.00

[HKEY_CURRENT_USERSoftwareMicrosoftWindowsCurrentVersionInternet Settings]
"MaxConnectionsPerServer"=dword:00000010
"MaxConnectionsPer1_0Server"=dword:0000010
</pre>
<p></code></p>
<a href="http://www.ajohnstone.com/tag/ajax" rel="tag">AJAX</a>, <a href="http://www.ajohnstone.com/tag/javascript" rel="tag">JavaScript</a>, <a href="http://www.ajohnstone.com/tag/mysql" rel="tag">MySQL</a>, <a href="http://www.ajohnstone.com/tag/php" rel="tag">PHP</a>]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/phpmysql-the-real-time-dilemma/feed/</wfw:commentRss>
		</item>
		<item>
		<title>Large Binary Data and Blob&#8217;s</title>
		<link>http://www.ajohnstone.com/archives/large-binary-data-and-blobs-2/</link>
		<comments>http://www.ajohnstone.com/archives/large-binary-data-and-blobs-2/#comments</comments>
		<pubDate>Thu, 19 Oct 2006 22:53:57 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[Db]]></category>

		<category><![CDATA[General]]></category>

		<category><![CDATA[PHP]]></category>
<category>Db</category><category>General</category><category>PHP</category>
		<guid isPermaLink="false">http://blog.ajohnstone.com/archives/large-binary-data-and-blobs-2/</guid>
		<description><![CDATA[I recently read &#8220;Binaries Belong in the Database Too&#8221; on sitepoint.com, and thought I would shed some light with regard to my experience of storing files in databases. I&#8217;m sure many of you have known this to be a taboo practice, and I would certainly agree depending on the database. A project I worked on [...]]]></description>
			<content:encoded><![CDATA[<p>I recently read &#8220;<a href="http://www.sitepoint.com/blogs/2006/10/15/binaries-belong-in-the-database-too/">Binaries Belong in the Database Too</a>&#8221; on <a href="http://www.sitepoint.com/">sitepoint.com</a>, and thought I would shed some light with regard to my experience of storing files in databases. I&#8217;m sure many of you have known this to be a taboo practice, and I would certainly agree depending on the database. A project I worked on for <a href="http://www.mtvne.com">MTV Networks Europe/International</a> required a completely shared nothing architecture. This meant that MTV&#8217;s hosting &#038; operations imposed that I stored files in the database, and expressed my hesitation.</p>
<h4>The platform:</h4>
<p> * Linux<br />
 * Apache<br />
 * MySQL<br />
 * PHP5</p>
<h4>The problems</h4>
<p>You typically get the common file upload problems with upload_max_filesize, max_input_time, execution time however you also have issues with mysql connections and max packet sizes, mysql chunked streams. Uploads via <a href="http://jupload.sourceforge.net/">JUpload</a> allows for large file uploads, however you still encounter TCP/IP connection interuptions and errors. Some of the more major issues I encountered were with the actual management of the data. Currently MySQL has no real support for handling Binary Large Objects, for example if you try to load data in from a file you generally encounter max packet size errors. Although the most fundamental issue is that the MySQL protocol does not send chunked streams for blobs and the client has to load the entire blob into memory. Admittedly memory limits on the server were not too much of an issue, as I was using a 8 CPU 16GB of ram server, however you may not have some of the infrastructure that I had available.</p>
<p>Whilst there were a number of limitations I had to resolve as described above, some of the expectations that I had not anticipated for were some user errors, such as trying to upload 4mb BMP files to be streamed as images for a website. Other factors were that hosting &#038; operations had not expected their adsales department to attempt to upload 120+ mb video files.</p>
<h4>DataTypes</h4>
<p>Firstly lets look at some of the limitations on the BLOB datatype in MySQL, as you can see there are length limitations on blobs.<br />
<code><br />
<b>TINYBLOB</b><br />
A BLOB column with a maximum length of 255 (2^8 - 1) bytes. </p>
<p><b>BLOB[(M)]</b><br />
A BLOB column with a maximum length of 65,535 (2^16 - 1) bytes.<br />
Beginning with MySQL 4.1, an optional length M can be given. MySQL will create the column as the smallest BLOB type largest enough to hold values M bytes long. </p>
<p><b>MEDIUMBLOB</b><br />
A BLOB column with a maximum length of 16,777,215 (2^24 - 1) bytes. </p>
<p><b>LONGBLOB</b><br />
A BLOB column with a maximum length of 4,294,967,295 or 4GB (2^32 - 1) bytes. Up to MySQL 3.23, the client/server protocol and MyISAM tables had a limit of 16MB per communication packet / table row. From MySQL 4.0, the maximum allowed length of LONGBLOB columns depends on the configured maximum packet size in the client/server protocol and available memory. </p>
<p></code></p>
<p>Alternatives for storing >4Gb BLOBs are:<br />
 * Compressing the BLOB so that it fits in 4Gb<br />
 * Splitting up the BLOB into 4Gb chunks as separate rows columns.</p>
<h4>Tips:</h4>
<p><b>Get Blob length</b><br />
To find the length in bytes of a stored BLOB. Simply use: SELECT LENGTH(blobcolumn) FROM table.</p>
<p><b>Get Blob fragment</b><br />
To retrieve large BLOBs by using repeatedly retrieving only fragments of a BLOB</p>
<p><code><br />
using substring, ie:<br />
  SELECT SUBSTRING(document, 1, 10240) FROM documents WHERE did=3;<br />
and then<br />
  SELECT SUBSTRING(document, 10241, 10240) FROM documents WHERE did=3;<br />
etc.</p>
<p></code></p>
<p><b>Inserting Blobs</b><br />
Inserting data into BLOBs. It has to be inserted in hex ie: &#8216;A&#8217; = 0&#215;41 and &#8216;AB&#8217; = 0&#215;4142 and so on. The prefix is a zero not a cap o.</p>
<p> If you want to insert binary data into a string column (such as a BLOB), the following characters must be represented by escape sequences:</p>
<p><code>
<pre>
NUL 	NUL byte (ASCII 0). Represent this character by ' ' (a backslash followed by an ASCII '0' character).
 	Backslash (ASCII 92). Represent this character by '\'.
' 	Single quote (ASCII 39). Represent this character by '''.
" 	Double quote (ASCII 34). Represent this character by '"'.

</pre>
<p></code><br />
When writing applications, any string that might contain any of these special characters must be properly escaped before the string is used as a data value in an SQL statement that is sent to the MySQL server, base64 encoding is a good option.</p>
<p><b>Indexing Blobs</b><br />
<quote><br />
<code><br />
Blobs can sometimes can be indexed, depending on the storage engine you’re using:<br />
MyISAM, InnoDB, and BDB tables support BLOB and TEXT indexing. However, you must specify a prefix size to be used for the index.This avoids creating index entries that might be huge and thereby defeat any benefits to be gained by that index.The exception is that prefixes are not used for FULLTEXT indexes on TEXT columns. FULLTEXT searches are based on the entire content of the indexed columns, so any prefix you specify is ignored.<br />
MEMORY tables do not support BLOB and TEXT indexes.This is because the MEMORY engine does not support BLOB or TEXT columns at all.</p>
<p>BLOB or TEXT columns may require special care:<br />
Due to the typical large variation in the size of BLOB and TEXT values, tables containing them are subject to high rates of fragmentation if many deletes and updates are done. If you’re using a MyISAM table to store BLOB or TEXT values, you can run OPTIMIZE TABLE periodically to reduce fragmentation and maintain good performance.</p>
<p>The max_sort_length system variable influences BLOB and TEXT comparison and sorting operations. Only the first max_sort_length bytes of each value are used. (For TEXT columns that use a multi-byte character set, this means that comparisons might involve fewer than max_sort_length characters.) If this causes a problem with the default max_sort_length value of 1024, you might want to increase the value before performing comparisons. If you’re using very large values, you might need to configure the server to increase the value of the max_allowed_packet parameter. See Chapter 11,“General MySQL Administration,” for more information.You will also need to increase the packet size for any client that wants to use very large values.The mysql and mysqldump clients support setting this value directly using a startup option.</p>
<p></code></quote></p>
<h4>Solution</h4>
<p>The solution ended up utilizing 2 <a href="http://www.danga.com/memcached/">memcached</a> servers that cached blobs and objects between the MySQL server, this saved streaming the content directly from MySQL upon each request. Then selecting chunks of data from a binary large object and concatenating the results alleviates maximum packet errors from MySQL. The only other aspects to alleviate are the initial upload, this is entirely upto you, how this is implemented whether it is via <a href="http://jupload.sourceforge.net/">JUpload</a>, SCP, FTP, or some other means. Finally increase the above settings. To import / export binary files I wrote a script that queried the database writing out the files, by chunking the data, this script did take a while to execute. </p>
<p>I have heard that Oracle has very good support for handling Binary Large Objects&#8230; Maybe thats something to look into..</p>
<p>Pointers.</p>
<p><a href="http://jeremy.zawodny.com/blog/archives/000078.html">http://jeremy.zawodny.com/blog/archives/000078.html</a><br />
<a href="http://jeremy.zawodny.com/blog/archives/000840.html">http://jeremy.zawodny.com/blog/archives/000840.html</a><br />
<a href="http://www.lentus.se/warehouse/SlidesDW.ppt">http://www.lentus.se/warehouse/SlidesDW.ppt</a><br />
<a href="http://sunsite.mff.cuni.cz/MIRRORS/ftp.mysql.com/doc/en/BLOB.html">http://sunsite.mff.cuni.cz/MIRRORS/ftp.mysql.com/doc/en/BLOB.html</a></p>
<a href="http://www.ajohnstone.com/tag/db" rel="tag">Db</a>, <a href="http://www.ajohnstone.com/tag/general" rel="tag">General</a>, <a href="http://www.ajohnstone.com/tag/php" rel="tag">PHP</a>]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/large-binary-data-and-blobs-2/feed/</wfw:commentRss>
		</item>
		<item>
		<title>More on SQL Injection&#8230;</title>
		<link>http://www.ajohnstone.com/archives/more-on-sql-injection/</link>
		<comments>http://www.ajohnstone.com/archives/more-on-sql-injection/#comments</comments>
		<pubDate>Tue, 01 Aug 2006 09:03:41 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://blog.ajohnstone.com/archives/more-on-sql-injection/</guid>
		<description><![CDATA[I wrote this a while ago, whilst playing with SQL Injection, however a little unfinished, the idea was to try to write out entire files, through an SQL Injection attack.
I thought I would expand, on my previous post Exceptions, Exceptions, Exceptions, and see what is possible with a simple a SQL Injection attack. I will [...]]]></description>
			<content:encoded><![CDATA[<p>I wrote this a while ago, whilst playing with SQL Injection, however a little unfinished, the idea was to try to write out entire files, through an SQL Injection attack.</p>
<p>I thought I would expand, on my previous post <a href="http://blog.ajohnstone.com/archives/exceptions-exceptions-exceptions/">Exceptions, Exceptions, Exceptions</a>, and see what is possible with a simple a SQL Injection attack. I will base this on the assumption, that if you&#8217;ve managed to overlook an arbitry SQL Injection attack, I will assume that there will be vulnerable output somewhere.</p>
<p><code>
<pre>
  $id = ($_REQUEST['id'])? (int) $_REQUEST['id'] : 0;
  $SearchTerm = (isset($_REQUEST['q']))? $_REQUEST['q'] : null;

  if ( $id!=0 &#038;&#038; !empty($SearchTerm) ) {
    $SQL = "SELECT id, StartDate, EndDate, Title FROM Table_One WHERE id={$id} AND Title='{$SearchTerm}';";
    $Query = mysql_query($SQL) or die('Query Error: '.mysql_error());
    $Row = mysql_fetch_array($Query, MYSQL_ASSOC);

    if(!empty($Row['Title'])) {
      print $Row['Title'];
    }
  }

<b>A couple experiments with sql injection.</b>

LOAD%20DATA%20INFILE%20&#8242;/home/httpd/vhosts/ajohnstone.com/httpdocs/index.php&#8217;%20INTO%20TABLE%20test.OUTRUN;
LOAD DATA INFILE &#8216;/root/.bash_profile&#8217; INTO TABLE test.OUTRUN;

string(96) &#8220;LOAD DATA INFILE &#8216;/home/httpd/vhosts/ajohnstone.com/httpdocs/index.php&#8217; INTO TABLE test(Output);&#8221;
Query Error: Access denied for user &#8216;user@&#8217;localhost&#8217; (using password: YES)
</pre>
<p></code></p>
<ul>
<li><a href="http://www.ilia.ws/archives/103-mysql_real_escape_string-versus-Prepared-Statements.html">mysql_real_escape_string versus Prepared Statements</a></li>
<li><a href="http://shiflett.org/archive/184">The addslashes() Versus mysql_real_escape_string() Debate</a></li>
</ul>
No Tags]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/more-on-sql-injection/feed/</wfw:commentRss>
		</item>
		<item>
		<title>Serialized Objects, Heredoc, and strings&#8230;</title>
		<link>http://www.ajohnstone.com/archives/119/</link>
		<comments>http://www.ajohnstone.com/archives/119/#comments</comments>
		<pubDate>Tue, 01 Aug 2006 08:43:40 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://blog.ajohnstone.com/archives/119/</guid>
		<description><![CDATA[It&#8217;s been some time since I last posted, due a rather hectic schedule and today I fly off to Loret De Mar, Spain for a week with the lads and two weeks after that to Miami. I&#8217;m sure this will help to take my mind of things.
As with anything you constantly learn from your mistakes [...]]]></description>
			<content:encoded><![CDATA[<p>It&#8217;s been some time since I last posted, due a rather hectic schedule and today I fly off to Loret De Mar, Spain for a week with the lads and two weeks after that to Miami. I&#8217;m sure this will help to take my mind of things.</p>
<p>As with anything you constantly learn from your mistakes and encounter various challenges and I have found that there are some problems with PHP itself from a recent and ongoing project.</p>
<p><strong>Serialized Objects, Heredoc, and strings. (PHP Version 4.3.2)</strong><br />
Writing a serialized object to a a flat published file can hold its benefits, however this can become exceptionally large. One of the fundamental issues I found was that a fairly large serialized object inside the Heredoc syntax, would halt execution of php. Apache would log memory limits in the apache logs, however using base64 encoding and decoding on a serialized object and placing it within string literals, would execute fine. From observation, the fault would incur directly during the closing chevron&#8217;s.</p>
<p><strong>URL&#8217;s with double slashe at the beginning</strong></p>
<p>A single slash at the root directory will point to the sites root directory. e.g.<br />
<code><br />
  &lt;a href="/root_script.php"&gt;/root&lt;/a&gt;<br />
</code></p>
<p>I&#8217;m quite surprised i&#8217;ve never encountered this issue before, however you can manage to get URL&#8217;s to point to http://directory_name/script_name.php when the target is actually pointing to http://www.thehistorychannel.co.uk//directory_name/script_name.php for example, (note the double slash at the root directory).</p>
<p>When the script&#8217;s action simply contains:</p>
<p><code><br />
< ?php<br />
  print "&lt;form action="{$_SERVER['PHP_SELF']}" method="post"&gt;";<br />
?><br />
</code></p>
<p>I&#8217;m not sure if anyone else has encountered this problem?</p>
No Tags]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/119/feed/</wfw:commentRss>
		</item>
		<item>
		<title>Switching to Mac</title>
		<link>http://www.ajohnstone.com/archives/switching-to-mac/</link>
		<comments>http://www.ajohnstone.com/archives/switching-to-mac/#comments</comments>
		<pubDate>Tue, 25 Apr 2006 19:54:23 +0000</pubDate>
		<dc:creator>Andrew Johnstone</dc:creator>
		
		<category><![CDATA[General]]></category>

		<guid isPermaLink="false">http://blog.ajohnstone.com/archives/switching-to-mac/</guid>
		<description><![CDATA[Well, I purchased a Mac Book Pro roughly a month ago and considering I have never used a Mac, i&#8217;m quite surprised as to how quickly I have adapted. Within a week I had a good grasp of the platform and now I have everything sync&#8217;d to or through my laptop, whether that be incoming [...]]]></description>
			<content:encoded><![CDATA[<p>Well, I purchased a <acronym title="MBP">Mac Book Pro</acronym> roughly a month ago and considering I have never used a Mac, i&#8217;m quite surprised as to how quickly I have adapted. Within a week I had a good grasp of the platform and now I have everything sync&#8217;d to or through my laptop, whether that be incoming phone calls on my mobile or syncronising data between multiple servers.</p>
<p>The fundamental aspects I like about Mac is the degree of control, you have over the OS with applescript or shell. As mentioned above, I am now making my Mac work for me with proximity detection via bluetooth, and/or trigger applescripts when motion is detected via the iSight. I find this extremely useful to actually log phone calls in iCal, or pause iTunes when on the phone. I&#8217;m also tempted to write some applescript to automatically log me onto the the intranet at work, and track my time keeping for me, with aspects such as this it just makes my life that little bit easier.</p>
<p>With that said, i&#8217;m not a complete Mac convert, as I have got my windows machine at work &#038; at home constantly shelled into the Mac, I&#8217;m finding that I am only using windows to test against IE 6 or as an extra keyboard. Despite some of its gains I am still yet to figure out how to get my Exchange account at work to work completely remotely with it using a reverse proxy, I guess thats simply time and patience I need to get that setup. I guess the key things now are finding out what else is possible, and making my development experience better.</p>
<p>Anyway, I found the following links useful.</p>
<p><a href="http://www.kernelthread.com/mac/osx/tools.html">www.kernelthread.com</a><br />
<a href="http://www.opensourcemac.org/">www.opensourcemac.org</a><br />
<a href="http://www.petermaurer.de/nasi.php?thema=witch">www.petermaurer.de</a></p>
<p>Oh, and a pointer from Eric&#8230;<br />
<a href="http://www.codingmonkeys.de/subethaedit/">SubEthaEdit from CodingMonkeys</a><br />
<a href="http://maczot.com/">BLOGZOT 2.0 on MacZOT.com</a><br />
MacZOT and TheCodingMonkeys will award $105,000 in Mac software, which seems pretty generous to me:).</p>
No Tags]]></content:encoded>
			<wfw:commentRss>http://www.ajohnstone.com/archives/switching-to-mac/feed/</wfw:commentRss>
		</item>
	</channel>
</rss>
