Development, Analysis And Research

The Project

I was recently working on a project to expose our trading systems via XmlRpc, Rest and SOAP. It was quite an interesting project, which took two of us three weeks to develop (Amongst other things).

This involved creating a testbed, that would automatically generate the payload and response for each protocol. The parameters are introspected for each class method capturing each parameters data type, allowing for user input via standard html forms. This is probably best described with a picture or two.

Most of the documentation was generated via reflection and comments within the docblocks, parameters, notes were also generated making it quick and simple to update. In addition to parsing the start and end line of each method for any applicable error codes/faults that may be returned.

Zend Framework

Using the Zend Framework for the first time in a commercial product was not exactly hassle free, and still has quite a few issues with its webservices implementation. Currently there seems to be quite a bit of confusion regarding its Rest implementation and whether it is to be merged, would be great if someone clarify this.

The main issue I found with the Zend Frameworks implementation of XmlRpc and Rest is that it assumes that the payload it receives is valid. During my development, I tended to mix the payloads from SOAP, XmlRpc and Rest, yet it would assume that simple_xml can parse the input.

For example $this->_sxml is assumed to be a valid object, if not you will either get invalid method call or an undefined index, which doesn’t render well for an xmlrpc server.

    /**
     * Constructor
     *
     * @param string $data XML Result
     * @return void
     */
    public function __construct($data)
    {
        $this->_sxml = simplexml_load_string($data);
    }

    /**
     * toString overload
     *
     * Be sure to only call this when the result is a single value!
     *
     * @return string
     */
    public function __toString()
    {
        if (!$this->getStatus()) {
            $message = $this->_sxml->xpath('//message');
            return (string) $message[0];
        } else {
            $result = $this->_sxml->xpath('//response');
            if (sizeof($result) > 1) {
                return (string) "An error occured.";
            } else {
                return (string) $result[0];
            }
        }
    }

One of the main issues with Rest was that it needed ksort when using the Rest client as the arguments were not necessarily passed in order. This can be “rest.php?method=x&arg1=1&arg0=0″ and it would interpret each arg in the order it received them. This should be sorted in the next release of the ZF.

As the webservices we are exposing needs to have quite good performance with the number of transactions it will be handling and the amount of reflection that Zend Server Reflection (Only noticed after I started profiling) performs and I wanted to optimize any overhead, which got me looking at Zend_XmlRpc_Server_Cache. First thing I did was profile Zend_XmlRpc_Server_Cache, which added a considerable amount of overhead. Looking at its implementation, it uses serialize, which is a relatively slow process and should be avoided, unless there is a large overhead in initializing objects. So most likely Zend_XmlRpc_Server_Cache will not add any benefit. And var_dump’ing out the reflection in XmlRpc spews out a shocking amount of information on some fairly large classes.

  if (!Zend_XmlRpc_Server_Cache::get($cacheFile, $server)) {

  }

Generating WSDL

I tried a number of WSDL generators including the implementation in incubator for ZF, which I found to be the best, yet I still had to write a large chunk of the WSDL by hand and adapt it.

The best way to debug is to run the soap client with verbose mode on, and it will typically tell you the issue straight away.

• Zend_Soap_AutoDiscover: Duplicates an operation in WSDL for methods with parameters that are optional. (ZF-2642)
• Zend_Soap_AutoDiscover: If missing the @return in your docblock the message response in the WSDL is not generated. (ZF-2643)
• AutoDiscover duplicates response if using set class multiple times. (ZF-2641 )
• One of my colleagues typically writes their docblocks with “@return int, comment.”, which the comma caused return types to be dropped with AutoDiscover, more of an issue with Zend Server Reflection

Other odd issues

Raw input bug

Some other obscurities I found was capturing the raw request data. In our local development environment reading the raw request input, and then once again within the Zend Frameworks appears to work fine. However in our pre-production environment it fails to read the second request to read the raw request. (PHP 5.2.2)


if (!isset($HTTP_RAW_POST_DATA)){
$HTTP_RAW_POST_DATA = file_get_contents('php://input');
}


It does seem a little odd that the XmlRpc does not check whether $HTTP_RAW_POST_DATA isset before attempting to re-read raw input.

Internal error: Wrong return type

Whilst running PHPUnit I noticed a very weird quirk in our local dev environment, which essentially did the following… You would expect this to output the contents of an array right? Well between the method call to x and return the result back to method y returns NULL. This is very obscure and i’ve never seen anything like it especially considering it is explicitly set. I had a number of colleagues check this, which had us all scratching our heads. Has anyone else seen anything similar to this?

class test {

  public function x() {
    $ret = array();
    for(...) {
      $ret[] = $row;
    }
    return $ret;
  }

  public function y() {
    $response = $this->x();
    var_dump($response);
  }
}

$t = new test();
$t->y();

Conclusion

Overall the project went pretty well, I’m confident it is now stable especially with the number of tests we ran against it. It is adaptable to other projects that we may need to expose via an API, in total there is about 6000 lines of code alone just testing the 3 different protocols it supports. I would have rather avoided the Rest implementation with ZF as it still needs a lot of work, however XmlRpc is a lot more stable and I would quite happily use again. As there is a lot of overhead with reflection it is not the fastest implementation and was contrasted to some of the heavier web pages we have for some simple functionality. It would be ideal to replace the reflection with something lighter such as an array with the corresponding methods, parameters and types, however I would have to look into that if performance did become a major issue.

PS. Just to note I used PHP’s in built soap server.

General

Popularity: 31% [?]

Just came across this whilst creating an XML HTTP object and creating a socket to a page that doesn’t exist. I couldn’t understand what was going on when I saw that.

<!--
- Unfortunately, Microsoft has added a clever new
- "feature" to Internet Explorer. If the text of
- an error's message is "too small", specifically
- less than 512 bytes, Internet Explorer returns
- its own error message. You can turn that off,
- but it's pretty tricky to find switch called
- "smart error messages". That means, of course,
- that short error messages are censored by default.
- IIS always returns error messages that are long
- enough to make Internet Explorer happy. The
- workaround is pretty simple: pad the error
- message with a big comment like this to push it
- over the five hundred and twelve bytes minimum.
- Of course, that's exactly what you're reading
- right now.
-->

Popularity: 2% [?]

Today I found an odd bug in MSXML.

<!--
  <xsl:comment> -----BEGIN call-template name="article" EMPTY PathID----- </xsl:comment>
-->

msxml3.dll error '80004005'

The stylesheet does not contain a document element. The stylesheet may be empty, or it may not be a well-formed XML document.

"C:\DOMAINS\HOSTELHOST.COM\WWWROOT\CLIENT\DK\EYEWITNESS\../includes/src/default.asp, line 43″

MSXML, XML, XSL

Popularity: 2% [?]

After toying with some XPath’s and a fair amount of XSL lately (& getting lost in XPath), I feel it would be wise to place something here to look back on…

Marcus Tucker, introduced me to Stylus Studio of which I am very thankful as XmlSpy didn’t help me to deduce the problem with one of the XPaths, (infact ‘Select Previous Link’)

So heres some snippets.

Select Previous Link - Select First Preceding Node @id with with specified PathID
//project/article[@id=$PathID]/preceding-sibling::article[1]/@id
Select Next Link - Select Following Sibling @id with with specified PathID
//project/article[@id=$PathID]/following-sibling::article/@id

Also using the following are very handy for helping you know where abouts you are:
<xsl:copy-of select="parent::*">
<xsl:copy-of select="position()">

<xsl:template match="*" mode="namespace">
   <xsl:for-each select="namespace::*">
      <xsl:if test="not(../ancestor::*[namespace::*[name() = name(current()) and . = current()]][last()])">
         <xsl:value-of select="name(..)" /> defines <xsl:text />
         <xsl:choose>
            <xsl:when test="name()">xmlns:<xsl:value-of select="name()" /></xsl:when>
            <xsl:otherwise>xmlns</xsl:otherwise>
         </xsl:choose>
         <xsl:text />="<xsl:value-of select="." />"<xsl:text />
      </xsl:if>
   </xsl:for-each>
</xsl:template>

  <!-- generate namespace declarations as needed for current
	element node and its attributes.
  -->
  <xsl:template name="findNamespace">
    <xsl:variable name="curnode" select="." />

    <xsl:for-each select=".|@*">
      <xsl:variable name="vName" select="substring-before(name(), ':')"/>
      <xsl:variable name="vUri" select="namespace-uri(.)"/>

      <xsl:variable name="vAncestNamespace">
        <xsl:call-template name="findAncNamespace">
          <xsl:with-param name="pName" select="$vName"/>
          <xsl:with-param name="pUri" select="$vUri"/>
          <xsl:with-param name="pNode" select="$curnode" />
        </xsl:call-template>
      </xsl:variable>

      <xsl:if test="not(number($vAncestNamespace))">
        <!-- not sure if "parent::*" should be "$curnode/parent::*" -->
        <xsl:if test="parent::* or namespace-uri() or contains(name(), ':')">
          <xsl:text> </xsl:text>
          <span class="namespace">
            <xsl:value-of select="'xmlns'"/>
            <xsl:if test="contains(name(), ':')">
              <xsl:value-of select="concat              (':', $vName)"/>
            </xsl:if>
          </span>
          <span class="markup">="</span>
          <span class="namespace">
            <xsl:value-of select="namespace-uri()"/>
          </span>
          <span class="markup">"</span>
        </xsl:if>
      </xsl:if>
    </xsl:for-each>
  </xsl:template>

  <!-- Return 1 if an ancestor of pNode (or an attribute thereof)
    has the same prefix (pName) and namespace-uri (pUri) as
    the current node, and there is no closer ancestor (or attribute
    thereof) that has the same prefix but a different namespace-uri. Return 0 otherwise. -->
  <xsl:template name="findAncNamespace">
    <xsl:param name="pNode" select="."/>
    <xsl:param name="pName" select="substring-before(name(), ':')"/>
    <xsl:param name="pUri" select="namespace-uri(.)"/>

     <xsl:choose>
      <xsl:when test="not($pNode/parent::*)">0</xsl:when>
      <xsl:otherwise>
        <xsl:choose>
          <xsl:when test="($pNode/.. | $pNode/../@*)
          [$pName = substring-before(name(), ':') and
          $pUri = namespace-uri()]">
            1
          </xsl:when>
          <xsl:when test="($pNode/.. | $pNode/../@*)
          [$pName = substring-before(name), ':')]">
          <!-- and $pUri !=namespace-uri() -->
            0
          </xsl:when>
          <xsl:otherwise>
            <xsl:call-template name="findAncNamespace">
              <xsl:with-param name="pNode" select="$pNode/.."/>
              <xsl:with-param name="pName" select="$pName"/>
              <xsl:with-param name="pUri" select="$pUri"/>
            </xsl:call-template>
          </xsl:otherwise>
        </xsl:choose>
      </xsl:otherwise>
    </xsl:choose>

  </xsl:template>

I also wrote up some XSL to retreive parameters from XSL being parsed on the client, now I haven’t tested this although it may be beneficial to someone trying to extract parameters on the client-side.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:eg="http://schemas.example.com/"
>
<xsl:output
  omit-xml-declaration="yes"
  method = "html"
  indent = "no"
  encoding = "UTF-8"

/>
<msxsl:script language="JScript" implements-prefix="eg">
  function GetParam(name) {
    var start=location.search.indexOf("?"+name+"=");
    if (start<0) start=location.search.indexOf("&"+name+"=");
    if (start<0) return '';
    start += name.length+2;
    var end=location.search.indexOf("&",start)-1;
    if (end<0) end=location.search.length;
    var result=location.search.substring(start,end);
    var result='';

    for(var i=start;i<=end;i++) {
      var c=location.search.charAt(i);
        result=result+(c=='+'?' ':c);
      }
      return unescape(result);
    }
</msxsl:script>

<xsl:template match="/">
  <xsl:value-of select="eg:GetParam('Query')" />
</xsl:template>
<xsl:stylesheet>

XML, XPATH, XSL

Popularity: 2% [?]

Lately I’ve been looking into a new topic.

There are a couple of different services of which can get quite easily mixed up. Firstly “WTVML is a markup language that provides a content authoring format for interactive television service applications based on Internet standards.” and also “Sky Active, which is an interactive entertainment channel that is a cross between a television channel and a magazine”.

The possiblities of SkyActive & SkyInteractive are not quite clear as yet, however finding the time todo further research & development into the above is holding me back at present. However it would be nice to hear back from others who have had the time to look into or are currently developing for either of the above.

For more information take a look at (skyinteractive.com / skyactive.tv)

Schemas:
http://waptv.com/xsd/v6/wtvml_6.0.xsd
DTD:
http://waptv.com/DTD/wtvml_1.0.dtd

More References

WTVML, XML

Popularity: 3% [?]

Whilst using the Zend Platform on a clustered server enviroment, I came across a number of limitations, however many of these have now been resolved with the beta release of the Zend Platform named Buran.

I am excited to hear that the following features are/have been implemented in the Zend Platform.

• PHP level session clustering (beta starting next week) - better performance than database or NFS solutions
• User defined alert triggers and alert hints to allow better drill-down into scripts with many includes.
• Reporting and graphs
• event aggregation by server groups - to prevent alert storms in large clusters
• PHP CLI/CGI mode monitoring

“Just to give you some history - as we were the first to bring out profiling for PHP (which doesn’t require any setting of flags in code) we have been working on other features until people started using it more. You’ll probably be astounded to know that many people don’t use profiling and don’t even know what it is….

Anyway, this is this second such request we’ve had in a week for improvements to the profiler so it’s now been moved up the priority list, I’m not sure whether it will be part of Zend Platform or part of Zend Studio, but I’ll collect some more details from you on Monday and we’ll take it from there.”

Please shout and scream to make this a priority!

• Zend Studio Server Error (eventest.php?action=test&event_id=993)
  In order to connect to HTTPS addresses, you must use PHP higher than 4.3.0
  —
  The current version of PHP on the production servers is PHP5.0.4
• Server & Vhost
  Server: mtvi-ws2.hos.ws.com
  VHost: lamp.ws.com
  —
  Desired VHost: lamp.ws.com/~user
  You don’t know which UserDir it is…
  Vhost is empty in some circumstances.
  (Apparently resolved in the beta)
• Performance | Studio Server tabs
  after selecting a server it pushes you back to enter a password; I assume this is because of insufficient privileges? However why doesn’t it state that, instead it states “invalid password‿.
• Currently all developers on the production server use a single user account on the Zend Platform (Not ideal really), however the developers account forbids any updates on events, ie close, ignore, delete.
  I’m still not sure whether events are logged globally and that each user account updates the status of that global event thus being the reason hosting & operations have disabled this feature?
• Grouping and aggregating of events. I am told this is in the Buran Beta, however my understanding is that it is somewhat limited.
• Comments to individual / grouped events would be a very handy feature forexample, steps to reproduce, fixes, or quick hacks that have been applied, hopefully will be added to the Zend Platform

Heres some API information that I have been sent, which is currently not live at present…

(more…)

Buran Beta, PHP, XML, Zend Platform

Popularity: 9% [?]