atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[TheLadders Engineering Stories]]></title>
  <link href="http://dev.theladders.com/atom.xml" rel="self"/>
  <link href="http://dev.theladders.com/"/>
  <updated>2015-05-15T09:54:12-04:00</updated>
  <id>http://dev.theladders.com/</id>
  <author>
    <name><![CDATA[TheLadders Engineering]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[Working in Harmony: ES6 in a nutshell]]></title>
    <link href="http://dev.theladders.com/2015/05/working-in-harmony-es6-in-a-nutshell/"/>
    <updated>2015-05-04T16:05:00-04:00</updated>
    <id>http://dev.theladders.com/2015/05/working-in-harmony-es6-in-a-nutshell</id>
    <content type="html"><![CDATA[<blockquote><p>&#8220;JavaScript is a language with more than its share of bad parts.&#8221;</p><footer><strong>&#8211;Douglas Crockford</strong> <cite>JavaScript: The Good Parts</cite></footer></blockquote>


<h1>Harmony</h1>

<p>JavaScript, also known as ECMAScript, has extended its life beyond the boundaries of the browser. It can be found in <a href="https://nodejs.org/">server side code</a>, in <a href="http://docs.unity3d.com/Manual/CreatingAndUsingScripts.html">video game engines</a>, even frameworks to run C programs in the <a href="http://asmjs.org/">browser</a>. Unfortunately, the language specification has not been updated since 2009, which in turn has led to different frameworks and tools to accommodate for the lack of features.  Luckily, TC-39, the committee for approving ECMAScript, has a targeted release date of June 2015 for the next version of ECMAScript (ES6). ECMAScript is a significant update to the language and brings a slew of new features.</p>

<p>Here are several of the important features that will make the ES6 release:</p>

<h1>Modules</h1>

<p>Modules provide private and public encapsulation to limit the creation of variables and functions on the global scope. They are part of a popular design pattern that has resulted in several frameworks such as <a href="http://requirejs.org/">RequireJS</a>, <a href="http://browserify.org/">Browserify</a>, and <a href="http://webpack.github.io/">Webpack</a>. ES6 provides a native solution that combines attributes of both CommonJS and AMD.</p>

<ul>
<li>Similar to CommonJS, native modules provide a compact syntax, a preference for single exports and support for cyclic dependencies.</li>
<li>Similar to AMD, they have direct support for asynchronous loading and configurable module loading.</li>
</ul>


<p>Let&rsquo;s look at an example of using native modules. Within a utility file, we have two functions, sumOfNumbers and logMessage. The module pattern allows us to choose which functions or attributes we want to expose to our consumers. There are two kinds of exports:</p>

<ul>
<li><strong>named exports</strong> (several per module), where the user specifies the name of the attribute on import.</li>
<li><strong>default exports</strong> (one per module), where the compiler picks the export item labeled with the default keyword when it cannot match a named export.</li>
</ul>


<p><em>Note:</em> You can run the following examples in babel&rsquo;s <a href="https://babeljs.io/repl/">live script editor</a>.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'><span class="c1">// lib/utility.js</span>
</span><span class='line'><span class="kr">export</span> <span class="kd">function</span> <span class="nx">sumOfNumbers</span><span class="p">(){</span>
</span><span class='line'> <span class="kd">var</span> <span class="nx">numbers</span> <span class="o">=</span> <span class="nb">Array</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">slice</span><span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="nx">arguments</span><span class="p">);</span>
</span><span class='line'> <span class="k">return</span> <span class="nx">numbers</span><span class="p">.</span><span class="nx">reduce</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">x</span><span class="p">,</span><span class="nx">y</span><span class="p">){</span> <span class="k">return</span> <span class="nx">x</span> <span class="o">+</span> <span class="nx">y</span> <span class="p">});</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kr">export</span> <span class="k">default</span> <span class="kd">function</span> <span class="nx">logMessage</span><span class="p">(</span><span class="nx">msg</span><span class="p">){</span>
</span><span class='line'>  <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">&#39;Message is &#39;</span><span class="p">,</span> <span class="nx">msg</span><span class="p">);</span>
</span><span class='line'><span class="p">};</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// app.js</span>
</span><span class='line'><span class="kr">import</span> <span class="p">{</span><span class="nx">logMessage</span><span class="p">,</span> <span class="nx">sumOfNumbers</span><span class="p">}</span> <span class="nx">from</span> <span class="s1">&#39;utility&#39;</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'><span class="nx">logMessage</span><span class="p">(</span><span class="nx">sumOfNumbers</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">));</span> <span class="c1">// Message is 15</span>
</span><span class='line'><span class="nx">logMessage</span><span class="p">(</span><span class="nx">sumOfNumbers</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">10</span><span class="p">));</span> <span class="c1">// Message is 19</span>
</span></code></pre></td></tr></table></div></figure>


<h1>Classes</h1>

<p>ES6 brings classes to the language. The class implementation still uses prototypical inheritance, so it is syntactic sugar to make it easier to implement and understand.  Using the <strong>extends</strong> keyword provides the following benefits:</p>

<ul>
<li>Binds the correct context when using the <em>extends</em> keyword.</li>
<li>Methods defined within the class block are defined on the prototype object.</li>
<li>Provides the <em>super</em> keyword to easily invoke parent methods.</li>
</ul>


<p>As a simple example of class inheritance, let&rsquo;s create a shape class with a triangle child class extending from shape.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'><span class="c1">//Traditional way to create inheritance</span>
</span><span class='line'><span class="kd">function</span> <span class="nx">Shape</span><span class="p">(</span><span class="nx">width</span><span class="p">,</span> <span class="nx">height</span><span class="p">){</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">width</span> <span class="o">=</span> <span class="nx">width</span><span class="p">;</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">height</span> <span class="o">=</span> <span class="nx">height</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nx">Shape</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">printObject</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
</span><span class='line'>  <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">&#39;Shape&#39;</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">width</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">height</span><span class="p">);</span>
</span><span class='line'><span class="p">};</span>
</span><span class='line'>
</span><span class='line'><span class="kd">function</span> <span class="nx">Triangle</span><span class="p">(</span><span class="nx">width</span><span class="p">,</span> <span class="nx">height</span><span class="p">){</span>
</span><span class='line'><span class="c1">//Note: not a convenient way to invoke parent constructor</span>
</span><span class='line'><span class="c1">// we have to call the parent object with Triangle&#39;s context.</span>
</span><span class='line'>  <span class="nx">Shape</span><span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nx">width</span><span class="p">,</span> <span class="nx">height</span><span class="p">);</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">sides</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nx">Triangle</span><span class="p">.</span><span class="nx">prototype</span> <span class="o">=</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="nx">Shape</span><span class="p">.</span><span class="nx">prototype</span><span class="p">);</span> <span class="c1">//prototype object now inheritances the properties of Shape</span>
</span><span class='line'>
</span><span class='line'><span class="nx">Triangle</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">printObject</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(){</span>
</span><span class='line'>  <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">&#39;Triangle&#39;</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">width</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">height</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">sides</span><span class="p">);</span>
</span><span class='line'><span class="p">};</span>
</span></code></pre></td></tr></table></div></figure>




<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'><span class="c1">//ES6 Way to create inheritance</span>
</span><span class='line'><span class="kr">class</span> <span class="nx">Shape</span><span class="p">{</span>
</span><span class='line'>  <span class="nx">width</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="c1">// properties of the class</span>
</span><span class='line'>  <span class="nx">height</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="c1">// note: they are NOT private</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">//function called with you call Shape with &#39;new&#39; keyword</span>
</span><span class='line'>  <span class="nx">constructor</span><span class="p">(</span><span class="nx">width</span><span class="p">,</span> <span class="nx">height</span><span class="p">){</span>
</span><span class='line'>    <span class="k">this</span><span class="p">.</span><span class="nx">width</span> <span class="o">=</span> <span class="nx">width</span><span class="p">;</span>
</span><span class='line'>    <span class="k">this</span><span class="p">.</span><span class="nx">height</span> <span class="o">=</span> <span class="nx">height</span><span class="p">;</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">//method part of shape</span>
</span><span class='line'>  <span class="nx">printObject</span><span class="p">(){</span>
</span><span class='line'>    <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">&#39;Shape&#39;</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">width</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">height</span><span class="p">);</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kr">class</span> <span class="nx">Triangle</span> <span class="kr">extends</span> <span class="nx">Shape</span><span class="p">{</span>
</span><span class='line'>  <span class="nx">sides</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="nx">constructor</span><span class="p">(</span><span class="nx">width</span><span class="p">,</span> <span class="nx">height</span><span class="p">){</span>
</span><span class='line'>    <span class="c1">//invokes the parent constructor by calling super</span>
</span><span class='line'>    <span class="c1">// automatically applies the correct context.</span>
</span><span class='line'>    <span class="kr">super</span><span class="p">(</span><span class="nx">width</span><span class="p">,</span> <span class="nx">height</span><span class="p">);</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">//Overrides the method on the parent object</span>
</span><span class='line'>  <span class="nx">printObject</span><span class="p">(){</span>
</span><span class='line'>    <span class="c1">//Draw the triangle</span>
</span><span class='line'>    <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">&#39;Triangle&#39;</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">width</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">height</span><span class="p">);</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>We can apply this new syntax to existing frameworks such as Backbone.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'><span class="c1">//current backbone implementation</span>
</span><span class='line'><span class="kd">var</span> <span class="nx">MyView</span> <span class="o">=</span> <span class="nx">Backbone</span><span class="p">.</span><span class="nx">View</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
</span><span class='line'>  <span class="nx">template</span><span class="o">:</span> <span class="s1">&#39;&lt;div&gt;Sample Template&lt;/div&gt;&#39;</span><span class="p">,</span>
</span><span class='line'>  <span class="nx">events</span><span class="o">:</span> <span class="p">{</span>
</span><span class='line'>    <span class="s1">&#39;click&#39;</span> <span class="o">:</span> <span class="s1">&#39;clickHandler&#39;</span>
</span><span class='line'>  <span class="p">},</span>
</span><span class='line'>  <span class="nx">render</span><span class="o">:</span> <span class="kd">function</span><span class="p">(){</span>
</span><span class='line'>    <span class="k">this</span><span class="p">.</span><span class="nx">$el</span><span class="p">.</span><span class="nx">html</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">template</span><span class="p">);</span>
</span><span class='line'>    <span class="k">return</span> <span class="k">this</span><span class="p">;</span>
</span><span class='line'>  <span class="p">},</span>
</span><span class='line'>  <span class="nx">clickHandler</span><span class="o">:</span> <span class="kd">function</span><span class="p">(){</span>
</span><span class='line'>    <span class="nx">alert</span><span class="p">(</span><span class="s1">&#39;My view was clicked&#39;</span><span class="p">);</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'><span class="p">});</span>
</span><span class='line'>
</span><span class='line'><span class="c1">//Backbone using classes</span>
</span><span class='line'><span class="kr">class</span> <span class="nx">MyView</span> <span class="kr">extends</span> <span class="nx">Backbone</span><span class="p">.</span><span class="nx">View</span> <span class="p">{</span>
</span><span class='line'>  <span class="nx">constructor</span><span class="p">(</span><span class="nx">options</span><span class="p">){</span>
</span><span class='line'>    <span class="k">this</span><span class="p">.</span><span class="nx">template</span> <span class="o">=</span> <span class="s1">&#39;&lt;div&gt;Sample Template&lt;/div&gt;&#39;</span><span class="p">;</span>
</span><span class='line'>    <span class="k">this</span><span class="p">.</span><span class="nx">events</span> <span class="o">=</span> <span class="p">{</span>
</span><span class='line'>      <span class="s1">&#39;click&#39;</span> <span class="o">:</span> <span class="s1">&#39;clickHandler&#39;</span>
</span><span class='line'>    <span class="p">};</span>
</span><span class='line'>    <span class="kr">super</span><span class="p">(</span><span class="nx">options</span><span class="p">);</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="nx">render</span><span class="p">(){</span>
</span><span class='line'>    <span class="k">this</span><span class="p">.</span><span class="nx">$el</span><span class="p">.</span><span class="nx">html</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">template</span><span class="p">);</span>
</span><span class='line'>    <span class="k">return</span> <span class="k">this</span><span class="p">;</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="nx">clickHandler</span><span class="p">(){</span>
</span><span class='line'>    <span class="nx">alert</span><span class="p">(</span><span class="s1">&#39;My view was clicked&#39;</span><span class="p">);</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h1>Arrow Functions</h1>

<p>ES6 has also added arrow functions, which provide a for more concise syntax for anonymous functions. However, there are a number of differences between arrow functions and traditional JavaScript functions:</p>

<ul>
<li><strong>Lexical <em>this</em> binding</strong> &ndash; The value of <em>this</em> inside of the function is determined by where the arrow function is defined, not where it is used.</li>
<li><strong>Not <em>new</em>-able</strong> &ndash; Arrow functions do not have a constructor method and therefore can not be used to create instances. Arrow functions throw an error when called with &ldquo;new&rdquo;.</li>
<li><strong>Cannot change the value of <em>this</em></strong> &ndash; The value of <em>this</em> remains the same value throughout the lifecycle of the function.</li>
<li><strong>No <em>arguments</em> object</strong> &ndash; Function arguments are not accessible through the arguments object; you must use named arguments or ES6 <a href="http://wiki.ecmascript.org/doku.php?id=harmony:rest_parameters">rest arguments</a>.</li>
</ul>


<p>Here is a simple example using arrow functions.  We are iterating of a list of items and printing out each number to the console.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'>  <span class="kd">var</span> <span class="nx">numbers</span> <span class="o">=</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">9</span><span class="p">,</span><span class="mi">10</span><span class="p">];</span>
</span><span class='line'>  <span class="c1">//Current javascript implementation</span>
</span><span class='line'>  <span class="nx">numbers</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">number</span><span class="p">){</span>
</span><span class='line'>    <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">number</span><span class="p">);</span>
</span><span class='line'>  <span class="p">};</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// ES6 - Arrow functions</span>
</span><span class='line'>  <span class="nx">numbers</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="nx">number</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">number</span><span class="p">);</span>
</span><span class='line'>  <span class="p">});</span>
</span></code></pre></td></tr></table></div></figure>


<p>Let&rsquo;s look at more practical example using an event handler:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'><span class="c1">//Current Javascript implementation</span>
</span><span class='line'><span class="kd">function</span> <span class="nx">Person</span><span class="p">(){</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">$el</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="s1">&#39;button&#39;</span><span class="p">);</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">text</span> <span class="o">=</span> <span class="s2">&quot;Hello I am a person&quot;</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">//Problem with current Javascript that we have to &quot;bind&quot; the person context </span>
</span><span class='line'>  <span class="c1">// in order to get the callback to work properly.  Without &quot;bind&quot;, the function</span>
</span><span class='line'>  <span class="c1">// would print out undefined.</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">$el</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">&#39;click&#39;</span><span class="p">,</span> <span class="kd">function</span><span class="p">(){</span>
</span><span class='line'>    <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">text</span><span class="p">);</span>
</span><span class='line'>  <span class="p">}.</span><span class="nx">bind</span><span class="p">(</span><span class="k">this</span><span class="p">));</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">//ES6</span>
</span><span class='line'><span class="kd">function</span> <span class="nx">Person</span><span class="p">(){</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">$el</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="s1">&#39;button&#39;</span><span class="p">);</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">text</span> <span class="o">=</span> <span class="s2">&quot;Hello I am a person&quot;</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">//No longer need to &quot;bind&quot; the function since it is set to where it is defined.</span>
</span><span class='line'>  <span class="k">this</span><span class="p">.</span><span class="nx">$el</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">&#39;click&#39;</span><span class="p">,</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">text</span><span class="p">);</span>
</span><span class='line'>  <span class="p">});</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Arrow functions also provide different types of optional syntax.  Here are several examples:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'> <span class="kd">var</span> <span class="nx">numbers</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">];</span>
</span><span class='line'> <span class="kd">var</span> <span class="nx">temp</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'> <span class="c1">//arrow functions return last value, no need for return statement.</span>
</span><span class='line'> <span class="nx">temp</span> <span class="o">=</span> <span class="nx">numbers</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">number</span> <span class="o">=&gt;</span> <span class="nx">number</span> <span class="o">+</span> <span class="mi">2</span><span class="p">);</span>
</span><span class='line'> <span class="c1">//need empty parentheses if no parameters are passed</span>
</span><span class='line'> <span class="nx">temp</span> <span class="o">=</span> <span class="nx">numbers</span><span class="p">.</span><span class="nx">map</span><span class="p">(()</span> <span class="o">=&gt;</span> <span class="mi">5</span><span class="p">);</span>
</span><span class='line'> <span class="c1">//multiple parameters </span>
</span><span class='line'> <span class="nx">temp</span> <span class="o">=</span> <span class="nx">numbers</span><span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">x</span><span class="p">,</span><span class="nx">y</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span> <span class="o">+</span> <span class="nx">y</span><span class="p">);</span>
</span></code></pre></td></tr></table></div></figure>


<h1>Destructing Variables</h1>

<p>Destructuring allows binding via pattern matching. ES6 supports matching for arrays and objects. It will try to match the variable directly and return undefined when no match is found.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'> <span class="kd">var</span> <span class="nx">numbers</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">];</span>
</span><span class='line'> <span class="kd">var</span> <span class="nx">obj</span> <span class="o">=</span> <span class="p">{</span> <span class="nx">a</span><span class="o">:</span> <span class="mi">4</span><span class="p">,</span> <span class="nx">b</span><span class="o">:</span> <span class="s1">&#39;Hello&#39;</span><span class="p">,</span> <span class="nx">c</span><span class="o">:</span> <span class="kc">false</span> <span class="p">};</span>
</span><span class='line'>
</span><span class='line'> <span class="c1">//first = 1, third = 3</span>
</span><span class='line'> <span class="kd">var</span> <span class="p">[</span><span class="nx">first</span><span class="p">,</span> <span class="p">,</span> <span class="nx">third</span><span class="p">]</span> <span class="o">=</span> <span class="nx">numbers</span><span class="p">;</span>
</span><span class='line'> <span class="kd">var</span> <span class="p">{</span><span class="nx">a</span><span class="p">,</span><span class="nx">c</span><span class="p">}</span> <span class="o">=</span> <span class="nx">obj</span><span class="p">;</span>
</span><span class='line'> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">a</span><span class="p">);</span> <span class="c1">//prints out 4</span>
</span><span class='line'> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">c</span><span class="p">);</span> <span class="c1">//prints out false</span>
</span><span class='line'>
</span><span class='line'> <span class="c1">//assigning value of &#39;b&#39; in object to variable &#39;anotherName&#39;</span>
</span><span class='line'> <span class="kd">var</span> <span class="p">{</span><span class="nx">b</span><span class="o">:</span> <span class="nx">anotherName</span><span class="p">}</span> <span class="o">=</span> <span class="nx">obj</span><span class="p">;</span>
</span><span class='line'> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">anotherName</span><span class="p">);</span> <span class="c1">//prints out &#39;Hello&#39;</span>
</span></code></pre></td></tr></table></div></figure>


<h1>Default, Rest, Spread</h1>

<p>ES6 provides the flexibility to set default values for function parameters. It lets you grab trailing values without using the arguments keyword. And it also allows you to turn an array into consecutive arguments for a function call. With these options, there is no longer a need to use the arguments variable.</p>

<h5>Default Parameter</h5>

<ul>
<li>Current values that are not set are given a value of undefined. ES6 provides new syntax to add default values for parameters.</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'>  <span class="kd">function</span> <span class="nx">add2</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span> <span class="o">=</span> <span class="mi">2</span><span class="p">){</span>
</span><span class='line'>     <span class="k">return</span> <span class="nx">a</span> <span class="o">+</span> <span class="nx">b</span><span class="p">;</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>  <span class="nx">add2</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">5</span><span class="p">);</span> <span class="c1">// returns 7</span>
</span><span class='line'>  <span class="nx">add2</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span> <span class="c1">//return 4</span>
</span></code></pre></td></tr></table></div></figure>


<h5>Rest parameter</h5>

<ul>
<li>Currently, in order to grab all argument values as an array, we need to explicitly convert the values using the arguments keyword.</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'>  <span class="c1">// Current Javascript</span>
</span><span class='line'>    <span class="c1">//need to convert arguments into an array object in order to use any useful method.</span>
</span><span class='line'>  <span class="kd">function</span> <span class="nx">sum</span><span class="p">(){</span>
</span><span class='line'>    <span class="kd">var</span> <span class="nx">numbers</span> <span class="o">=</span> <span class="nb">Array</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">slice</span><span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="nx">arguments</span><span class="p">);</span>
</span><span class='line'>    <span class="k">return</span> <span class="nx">numbers</span><span class="p">.</span><span class="nx">reduce</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">y</span><span class="p">){</span> <span class="k">return</span> <span class="nx">x</span> <span class="o">+</span> <span class="nx">y</span><span class="p">;</span> <span class="p">});</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// ES6</span>
</span><span class='line'>    <span class="c1">// The conversion happens automatically. </span>
</span><span class='line'>    <span class="kd">function</span> <span class="nx">sum</span><span class="p">(...</span><span class="nx">numbers</span><span class="p">){</span>
</span><span class='line'>      <span class="k">return</span> <span class="nx">numbers</span><span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">x</span><span class="p">,</span><span class="nx">y</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span> <span class="o">+</span> <span class="nx">y</span><span class="p">);</span>
</span><span class='line'>    <span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<h5>Spread parameter</h5>

<ul>
<li>Spread functionality allows us to convert an array into consecutive arguments to pass into a function. Currently, we have to use the <em>apply</em> keyword in order to provide the same functionality.</li>
</ul>


<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'>  <span class="c1">//current implementation</span>
</span><span class='line'>  <span class="kd">var</span> <span class="nx">numbers</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">];</span>
</span><span class='line'>  <span class="nb">Math</span><span class="p">.</span><span class="nx">max</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nb">Math</span><span class="p">,</span> <span class="nx">numbers</span><span class="p">);</span> <span class="c1">//spreads the array and returns 5</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">//Future implementation</span>
</span><span class='line'>  <span class="kd">var</span> <span class="nx">numbers</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">];</span>
</span><span class='line'>  <span class="nb">Math</span><span class="p">.</span><span class="nx">max</span><span class="p">(...</span><span class="nx">numbers</span><span class="p">);</span> <span class="c1">//performs the same action</span>
</span></code></pre></td></tr></table></div></figure>


<h1>Current Alternatives</h1>

<p>The JavaScript specification is targeted to release in June, but it could take browsers months to <a href="http://kangax.github.io/compat-table/es6/">implement</a> these new features and years before users decide it&rsquo;s a good time to upgrade their browsers. Luckily, there are solutions which allow us to take advantage of these features right now. Here are several options:</p>

<h3><a href="https://babeljs.io/"><img src="https://raw.githubusercontent.com/babel/logo/master/logo.png" alt="Babel" width="250px"></a></h3>

<ul>
<li>A compiler for converting ES6 syntax to ES5-compatible javascript. Babel is among the most mature compilers for ES6 conversion with the options of extensible plugins as well as framework and tool integration.</li>
</ul>


<h3><a href="https://github.com/google/traceur-compiler"><img src="https://google.github.com/traceur-compiler/logo/tc.svg" alt="Traceur logo" width="200px"> Traceur</a></h3>

<ul>
<li>Google&rsquo;s version of Babel. Traceur is a JavaScript.next-to-JavaScript-of-today compiler that allows you to use features from the future today. Traceur supports ES6 as well as some experimental ES.next features.</li>
</ul>


<h3><a href="http://www.typescriptlang.org/"><img src="http://www.typescriptlang.org/content/images/logo_small.png" alt="Typescript"></a></h3>

<ul>
<li>Typescript is a typed superset of JavaScript that compiles to plain JavaScript. It also provides the option of applying types to variables and functions at compile time. TypeScript provides the flexibility of using ES6 features such as classes, modules and arrow function which ultimately compile to ES5 compatible JavaScript. Similar to Babel, Typescript has integrations with many development environments and definition files for existing frameworks.</li>
</ul>


<h3><a href="http://flowtype.org/"><img src="http://flowtype.org/static/flow-hero-logo.png" alt="Flow" width="200px"></a></h3>

<ul>
<li>Flow is a new static type checker for JavaScript. Flow is similar to Typescript in that it will compile down to plain JavaScript. The difference is that it provides valid checking for null variable type. Flow integrates well with other Facebook tools such as React and Flux.</li>
</ul>


<h1>Conclusion</h1>

<p>ECMAScript 6 provides an array of exciting tools and libraries that we can leverage to make development both easier and more exciting. Though we know that it could be months before we can use these tools natively, we have alternatives that allow developers to expand their toolsets and think of efficient and abstract solutions. If interested in learning more, here are several links:</p>

<h4>Resources:</h4>

<ul>
<li><a href="https://github.com/lukehoban/es6features">All ES6 Features</a></li>
<li><a href="https://leanpub.com/understandinges6/">Understanding ES6 &ndash; Nicolas Zakas</a></li>
<li><a href="https://babeljs.io/docs/learn-es6/">Try ES6 Compilation using Babel</a></li>
<li><a href="https://www.youtube.com/watch?v=DqMFX91ToLw">Evolution of Javascript &ndash; Netflix Javascript Talks</a></li>
<li><a href="https://people.mozilla.org/~jorendorff/es6-draft.html">EcmaScript 6 Draft</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Writing better API tests]]></title>
    <link href="http://dev.theladders.com/2015/04/writing-better-api-tests/"/>
    <updated>2015-04-27T17:41:00-04:00</updated>
    <id>http://dev.theladders.com/2015/04/writing-better-api-tests</id>
    <content type="html"><![CDATA[<blockquote><p>It is only through labor and painful effort, by grim energy and resolute courage, that we move on to better things.</p><footer><strong>&#8211;Theodore Roosevelt</strong></footer></blockquote>


<h1>The problem</h1>

<p>When I was first tasked with writing the web service for our <a href="http://www.theladders.com/mobile/">jobseeker iOS app</a>, I wanted to be able to write tests against the API without diving into the world of iOS automated functional testing.  I really didn’t see a need to fire up an iOS client just so I could make calls to a service that was under heavy development and in a constant state of change.  So how could I write tests that would exercise the API in a way similar to an iOS client?</p>

<p>Fortunately for me, engineers at TheLadders have always emphasized testing in one form or another.  Sometimes this means a unit test for a particular method on a class.  Other times it means an automated functional test (AFT) verifying client/service interactions for a particular feature.  And in some instances it may even mean, f**k it, ship it and see if it blows up in production (a rare occurrence of course).</p>

<p>It turns out that before I even started working on the mobile service, people before me were asking similar questions about testing their APIs, and they had come up with a slightly different type of test that we now call &ldquo;web tests&rdquo;.  A &ldquo;web test&rdquo; is basically a way to fire up an instance of the service for each test and, using a Jersey client, programmatically make calls to that service within the test itself.  This was great because it allowed us to write tests against our APIs without needing to fire up an external tool such as a web browser.  The following code is an example of such a test:</p>

<figure class='code'><figcaption><span>Basic web test</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">TestRestClient</span> <span class="n">client</span> <span class="o">=</span> <span class="n">createTestRestClient</span><span class="o">();</span>
</span><span class='line'><span class="n">Response</span> <span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="err">“</span><span class="n">api</span><span class="o">/</span><span class="n">jobs</span><span class="err">”</span><span class="o">);</span>
</span><span class='line'><span class="n">assertEquals</span><span class="o">(</span><span class="mi">200</span><span class="o">,</span> <span class="n">response</span><span class="o">.</span><span class="na">getStatus</span><span class="o">());</span>
</span></code></pre></td></tr></table></div></figure>


<p>This code makes a call to retrieve a list of jobs and verifies the response status code is 200.</p>

<p>The idea of the “web” or “API test” was exactly what I needed.  This gave me a starting point.  But I knew these web tests could be improved upon, making it easier to test our APIs.  So I set about trying to figure out what I wanted for testing the mobile service and came up with a list of requirements.</p>

<h1>The requirements</h1>

<p>There were some requirements I wanted to establish that would help guide my decision-making process when writing web tests.  These requirements included the following:</p>

<ul>
<li>The tests should mimic how an iOS client would navigate the API</li>
<li>Anyone reading the test code should be able to easily understand what is being tested</li>
<li>The code should be easy to reuse, reducing duplication where possible</li>
</ul>


<p>These really aren’t groundbreaking ideas, but they were helpful to keep in mind at all times.  They served as guideposts throughout my decision-making process.  With these requirements defined, I&rsquo;m going to give you a quick rundown of some key elements of the API before diving into implementation details.</p>

<h1>The API</h1>

<p>The API for the mobile service is a RESTful API that uses <a href="http://en.wikipedia.org/wiki/HATEOAS">hypermedia as the engine of application state (HATEOAS)</a>.  We use the HTTP methods GET, DELETE, POST, and PUT against URLs representing resources.  Everything is stateless and we leverage headers for things such as authorization and versioning.  The following figure illustrates sample request and response against this API:</p>

<p><img class="center medium" src="http://dev.theladders.com/images/api-tests/api-request-response.png"></p>

<p>The above example illustrates a request for the details of an individual job.  One important thing to note here is the “links” element in the JSON.  This is the HATEOAS piece I mentioned earlier. If a user taps the “like” button on the app for this particular job, the client code would send a request to the URL associated with the “like” action.  If the job was already liked when it was retrieved, then an “unlike” link would have been present instead.</p>

<p>We use hypermedia links extensively in the API.  In fact, the only URL that is hard-coded in the iOS application is what we call a “bootstrap” URL.  This URL gives the client all of the hypermedia links needed to get started with the app, and would look something like the following:</p>

<figure class='code'><figcaption><span>Bootstrap JSON response</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='json'><span class='line'><span class="p">{</span>
</span><span class='line'>  <span class="nt">&quot;links&quot;</span> <span class="p">:</span> <span class="p">[{</span>
</span><span class='line'>    <span class="nt">&quot;action&quot;</span> <span class="p">:</span> <span class="s2">&quot;authenticate&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;method&quot;</span> <span class="p">:</span> <span class="s2">&quot;POST&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;uri&quot;</span> <span class="p">:</span> <span class="s2">&quot;https://localhost:54837/authenticate&quot;</span><span class="p">,</span>
</span><span class='line'>  <span class="p">},</span> <span class="p">{</span>
</span><span class='line'>    <span class="nt">&quot;action&quot;</span> <span class="p">:</span> <span class="s2">&quot;lookupIndustries&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;method&quot;</span> <span class="p">:</span> <span class="s2">&quot;GET&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;uri&quot;</span> <span class="p">:</span> <span class="s2">&quot;https://localhost:54837/api/lookup/industries&quot;</span><span class="p">,</span>
</span><span class='line'>  <span class="p">},</span> <span class="p">{</span>
</span><span class='line'>    <span class="nt">&quot;action&quot;</span> <span class="p">:</span> <span class="s2">&quot;autocompleteTitle&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;method&quot;</span> <span class="p">:</span> <span class="s2">&quot;GET&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;uri&quot;</span> <span class="p">:</span> <span class="s2">&quot;https://localhost:54837/api/autocomplete/title?q={toMatch}&quot;</span><span class="p">,</span>
</span><span class='line'>  <span class="p">},</span> <span class="p">{</span>
</span><span class='line'>    <span class="nt">&quot;action&quot;</span> <span class="p">:</span> <span class="s2">&quot;register&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;method&quot;</span> <span class="p">:</span> <span class="s2">&quot;POST&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;uri&quot;</span> <span class="p">:</span> <span class="s2">&quot;https://localhost:54837/register&quot;</span><span class="p">,</span>
</span><span class='line'>  <span class="p">}]</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>From here, the client has the links for retrieving some lookup lists, maybe a URL for auto-completing the job title field, a link for logging in an existing user, and a link for registering a new user.  If the link to login is invoked, then new response JSON will be returned containing “links” for valid actions that user can perform, and so on and so forth.</p>

<p>There is more to the API and I’ve glossed over some details.  But this should give you enough of an idea of how things work to understand the implementation.</p>

<h1>The implementation</h1>

<p>It’s important to keep in mind that the following implementation has evolved over time.  What I present to you here is an example that identifies the key components of the current state.  I was always going back-and-forth with different ideas, trying out various implementations.  And I’m pretty certain the team will continue to iterate on what we have today.</p>

<p>But in its current state, the implementation has the following four components:</p>

<ol>
<li>Class representing request/response JSON</li>
<li>Class for performing asserts against a body of JSON</li>
<li>Class representing the actions that can be taken on a body of JSON (such as calling hypermedia links or accessing an item in a list)</li>
<li>Class representing a user, their state, and their “connection” to the API.</li>
</ol>


<p>These four components and how they relate to the API can be seen in the following figure:</p>

<p><img class="center medium" src="http://dev.theladders.com/images/api-tests/api-implementation-components.png"></p>

<p>In addition to these four components, I did my  best to provide some “syntactic sugar” for allowing us to write tests in the given/when/then style.  So let’s dive into each of the four components and give you some concrete examples to work with.</p>

<h2>Class representing request/response JSON</h2>

<p>We have what are called “representation” classes whose sole purpose is to “represent” incoming or outgoing JSON.  These classes are annotated with JAXB annotations for serialization/deserialization purposes.  These classes also have methods for accessing the hypermedia links (some of which may be optional).  So the following JSON:</p>

<figure class='code'><figcaption><span>Job JSON</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='json'><span class='line'><span class="p">{</span>
</span><span class='line'>  <span class="nt">&quot;details&quot;</span> <span class="p">:</span> <span class="p">{</span>
</span><span class='line'>    <span class="nt">&quot;title&quot;</span> <span class="p">:</span> <span class="s2">&quot;Software Engineer&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;company&quot;</span> <span class="p">:</span> <span class="s2">&quot;TheLadders&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;location&quot;</span> <span class="p">:</span> <span class="s2">&quot;New York, NY&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;postedOn&quot;</span> <span class="p">:</span> <span class="s2">&quot;2015-04-17&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;salary&quot;</span> <span class="p">:</span> <span class="s2">&quot;$100k&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;description&quot;</span> <span class="p">:</span> <span class="s2">&quot;Work with us!&quot;</span>
</span><span class='line'>  <span class="p">},</span>
</span><span class='line'>  <span class="nt">&quot;links&quot;</span> <span class="p">:</span> <span class="p">[</span> <span class="p">{</span>
</span><span class='line'>    <span class="nt">&quot;action&quot;</span> <span class="p">:</span> <span class="s2">&quot;like&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;method&quot;</span> <span class="p">:</span> <span class="s2">&quot;POST&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;uri&quot;</span> <span class="p">:</span> <span class="s2">&quot;https://localhost:54837/api/jobs/e-1/like&quot;</span>
</span><span class='line'>  <span class="p">},</span> <span class="p">{</span>
</span><span class='line'>    <span class="nt">&quot;action&quot;</span> <span class="p">:</span> <span class="s2">&quot;get&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;method&quot;</span> <span class="p">:</span> <span class="s2">&quot;GET&quot;</span><span class="p">,</span>
</span><span class='line'>    <span class="nt">&quot;uri&quot;</span> <span class="p">:</span> <span class="s2">&quot;https://localhost:54837/api/jobs/e-1&quot;</span>
</span><span class='line'>  <span class="p">}</span> <span class="p">]</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Would be represented by a class that looks like:</p>

<figure class='code'><figcaption><span>JobRepresentation.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="nd">@XmlRootElement</span>
</span><span class='line'><span class="nd">@XmlAccessorType</span><span class="o">(</span><span class="n">XmlAccessType</span><span class="o">.</span><span class="na">FIELD</span><span class="o">)</span>
</span><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">JobRepresentation</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="kd">public</span> <span class="n">JobDetailRepresentation</span> <span class="n">details</span><span class="o">;</span>
</span><span class='line'>  <span class="kd">public</span> <span class="n">HypermediaLinks</span> <span class="n">links</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">Optional</span><span class="o">&lt;</span><span class="n">HypermediaLink</span><span class="o">&gt;</span> <span class="nf">likeLink</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">Optional</span><span class="o">.</span><span class="na">ofNullable</span><span class="o">(</span><span class="n">links</span><span class="o">.</span><span class="na">forAction</span><span class="o">(</span><span class="err">“</span><span class="n">like</span><span class="err">”</span><span class="o">));</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">Optional</span><span class="o">&lt;</span><span class="n">HypermediaLink</span><span class="o">&gt;</span> <span class="nf">unlikeLink</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">Optional</span><span class="o">.</span><span class="na">ofNullable</span><span class="o">(</span><span class="n">links</span><span class="o">.</span><span class="na">forAction</span><span class="o">(</span><span class="err">“</span><span class="n">unlike</span><span class="err">”</span><span class="o">));</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">HypermediaLink</span> <span class="nf">getLink</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">links</span><span class="o">.</span><span class="na">forAction</span><span class="o">(</span><span class="err">“</span><span class="n">get</span><span class="err">”</span><span class="o">);</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>The methods for the links are particularly useful when navigating the API in a way an actual client would in tests.  Having defined a class representing a body of JSON, let&rsquo;s move on to the class for performing assertions against this body of JSON.</p>

<h2>Class for performing asserts against a body of JSON</h2>

<p>Having a class for performing assertions against a particular body of JSON helps us achieve two goals:  1) write more expressive test code and 2) have assertions that can be reused across tests.  For example, there may be several tests that require asserting the title for a particular job returned by the API.  We may also want to verify the presence or absence of certain hypermedia links.  The following code illustrates an “asserts” class for a job representation.</p>

<figure class='code'><figcaption><span>JobRepresentationAsserts.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">JobRepresentationAsserts</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="kd">private</span> <span class="kd">final</span> <span class="n">JobRepresentation</span> <span class="n">representation</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">hasTitleOf</span><span class="o">(</span><span class="n">String</span> <span class="n">title</span><span class="o">)</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">assertEquals</span><span class="o">(</span><span class="n">title</span><span class="o">,</span> <span class="n">representation</span><span class="o">.</span><span class="na">details</span><span class="o">.</span><span class="na">title</span><span class="o">);</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">hasLikeLink</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">assertTrue</span><span class="o">(</span><span class="n">representation</span><span class="o">.</span><span class="na">likeLink</span><span class="o">().</span><span class="na">isPresent</span><span class="o">());</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">doesNotHaveLikeLink</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">assertFalse</span><span class="o">(</span><span class="n">representation</span><span class="o">.</span><span class="na">likeLink</span><span class="o">().</span><span class="na">isPresent</span><span class="o">());</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">hasUnlikeLink</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">assertTrue</span><span class="o">(</span><span class="n">representation</span><span class="o">.</span><span class="na">unlikeLink</span><span class="o">().</span><span class="na">isPresent</span><span class="o">());</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">doesNotHaveUnlikeLink</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">assertFalse</span><span class="o">(</span><span class="n">representation</span><span class="o">.</span><span class="na">unlikeLink</span><span class="o">().</span><span class="na">isPresent</span><span class="o">());</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>In addition to these “asserts” classes, I created a class with static builder methods to provide some syntactic sugar.  This allows us to write our tests in a given/when/then fashion.  This builder class is as follows:</p>

<figure class='code'><figcaption><span>AssertsBuilder.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">AssertsBuilder</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kd">static</span> <span class="n">JobRepresentationAsserts</span> <span class="nf">thenThe</span><span class="o">(</span><span class="n">JobRepresentation</span> <span class="n">representation</span><span class="o">)</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="k">new</span> <span class="nf">JobRepresentationAsserts</span><span class="o">(</span><span class="n">representation</span><span class="o">);</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>This results in something like the following in a test:</p>

<figure class='code'><figcaption><span>Sample job test</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kn">import</span> <span class="nn">static</span> <span class="n">com</span><span class="o">.</span><span class="na">theladders</span><span class="o">.</span><span class="na">AssertsBuilder</span><span class="o">.*;</span>
</span><span class='line'>
</span><span class='line'><span class="o">...</span>
</span><span class='line'>
</span><span class='line'><span class="n">JobRepresentation</span> <span class="n">job</span> <span class="o">=</span> <span class="n">apiCallToGetJob</span><span class="o">();</span>
</span><span class='line'><span class="n">thenThe</span><span class="o">(</span><span class="n">job</span><span class="o">).</span><span class="na">hasTitleOf</span><span class="o">(</span><span class="err">“</span><span class="n">Software</span> <span class="n">Engineer</span><span class="err">”</span><span class="o">);</span>
</span><span class='line'><span class="n">thenThe</span><span class="o">(</span><span class="n">job</span><span class="o">).</span><span class="na">hasLikeLink</span><span class="o">();</span>
</span></code></pre></td></tr></table></div></figure>


<p>We’re not there yet, but we’re making progress.  So far we have the following:</p>

<ul>
<li>Class for representing incoming/outgoing bodies of JSON</li>
<li>Class for performing assertions against a particular body of JSON</li>
<li>Some syntactic sugar for creating assert-related objects.</li>
</ul>


<p>The next piece of the puzzle is a class representing the actions that can be taken on a body of JSON.</p>

<h2>Class representing the actions that can be taken on a body of JSON</h2>

<p>What exactly do I mean by &ldquo;taking actions on a body of JSON&rdquo;.  &ldquo;Actions&rdquo; can include anything from selecting an individual item in a list (a job in a list of jobs) to calling a URL associated with a hypermedia link.  So why the need for a separate class encapsulating these &ldquo;actions&rdquo; when we already have a representation class for a body of JSON (which includes hypermedia links)?  The reason is rooted in the fact that our representation classes are used in production code for serialization purposes (a topic worthy of its own blog post), and I didn&rsquo;t want to mix in testing concerns with our representation classes for something like calling the API via TestRestClient.  So I decided to create a different type of class for this purpose.  I called this the &ldquo;actions&rdquo; class.</p>

<p>Let&rsquo;s take a look at JSON containing a list of jobs as an example.  If a user is viewing a list of jobs, it&rsquo;s reasonable to expect they might want to &ldquo;do something&rdquo; with a particular job.  This would be captured with something like the following:</p>

<figure class='code'><figcaption><span>JobListActions.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">JobListActions</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="kd">public</span> <span class="n">UserAndClient</span> <span class="n">userAndClient</span><span class="o">;</span> <span class="c1">// this class represents a user and their connection to the API and will be discussed next.</span>
</span><span class='line'>  <span class="kd">public</span> <span class="n">JobListRepresentation</span> <span class="n">representation</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">JobActions</span> <span class="nf">actingOnJobAtIndex</span><span class="o">(</span><span class="kt">int</span> <span class="n">index</span><span class="o">)</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="k">new</span> <span class="nf">JobActions</span><span class="o">(</span><span class="n">userAndClient</span><span class="o">,</span> <span class="n">representation</span><span class="o">.</span><span class="na">jobs</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">index</span><span class="o">));</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>JobListActions by itself isn’t necessarily interesting, but it does provide a nice way for indicating what job a user is acting upon in a test.  The next code listing is more interesting, providing methods for performing various “actions” an individual job.</p>

<figure class='code'><figcaption><span>JobActions.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">JobActions</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="kd">private</span> <span class="kd">final</span> <span class="n">UserAndClient</span> <span class="n">userAndClient</span><span class="o">;</span> <span class="c1">// this class represents a user and their connection to the API and will be discussed next</span>
</span><span class='line'>  <span class="kd">private</span> <span class="kd">final</span> <span class="n">JobRepresentation</span> <span class="n">representation</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">Response</span><span class="o">&lt;</span><span class="n">JobRepresentation</span><span class="o">&gt;</span> <span class="nf">like</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">Optional</span><span class="o">&lt;</span><span class="n">HypermediaLink</span><span class="o">&gt;</span> <span class="n">link</span> <span class="o">=</span> <span class="n">representation</span><span class="o">.</span><span class="na">likeLink</span><span class="o">();</span>
</span><span class='line'>    <span class="k">return</span> <span class="k">new</span> <span class="n">Response</span><span class="o">&lt;&gt;(</span><span class="n">userAndClient</span><span class="o">.</span><span class="na">client</span><span class="o">.</span><span class="na">post</span><span class="o">(</span><span class="n">link</span><span class="o">.</span><span class="na">get</span><span class="o">()),</span> <span class="n">JobRepresentation</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">Response</span><span class="o">&lt;</span><span class="n">JobRepresentation</span><span class="o">&gt;</span> <span class="nf">unlike</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">Optional</span><span class="o">&lt;</span><span class="n">HypermediaLink</span><span class="o">&gt;</span> <span class="n">link</span> <span class="o">=</span> <span class="n">representation</span><span class="o">.</span><span class="na">unlikeLink</span><span class="o">();</span>
</span><span class='line'>    <span class="k">return</span> <span class="k">new</span> <span class="n">Response</span><span class="o">&lt;&gt;(</span><span class="n">userAndClient</span><span class="o">.</span><span class="na">client</span><span class="o">.</span><span class="na">post</span><span class="o">(</span><span class="n">link</span><span class="o">.</span><span class="na">get</span><span class="o">()),</span> <span class="n">JobRepresentation</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>This class contains methods for liking and un-liking a job.  This is done by using the TestRestClient found in UserAndClient (more on this next) to make calls to the API via the hypermedia links.  The Response class that is returned from these methods is a simple wrapper class for a JAX-RS response and looks like the following:</p>

<figure class='code'><figcaption><span>Response.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Response</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="kd">private</span> <span class="kd">final</span> <span class="n">javax</span><span class="o">.</span><span class="na">ws</span><span class="o">.</span><span class="na">rs</span><span class="o">.</span><span class="na">core</span><span class="o">.</span><span class="na">Response</span> <span class="n">response</span><span class="o">;</span>
</span><span class='line'>  <span class="kd">private</span> <span class="kd">final</span> <span class="n">Class</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">responseBodyType</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">int</span> <span class="nf">status</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="na">getStatus</span><span class="o">();</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">T</span> <span class="nf">body</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="na">readEntity</span><span class="o">(</span><span class="n">responseBodyType</span><span class="o">);</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>So we have all of these separate pieces:  1) representation class, 2) representation asserts class, and 3) class for performing actions against a representation.  By themselves, these three pieces really aren’t that useful.  We need something to pull them together.  This is where the UserAndClient alluded to earlier comes in.</p>

<h2>Class representing a user, their state, and their “connection” to the API</h2>

<p>This class is used for encapsulating a user and how they make calls to the API.  State for a particular user is kept here.  This class acts as the launching point for navigating through the API.  The code in the following listing provides methods for viewing job matches and being able to act upon the viewed job matches.</p>

<figure class='code'><figcaption><span>UserAndClient.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">UserAndClient</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="kd">public</span> <span class="n">AuthenticatedUserRepresentation</span> <span class="n">user</span><span class="o">;</span>
</span><span class='line'>  <span class="kd">public</span> <span class="n">TestRestClient</span> <span class="n">client</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">JobListRepresentation</span> <span class="n">jobMatches</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">JobListRepresentation</span> <span class="nf">viewsJobMatches</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">HypermediaLink</span> <span class="n">link</span> <span class="o">=</span> <span class="n">user</span><span class="o">.</span><span class="na">jobMatchesLink</span><span class="o">();</span>
</span><span class='line'>    <span class="n">jobMatches</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">post</span><span class="o">(</span><span class="n">link</span><span class="o">).</span><span class="na">readEntity</span><span class="o">(</span><span class="n">JobListRepresentation</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">jobMatches</span><span class="o">;</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">JobListActions</span> <span class="nf">actingOn</span><span class="o">(</span><span class="n">JobListRepresentation</span> <span class="n">representation</span><span class="o">)</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="k">new</span> <span class="nf">JobListAction</span><span class="o">(</span><span class="k">this</span><span class="o">,</span> <span class="n">representation</span><span class="o">);</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>With some additional syntactic sugar, we are able to use UserAndClient to write very expressive tests with decoupled and reusable components.  Our base test class provides this syntactic sugar:</p>

<figure class='code'><figcaption><span>RegisteredUserWebTest.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">RegisteredUserWebTest</span> <span class="kd">extends</span> <span class="n">WebTest</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="kd">protected</span> <span class="n">UserAndClient</span> <span class="n">me</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="nd">@Before</span>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">setUp</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">me</span> <span class="o">=</span> <span class="n">createNewUserAndClient</span><span class="o">();</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">UserAndClient</span> <span class="nf">given</span><span class="o">(</span><span class="n">UserAndClient</span> <span class="n">userAndClient</span><span class="o">)</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">userAndClient</span><span class="o">;</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">UserAndClient</span> <span class="nf">when</span><span class="o">(</span><span class="n">UserAndClient</span> <span class="n">userAndClient</span><span class="o">)</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">userAndClient</span><span class="o">;</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>We are then able to put it all together in a test such as the following:</p>

<figure class='code'><figcaption><span>LikeJobWebTest.java</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">LikeJobWebTest</span> <span class="kd">extends</span> <span class="n">RegisteredUserWebTest</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="nd">@Test</span>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">likingJobReturnsResultWithUnlikeLink</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="n">given</span><span class="o">(</span><span class="n">me</span><span class="o">).</span><span class="na">viewsJobMatches</span><span class="o">();</span>
</span><span class='line'>    <span class="n">Response</span><span class="o">&lt;</span><span class="n">JobRepresentation</span><span class="o">&gt;</span> <span class="n">response</span> <span class="o">=</span> <span class="n">when</span><span class="o">(</span><span class="n">me</span><span class="o">).</span><span class="na">actingOn</span><span class="o">(</span><span class="n">me</span><span class="o">.</span><span class="na">jobMatches</span><span class="o">)</span>
</span><span class='line'>                                                   <span class="o">.</span><span class="na">actingOnJobAtIndex</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span>
</span><span class='line'>                                                   <span class="o">.</span><span class="na">like</span><span class="o">();</span>
</span><span class='line'>    <span class="n">thenThe</span><span class="o">(</span><span class="n">response</span><span class="o">).</span><span class="na">hasStatusOfOk</span><span class="o">();</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">JobRepresentation</span> <span class="n">job</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="na">body</span><span class="o">();</span>
</span><span class='line'>    <span class="n">thenThe</span><span class="o">(</span><span class="n">job</span><span class="o">).</span><span class="na">doesNotHaveLikeLink</span><span class="o">();</span>
</span><span class='line'>    <span class="n">thenThe</span><span class="o">(</span><span class="n">job</span><span class="o">).</span><span class="na">hasUnlikeLink</span><span class="o">();</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Looking at the above test, it’s pretty clear what is happening.  A user views their job matches.  From there, they like a single job in the list.  We are then verifying that the response of that like contains JSON without a like link (so they cannot “like” a job two times in a row).</p>

<h1>Conclusion</h1>

<p>Overall the results have been positive.  We have fairly reusable test code that minimizes duplication.  At first glance it may seem like a lot of unnecessary classes (representation asserts &amp; a class for encapsulating “actions”).  But creating these classes forces us to really think about the API as we develop.  An “asserts” class forces us to zero in on the needs of the client.  The “actions” class gives us a feel for whether or not we are providing the best API in terms of hypermedia links and navigating between URLs.</p>

<p>At the end of the day, we wanted something that would result in an API that met the needs of it’s customer (the iOS client), and I think we’ve achieved that.  By “eating our own dog food” via navigating the hypermedia links in our web tests, we are putting ourselves in the shoes of the iOS client, and the result has been increased test coverage with tests that are easy to extend and understand.</p>

<p>Hopefully you have found this useful.  I realize I&rsquo;ve glossed over some of the finer details, such as &ldquo;what is actually inside of the TestRestClient&rdquo; class?  This blog post wasn&rsquo;t meant to give you the exact template for doing what we do here at TheLadders.  It was meant to get you thinking about how you are testing your APIs and possible ways to improve upon how that is done.  I&rsquo;m sure what we have today will look much different six months for now.  While not perfect, it has resulted in what I think are more stable APIs, and at the end of the day, that&rsquo;s a win in my book.</p>

<p>Now time to go test that new Storm topology of mine in production and see if it works&hellip;</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[CasperJS the Friendly Testing Framework]]></title>
    <link href="http://dev.theladders.com/2015/03/casperjs-the-friendly-testing-framework/"/>
    <updated>2015-03-25T16:30:00-04:00</updated>
    <id>http://dev.theladders.com/2015/03/casperjs-the-friendly-testing-framework</id>
    <content type="html"><![CDATA[<blockquote><p>If you don&#8217;t like testing your product, most likely your customers won&#8217;t like to test it either.</p><footer><strong>&#8211; Anonymous</strong></footer></blockquote>


<p>When we started our new <a href="https://www.theladders.com/careers/search">job market guide</a>
project (a site where career-minded professionals can check out open positions
and stats such as average compensation), the question of testing came up pretty
quickly. We had used <a href="http://jasmine.github.io/">Jasmine</a> for
JavaScript unit testing on a previous project so we kept using it, but it wasn&rsquo;t enough.
Our Jasmine tests could pass, but the site might not actually work. We needed
end-to-end tests.</p>

<p>One of the new tools we found was <a href="http://casperjs.org/">CasperJS</a>. It&rsquo;s a neat
JavaScript project that uses <a href="http://phantomjs.org/">PhantomJS</a> to open a headless
browser, go to your site, take some actions and then make assertions. We&rsquo;re using
CasperJS for end-to-end testing and we&rsquo;re automating the test runs with
<a href="http://gruntjs.com/">Grunt</a>.</p>

<p>Here&rsquo;s a snippet of a test for our &lsquo;share&rsquo; feature:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class='javascript'><span class='line'><span class="kd">function</span> <span class="nx">emailCaptureTopFormTest</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">casper</span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">givenCareersPage</span><span class="p">);</span>
</span><span class='line'>    <span class="nx">casper</span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">whenEmailFormIsFilledOut</span><span class="p">);</span>
</span><span class='line'>    <span class="nx">casper</span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">thenConfirmationIsShown</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">function</span> <span class="nx">givenCareersPage</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">casper</span><span class="p">.</span><span class="nx">open</span><span class="p">(</span><span class="nx">DOMAIN</span> <span class="o">+</span> <span class="s1">&#39;careers/NY/sales/1&#39;</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">function</span> <span class="nx">whenEmailFormIsFilledOut</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">casper</span><span class="p">.</span><span class="nx">click</span><span class="p">(</span><span class="s1">&#39;[data-js=&quot;email-capture__top-link-test&quot;]&#39;</span><span class="p">);</span>
</span><span class='line'>    <span class="nx">casper</span><span class="p">.</span><span class="nx">sendKeys</span><span class="p">(</span><span class="nx">topFormInputSelector</span><span class="p">,</span> <span class="s1">&#39;jblocktest&#39;</span> <span class="o">+</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">().</span><span class="nx">getTime</span><span class="p">()</span> <span class="o">+</span> <span class="s1">&#39;@theladders.net&#39;</span><span class="p">);</span>
</span><span class='line'>    <span class="nx">casper</span><span class="p">.</span><span class="nx">click</span><span class="p">(</span><span class="s1">&#39;[data-js=&quot;email-capture__top-submit&quot;]&#39;</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">function</span> <span class="nx">thenConfirmationIsShown</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="nx">casper</span><span class="p">.</span><span class="nx">waituntilVisible</span><span class="p">(</span><span class="s1">&#39;[data-js=&quot;email-capture__confirmation&quot;]&#39;</span><span class="p">,</span>
</span><span class='line'>            <span class="nx">util</span><span class="p">.</span><span class="nx">success</span><span class="p">(</span><span class="s2">&quot;email confirmation ok&quot;</span><span class="p">),</span>
</span><span class='line'>            <span class="nx">util</span><span class="p">.</span><span class="nx">failWithScreenshot</span><span class="p">(</span><span class="s2">&quot;email-confirmation-fail&quot;</span><span class="p">,</span>
</span><span class='line'>                                    <span class="s2">&quot;email confirmation didn&#39;t appear&quot;</span><span class="p">,</span>
</span><span class='line'>                                    <span class="nx">SCREENSHOT_OPTIONS</span><span class="p">));</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p></p>

<p>The test opens up the page, fills out the form, and makes sure the confirmation
window appears. If it doesn&rsquo;t appear, the test takes a screenshot and reports a failure.</p>

<p>The tests are generally pretty fast. We use Grunt to kick off the test suites
so that all you need to do to run them is type <code>grunt test</code>. (That&rsquo;s a lot
easier to remember than <code>casperjs --ssl-protocol=any --ignore-ssl-errors=true
test path/to/tests</code>!) Simpler tests are typically less than a second to run,
but there are a few slower tests that rely on external services, which can
take as long as 15 seconds to run. This led to concerns about the test run
time. We want to run the whole suite of tests frequently, but we don&rsquo;t want it
to take a couple of minutes each time.</p>

<p>The solution I went with was running them in parallel. They&rsquo;re all independent
so there&rsquo;s no need for any test to wait for any other test to finish. CasperJS
doesn&rsquo;t officially support parallelization so I jury-rigged something together
with a shell script. It gets each test file, runs them all as background processes
and redirects their output to temporary files. Once that&rsquo;s done, it cats all the
output in order and then uses <code>grep</code> to print failures at the end.</p>

<p>Here&rsquo;s some sample output after the test suite has run:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='text'><span class='line'>Test file: /casper/tests/no_search_results_test.js
</span><span class='line'># search no results test
</span><span class='line'>PASS no-jobs-search-ok: Saw no jobs container
</span><span class='line'>PASS 1 test executed in 2.227s, 1 passed, 0 failed, 0 dubious, 0 skipped.
</span><span class='line'>
</span><span class='line'>Done, without errors.
</span><span class='line'>*********************************
</span><span class='line'>
</span><span class='line'>FAIL PremiumSignupTestfail:  Expected pricing form. Didn&#39;t see it.
</span><span class='line'>#    file: /casper/tests/premium_signup_test.js
</span><span class='line'>FAIL 1 test executed in 13.195s, 0 passed, 1 failed, 0 dubious, 0 skipped.
</span><span class='line'>
</span><span class='line'>real  0m25.646s
</span><span class='line'>user  1m16.980s
</span><span class='line'>sys   0m7.416s
</span><span class='line'>^ Time it took for this task.
</span></code></pre></td></tr></table></div></figure>


<p>I used the <code>time</code> command to print out how long the suite takes. It&rsquo;s now
around 25s instead of 90s+. That is, the run time is the slowest test&rsquo;s run
time plus some overhead. That&rsquo;s a big improvement over the sum of all the tests&#8217;
 run times!</p>

<p>This was great when we only had a few tests, but as the suite grew larger, I
noticed the server was starting to struggle. It could handle five connections
 opening at once, but a hundred was causing tests to time out. My solution for
this was to split the tests into smaller batches. Instead of running 100 tests
all at once and bringing the server down, I can run two sets of 50. It&rsquo;s a
little slower than it would be if they could all run at once, but it&rsquo;s definitely
faster than having some tests randomly time out and fail.</p>

<p>Now that the casper tests are quick and easy to run, they&rsquo;re being used more
frequently and catching errors faster. Some developers are writing casper tests
before they write the actual code, too.</p>

<p>While CasperJS is a great tool for testing interactions and catching errors (like
forms not submitting correctly), it doesn&rsquo;t particularly care about how the page
looks. The casper tests will happily pass even if no CSS loads on the page. A
human would obviously see something is broke, but the casper tests won&rsquo;t. We
wanted to be able to catch problems like that without manually looking at
every page. Fortunately, there&rsquo;s a tool for that:
<a href="https://github.com/Huddle/PhantomCSS">PhantomCSS</a>.</p>

<p>PhantomCSS builds on top of CasperJS. It takes screenshots of your
site and then after you&rsquo;ve done work, it takes new screenshots. It then compares
the two and highlights any changes. This can be incredibly useful. For example,
suppose you&rsquo;re changing the header on one pager to be centered. If this accidentally
center headers on other pages, that will show up as a failure in PhantomCSS.</p>

<p>Since PhantomCSS tests run the same way as the casper tests, I was able to use the
same method to run them in parallel. Like with casper tests, individually they&rsquo;re
pretty quick, but running them all sequentially can be slow. Running them in
parallel is a big time saver.</p>

<p>Now that we are using CasperJS and PhantomJS, our confidence when releasing has
gone way up. We no longer need to manually check styles on every page because
PhantomCSS will show us every change. We don’t need to click through flows because
CasperJS does that for us. We&rsquo;ve been releasing new bug-free features at a consistent
rate that wouldn&rsquo;t be possible if we were still manually testing.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Conclave]]></title>
    <link href="http://dev.theladders.com/2015/02/conclave/"/>
    <updated>2015-02-13T13:13:00-05:00</updated>
    <id>http://dev.theladders.com/2015/02/conclave</id>
    <content type="html"><![CDATA[<blockquote><p>For once in my life I&#8217;ve got something to say<br/> I wanna say it now for now is today<br/> A love has been given so why not enjoy<br/> So let&#8217;s all grab and let&#8217;s all enjoy</p><footer><strong>&#8211; Sham 69 &#8220;If The Kids Are United&#8221;</strong></footer></blockquote>


<h2>If We All Stand Together It Will Just Be A Start</h2>

<p>Our engineering culture at TheLadders celebrates curiosity, teamwork and knowledge sharing.  The very nature of the loosely-coupled systems that we&rsquo;re building leads to a tendency for our teams to become isolated from each other.  We look to Friday Conclave to combat our isolation and stimulate our collective intellectual curiosity.</p>

<p>What is Friday Conclave you ask?  It&rsquo;s a chance for our engineers to learn something outside of their day-to-day work.  It&rsquo;s an opportunity for us to reconnect as a larger team.  It&rsquo;s a venue for us to share our intellectual curiosity with our colleagues.  Occasionally, it&rsquo;s a means for us to share domain knowledge and keep everyone in the loop on each group&rsquo;s direction and progress.</p>

<h2>Freedom Is Given, Speak How You Feel</h2>

<p>The format of Friday Conclave isn&rsquo;t set in stone, but the sessions tend to fall into one of the following categories:</p>

<ul>
<li>focused talk</li>
<li>lightning talks</li>
<li>very special guest</li>
<li>video with discussion</li>
<li>round table</li>
</ul>


<h3>Focused Talks</h3>

<p>Our longer talks are often the result of someone learning something in the course of doing their job, and then sharing it with the rest of the team.  The talks generally last 45 minutes to an hour and include a question and answer period at the end. Recently an engineer told the story of a bug that he found and fixed that involved a complex interaction between our caching system and one of our business rules. In the process of preparing for the talk he learned even more about the system as he worked out how to explain the issue to people who may not have been familiar with the parts involved.  Past talks have included an overview of HTTP by <a href="http://dev.theladders.com/ourteam/kylewinter">Kyle Winter</a> and a preview of an upcoming <a href="https://twitter.com/paperswelovenyc">Papers We Love NYC</a> talk on the <a href="https://github.com/papers-we-love/papers-we-love/blob/master/garbage_collection/incremental_mature_garbage_collection_using_the_train_algorithm.pdf?raw=true">Train Algorithm</a> by <a href="http://dev.theladders.com/ourteam/andrewturley">Andrew Turley</a>.</p>

<p>Some of our focused talks have nothing to do with people&rsquo;s current work.  Our engineering team comes from an astonishingly wide variety of backgrounds and team members routinely draw on their prior experience or research as conclave topics.  <a href="http://www.linkedin.com/in/shanioffen">Shani Offen</a> has spoken to the team about Dimensionality Reduction, Singular Value Decompositions, Eigenvectors, and Principal Component Analysis and <a href="http://dev.theladders.com/ourteam/wamiqahmed">Wamiq Ahmed</a> shared his <a href="http://www.ncbi.nlm.nih.gov/pubmed/23060342">NIH research</a> into the Classification of Bacterial Contamination.</p>

<h3>Lightning Talks</h3>

<p>Lightning talks are ten minute talks about any topic. We&rsquo;ve had people present on topics as ranging from great figures in the history of science, and the evolution of soccer defensive formations, to esoteric programming languages, and Hindu guilt. As with the focused talks, lightning talks vary greatly in their style and format. Sometimes an attempt is made to bring the topic of discussion back around to software engineering, but more often than not they serve as windows into the way our fellow engineers think and the things they hold dear.</p>

<h3>Very Special Guests</h3>

<p>From time to time we bring in guest presenters &mdash; engineers from outside the company who talk about their experience and expertise. <a href="http://whilefalse.blogspot.com/">Camille Fournier</a> of <a href="https://www.renttherunway.com/rtr_home">Rent the Runway</a> spoke to us about <a href="http://zookeeper.apache.org/">Zookeeper</a>, and <a href="https://twitter.com/zeeshanlakhani">Zeeshan Lakhani</a> gave a presentation on how he and his coworkers have used property-based testing in <a href="http://clojure.org">Clojure</a>. These are great ways to meet other engineers and it introduces us to skills and practices with which we may not be familiar.</p>

<h3>Videos with Discussion</h3>

<p>When we discover a talk that we feel is relevant to our work, we&rsquo;ll occasionally watch the video together and discuss as a group.  This is particularly applicable when someone returns from a conference and one of the presentations really stuck with them.  Past examples include Jeff Dean&rsquo;s talk from Ricon West 2013, <a href="https://www.youtube.com/watch?v=C_PxVdQmfpk">The Tail at Scale: Achieving Rapid Response Times in Large Online Services</a> and Donald Reinertsen&rsquo;s talk from Lean Software and Systems Conference 2012, <a href="http://vimeo.com/45947817">Decentralizing Control: How Aligned Initiative Conquers Uncertainty</a>.</p>

<h3>Round Table</h3>

<p>Sometimes we don&rsquo;t have an agenda. This isn&rsquo;t a bad thing. We still sit down and talk. If people are feeling particularly reticent we may start by going around and asking everyone to say something about one thing they worked on that week. Other times someone will come in with a story, a complaint, a question, or an issue that needs addressing. Early last year a few of us were trying to figure out how to deal with content caching on our new website. Since ops, platform, and the website team were all sitting there together we decided to hash out the details. Three hours later we emerged with a plan.</p>

<h2>Understand Them and They&rsquo;ll Understand You</h2>

<p>Friday Conclave is an excellent manifestation of the engineering culture here at TheLadders.  It provides us a forum to discuss how we build our products, explore new ideas, share points of view, learn about new and &ldquo;new-to-us&rdquo; technologies, and engage face-to-face rather than keyboard-to-keyboard.  We always welcome guest presenters and participants.  If you&rsquo;d like to attend or present at one of our Friday Conclaves in the future, let us know <a href="https://twitter.com/theladdersdev">@TheLaddersDev</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[MMR is not Measles, Mumps, and Rubella]]></title>
    <link href="http://dev.theladders.com/2015/01/not-measles/"/>
    <updated>2015-01-22T15:58:00-05:00</updated>
    <id>http://dev.theladders.com/2015/01/not-measles</id>
    <content type="html"><![CDATA[<blockquote><p>Communication Breakdown, It&#8217;s always the same,<br/>I&#8217;m having a nervous breakdown, Drive me insane!</p><footer><strong>&#8211;Led Zeppelin</strong></footer></blockquote>


<p>tl;dr &mdash; Use this <a href="http://github.com/TheLadders/monitor-ds-replication">plugin</a> to monitor 389 Directory Server replication</p>

<p>We&rsquo;ve been bitten in the past when the multi-master replication between our authentication servers stops functioning properly and we don&rsquo;t find out about it immediately.  This usually manifests itself as users complaining that they&rsquo;re intermittently unable to authenticate against certain services, which results in a bunch of troubleshooting effort only to discover that the real problem is the user not existing on all IPA servers.</p>

<p>We use <a href="http://freeipa.org">freeIPA</a> internally as our centralized user management system.  freeIPA combines several standard open source components to provide an &ldquo;integrated security information management solution&rdquo;.  These components include <a href="http://directory.fedoraproject.org/">389 Directory Server</a>, <a href="http://k5wiki.kerberos.org/wiki/Main_Page">MIT Kerberos</a>, <a href="http://www.ntp.org">NTP</a>, <a href="http://fedorahosted.org/bind-dyndb-ldap/">DNS</a>, <a href="http://pki.fedoraproject.org/">Dogtag certificate system</a>, <a href="http://fedorahosted.org/sssd/">SSSD</a> as well as several others.  In the absence of custom configuration, freeIPA utilizes two instances of 389 Directory Server &ndash; one for traditional directory information on the standard port 389, and one for <a href="http://en.wikipedia.org/wiki/Public_key_infrastructure">PKI/CA</a> on port 7389.  389 Directory Server&rsquo;s multi-master replication (MMR) support ensures that directory and certificate data is available from any node in the cluster.</p>

<p>To prevent this unfortunate scenario in the future, we developed a simple <a href="http://www.nagios.org">nagios</a>/<a href="http://www.icinga.org">icinga</a> plugin to assess replication health within the 389 Directory Server cluster.  Fortunately, information including structure of the cluster and status of replication is stored within the LDAP schema itself.  In developing the plugin, we hoped to avoid storing any authentication details in the plugin or the nagios/icinga configuration.  This required enabling anonymous read-only querying of the replication agreement data.  Daniel James Scott&rsquo;s <a href="http://danieljamesscott.org/11-articles/application-guides/26-freeipa-replication-monitoring.html">blog post</a> provided very clear instructions for enabling anonymous read/search/compare access to the replication agreements.  Our <a href="http://github.com/TheLadders/monitor-ds-replication">plugin</a> uses the <a href="http://rubygems.org/gems/net-ldap">Net::LDAP</a> Ruby gem to interact with a 389 Directory Server instance to discover all of the downstream replicas and their respective status.  We query the ldap server with base <code>cn=config</code> and filter on <code>(objectclass=nsds5replicationagreement)</code>.  The equivalent command line query is:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>ldapsearch -x -h openldap_server.example.com -b cn=config '(objectclass=nsds5replicationagreement)'</span></code></pre></td></tr></table></div></figure>


<p>This yields data similar to:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
</pre></td><td class='code'><pre><code class=''><span class='line'># extended LDIF
</span><span class='line'>#
</span><span class='line'># LDAPv3
</span><span class='line'># base &lt;cn=config&gt; with scope subtree
</span><span class='line'># filter: (objectclass=nsds5replicationagreement)
</span><span class='line'># requesting: ALL
</span><span class='line'>#
</span><span class='line'>
</span><span class='line'># meToipa-1.example.com, replica, dc\3Dexample\2Cdc\3com, mapping tree, config
</span><span class='line'>dn: cn=meToipa-1.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
</span><span class='line'>cn: meToipa-1.example.com
</span><span class='line'>objectClass: nsds5replicationagreement
</span><span class='line'>objectClass: top
</span><span class='line'>nsDS5ReplicaTransportInfo: LDAP
</span><span class='line'>description: me to ipa-1.example.com
</span><span class='line'>nsDS5ReplicaRoot: dc=example,dc=com
</span><span class='line'>nsDS5ReplicaHost: ipa-1.example.com
</span><span class='line'>nsds5replicaTimeout: 120
</span><span class='line'>nsDS5ReplicaPort: 389
</span><span class='line'>nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserialentryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
</span><span class='line'>nsDS5ReplicaBindMethod: SASL/GSSAPI
</span><span class='line'>nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
</span><span class='line'>nsds5replicareapactive: 0
</span><span class='line'>nsds5replicaLastUpdateStart: 20150121214458Z
</span><span class='line'>nsds5replicaLastUpdateEnd: 20150121214501Z
</span><span class='line'>nsds5replicaChangesSentSinceStartup:: MTM6MjAwMzUxNy8wIDY6NC8wIDE0OjQ0MjkvMCA=
</span><span class='line'>nsds5replicaLastUpdateStatus: 1 Can't acquire busy replica
</span><span class='line'>nsds5replicaUpdateInProgress: FALSE
</span><span class='line'>nsds5replicaLastInitStart: 0
</span><span class='line'>nsds5replicaLastInitEnd: 0
</span><span class='line'>
</span><span class='line'># meToipa-2.example.com, replica, dc\3Dexample\2Cdc\3Dcom, mapping tree, config
</span><span class='line'>dn: cn=meToipa-2.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
</span><span class='line'>cn: meToipa-2.example.com
</span><span class='line'>objectClass: nsds5replicationagreement
</span><span class='line'>objectClass: top
</span><span class='line'>nsDS5ReplicaTransportInfo: LDAP
</span><span class='line'>description: me to ipa-2.example.com
</span><span class='line'>nsDS5ReplicaRoot: dc=example,dc=com
</span><span class='line'>nsDS5ReplicaHost: ipa-2.example.com
</span><span class='line'>nsds5replicaTimeout: 120
</span><span class='line'>nsDS5ReplicaPort: 389
</span><span class='line'>nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserialentryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
</span><span class='line'>nsDS5ReplicaBindMethod: SASL/GSSAPI
</span><span class='line'>nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
</span><span class='line'>nsds5replicareapactive: 0
</span><span class='line'>nsds5replicaLastUpdateStart: 20150121214628Z
</span><span class='line'>nsds5replicaLastUpdateEnd: 0
</span><span class='line'>nsds5replicaChangesSentSinceStartup:: Njo0MzEyLzAgMTM6NDAzMjMzOS8wIA==
</span><span class='line'>nsds5replicaLastUpdateStatus: 0 Replica acquired successfully: Incremental update started
</span><span class='line'>nsds5replicaUpdateInProgress: TRUE
</span><span class='line'>nsds5replicaLastInitStart: 20141215154802Z
</span><span class='line'>nsds5replicaLastInitEnd: 20141215154807Z
</span><span class='line'>nsds5replicaLastInitStatus: 0 Total update succeeded
</span><span class='line'>
</span><span class='line'># search result
</span><span class='line'>search: 2
</span><span class='line'>result: 0 Success
</span><span class='line'>
</span><span class='line'># numResponses: 3
</span><span class='line'># numEntries: 2</span></code></pre></td></tr></table></div></figure>


<p>We&rsquo;re primarily concerned with how far in the past each replica successfully performed an update.  As you can see from the output above, the replication agreement with ipa-2.example.com is in the middle of an incremental update and shows a last update end of <code>0</code>.  This does not necessarily mean that replication is broken.  For better or worse, when the server begins an update, it clears the last end time.  To avoid constantly alerting when we&rsquo;re unable to retrieve meaningful replication data, the plugin maintains a state file that tracks the last valid update completion time and how many times a check has resulted in a last update completion of <code>0</code>.  The number of successive zero responses and acceptable number of minutes since last successful update completion are configurable parameters with the ability to set distinct warning and critical thresholds.</p>

<p>Since putting this monitoring in place, we&rsquo;ve moved to newer freeIPA servers using replication to seamlessly migrate data from the old servers to the new.  This plugin has already served to identify a breakdown in our replication that was easily remedied because the nodes had not yet significantly diverged.  Other aspects of the health and performance of the IPA cluster are available via SNMP.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Pipes, Tubes, and Email]]></title>
    <link href="http://dev.theladders.com/2014/11/pipes-tubes-and-email/"/>
    <updated>2014-11-18T09:00:00-05:00</updated>
    <id>http://dev.theladders.com/2014/11/pipes-tubes-and-email</id>
    <content type="html"><![CDATA[<blockquote><p>It&#8217;s not a big truck.  It&#8217;s a series of tubes.  And if you don&#8217;t understand, those tubes can be filled and if they are filled, when you put your message in, it gets in line and it&#8217;s going to be delayed.</p><footer><strong>&#8211;US Senator Ted Stevens (R-Alaska)</strong></footer></blockquote>


<h2>Plumbing 101</h2>

<p>Sending email is a major part of our business.  Over the years, TheLadders has moved through several iterations of getting emails out to our millions of job seekers and recruiters.  We&rsquo;ve both built in-house solutions and utilized <a href="http://en.wikipedia.org/wiki/Email_service_provider_(marketing)">Email Service Providers (ESPs)</a>.</p>

<p>Recently, we transitioned onto a new ESP, <a href="http://sendgrid.com">SendGrid</a>.  SendGrid offers us the choice of handing off email via their <a href="https://sendgrid.com/docs/API_Reference/Web_API/index.html">HTTP</a> or <a href="https://sendgrid.com/docs/API_Reference/SMTP_API/index.html">SMTP</a> APIs.  We selected <a href="http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">SMTP</a> as our transport mechanism because it allows us the luxury of inserting a layer in our infrastructure to handle queueing and resilience of mail transport before handing messages across to the ESP.  We are also able to achieve higher overall throughput with SMTP than would be possible with the HTTP API.  We&rsquo;re using <a href="http://www.postfix.org">Postfix</a> as our mail transport agent because of its scalability properties, the flexibility of its configuration options and our team&rsquo;s familiarity with running and maintaining the application.</p>

<p>To ensure that our subscribers receive the emails we send them, we divide the different types of messages across multiple sub-user accounts at SendGrid.  This spreads our outbound email across different source IPs depending on the chosen sub-user account and allows us to control the reputation of each IP group.  In addition to the usual challenges that go along with deploying any new infrastructure &mdash; monitor it, scale it, make it highly available &mdash; SendGrid&rsquo;s use of SMTP authentication credentials to determine which sub-user account will handle the sending of a particular message created a new and interesting problem to solve.  How could we allow our application to instruct Postfix which sub-user account to use without building a Postfix cluster for each sub-user account?  Moreover, how could we make that process invisible to our end recipient?</p>

<h2>Don&rsquo;t Clog the Tubes!</h2>

<p>The majority of our email traffic is created by various <a href="http://dev.theladders.com/categories/storm/">Storm</a> topologies crunching through our data.  Storm provides the ability to parallelize each step in the process, resulting in fairly rapid generation of email traffic.  We utilize our F5 load balancers in front of a pool of Postfix servers to make our mail transport layer both fault-tolerant and scalable.  During our larger sends, we prefer messages to be sent out with minimal queueing.  We&rsquo;ve found after tuning that we can sustain roughly 7000 messages per second through a single Postfix server before messages begin to queue.  We can easily scale out to additional Postfix servers behind the load balancer to increase our total throughput.</p>

<p><img class="center medium" src="http://dev.theladders.com/images/pipes-tubes-and-email/animated_email_flow.gif"></p>

<h2>Pick a Tube</h2>

<p>Postfix provides the <a href="http://www.postfix.org/postconf.5.html#smtp_sasl_password_maps">SASL Password Map</a> mechanism to look up SMTP credentials based on remote hostname or domain.  When coupled with <a href="http://www.postfix.org/postconf.5.html#smtp_sender_dependent_authentication">Sender-dependent Authentication</a>, that lookup can be performed based on the sender address.  We leveraged this combination of options along with <a href="http://tools.ietf.org/html/rfc5233#page-2">plus-sign subaddressing</a> to encode the sub-user account in the message&rsquo;s source address, so an email from <code>user@example.com</code> would actually be sent from <code>user+account@example.com</code>.</p>

<p>We utilize a regular expression map for the password selection that matches the subaddress portion of the sender address and returns the appropriate sub-user account credentials to postfix, with a default for messages that come through without a subaddress.</p>

<figure class='code'><figcaption><span>SASL Password Map</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>/^.*\+user1@example.com$/   user1:password1
</span><span class='line'>/^.*\+user2@example.com$/   user2:password2
</span><span class='line'>/^.*$/                      defaultuser:defaultpassword</span></code></pre></td></tr></table></div></figure>


<p><span class='caption-wrapper center medium'><img class='caption' src='http://dev.theladders.com/images/pipes-tubes-and-email/pick-a-tube.jpg' width='' height='' alt='[Photo](https://www.flickr.com/photos/biscuitsmlp/2431615179) by [smlp.co.uk](https://www.flickr.com/photos/biscuitsmlp/) / [CC BY 2.0](http://creativecommons.org/licenses/by/2.0/)' title='[Photo](https://www.flickr.com/photos/biscuitsmlp/2431615179) by [smlp.co.uk](https://www.flickr.com/photos/biscuitsmlp/) / [CC BY 2.0](http://creativecommons.org/licenses/by/2.0/)'><span class='caption-text'><a href="https://www.flickr.com/photos/biscuitsmlp/2431615179">Photo</a> by <a href="https://www.flickr.com/photos/biscuitsmlp/">smlp.co.uk</a> / <a href="http://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></span></span></p>

<h2>Keep Your Tubes Straight</h2>

<p>While our solution works great for allowing our applications to properly route messages to the correct sub-user account, it doesn&rsquo;t necessarily provide the best customer facing appearance.  Our customers don&rsquo;t care what ESP we use, how we perform our sub-user account selection, or which sub-user account sourced their email.  Furthermore, a sudden change in source email address could cause our messages to be filtered incorrectly on the recipient side.  Perhaps a customer has a filter in place to always drop our messages into a specific folder for them to read later.  Maybe they have a strict policy stating that only mail from known addresses will end up in their inbox.  We periodically have to adjust how our outbound email is processed and we have to make those changes as transparent as possible for our customers.  To that end, we are also utilizing Postfix&rsquo;s address rewriting capabilities to ensure that our email source addresses remain consistent.</p>

<p>Postfix provides several opportunities and methods to transform email envelope information <a href="http://www.postfix.org/ADDRESS_REWRITING_README.html">for a variety of purposes</a>.  Our requirement was to transform messages sent from <code>user+account@example.com</code> so that the customer sees the message sourced from <code>user@example.com</code>.  <a href="http://www.postfix.org/postconf.5.html#sender_canonical_maps">Sender canonical maps</a> initially seemed like the ideal solution to this problem.  It worked perfectly to transform the sender address as desired however, the transformation occurred so early in the process that addresses were being rewritten before the subaccount selection was performed.  We finally settled on <a href="http://www.postfix.org/postconf.5.html#smtp_generic_maps">SMTP generic maps</a> as the correct solution since it performs its transformation when mail leaves the machine via SMTP, after all other processing has taken place.  We again use a regular expression to strip the subaccount information from source addresses.</p>

<figure class='code'><figcaption><span>SMTP Generic Map</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>/^(.*)\+(.*)@(.*)$/ ${1}@${3}</span></code></pre></td></tr></table></div></figure>


<p><span class='caption-wrapper center medium'><img class='caption' src='http://dev.theladders.com/images/pipes-tubes-and-email/the-internet.jpg' width='' height='' alt='[Photo](https://www.flickr.com/photos/wheresmysocks/205710716) by [Kendrick Erickson](https://www.flickr.com/photos/wheresmysocks/) / [CC BY 2.0](http://creativecommons.org/licenses/by/2.0/)' title='[Photo](https://www.flickr.com/photos/wheresmysocks/205710716) by [Kendrick Erickson](https://www.flickr.com/photos/wheresmysocks/) / [CC BY 2.0](http://creativecommons.org/licenses/by/2.0/)'><span class='caption-text'><a href="https://www.flickr.com/photos/wheresmysocks/205710716">Photo</a> by <a href="https://www.flickr.com/photos/wheresmysocks/">Kendrick Erickson</a> / <a href="http://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></span></span></p>

<h2>Pipes or Tubes</h2>

<p>Pulling it all together with the <a href="https://sendgrid.com/docs/Integrate/Mail_Servers/postfix.html">recommended Postfix config</a> from SendGrid, results in <code>/etc/postfix/main.cf</code> containing:</p>

<figure class='code'><figcaption><span>main.cf</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>smtp_sasl_auth_enable = yes
</span><span class='line'>smtp_sender_dependent_authentication = yes
</span><span class='line'>smtp_sasl_password_maps = pcre:/path/to/sasl_password_map
</span><span class='line'>smtp_generic_maps = pcre:/path/to/smtp_generic_map
</span><span class='line'>smtp_sasl_security_options = noanonymous
</span><span class='line'>smtp_tls_security_level = encrypt
</span><span class='line'>header_size_limit = 4096000</span></code></pre></td></tr></table></div></figure>


<p>Find this post interesting? Join the discussion over on <a href="https://news.ycombinator.com/item?id=8623846">Hacker News</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Speeding Up R]]></title>
    <link href="http://dev.theladders.com/2014/10/speeding-up-r/"/>
    <updated>2014-10-02T14:51:51-04:00</updated>
    <id>http://dev.theladders.com/2014/10/speeding-up-r</id>
    <content type="html"><![CDATA[<p>If you ever speak to a data scientist about what tools they use, you’re sure to hear a whole slew of complaints about how slow R can be. R is a programming language used for data processing, analysis, model building, and data visualization. It isn’t really designed for speed and most statistics courses don’t spend much time teaching students how to make their code run quicker.</p>

<p>A few months ago, we encountered this issue directly. We started working on an algorithm that would analyze any one of our six million jobseekers and find other jobseekers that they’re most similar to. To come up with this list, we’d look through several of the data points we had about the jobseeker, such as their career history and which jobs on our site they had been interested in, and looked for other members who had a similar set of data points. Our first pass at this algorithm in R took about thirty seconds to run for each jobseeker. If we wanted to run it for all six million of our users… well, we couldn’t really wait until 2019 to see the results. We’ve spent the subsequent few weeks speeding up the process so that it now takes less than half a second per jobseeker, and we’ve learned a bunch of helpful tips along the way.</p>

<p>(Note: Whenever I need to speed up my code, I look back at Hadley Wickham’s <a href="http://adv-r.had.co.nz/Performance.html">Advanced R</a> which is where a lot of these ideas come from and is a fantastic resource.)</p>

<h3>1. Use lineprof to find the bottleneck</h3>

<p>Hadley Wickham created a <a href="https://github.com/hadley/lineprof">useful and easy to use package</a> that you can use to figure out which part of your code is causing the biggest issues and drill down to the specific lines that are taking up most of the runtime. Figuring out which parts of your code are problematic is the first step towards fixing that problem.</p>

<h3>2. Attack the slowest cases, but don’t forget about the rest</h3>

<p>After making some improvements, we found that our algorithm could take anywhere between 0.5 and 6 seconds to run on a jobseeker. For the next iteration, we focused on the slowest 10% and tried to speed each of them up by 1-2 seconds. We repeated this process a few more times, taking a look at the slowest cases and seeing if we could make some big improvements there. However, we also stumbled on an easier but equally effective approach: making small improvements that impact all cases. Speeding up the process by 0.2 seconds for all jobseekers has the same impact of cutting off 2 seconds from just 10% of jobseekers and may be easier to do.</p>

<h3>3. Using data.table can speed up almost any merge or aggregation</h3>

<p>Our algorithm requires merging together a bunch of big data frames (for example, ones covering hundreds of thousands of user clicks or job applications) so we quickly searched for a package that could speed things up. We found the <a href="http://datatable.r-forge.r-project.org/">data.table package</a> which was orders of magnitude quicker than the base merge in some cases. However, we realized that there was no reason to limit our use of data.table as speed improvements could be found even on merges of much smaller data frames. Furthermore, any time we needed to run an aggregation over a data frame (usually done with aggregate or tapply), data.table could cut out valuable milliseconds.</p>

<h3>4. Use microbenchmark to rigorously test out your code</h3>

<p>The <a href="http://cran.r-project.org/web/packages/microbenchmark/index.html">microbenchmark package</a> is a far better way to test your R code’s speed than the standard system.time function. Microbenchmark runs each line of code you give it 100 times (by default) and computes summary statistics of those runtimes, allowing you to be confident that any speed improvements you find are not merely the result of random variation.</p>

<h3>5. Don’t overfit to a single case</h3>

<p>Good data scientists are cognizant of the tradeoff between training error and test error when building a model. A similar phenomenon can occur when trying to optimize an algorithm for speed. As we were refining our code, we typically tested things out on our own accounts. However, this was a bit dangerous, since we are by no means a representative sample of users of our site. We are unusually active on the site and all have somewhat similar career histories to each other. A change that improves runtime when generating matches for one of us may not have the same impact on users who are less active or belong to a different job function. Make sure to test out any changes on a large, random sample before concluding that your changes have done the trick.</p>

<h3>6. The clearest code is not always the fastest code</h3>

<p>Imagine you have a data frame that’s 100,000 rows long where the head looks like this:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>&gt; head(my.df, 10)
</span><span class='line'>           a         b         c         d
</span><span class='line'>1         NA        NA 0.4885687        NA
</span><span class='line'>2         NA        NA 0.1666159        NA
</span><span class='line'>3         NA        NA 0.3176160        NA
</span><span class='line'>4         NA 0.9568433 0.4848375        NA
</span><span class='line'>5  0.8441578 0.3022814        NA        NA
</span><span class='line'>6  0.4223605        NA        NA        NA
</span><span class='line'>7         NA 0.7998102        NA        NA
</span><span class='line'>8         NA 0.8183823        NA        NA
</span><span class='line'>9         NA        NA        NA 0.1550433
</span><span class='line'>10 0.1511128 0.4783680        NA        NA</span></code></pre></td></tr></table></div></figure>


<p>Let’s say what you want to do is replace all of the NA’s with 0’s. So a pretty straightforward function to use would be defined as:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>replaceNA &lt;- function(df){
</span><span class='line'>  df[is.na(df)] &lt;- 0
</span><span class='line'>  return(df)
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>That’s concise, easy to understand, and flexible enough to handle most changes that could occur to my.df in the future. Now, take a look at this function:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>replaceNAUgly &lt;- function(df){
</span><span class='line'>  df$a[is.na(df$a)] &lt;- 0
</span><span class='line'>  df$b[is.na(df$b)] &lt;- 0
</span><span class='line'>  df$c[is.na(df$c)] &lt;- 0
</span><span class='line'>  df$d[is.na(df$d)] &lt;- 0
</span><span class='line'>  return(df)
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>It’s not concise, is a little difficult to understand, and is not very flexible, but it’s actually quite a bit quicker:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>&gt; microbenchmark(
</span><span class='line'>+   replaceNA(my.df),
</span><span class='line'>+   replaceNAUgly(my.df)
</span><span class='line'>+ )
</span><span class='line'>Unit: milliseconds
</span><span class='line'>                           min       lq   median       uq      max neval
</span><span class='line'>     replaceNA(my.df) 219.7503 241.4529 249.8200 271.3922 316.8102   100
</span><span class='line'> replaceNAUgly(my.df) 148.9356 159.6738 167.7006 178.1716 234.8962   100</span></code></pre></td></tr></table></div></figure>


<p>In the majority of cases, the first function is fine or even preferable. However, if speed is your only concern, you may be able to speed things up if you’re willing to forgo having pretty code.</p>

<h3>7. Think about applying some filters</h3>

<p>We categorize each of our jobseekers into one of fourteen overarching job functions, such as sales, marketing, technology, or finance. If I’m a software engineer, almost all of the jobseekers who are similar to me are likely to belong in technology. Sure, we could think of some cases where someone shares attributes with people in a couple of different job functions, but this is more of an edge case. By comparing each jobseeker with the half-million other jobseekers in their job function instead of the entire set of six million, we could make the algorithm about ten times faster while still capturing the vast majority of the same results, a tradeoff we deemed totally acceptable.</p>

<h3>8. Preprocess, preprocess, preprocess</h3>

<p>One of the steps in our algorithm is to compare the set of titles a jobseeker has had in his/her career with the sets for all other jobseekers. However, sometimes two jobseekers will have had the same title but used different punctuation or capitalization when entering their title (not to mention misspellings or synonyms). As a result, we needed to remove the case and punctuation from each title before a comparison could be done. Our initial instinct was to run this text cleaning process right before doing the comparison, but this meant that we would end up cleaning the same text hundreds of times. Instead, we cleaned all the titles once, stored the results, and did the comparisons on these cleaned titles. You should constantly be on the lookout for places where the algorithm repeats a step multiple times and see if there’s a way to do so just once and then just look up those results when needed.</p>

<h3>9. Rcpp for when you’re really stuck</h3>

<p>If you’ve narrowed down your issue to a single operation and you can’t find a way around it, the <a href="http://dirk.eddelbuettel.com/code/rcpp.html">Rcpp package</a> may be able to help. With Rcpp, you can run parts of your code in C++ without interrupting the flow of the rest of your code. There’s a bit of a learning curve if you’re not familiar with C++, but the improvements could be massive enough to justify the effort. Alternatively, with some diligent Googling, you may be able to find someone who’s already solved your problem using Rcpp.</p>

<h3>10. Parallelize!</h3>

<p>For R users who have never done any sort of parallel computing before, this may seem a bit daunting. However, depending on how your code is structured, it can actually be quite trivial to run tasks on all the cores of your machine in parallel. For instance, if you’re using Linux or a Mac and part of your code is run using lapply, forcing it to run in parallel is as simple as loading the parallel package and replacing lapply with mclapply (i.e. multicore lapply). Parallelization doesn’t guarantee speed improvements, but testing it out shouldn’t require too much effort and is worth considering.</p>

<p>Along the same lines, you can force R to utilize all the cores of your machine when doing any linear algebra operations by using OpenBLAS or ATLAS, though the process of getting it up and running is not as easy. <a href="http://www.stat.cmu.edu/~nmv/2013/07/09/for-faster-r-use-openblas-instead-better-than-atlas-trivial-to-switch-to-on-ubuntu/">Nathan VanHoudnos&#8217; blog post on the topic</a> is a good place to start.</p>

<h3>11. Make sure this is all worth it</h3>

<p>While speeding things up is always a good idea, make sure you’re not spending hours of work to just speed things up by a few minutes total. Your time is better spent doing some actual analysis or model building. If this means that you need to let something run in the background of your computer while you do other stuff (or even that it needs to run overnight), so be it.</p>

<p>Have a question or comment? <a href="http://redd.it/2i4807">Join the discussion on reddit</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Thunder and Lightning]]></title>
    <link href="http://dev.theladders.com/2014/07/thunder-and-lightning/"/>
    <updated>2014-07-23T09:00:00-04:00</updated>
    <id>http://dev.theladders.com/2014/07/thunder-and-lightning</id>
    <content type="html"><![CDATA[<blockquote><p>The simplest of explanations is more likely to be correct than any other.</p><footer><strong>&#8211;Occam&#8217;s Razor</strong></footer></blockquote>


<h2>Braving the Storm</h2>

<p>At TheLadders, we operate a fully virtualized environment consisting of slightly less than a thousand virtual machines across our Development, QA, and Production environments.  We manage these systems with <a href="http://puppetlabs.com">Puppet</a> and <a href="http://theforeman.org">Foreman</a>, which enables us to rapidly deploy new systems when necessary, as well as maintain our systems predictably throughout their lifecycles.</p>

<p>One of the tools we <a href="http://dev.theladders.com/categories/storm/">rely on heavily</a> is <a href="https://storm.incubator.apache.org">Storm</a>, a distributed, real-time computation system.  Storm provides us with a framework that we use to constantly crunch data about the millions of job seekers and jobs in our environment.  This enables us to provide our job seekers with relevant information about the best jobs available to them at any given time.  When we build new Storm nodes, Puppet takes a very minimal OS install, lays down our standard configuration, then installs Storm, starts the Storm process and ensures it will start after reboot.</p>

<h2>Dark Skies Ahead</h2>

<p>We operate several Storm clusters across QA and Production spanning Storm versions 0.8 and 0.9.  Over the past several months, we&rsquo;ve experienced intermittent issues within the clusters where individual nodes stopped behaving properly.  The issues occur more frequently in our Production environment, which we attribute to the orders of magnitude higher volumes of traffic traversing Production.  We&rsquo;ve also seen this particular issue in both our 0.8 and 0.9 clusters.  Until recently, the problem has occurred so infrequently that it was much quicker and easier to shut down and rebuild the problem nodes than invest significant time nailing down the root cause.</p>

<p>Last month, we rebuilt our 0.9 Production cluster from the ground up and immediately began seeing topologies fail to start on multiple workers.  The issue was clouded by the fact that we saw several different errors occurring, including too many files open, DNS resolution failures, Java class not found errors, heartbeat file not found, etc.</p>

<p><span class='caption-wrapper center medium'><img class='caption' src='http://dev.theladders.com/images/thunder-and-lightning/stormy-city.jpg' width='' height='' alt='[Photo](https://www.flickr.com/photos/29311691@N05/7653430352) by [H.L.I.T.](https://www.flickr.com/photos/29311691@N05/) / [CC BY 2.0](http://creativecommons.org/licenses/by/2.0/)' title='[Photo](https://www.flickr.com/photos/29311691@N05/7653430352) by [H.L.I.T.](https://www.flickr.com/photos/29311691@N05/) / [CC BY 2.0](http://creativecommons.org/licenses/by/2.0/)'><span class='caption-text'><a href="https://www.flickr.com/photos/29311691@N05/7653430352">Photo</a> by <a href="https://www.flickr.com/photos/29311691@N05/">H.L.I.T.</a> / <a href="http://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></span></span></p>

<h2>When it Rains, it Pours</h2>

<p>Since we were seeing multiple errors occurring without any obviously predictable pattern, we started the investigation by trying to reproduce or verify the individual errors on the nodes where we saw the issues in the Nimbus interface.  Despite complaints from Storm of DNS resolution issues, we were unable to find any issues with our DNS system or name resolution on any of the nodes in the cluster, even when performing many lookups in rapid succession.</p>

<p>After eliminating DNS as a root cause, we surmised that the real problem was limits on open file handles and that the other errors &mdash; Java class not found, heartbeat file not found and DNS resolution failure &mdash; were just different manifestations of the process’ inability to open a file handle or socket.  One change that Puppet makes to our systems is to increase the open file handle limits for the storm user/process from the default of 1024 to 256k.  We do this by setting the <code>nofile</code> option in <code>/etc/security/limits.conf</code>.  We verified on every host that the Storm user had properly set file handle limits.  Observing the workers when they were experiencing the issue proved difficult because Storm dynamically assigns workers to nodes at process startup and processes are not sticky.  This means that in our situation, where processes were starting and dying very quickly, it was extremely challenging to be logged into the host watching the process and gathering useful data in the few seconds between startup and death.  One approach to avoid this problem was to shut down workers to eliminate unused worker slots, thus limiting the potential destinations for new processes.  After a prolonged struggle with observing a process as it died, we were finally able to see that the worker process itself was limited to the default 1024 file handles.  We confirmed this suspicion by watching <code>/proc/&lt;PID&gt;/limits</code> to confirm that all Storm related processes were limited to 1024 open file handles on the affected hosts.</p>

<h2>Every Cloud has a Silver Lining</h2>

<p>Now that we had observed a worker process with a 1024 open file handle limit, we moved on to determining how this could happen and why it seemed to occur only on certain nodes.  We noted that rebooting a host did not resolve the issue and further that rebooting a working node caused it to cease functioning properly.  After quite a bit of experimentation, we found that manually restarting the Storm supervisor on an affected host allowed the node to function properly again, at least until the next reboot.</p>

<p>We recently altered our new machine deployment to reboot the host between running Puppet and putting the machine in service.  Whereas previously the Storm supervisor would be started by Puppet and function normally, the supervisor is now being started by init on boot.</p>

<p>We ultimately determined that the root cause of this issue is that processes started by init don&rsquo;t go through pam, so limits set in <code>/etc/security/limits.conf</code>, which is utilized by <code>pam_limits.so</code>, are not applied to processes started on boot.</p>

<p>We chose to solve this issue by following the RHEL convention of configuration in /etc/sysconfig and modifying the storm supervisor init script to load <code>/etc/sysconfig/storm</code> if it exists.  Our <code>/etc/sysconfig/storm</code> contains a single line for the time being, increasing the <code>nofile</code> limit to 256k.  This method provides us with the flexibility to augment the configuration in the future with minimal impact.</p>

<p>Once Puppet deployed this change to our entire environment, we verified via <code>/proc/&lt;PID&gt;/limits</code> that the Storm supervisors had picked up the changes both when started by hand/Puppet and when started on boot.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Monitoring RabbitMQ at TheLadders]]></title>
    <link href="http://dev.theladders.com/2014/07/monitoring-rabbitmq-at-theladders/"/>
    <updated>2014-07-15T10:24:00-04:00</updated>
    <id>http://dev.theladders.com/2014/07/monitoring-rabbitmq-at-theladders</id>
    <content type="html"><![CDATA[<blockquote><p>Alice:        “How long is forever?”</p><p>White Rabbit:     “Sometimes, just one second.&#8221;</p><footer><strong>&#8211;Lewis Carroll</strong> <cite>Alice in Wonderland</cite></footer></blockquote>


<h2><em>Motivation</em></h2>

<p>TheLadders makes heavy use of <a href="http://www.rabbitmq.com">RabbitMQ</a> in its operations. RabbitMQ is a messaging broker that we use to connect our applications. Sending and receiving are separate, so the messaging is asynchronous. The asynchronous messaging decouples one application from another. We use RabbitMQ to publish events to multiple subscribers and to queue up work.</p>

<ul>
<li>In some cases, events in our systems are ad hoc, driven by the various activities of our users. Examples of such events include the addition of a job seeker or a job to the system. These events are realized as RabbitMQ messages posted to exchanges. Systems interested in these events read them from queues bound to the exchanges.</li>
<li>In other cases, a scheduled process posts RabbitMQ messages to an exchange so that a specific system (say, a <a href="https://storm.incubator.apache.org">Storm</a> topology) can process them. An example of this is a system that calculates matches between jobs and job seekers and puts those matches on a queue. A Storm topology reads those matches from the queue and transforms them into emails to be sent by an external email service.</li>
</ul>


<p>In the first scenario above, we are mostly interested in making sure that the queue of ad hoc events has not grown too large, which would indicate a problem with the system that is consuming the messages.</p>

<p>In the second scenario, we want to be sure that messages are being published to the queue at a suitable rate, and that the topology that consumes the messages is processing them quickly enough to get the emails out by a particular time.</p>

<p>We did not find a suitable ready-to-use solution for this need. As you will see below, we eventually arrived at a solution that combined a <a href="http://clojure.org/">Clojure</a> library for accessing RabbitMQ queue metrics with a <a href="http://riemann.io">Riemann</a> server to process events produced by the library.</p>

<h2><em>First Attempts</em></h2>

<p>Our first attempts at a monitoring solution revolved around an assortment of shell scripts that made HTTP requests to the RabbitMQ Management API to discover the lengths of important queues at particular times. If certain thresholds were crossed, alerts were sent to on-call personnel. This approach had a number of drawbacks:</p>

<ul>
<li>Brittleness: The times of the Management API queries were controlled by when the script was run by cron. The queue length thresholds were often dependent on the sizes of the datasets involved and the speed of the systems which published messages or read them from the queues.</li>
<li>Measuring Queue Lengths at a Single Point in Time: Checking the queue length at a particular point in time could not provide the information necessary to answer questions such as &ldquo;When will this queue be drained?&rdquo; and &ldquo;Are messages being placed on the queue at a rate sufficient to meet a particular processing schedule?&rdquo;</li>
</ul>


<h2><em>How We Solved the Problem</em></h2>

<p>We created a <a href="http://clojure.org/">Clojure</a> library called <a href="https://github.com/TheLadders/monitor-rabbitmq">monitor-rabbitmq</a>, available on GitHub.
It gathers statistics on RabbitMQ queues and nodes, and packages them as <a href="http://riemann.io">Riemann</a> events. <em>monitor-rabbitmq</em> takes care of acquiring RabbitMQ statistics from the RabbitMQ cluster, using the RabbitMQ’s HTTP based <a href="http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_4/priv/www/api/index.html">Management API</a>. Riemann takes care of aggregating the events and performing the calculations necessary to determine whether an alert should be triggered.</p>

<hr />

<p><em>A Bit More Detail</em></p>

<p><strong>What’s the data flow?</strong></p>

<p><img class="center" src="http://dev.theladders.com/images/monitor-rabbitmq2.png" title="'Data Flow graphic'" ></p>

<p><strong>What queue data does</strong> <em>monitor-rabbitmq</em> <strong>gather?</strong> [all rates are messages per second]</p>

<ul>
<li>Queue Length: number of messages on the queue</li>
<li>Ack rate: messages acknowledged by the client</li>
<li>Deliver rate: notifications to the client that there is a message</li>
<li>Deliver-get rate: combination of deliver and get rates</li>
<li>Deliver-no-ack rate: notifications to the client that there is a message which does not require acknowledgement</li>
<li>Get rate: messages synchronously requested by client</li>
<li>Get-no-ack rate: messages sent to client with no acknowledgement required</li>
<li>Publish rate: messages published to the queue</li>
<li>Redeliver rate: messages redelivered to the client after being delivered and not acked</li>
</ul>


<p><strong>What node data does</strong> <em>monitor-rabbitmq</em> <strong>gather?</strong></p>

<ul>
<li>fd_used: file descriptors used</li>
<li>fd_total: file descriptors total</li>
<li>sockets_used</li>
<li>sockets_total</li>
<li>mem_used</li>
<li>mem_limit</li>
<li>mem_alarm: memory high water mark alarm</li>
<li>disk_free</li>
<li>disk_free_limit</li>
<li>disk_free_alarm: disk free low water mark alarm</li>
<li>proc_used</li>
<li>proc_total</li>
</ul>


<p><strong>What does each</strong> Riemann <strong>event look like?</strong>
Here is the Clojure representation (a map):</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='clj'><span class='line'><span class="p">{</span><span class="ss">:time</span> <span class="mi">1390593087006</span>,
</span><span class='line'>    <span class="ss">:host</span> <span class="s">&quot;our-rabbitmq.super.awesome.queue&quot;</span>,
</span><span class='line'>    <span class="ss">:service</span> <span class="s">&quot;publish.rate&quot;</span>,
</span><span class='line'>    <span class="ss">:metric</span> <span class="mf">0.0</span>,
</span><span class='line'>    <span class="ss">:state</span> <span class="s">&quot;ok&quot;</span>,
</span><span class='line'>    <span class="ss">:tags</span> <span class="p">[</span><span class="s">&quot;rabbitmq&quot;</span><span class="p">]}</span>
</span></code></pre></td></tr></table></div></figure>


<p>This event, created by the Clojure library, includes a <code>:host</code> member which is formed by taking a <code>rmq-display-name argument</code> (“our-rabbitmq”) and composing it with the queue (or node) name: “super.awesome.queue”</p>

<h2><em>An Example</em></h2>

<p>Let&rsquo;s say that we have a queue which is read by a Storm topology. That queue contains messages which hold matches between jobs and job seekers. The topology must process the messages so that emails notifying job seekers of these job opportunities are composed and sent to an external email service. We want to be alerted if the messages are being consumed from the queue at a rate such that the queue will not be cleared by a certain deadline, say 9:00 AM.</p>

<p>We create a simple Clojure application that calls the <code>send-nodes-stats-to-riemann</code> function of our library and then exits. We call this application once a minute. Each call results in a full set of queue statistics being sent to our Riemann server.</p>

<p>Meanwhile, on the Riemann side of things, Riemann has been configured to watch the Ack rate of our queue. Riemann accumulates events over a time interval, smoothing out minor variations in the Ack rate, and calculates a projected time for the queue to be emptied. If the Ack rate dips below a certain threshold, it triggers an alert using, in our environment, the <a href="https://www.icinga.org">Icinga</a> monitoring system.</p>

<h2><em>Wrapping Up</em></h2>

<p>With <em>monitor-rabbitmq</em>, we supply a regular flow of events to our Riemann server containing data about our RabbitMQ queues and nodes. By choosing a good base set of statistics to request from the RabbitMQ Management API, a change to our monitoring and alerting requirements usually results in only a change to our Riemann configuration.</p>

<p>Find this useful? Join the discussion at <a href="https://news.ycombinator.com/item?id=8036593">Hacker News</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[A Brewing Storm]]></title>
    <link href="http://dev.theladders.com/2013/12/a-brewing-storm/"/>
    <updated>2013-12-19T11:45:00-05:00</updated>
    <id>http://dev.theladders.com/2013/12/a-brewing-storm</id>
    <content type="html"><![CDATA[<blockquote><p>If there&#8217;s a book that you want to read, but it hasn&#8217;t been written yet, then you must write it</p><footer><strong>&#8211;Toni Morrison</strong></footer></blockquote>


<p>Once upon a time there was a script. Each morning it sent new jobs to our job seeking customers. It was simple and the data it operated on was relatively small.  As the months turned into years this script grew in complexity and the data grew in size.  And more scripts came to the party and they had data of their own and these new scripts beget data for other scripts, and this small group of scripts extracted and transformed, processed and prepared, working tirelessly every night.</p>

<p>Some worked alone while others fed each other, all growing increasingly more complex.  Occasionally at first and more frequently over time, the scripts started to interfere with each other by competing for resources or not finishing in time for a child script to consume it’s parents data, or worst of all, deadlocking for no apparent reason.</p>

<p>Let’s bring some order we said and used cron to wrangle them at first, then moving to more complex “enterprise” scheduling systems to try and tame the beast, and keep the scripts from clobbering each other or grinding themselves to a crawl.</p>

<p>But the data grew and grew and with it came longer and longer run times.  At this same time our script was getting slower, our users’ expectations and our own ambitions grew.  Could we alert a job seeker the minute a new relevant job entered our system?  Could we alert an employer as soon as job seeker they were interested in updated their profile?</p>

<p>The network of interactions/activity/entities we wanted to grow, monitor and react to and then extract value from was exploding in complexity.  You can picture all of our job seekers, employers and jobs as nodes in a graph, and imagine our teams are furiously trying to connect them in new and interesting ways.  They are using machine learning to light up edges between these nodes, indicating which jobs a particular job seeker might be interested in and which job seekers an employer might like.  Or clustering these entities into similar groups.</p>

<p>Doing all of this between 12am and 6am was getting hard. Hadoop was an option, but processing speed alone wasn’t just the issue.  Some things we wanted to do had to be done in real time, not batch.  A user isn’t going to wait until the nightly run for us to calculate what jobs are appropriate for them.  In addtion, as TheLadders moves into mobile and speed becomes more and more a concern we want the Graph of our Ecosystem to be as current as possible not something that gets updated once a day.</p>

<p>There comes this thing. Storm. It fits into our infrastructure; working nicely with RabbitMQ and interop’ing well with our existing code base.</p>

<p>However, Storm is young and fresh out of Twitter.  Getting it to work, quickly and reliably can be painful as can figuring out the best practices to manage it.  All of this took us some time and some hard lessons had to be learnt.  A lot of late nights and head scratching, time spent hanging out in irc channels, reading blog posts, watching conference talks and just plain ole trial and error and occassionally grabbing Nathan Marz when he was speaking to ask “How do we deal with X?” brought us to our current place.</p>

<p>Today Storm runs most of the backend at TheLadders, keeping that graph of our ecosystem fresh so our experience teams can ship great new features to our customers.  And we hope you can find a use for Storm and maybe get there quicker than we did.  Manning publications reached out to us and I’m incredibly excited and very proud to announce that three of our top engineers, Sean Allen, Matt Jankowski and Peter Pathirana produced a book, Storm Applied, a book designed to help you leverage some of the hard earned knowledge we’ve acquired here at TheLadders. Storm Applied will be available in the <a href="http://www.manning.com/sallen/">Manning Early Access Program</a> any day now.  We hope you find it useful and we look forward to your feedback on the book.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Responsive Design: Keeping our thick client skinny]]></title>
    <link href="http://dev.theladders.com/2013/10/responsive-design-keeping-our-thick-client-skinny/"/>
    <updated>2013-10-30T01:37:00-04:00</updated>
    <id>http://dev.theladders.com/2013/10/responsive-design-keeping-our-thick-client-skinny</id>
    <content type="html"><![CDATA[<blockquote><p>Recognizing the need is the primary condition for design.</p><footer><strong>&#8211;Charles Eames</strong></footer></blockquote>


<p>You may or may not have noticed, but <a href="https://www.theladders.com/">TheLadders.com</a> just got a huge make over. We rewrote the website, putting it through a diet and a facelift. The team cut out excess network calories, trimmed DOM fat, and ironed out sloppy CSS wrinkles. We started 2013 with a shiny new single-page thick web client that is not only faster and cleaner, but uses some pretty provocative polishes like backbone.js, require.js, and SASS with Compass. But my favorite part: it’s responsive and looks great no matter where it goes.
iPhone? Check. iPad? Check. MacBook Air? Check. Ginormous iMac screen? Check.</p>

<p><span class='caption-wrapper left small'><img class='caption' src='http://dev.theladders.com/images/responsive/ladders-wide.jpeg' width='' height='' alt='New TheLadders.com on a desktop screen with the wide experience' title='New TheLadders.com on a desktop screen with the wide experience'><span class='caption-text'>New TheLadders.com on a desktop screen with the wide experience</span></span></p>

<p><span class='caption-wrapper left small'><img class='caption' src='http://dev.theladders.com/images/responsive/ladders-skinny.png' width='' height='' alt='New TheLadders.com on mobile with the skinny experience. Left: main content screen, right: [off canvas](http://jasonweaver.name/lab/offcanvas/) sidebar' title='New TheLadders.com on mobile with the skinny experience. Left: main content screen, right: [off canvas](http://jasonweaver.name/lab/offcanvas/) sidebar'><span class='caption-text'>New TheLadders.com on mobile with the skinny experience. Left: main content screen, right: <a href="http://jasonweaver.name/lab/offcanvas/">off canvas</a> sidebar</span></span></p>

<h2>Adapt to survive</h2>

<p><span class='caption-wrapper left small'><img class='caption' src='http://dev.theladders.com/images/responsive/messy-windows.jpg' width='' height='' alt='Look familiar? Users will squish, stretch, enlarge, and shrink their windows. Responsive design is not just for mobile devices, it’s for the wild jungle of user behavior out there.' title='Look familiar? Users will squish, stretch, enlarge, and shrink their windows. Responsive design is not just for mobile devices, it’s for the wild jungle of user behavior out there.'><span class='caption-text'>Look familiar? Users will squish, stretch, enlarge, and shrink their windows. Responsive design is not just for mobile devices, it’s for the wild jungle of user behavior out there.</span></span>
Mobile is here, and has been for years. It is only a matter of time before we are forced to adapt the front-end code design in order to survive on all screens. To me, responsive design is not an option; it’s mandatory.</p>

<p>It&rsquo;s not just about fitting inside of a phone screen either. Our site should be flexible and sturdy enough to undergo all sorts crazy behavior: windows shrinking, windows expanding, zoom in, zoom out, and so forth. It’s a wilderness of user actions with legions of new mobile devices to support &hellip; how the heck do we keep up?</p>

<h2>One size fits most</h2>

<p>TheLadders’ responsive support subscribes to the majority rule. We prioritize the user experience for the majority, and the minority cases are “overlooked.” This means the website is not tested on every phone and tablet &ndash; even newer ones &ndash; because there isn’t enough site traffic. We sacrificed this level of precision hoping to concentrate instead on improving the experience for the majority.
<span class='caption-wrapper right extra-small'><img class='caption' src='http://dev.theladders.com/images/responsive/device-breakdown.jpeg' width='' height='' alt='Google Analytics device breakdown. Blue - iPhone, Green - iPad, Gray - undetermined. Other slivers are mostly Samsung Android devices.' title='Google Analytics device breakdown. Blue - iPhone, Green - iPad, Gray - undetermined. Other slivers are mostly Samsung Android devices.'><span class='caption-text'>Google Analytics device breakdown. Blue &ndash; iPhone, Green &ndash; iPad, Gray &ndash; undetermined. Other slivers are mostly Samsung Android devices.</span></span></p>

<p><strong>First</strong>, we decided what we were going to support &ndash; devices and screen size ranges:</p>

<ul>
<li>Devices

<ul>
<li>iPhone 4S and up (incl. iPad mini)</li>
<li>iPad 2 and up</li>
</ul>
</li>
<li>Screen size ranges

<ul>
<li>20em (320px) to 48em (768px) for skinny experience support</li>
<li>50em (800px) and up for wide experience support</li>
</ul>
</li>
</ul>


<p>Together these Apple devices compose approximately 17% of our total page visits (includes hits from both browsers and devices). Of course you should look at your own device and browser breakdowns before deciding on a strategy. But do it early, otherwise you will be spending a lot of time debugging for ~1% of your users.</p>

<p><strong>Second</strong>, we decided on our limitations. I think this saved hours of designer/developer time by acknowledging that not everything will interact exactly as it would on a desktop screen:</p>

<ul>
<li>Can’t support every mobile device and every screen width.</li>
<li>Can’t have 100% feature parity between a mobile device and a desktop browser.</li>
<li>Support for touch gestures (like swipe) costs extra code.</li>
<li>Modals are not small screen friendly, especially certain date pickers.</li>
<li>Hover states will be awkward on a touch screen. (If there is a hover state, the first tap is hover and the second tap is click.)</li>
<li>Performance will suffer on mobile connections.</li>
</ul>


<p><strong>Third</strong>, we decided what techniques to use. Truthfully, each of these points needs its own blog topic, but here’s a quick list for now:</p>

<ul>
<li><a href="http://jasonweaver.name/lab/offcanvas">Off canvas</a> layout for our “skinny” experience.</li>
<li><a href="https://developer.mozilla.org/en-US/docs/CSS/Media_queries">Media queries</a> to trigger the skinny vs. wide experience. Used SASS to implement <a href="http://thesassway.com/intermediate/responsive-web-design-in-sass-using-media-queries-in-sass-32">reusable media query break points</a>, making our lives MUCH easier.</li>
<li>Set <a href="https://developer.mozilla.org/en-US/docs/Mobile/Viewport_meta_tag">width=device-width</a> in viewport metatag.</li>
<li><a href="http://css-tricks.com/examples/PerfectFluidWidthLayout">Elastic layout</a> for widths.  Use % widths to create a fluid main content area, and a fixed width for the sidebar area.</li>
<li>Use em. First <a href="http://css-tricks.com/css-font-size">understand</a> what measurement units are out there. There are many reasons to use em, but <a href="http://blog.cloudfour.com/the-ems-have-it-proportional-media-queries-ftw">this article</a> made the best case.</li>
<li>SVG for responsive images.  We chose to use SVG logos and icons that fall back to PNG sprites if SVG is not supported.</li>
</ul>


<p>Now we should never, ever see a horizontal scroll bar. Scrollbars mean users can’t easily see precious content and get a unfulfilling experience.</p>

<p><span class='caption-wrapper center medium'><img class='caption' src='http://dev.theladders.com/images/responsive/sad-scrollbar.png' width='' height='' alt='Ewww horizontal scrollbar' title='Ewww horizontal scrollbar'><span class='caption-text'>Ewww horizontal scrollbar</span></span></p>

<h2>Mobile First?</h2>

<p>Mobile first means to implement your website for mobile devices first, and desktop, etc., second. It helps set boundaries for page weight and complicated layouts that are unfriendly to handhelds.</p>

<p>In theory, mobile first for design and development is a good idea.  It’s a sure-fire way to keep your pages as lightweight as possible. But in practice, our designer created mobile designs simultaneously with desktop designs, remembering to remove superfluous visual elements, ensuring a similar design would work for a small screen. As for development, we coded for the desktop first because we still support IE8 (IE8 doesn’t support media queries).  We built a desktop version first and used media queries to adjust for the mobile version. This way, IE8 screens work without extra media queries. Only mobile and the latest browsers are responsible for triggering media queries.</p>

<p>So we aren’t really mobile first, but it heavily influences how we design the site. This practice works well if the designs are clean, and you are innately stingy with markup.</p>

<h2>Anyway …</h2>

<p><span class='caption-wrapper right extra-small'><img class='caption' src='http://dev.theladders.com/images/responsive/rabbit-pair.jpg' width='' height='' alt='Collaborating like arctic hares' title='Collaborating like arctic hares'><span class='caption-text'>Collaborating like arctic hares</span></span>
My favorite side-effect of going responsive is the close collaboration between designers and developers. It’s hard to articulate screen reactions and device behavior through design comps. During this rewrite our front-end team and design team were pairing regularly throughout the day to determine what responsive meant for TheLadders.</p>

<p>Also, it is important to recognize a responsive website is NOT a substitute for a native phone application.  A browser page and native phone application have completely different rules of conduct, so just because your website is responsive doesn’t mean you don’t need a native application.</p>

<h2>Results</h2>

<p>Our new responsive website is still young, it was only released earlier this year. We’ve just begun to throw serious traffic at it, but I’m confident that it performs and looks better than the old, non-responsive site. We are closely monitoring customer satisfaction and page visits from mobile devices. Hopefully in a couple months we’ll have flattering statistics and feedback to post.</p>

<p>Until then, check it out for yourself: <a href="https://www.theladders.com/">www.theladders.com</a>. While it is by no means perfect&mdash;there are some devices and screen configurations where the experience is less than optimal&mdash;it is far better than the original site both in terms of its responsiveness and clean design.</p>

<p>Join the discussion at <a href="https://news.ycombinator.com/item?id=6641857">Hacker News</a>.</p>

<p><em>Note: In an effort to most accurately describe the current state of our site&rsquo;s responsiveness, this post was edited and updated by Daniel Wislocki. It was originally written by <a href="https://twitter.com/logtailer">Christina Kung</a> in early April, when design was unfinished.</em></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[We're Hosting the first NYC Erlang Factory Lite on September 14th]]></title>
    <link href="http://dev.theladders.com/2013/08/were-hosting-erlang-factory-lite/"/>
    <updated>2013-08-07T16:06:00-04:00</updated>
    <id>http://dev.theladders.com/2013/08/were-hosting-erlang-factory-lite</id>
    <content type="html"><![CDATA[<p>Thanks to one of our Lead Software Engineers, Derek Brown, TheLadders will be hosting the first <a href="http://www.erlang-factory.com/conference/NYC2013">NYC Erlang Factory Lite</a> on the 14th of September.</p>

<p>If you haven’t heard about Erlang before, it is an open-source programming language that was designed for massive concurrency, making it an ideal candidate for scalability on multicore architectures. The language was invented for telecoms, but due to its unique features it quickly made its way into a range of other verticals with similar requirements – from financial switches, through instant messaging servers, HTTP servers and web frameworks, database engines and so forth. It is used for highly available systems, providing uptime of five nines and allowing live code updates, all at a fraction of the effort required by conventional languages such as C++ and Java. Ericsson, Aol, WhatsApp, Nokia, T-Mobile, Heroku and Basho are a few of the companies that employ Erlang.</p>

<p>At TheLadders, we’re currently using two of the most popular software products written in Erlang: the document-oriented database system Couchbase, and RabbitMQ, an AMQP messaging service. I asked Derek how he found out about Erlang in the first place: &ldquo;I became interested in Erlang while first reading about functional programming, and was drawn to its fault-tolerant capabilities and its unique syntax. Erlang&rsquo;s functional nature is a big draw, coming from an imperative background. On top of that, the Erlang community is active and very open. So a few years later here I am co-organizing the NYC Erlang Meetup and now co-organizing the NYC Erlang Factory Lite.&rdquo;</p>

<p>On the 14th of September New York developers will not only get to find out more about Erlang and its applications to distributed environments, but also to meet two of the language’s inventors. NYC Erlang Factory Lite will also be a chance for the East Coast Erlang community to showcase Erlang and its use in the real world. Adam Denenberg, VP of Engineering at the Huffington Post will talk about the architecture and setup built by the Huffington Post to power their real-time WebSocket-based commenting framework.  Leveraging RabbitMQ, SockJS, Cowboy and Erlang, they were able to build and design a highly scalable, customizable solution to power real-time comments. Christopher Meiklejohn, software engineer at Basho, will provide an introduction to building Riak Core applications. Fred Hebert, engineer on Heroku&rsquo;s Routing Team, will talk about Heroku’s cloud platform composed of heterogeneous components (written in Ruby, Erlang and Go, to name a few) and why a large part of its routing stack is written in Erlang.  Other speakers include Christian Gribneau, CTO of Ology Media, and Mahesh Paolini Subramania, Creator of the first Erlang Cloud PBX. For more info check out the program on the ErlangFactory website: <a href="http://www.erlang-factory.com/conference/NYC2013/programme">http://www.erlang-factory.com/conference/NYC2013/programme</a></p>

<p>The high point of the day will the keynotes of Joe Armstrong and Robert Virding, two of the three inventors of the language. Joe was part of the team who designed and implemented the first version of Erlang and has written several Erlang books including Programming Erlang Software for a Concurrent World. He will be talking about the OTP rationale. OTP is a number of different things: a delivery platform for Erlang and the Erlang VM, a large set of general-purpose libraries and a set of design principles. Robert took part in the original system design and contributed much of the original libraries, as well as to the current compiler. While at the lab he also did a lot of work on the implementation of logic and functional languages and on garbage collection. He also taught Erlang to literally hundreds of developers.  In his talk he will look at the rationale behind Erlang and try to show how many of its important features combine to work together.</p>

<p>For those interested in learning Erlang, Robert Virding will teach a three-day course immediately prior to the Erlang Factory Lite. The course contains all the Erlang basics such as sequential and concurrent programming, alongside error handling. More details on all modules here: <a href="http://www.erlang-factory.com/conference/NYC2013/university/ErlangExpress">http://www.erlang-factory.com/conference/NYC2013/university/ErlangExpress</a></p>

<p>Early Bird tickets are available at $70 until the 16th of August. After the 16th, the full-price ticket is $90. Half-price student discounts are also available. Places are limited, so if you want to come you should get your ticket now.</p>

<p>If you want to find out more about Erlang, here’s a great introduction to Erlang ‘from behind the trenches’ video: <a href="https://vimeo.com/19806728">https://vimeo.com/19806728</a>. A lot more videos and slides from various Erlang Factory events around the world can be found on the event website: <a href="http://www.erlang-factory.com/">http://www.erlang-factory.com/</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Denormalize the Datas for Great Good]]></title>
    <link href="http://dev.theladders.com/2013/07/denormalize-the-datas-for-great-good/"/>
    <updated>2013-07-08T16:02:00-04:00</updated>
    <id>http://dev.theladders.com/2013/07/denormalize-the-datas-for-great-good</id>
    <content type="html"><![CDATA[<blockquote><p>Normal is not something to aspire to, it&#8217;s something to get away from.</p><footer><strong>&#8211;Jodie Foster</strong></footer></blockquote>


<h2>Scout reads go slow</h2>

<p>A few weeks ago, as we were about to launch our <a href="http://app.appsflyer.com/id654867487?pid=TLC_organic">iPhone app</a>, we discovered that one of its core features, Scout, frequently took seconds to render.</p>

<center>
<span class='caption-wrapper small'><img class='caption' src='http://dev.theladders.com/images/denormalize-the-datas-for-great-good/scout-screenshot.png' width='' height='' alt='Scout' title='Scout'><span class='caption-text'>Scout</span></span>
</center>


<p>For a little background as to what Scout is, at TheLadders our mission is to find the right person for the right job. One of the ways we strive to deliver on that promise is to provide jobseekers information about jobs they’ll find nowhere else. Serving that mission is Scout, which in a nutshell allows jobseekers to view anonymized information about applicants who have applied to the job they are viewing. Salary, education, career history: we present a lot of useful information to jobseekers about their competition for any given job.</p>

<p>Over time, some attractive jobs accumulate on the order of 30 to 60 applicants, yielding response times of over 1 second (due to multiple synchronous requests, done serially, just to serve <em>one</em> Scout view request).  In cases of higher load, sometimes request times take well over that.</p>

<center>
<span class='caption-wrapper small'><img class='caption' src='http://dev.theladders.com/images/denormalize-the-datas-for-great-good/scout-screenshot-many-applies.png' width='' height='' alt='Scout view of a job with many applicants' title='Scout view of a job with many applicants'><span class='caption-text'>Scout view of a job with many applicants</span></span>
</center>


<p>That brings Scout into unusably slow country, as the Graphite chart below indicates:</p>

<center>
<span class='caption-wrapper medium'><img class='caption' src='http://dev.theladders.com/images/denormalize-the-datas-for-great-good/before-graphite.png' width='' height='' alt='95th percentile of response times for Scout in seconds' title='95th percentile of response times for Scout in seconds'><span class='caption-text'>95th percentile of response times for Scout in seconds</span></span>
</center>


<p>The graph shows the time it takes to form a response to a view-job request issued by our iPhone app. It’s the 95th percentile, which means that 5% of requests had times of the lines in the graph or higher for any given date.  One in twenty requests took this long or longer. There are many lines because we have a horizontally scalable architecture, so there are many backend app nodes.</p>

<p>We managed to bring those seconds down to milliseconds, with about a 1000x decrease in times of high load.  Below I’ll describe the changes in our architecture that enabled us to make such a huge improvement.</p>

<hr />

<h2>Architecture</h2>

<p>In its initial implementation, Scout’s applicant information was gathered and assembled on the fly for each and every request. Driving the iPhone app, we have a backend app server, which is essentially just a number of RESTful endpoints against which our iPhone app issues requests.  Below is a quick rundown of the architecture before I trace a request through our architecture.</p>

<center>
<span class='caption-wrapper medium'><img class='caption' src='http://dev.theladders.com/images/denormalize-the-datas-for-great-good/front-end-orchestration.png' width='' height='' alt='iPhone app talks to the backend app server' title='iPhone app talks to the backend app server'><span class='caption-text'>iPhone app talks to the backend app server</span></span>
</center>


<p>Below this backend server there are a number of RESTful entity servers with which the app server is interacting via HTTP.</p>

<p><span class='caption-wrapper center medium'><img class='caption' src='http://dev.theladders.com/images/denormalize-the-datas-for-great-good/front-end-orchestration-entity.png' width='' height='' alt='Backend app server relies on entity servers' title='Backend app server relies on entity servers'><span class='caption-text'>Backend app server relies on entity servers</span></span></p>

<p>These entity servers in turn query each other and the canonical data store, in our case Clustrix, and that’s that.</p>

<p><span class='caption-wrapper center medium'><img class='caption' src='http://dev.theladders.com/images/denormalize-the-datas-for-great-good/front-end-orchestration-entity-clustrix.png' width='' height='' alt='Entity servers query the db' title='Entity servers query the db'><span class='caption-text'>Entity servers query the db</span></span></p>

<p>So when a user of our iPhone app taps on a job, a request is sent to the backend app server&hellip;</p>

<p><img class="center medium" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/mobile-orchestration-request.png" title="iPhone app makes a request" ></p>

<p>&hellip;which then issues a request to our job application service for all job applications for that job. The response contains a number of links to the where those job applications may be retrieved.</p>

<p><img class="center medium" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/mobile-orchestration-service-request.png" title="backend server queries the job application service for all applications to a job" ></p>

<p>The backend server iterates over those links, requesting the job applications themselves one at a time. Just as before, adhering to hypermedia design, the response contains a link to the jobseeker who applied to the job. For your sanity, I’ve simplified the response to contain only the job seeker link:</p>

<p><img class="center medium" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/mobile-orchestration-service-request2.png" title="backend retrieves each application" ></p>

<p>Finally with that result set, the orchestration service then issues a number of requests to the job seeker service for information about the job seekers who have applied to the job being viewed.  In its initial implementation all of the requests were synchronous and in series as I mentioned earlier. We eventually parallelized them, as you can see in the Graphite chart where the big spikes left diminish towards the right.</p>

<p><img class="center medium" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/mobile-orchestration-service-request3.png" title="backend retrieves each application" ></p>

<p>The iPhone app backend server then extracts the relevant information from those job seekers’ profiles, and returns them as a JSON array of applicants to the mobile app.</p>

<p><img class="center medium" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/mobile-orchestration-response.png" title="backend retrieves each application" ></p>

<p>That is not just a lot of words and diagrams, that is a lot of work!</p>

<p>The workflow includes multiple objects serializing and deserializing, HTTP transfers, hitting the canonical store etc. Why does each request need to assemble this data itself? Why bother hitting the database? Is there an alternative? It seems like a natural fit for a document-oriented database, as the data we are passing back to the client is just a JSON object containing an array of applicants.  We could stand a <a href="http://dev.theladders.com/2013/05/varnish-in-five-acts/">Varnish cache</a> in front of the Scout endpoints on the orchestration service, but then we’d be trading freshness for speed. On the platform team we like to deliver data fast and fresh (and furious).</p>

<p><img class="center" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/tokyo-drift-o.gif" title="how we roll at the Democratic Republic of Platformia" ></p>

<hr />

<h2>Scout reads go fast</h2>

<p>Principal Architect <a href="http://twitter.com/SeanTAllen">Sean T Allen</a> set <a href="http://twitter.com/casio_juarez">Andy Turley</a> and me to improving Scout’s performance. The architecture is surprisingly simple: stick the data in <a href="http://www.couchbase.com/">Couchbase</a> and have the iPhone app backend query that instead. How would we keep this data up-to-date? The first step is to have the job application entity service emit a RabbitMQ event when it receives an application from a job seeker to a particular job (a PUT returning a 201).  On the other end of that message queue there is a  <a href="http://dev.theladders.com/2013/03/riders-on-the-storm-take-a-long-holiday-let-your-children-play/">Storm</a> topology that should listen for that message. The RabbitMQ message would be the entry point into the spout.</p>

<p>The message contains a link to the job seeker who applied to the job, as well as the ID for the job to which she applied.   The message isn’t actually encoded as JSON and transmitted over the wire, but for clarity I’ve displayed the RabbitMQ message as JSON.</p>

<p><img class="center" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/rabbitmq-storm.png" title="RabbitMQ passes along a job-application message to a listening Storm topology" ></p>

<p>The second step, after having received the RabbitMQ message, fetches the job seeker profile from the jobseeker service, and passes that information to the next step.</p>

<p><img class="center" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/rabbitmq-storm-jobseeker.png" title="The Storm topology extracts the job seeker link from the messages and retrieves information about the job seeker who just applied to the job." ></p>

<p>This third step is responsible for persisting the applicant information to a Couchbase bucket. It uses the job ID as the key, and it does a create or update operation on the document corresponding to that key depending on whether there are applicants already in the bucket for that job.</p>

<p><img class="center" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/rabbitmq-storm-couchbase.png" title="The final step is that the topology persists the relevant job" ></p>

<p>That last diagram is a bit of a simplification. Although Couchbase is &ldquo;JSON-aware&rdquo;, it lacks the ability to perform certain operations on the JSON documents it stores.  For example, if the document being stored is an Array, and the client&rsquo;s append method is called, we hoped that Couchbase would add that element to the end of the Array. Instead, it&rsquo;s just a blind String.append, resulting in an invalid JSON document. As a result, we had to implement our own append operation by reading the document (if it exists), adding an item to a list if it’s not already there, and then writing the document.  So it’s more like two operations than one.</p>

<p><em>Now</em> when TheLadders mobile service gets a request for Scout information for a job, all it does is a lookup in Couchbase with that job ID and returns the applicants associated with that key.</p>

<p><img class="center" src="http://dev.theladders.com/images/denormalize-the-datas-for-great-good/mobile-orchestration-couchbase.png" title="iPhone issues a request for Scout information, backend just retrieves it from couchbase" ></p>

<p><span class='caption-wrapper center'><img class='caption' src='http://dev.theladders.com/images/denormalize-the-datas-for-great-good/before-couchbase-after-no-lines.png' width='' height='' alt='95th percentile response time for Scout data, before and after moving to the read view' title='95th percentile response time for Scout data, before and after moving to the read view'><span class='caption-text'>95th percentile response time for Scout data, before and after moving to the read view</span></span></p>

<p>Dramatically faster, even at the 95th percentile.</p>

<hr />

<p>SOA is no panacea. There are many instances where querying a number of backend servers to assemble and aggregate data returned from a database simply doesn&rsquo;t make sense. In those cases, you may do well to denormalize that data and put it in a store that&rsquo;s more efficient for retrieval.</p>

<p>If you find this post interesting, join the dicussion over on <a href="https://news.ycombinator.com/item?id=6015123">Hacker News</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[The Evolution of QA at TheLadders]]></title>
    <link href="http://dev.theladders.com/2013/05/the-evolution-of-qa-at-theladders/"/>
    <updated>2013-05-16T09:04:00-04:00</updated>
    <id>http://dev.theladders.com/2013/05/the-evolution-of-qa-at-theladders</id>
    <content type="html"><![CDATA[<blockquote><p>Change means that what was before wasn&#8217;t perfect. People want things to be better.</p><footer><strong>&#8211;Esther Dyson</strong></footer></blockquote>


<p>Looking back on my 6+ years at TheLadders, I realize how much has changed. When I first joined the company at the start of 2007, there were five other people on the QA team supporting a team of 12 software developers. We followed a strict waterfall process. Product ideas were generated, researched and solidified before any conversation with the development or QA teams. Product managers would spend a large portion of their time hunkered down at their desks writing specification documents, complete with statements on purpose and scope, workflows, wireframes and interaction rules.</p>

<p>After these documents were approved by business owners, the development, QA, and design teams would meet to walk through the document with the goal of estimating the implementation time. QA played the role of a customer advocate and tried to ensure that there was as little uncertainty in the documentation as possible. Instead of walking out of the meeting with estimates, the Product Manager would usually leave with a list of missing workflows and use cases that they would need to add to the spec before development began.</p>

<p>Eventually a release date would be set. Spec and design mocks in hand, the development team would start building the feature.  The QA team would write test cases. We created long lists of actions and expected outcomes that we would use to manually test the development team’s work. Test automation &ndash; unit, integration, functional, load, et al &ndash; was a distant dream. Our primary interaction with the development team was proving the existence of bugs that they couldn’t reproduce.</p>

<p><img class="center" src="http://dev.theladders.com/images/the-evolution-of-qa-at-theladders/it-works-on-my-machine.png"></p>

<p>Even though both the development and QA teams worked off of the same documents, the software never matched the expected behavior according to the test case. Preparing for a release was a never-ending cycle of testing, bug-finding, bug-fixing, and retesting. Fixed release dates and not testing until the end of the development cycle led to releases that were huge events with late, stressful nights leading up to release and late, stressful nights spent cleaning up after it was over!</p>

<p>Between major releases, QA was either responding to customer issues, estimating how long it would take to test an upcoming project, or maintaining our set of regression test case documents in Excel.</p>

<h2>Fast forward to 2009.</h2>

<p>The development team had grown to 35. The QA team had shrunk to 4. We’d transitioned to Scrum a few months previous and shuffled the production teams around to form 4 cross-functional ones. Each team consisted of developers, a QA lead, a designer, a copy editor, and a product manager. The move to Scrum was bumpy as we molded the framework into a process that worked for us but teams had started to find a groove. Ideas were generated and defined within each team, we were checking in with each other often, and delivering value to our customers faster. The teams were working like well-oiled machines&hellip; almost.</p>

<p>Selecting user stories to be undertaken during an iteration had become a team decision, however then the Product Manager and QA rep would work together, without the rest of the team, to define the requirements for each story and flesh them out into use case documents. QA still played the role of customer advocate: working to clearly define expected behavior of each user story in terms of the use case document.</p>

<p>The developers would use these documents to build each user story. Sometimes, they&rsquo;d also write unit tests to cover a few of the more important use case scenarios. Before a story was considered &ldquo;done&rdquo;, a code review would need to be completed by at least two developers. In order to make sure that each story met customer expectations, QA would need to manually test the feature. The need for manual testing didn’t end there. At the end of each iteration, QA would run through the entire regression suite before releasing.</p>

<p>We had made some improvements, but it was clear that we could improve the process if we increased communication of requirements and lessened the amount of manual testing that needed to be done.</p>

<p>We were releasing every two weeks. They were smaller and less stressful than in the past. QA still spent a lot of time manually testing, but it was done throughout the iteration instead of being squeezed in at the end. This meant that QA had a lot more downtime than before. While some of this time was spent doing the work we had always done, we also made sure to spend time learning about new tools and best practices. Over time, we learned a lot about automated functional testing. We experimented with different frameworks, starting with a simple record-and-playback tool, until we learned more about what we needed and which tools might best fit that need. I’ll write a post later to shed some light on what that journey was like, but suffice it to say that it took quite a bit of trial and error!</p>

<h2>Fast forward to now.</h2>

<p>We&rsquo;ve moved beyond Scrum to an adapted Agile process that emphasises cross-functional, independent, and highly collaborative teams. Each team is free to adopt practices that work for them. They have free reign to define the way that they work amongst themselves. There are certain commonalities that hold true across all our teams:</p>

<p><strong>QA plays the role of customer advocate</strong>. We know our customers well and our applications even better. This gives us a unique perspective when defining new user stories that helps our teams deliver better products for our users.</p>

<p><strong>The teams define user stories together</strong>. QA still helps flesh out requirements and acceptance criteria, but with the participation of the rest of the team. Defining requirements as a team helps us make sure that we&rsquo;re all on the same page which in turn helps us work better and faster.</p>

<p><strong>We aren&rsquo;t heavy-handed with the documentation</strong>. We&rsquo;ve found that most of our previous attempts at documentation were wasteful &ndash; and often not as useful to the developers as we thought. Instead, after each team conversation around story requirements QA either writes notes and acceptance criteria on the back of the story card (if the team uses a whiteboard to manage their backlog) or in comments attached to the story in Trello.</p>

<p><strong>We automate tests when it&rsquo;s the smart thing to do</strong>. Manually testing everything definitely was not the best thing for us to do, but <a href="http://xkcd.com/1205/">neither is automating every test</a>. Instead, QA and the developers have frequent conversations around how to appropriately test each user scenario: Can it be sufficiently covered in a unit or integration test? Does it make sense to write an expensive automated functional test that checks the functionality through the browser? How important is this functionality? What&rsquo;s the risk of it breaking? We also keep test coverage at the front of our minds and use different <a href="http://dev.theladders.com/2013/02/mutation-testing-with-pit-a-step-beyond-normal-code-coverage/">strategies</a> to make sure that we can trust the tests that we do automate.</p>

<p><strong>Developers write most of our automated tests</strong>. Whether unit, integration, functional, or load tests, usually it&rsquo;s a developer that&rsquo;s going to write it. QA does pitch in when they can but with one QA lead per team, it&rsquo;s not feasible for them to write all of the automated tests without becoming a bottleneck.</p>

<p><strong>We peer review code</strong> with QA actively participating.</p>

<p><strong>We&rsquo;ve set up continuous integration builds</strong>. All of our unit-level tests run after each commit and all of our automated functional regression tests must pass before each deployment. If a test fails, we fix it.</p>

<p>And of course, QA is still responsible for investigating and responding to customer issues. Some things will never change.</p>

<p>We&rsquo;ve come a long way! But that doesn&rsquo;t mean that we can&rsquo;t get better. As a wise man* once said, consistency requires you to be as ignorant today as you were a year ago. Here at TheLadders, we&rsquo;re always learning.</p>

<p>*Bernard Berenson</p>

<p>Join the discussion over at <a href="http://www.softwaretestingclub.com/forum/topics/how-has-the-role-of-qa-evolved">Software Testing Club</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Varnish in Five Acts]]></title>
    <link href="http://dev.theladders.com/2013/05/varnish-in-five-acts/"/>
    <updated>2013-05-03T13:21:00-04:00</updated>
    <id>http://dev.theladders.com/2013/05/varnish-in-five-acts</id>
    <content type="html"><![CDATA[<blockquote><p>“Take a load off Annie. Take a load for free. Take a load off Annie. And you put the load right on me.” &#8211; Robbie Robertson</p></blockquote>


<h2>Act I: The Players</h2>

<p>At TheLadders, we have a number of entity services that clients access via HTTP. Some examples are the job service, the job application service and the topic of this post: the job seeker service. Each service manages the lifecycle of a core entity in our domain. On each request, the service gathers data from multiple data sources to build a complete entity and then serializes that entity to JSON. This is done for every request and is incredibly wasteful when you consider that most of these entities don’t change that often. This was a major bottleneck in our infrastructure. In the case of the job seeker service, the same entity can be requested multiple times per request from our consumer website.</p>

<p>All this repeated, unnecessary entity object assembly and JSON serialization created scaling problems. Making matters worse, we periodically have batch processes that can increase service load by an order of magnitude. Caching is an easy win here. The question is how.</p>

<p>Initial attempts to cache these entities were done inside the service JVMs, using familiar and popular JVM based caches like EHcache and calls out to memcache. Unfortunately, this left us operating at JVM speeds and the caches were competing with the service logic for memory and threads.</p>

<p>In addition, our service code was muddled with messy caching logic.  Making the code harder to reuse, and more annoyingly, changes just affecting caching forced us to re-release the entire service.  We didn’t like this mixing of concerns.</p>

<p>We thought we could do better if we used external read through caches. It&rsquo;s easy to slide them between existing clients and services. With caching outside the service, it get released only when their logic changes not because we’re tuning caching.</p>

<p>For reasons too numerous to cover in this post we chose <a href="https://www.varnish-cache.org/about">Varnish</a> as our read through cache.</p>

<hr />

<h2>Act II: The Architecture</h2>

<p>When we introduced Varnish to our architecture, we wanted to make sure we were not adding a single point of failure. Simply put, if the cache layer goes down, our infrastructure should continue running. Performance might be degraded, but we should continue to be able to serve content to clients.</p>

<p>The diagram below shows a typical setup. In a normal scenario, a client accesses Varnish via a load balancer. Varnish in turn farms out the work in round robin fashion to one of four job seeker service nodes. Should Varnish become unavailable, the load balancer stops sending traffic to Varnish and reroutes it to the four job seeker service nodes.</p>

<p><img class="center" src="http://dev.theladders.com/images/varnish-in-five-acts/varnish-flow.png"></p>

<p>Of all our entity services, the job seeker services carries the highest median load. The graph below is the 1 minute request rate on 4 jobseeker service nodes over the 36 hour period before and after Varnish was turned on.</p>

<p><img class="center" src="http://dev.theladders.com/images/varnish-in-five-acts/before-after.png"></p>

<hr />

<h2>Act III: Cache Invalidation</h2>

<p>Cache invalidation is one of the 2 hard problems in computer science along with naming things and off by one errors.</p>

<p>We cache job seeker entity representations until some point in the “far future”, which is great until something about that job seeker changes, then we must invalidate the cached entry. So, how do we do that?</p>

<p>Two ways.</p>

<h3>Via Header:</h3>

<p>All requests that change the state of a job seeker that are made via the service attach a header in the response called &ldquo;x-invalidates&rdquo; that looks something like:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>x-invalidates: /jobseekers/123</span></code></pre></td></tr></table></div></figure>


<p>Varnish, when it sees this header, turns the value into a content expiring regular expression. My team mate <a href="http://twitter.com/johnconnolly">@johnconnolly</a> learned about this general technique from <a href="https://twitter.com/kevburnsjr">Kevin Burns Jr.</a> at <a href="RESTFest">http://restfest.org</a> 2012.  I used Kevin’s post on the <a href="http://blog.kevburnsjr.com/tagged-cache-invalidation">subject</a> as a jumping off point for our implementation.</p>

<h3>Via Magic:</h3>

<p>Once upon a time, we had a database administrator named Gennady. Gennady wrote a <a href="http://www.dinodigusa.com/images/Magic1.gif">PHP script that reads MySQL’s binary logs</a>, looking for changes to a set of predefined tables. When it sees an update, it finds the primary key for the row and fires off an invalidation request. In our case, a purge of the cached jobseeker url in Varnish. This allows us to invalidate cached job seeker entities even when the update was performed via legacy code that interacts directly with the database rather than through the service.</p>

<p>If you were to do this manually, it would look something like:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>curl -X PURGE varnish-jobseeker/jobseekers/123</span></code></pre></td></tr></table></div></figure>


<hr />

<h2>Act IV: Configuration Spelunking</h2>

<p>So, how did we do it? I’m going to break down our configuration into its parts and cover the general role each part plays. From here on out, I’m assuming you understand the basics of how Varnish works and how you configure it. Also, there is some repetition in our configuration that isn’t required, it just makes it easier for our configuration management tool, puppet, to create the final output.</p>

<h3>Load Balancing</h3>

<p>We have four service servers behind varnish so we create four backend entries and then set up a director to round robin between them. Then in vcl_recv, we set our director named &lsquo;nodes&rsquo; to be the backend that we will use to fetch content.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>backend JS1 <span class="o">{</span>
</span><span class='line'>  .host  <span class="o">=</span> <span class="s2">&quot;JS1&quot;</span>;
</span><span class='line'>  .port  <span class="o">=</span> <span class="s2">&quot;8080&quot;</span>;
</span><span class='line'>
</span><span class='line'>  ...
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>backend JS2 <span class="o">{</span>
</span><span class='line'>  .host  <span class="o">=</span> <span class="s2">&quot;JS2&quot;</span>;
</span><span class='line'>  .port  <span class="o">=</span> <span class="s2">&quot;8080&quot;</span>;
</span><span class='line'>
</span><span class='line'>  ...
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>backend JS3 <span class="o">{</span>
</span><span class='line'>  .host  <span class="o">=</span> <span class="s2">&quot;JS3&quot;</span>;
</span><span class='line'>  .port  <span class="o">=</span> <span class="s2">&quot;8080&quot;</span>;
</span><span class='line'>
</span><span class='line'>  ...
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>backend JS4 <span class="o">{</span>
</span><span class='line'>  .host  <span class="o">=</span> <span class="s2">&quot;JS4&quot;</span>;
</span><span class='line'>  .port  <span class="o">=</span> <span class="s2">&quot;8080&quot;</span>;
</span><span class='line'>
</span><span class='line'>  ...
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>director nodes round-robin <span class="o">{</span>
</span><span class='line'>  <span class="o">{</span> .backend <span class="o">=</span> JS1 ; <span class="o">}</span>
</span><span class='line'>  <span class="o">{</span> .backend <span class="o">=</span> JS2 ; <span class="o">}</span>
</span><span class='line'>  <span class="o">{</span> .backend <span class="o">=</span> JS3 ; <span class="o">}</span>
</span><span class='line'>  <span class="o">{</span> .backend <span class="o">=</span> JS4 ; <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_recv <span class="o">{</span>
</span><span class='line'>  <span class="nb">set </span>req.backend <span class="o">=</span> nodes;
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="c"># store in cache only by url, not backend host</span>
</span><span class='line'>sub vcl_hash <span class="o">{</span>
</span><span class='line'>  hash_data<span class="o">(</span>req.url<span class="o">)</span>;
</span><span class='line'>  <span class="k">return</span> <span class="o">(</span><span class="nb">hash</span><span class="o">)</span>;
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>Degraded</h3>

<p>Each backend is setup with a probe url that we use to check its health. If the probe url doesn&rsquo;t return at least one HTTP 200 response within a fifteen second period, we mark that backend as unhealthy.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>backend ... <span class="o">{</span>
</span><span class='line'>   ...
</span><span class='line'>
</span><span class='line'>  .probe <span class="o">=</span> <span class="o">{</span>
</span><span class='line'>    .url <span class="o">=</span> <span class="s2">&quot;/donjohnson/pulse&quot;</span>;
</span><span class='line'>    .interval <span class="o">=</span> 5s;
</span><span class='line'>    .timeout <span class="o">=</span> 250ms;
</span><span class='line'>    .window <span class="o">=</span> 3;
</span><span class='line'>    .threshold <span class="o">=</span> 2;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Varnish has the concept of a grace period, wherein, we can keep content alive in our cache past the TTL based on the health status of our backends. In our case, when the all backends are down, we keep cached items alive for an extra hour. During this time, we operate in a degraded status. Read requests for cached items will be handled while write requests will fail because there is no backend service to handle them.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>sub vcl_fetch <span class="o">{</span>
</span><span class='line'>  <span class="c"># max time to keep an item in the cache past its ttl</span>
</span><span class='line'>  <span class="c"># used in conjunction with code in vcl_recv to</span>
</span><span class='line'>  <span class="c"># deal with &#39;sick&#39; backends</span>
</span><span class='line'>  <span class="nb">set </span>beresp.grace <span class="o">=</span> 1h;
</span><span class='line'>
</span><span class='line'>  ...
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_recv <span class="o">{</span>
</span><span class='line'>  ...
</span><span class='line'>
</span><span class='line'>  <span class="c"># extra ttl for cached objects based on backend health</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>!req.backend.healthy<span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="nb">set </span>req.grace <span class="o">=</span> 1h;
</span><span class='line'>  <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class='line'>    <span class="nb">set </span>req.grace <span class="o">=</span> 15s;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<h3>Invalidation</h3>

<p>We do two types of invalidation:</p>

<ul>
<li>invalidation based on the &lsquo;x-invalidates&rsquo; header that comes back with a response</li>
<li>&lsquo;manual&rsquo; invalidation based on sending the HTTP PURGE verb to a url in the Varnish cache.</li>
</ul>


<p>The ability to do a manual purge is limited to a small set of IP addresses that we validate against when a purge request is received.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>acl purge <span class="o">{</span>
</span><span class='line'>  <span class="s2">&quot;localhost&quot;</span>;
</span><span class='line'>  <span class="s2">&quot;10.10.10.10&quot;</span>;
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_recv <span class="o">{</span>
</span><span class='line'>  ...
</span><span class='line'>
</span><span class='line'>  <span class="c"># &#39;manual&#39; purge</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>req.request <span class="o">==</span> <span class="s2">&quot;PURGE&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(</span>client.ip ~ purge<span class="o">)</span> <span class="o">{</span>
</span><span class='line'>       <span class="k">return</span><span class="o">(</span>lookup<span class="o">)</span>;
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>    error 405 <span class="s2">&quot;Not allowed.&quot;</span>;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  ...
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>The actual mechanics of doing the purge are fairly simple. If the url attempted to be purged exists, purge it and return a 200.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>sub vcl_hit <span class="o">{</span>
</span><span class='line'>  <span class="c"># &#39;manual&#39; purge</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>req.request <span class="o">==</span> <span class="s2">&quot;PURGE&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    purge;
</span><span class='line'>    error 200 <span class="s2">&quot;Purged.&quot;</span>;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>If it doesn&rsquo;t, return a 404 response code:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>sub vcl_miss <span class="o">{</span>
</span><span class='line'>  <span class="c"># &#39;manual&#39; purge</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>req.request <span class="o">==</span> <span class="s2">&quot;PURGE&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    purge;
</span><span class='line'>    error 404 <span class="s2">&quot;Not in cache.&quot;</span>;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Update requests include invalidation-related headers. Every request we fetch has, inside of Varnish, its request url stored in a special x-url header. This will be used as the url to check the x-invalidates header against. As this header is purely for our internal use, we remove it before delivering items to a client:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>sub vcl_fetch <span class="o">{</span>
</span><span class='line'>  ...
</span><span class='line'>
</span><span class='line'>  <span class="nb">set </span>beresp.http.x-url <span class="o">=</span> req.url;
</span><span class='line'>
</span><span class='line'>  ...
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_deliver <span class="o">{</span>
</span><span class='line'>  <span class="c"># clear internal cache invalidation header before sending to client</span>
</span><span class='line'>  <span class="nb">unset </span>resp.http.x-url;
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Any &lsquo;successful&rsquo; PUT, POST, DELETE or PATCH response will have its x-invalidates header used as a regular expression to invalidate existing content whose x-url header matches the x-invalidates regex.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>sub vcl_fetch <span class="o">{</span>
</span><span class='line'>  ...
</span><span class='line'>
</span><span class='line'>  <span class="c"># cache invalidation</span>
</span><span class='line'>  <span class="nb">set </span>beresp.http.x-url <span class="o">=</span> req.url;
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>req.request <span class="o">==</span> <span class="s2">&quot;PUT&quot;</span> <span class="o">||</span> req.request <span class="o">==</span> <span class="s2">&quot;POST&quot;</span> <span class="o">||</span> req.request <span class="o">==</span> <span class="s2">&quot;DELETE&quot;</span> <span class="o">||</span> req.request <span class="o">==</span> <span class="s2">&quot;PATCH&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="k">if</span>  <span class="o">(</span>beresp.status &gt;<span class="o">=</span> 200 <span class="o">&amp;&amp;</span> beresp.status &lt; 400<span class="o">)</span> <span class="o">{</span>
</span><span class='line'>     ban<span class="o">(</span><span class="s2">&quot;obj.http.x-url ~ &quot;</span> + beresp.http.x-invalidates<span class="o">)</span>;
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<hr />

<h2>Act V: The final product</h2>

<p>And finally, we put it all together into a complete file (note, we use Varnish 3, the semantics around ban/purge changed from v2 to v3):</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
<span class='line-number'>75</span>
<span class='line-number'>76</span>
<span class='line-number'>77</span>
<span class='line-number'>78</span>
<span class='line-number'>79</span>
<span class='line-number'>80</span>
<span class='line-number'>81</span>
<span class='line-number'>82</span>
<span class='line-number'>83</span>
<span class='line-number'>84</span>
<span class='line-number'>85</span>
<span class='line-number'>86</span>
<span class='line-number'>87</span>
<span class='line-number'>88</span>
<span class='line-number'>89</span>
<span class='line-number'>90</span>
<span class='line-number'>91</span>
<span class='line-number'>92</span>
<span class='line-number'>93</span>
<span class='line-number'>94</span>
<span class='line-number'>95</span>
<span class='line-number'>96</span>
<span class='line-number'>97</span>
<span class='line-number'>98</span>
<span class='line-number'>99</span>
<span class='line-number'>100</span>
<span class='line-number'>101</span>
<span class='line-number'>102</span>
<span class='line-number'>103</span>
<span class='line-number'>104</span>
<span class='line-number'>105</span>
<span class='line-number'>106</span>
<span class='line-number'>107</span>
<span class='line-number'>108</span>
<span class='line-number'>109</span>
<span class='line-number'>110</span>
<span class='line-number'>111</span>
<span class='line-number'>112</span>
<span class='line-number'>113</span>
<span class='line-number'>114</span>
<span class='line-number'>115</span>
<span class='line-number'>116</span>
<span class='line-number'>117</span>
<span class='line-number'>118</span>
<span class='line-number'>119</span>
<span class='line-number'>120</span>
<span class='line-number'>121</span>
<span class='line-number'>122</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>backend JS1 <span class="o">{</span>
</span><span class='line'>  .host  <span class="o">=</span> <span class="s2">&quot;JS1&quot;</span>;
</span><span class='line'>  .port  <span class="o">=</span> <span class="s2">&quot;8080&quot;</span>;
</span><span class='line'>  .probe <span class="o">=</span> <span class="o">{</span>
</span><span class='line'>    .url <span class="o">=</span> <span class="s2">&quot;/donjohnson/pulse&quot;</span>;
</span><span class='line'>    .interval <span class="o">=</span> 5s;
</span><span class='line'>    .timeout <span class="o">=</span> 250ms;
</span><span class='line'>    .window <span class="o">=</span> 3;
</span><span class='line'>    .threshold <span class="o">=</span> 2;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>backend JS2 <span class="o">{</span>
</span><span class='line'>  .host  <span class="o">=</span> <span class="s2">&quot;JS2&quot;</span>;
</span><span class='line'>  .port  <span class="o">=</span> <span class="s2">&quot;8080&quot;</span>;
</span><span class='line'>  .probe <span class="o">=</span> <span class="o">{</span>
</span><span class='line'>    .url <span class="o">=</span> <span class="s2">&quot;/donjohnson/pulse&quot;</span>;
</span><span class='line'>    .interval <span class="o">=</span> 5s;
</span><span class='line'>    .timeout <span class="o">=</span> 250ms;
</span><span class='line'>    .window <span class="o">=</span> 3;
</span><span class='line'>    .threshold <span class="o">=</span> 2;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>backend JS3 <span class="o">{</span>
</span><span class='line'>  .host  <span class="o">=</span> <span class="s2">&quot;JS3&quot;</span>;
</span><span class='line'>  .port  <span class="o">=</span> <span class="s2">&quot;8080&quot;</span>;
</span><span class='line'>  .probe <span class="o">=</span> <span class="o">{</span>
</span><span class='line'>    .url <span class="o">=</span> <span class="s2">&quot;/donjohnson/pulse&quot;</span>;
</span><span class='line'>    .interval <span class="o">=</span> 5s;
</span><span class='line'>    .timeout <span class="o">=</span> 250ms;
</span><span class='line'>    .window <span class="o">=</span> 3;
</span><span class='line'>    .threshold <span class="o">=</span> 2;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>backend JS4 <span class="o">{</span>
</span><span class='line'>  .host  <span class="o">=</span> <span class="s2">&quot;JS4&quot;</span>;
</span><span class='line'>  .port  <span class="o">=</span> <span class="s2">&quot;8080&quot;</span>;
</span><span class='line'>  .probe <span class="o">=</span> <span class="o">{</span>
</span><span class='line'>    .url <span class="o">=</span> <span class="s2">&quot;/donjohnson/pulse&quot;</span>;
</span><span class='line'>    .interval <span class="o">=</span> 5s;
</span><span class='line'>    .timeout <span class="o">=</span> 250ms;
</span><span class='line'>    .window <span class="o">=</span> 3;
</span><span class='line'>    .threshold <span class="o">=</span> 2;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>director nodes round-robin <span class="o">{</span>
</span><span class='line'>  <span class="o">{</span> .backend <span class="o">=</span> JS1 ; <span class="o">}</span>
</span><span class='line'>  <span class="o">{</span> .backend <span class="o">=</span> JS2 ; <span class="o">}</span>
</span><span class='line'>  <span class="o">{</span> .backend <span class="o">=</span> JS3 ; <span class="o">}</span>
</span><span class='line'>  <span class="o">{</span> .backend <span class="o">=</span> JS4 ; <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="c"># what machines can institute a &#39;manual&#39; purge</span>
</span><span class='line'>acl purge <span class="o">{</span>
</span><span class='line'>  <span class="s2">&quot;localhost&quot;</span>;
</span><span class='line'>  <span class="s2">&quot;192.1.1.4&quot;</span>;
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="c"># store in cache only by url, not backend host</span>
</span><span class='line'>sub vcl_hash <span class="o">{</span>
</span><span class='line'>  hash_data<span class="o">(</span>req.url<span class="o">)</span>;
</span><span class='line'>  <span class="k">return</span> <span class="o">(</span><span class="nb">hash</span><span class="o">)</span>;
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_fetch <span class="o">{</span>
</span><span class='line'>  <span class="c"># max time to keep an item in the cache past its ttl</span>
</span><span class='line'>  <span class="c"># used in conjunction with code in vcl_recv to</span>
</span><span class='line'>  <span class="c"># deal with &#39;sick&#39; backends</span>
</span><span class='line'>  <span class="nb">set </span>beresp.grace <span class="o">=</span> 1h;
</span><span class='line'>
</span><span class='line'>  <span class="c"># cache invalidation</span>
</span><span class='line'>  <span class="nb">set </span>beresp.http.x-url <span class="o">=</span> req.url;
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>req.request <span class="o">==</span> <span class="s2">&quot;PUT&quot;</span> <span class="o">||</span> req.request <span class="o">==</span> <span class="s2">&quot;POST&quot;</span> <span class="o">||</span> req.request <span class="o">==</span> <span class="s2">&quot;DELETE&quot;</span> <span class="o">||</span> req.request <span class="o">==</span> <span class="s2">&quot;PATCH&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="k">if</span>  <span class="o">(</span>beresp.status &gt;<span class="o">=</span> 200 <span class="o">&amp;&amp;</span> beresp.status &lt; 400<span class="o">)</span> <span class="o">{</span>
</span><span class='line'>     ban<span class="o">(</span><span class="s2">&quot;obj.http.x-url ~ &quot;</span> + beresp.http.x-invalidates<span class="o">)</span>;
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_recv <span class="o">{</span>
</span><span class='line'>  <span class="nb">set </span>req.backend <span class="o">=</span> nodes;
</span><span class='line'>
</span><span class='line'>  <span class="c"># &#39;manual&#39; purge</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>req.request <span class="o">==</span> <span class="s2">&quot;PURGE&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">(</span>client.ip ~ purge<span class="o">)</span> <span class="o">{</span>
</span><span class='line'>       <span class="k">return</span><span class="o">(</span>lookup<span class="o">)</span>;
</span><span class='line'>    <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>    error 405 <span class="s2">&quot;Not allowed.&quot;</span>;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="c"># extra ttl for cached objects based on backend health</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>!req.backend.healthy<span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    <span class="nb">set </span>req.grace <span class="o">=</span> 1h;
</span><span class='line'>  <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
</span><span class='line'>    <span class="nb">set </span>req.grace <span class="o">=</span> 15s;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_deliver <span class="o">{</span>
</span><span class='line'>  <span class="c"># clear internal cache invalidation header before sending to client</span>
</span><span class='line'>  <span class="nb">unset </span>resp.http.x-url;
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_hit <span class="o">{</span>
</span><span class='line'>  <span class="c"># &#39;manual&#39; purge</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>req.request <span class="o">==</span> <span class="s2">&quot;PURGE&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    purge;
</span><span class='line'>    error 200 <span class="s2">&quot;Purged.&quot;</span>;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'>sub vcl_miss <span class="o">{</span>
</span><span class='line'>  <span class="c"># &#39;manual&#39; purge</span>
</span><span class='line'>  <span class="k">if</span> <span class="o">(</span>req.request <span class="o">==</span> <span class="s2">&quot;PURGE&quot;</span><span class="o">)</span> <span class="o">{</span>
</span><span class='line'>    purge;
</span><span class='line'>    error 404 <span class="s2">&quot;Not in cache.&quot;</span>;
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<h2>Hidden Track Bonus Act:</h2>

<p><img class="center" src="http://dev.theladders.com/images/varnish-in-five-acts/varnish-all-the-things.jpg"></p>

<p>Join the discussion over at <a href="https://news.ycombinator.com/item?id=5651874">Hacker News</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Getting Some Clojure: nREPL in a Spring App]]></title>
    <link href="http://dev.theladders.com/2013/04/getting-some-clojure-nrepl-in-a-spring-app/"/>
    <updated>2013-04-02T16:04:00-04:00</updated>
    <id>http://dev.theladders.com/2013/04/getting-some-clojure-nrepl-in-a-spring-app</id>
    <content type="html"><![CDATA[<blockquote><p>You don&#8217;t have to be great to start, but you have to start to be great.</p><footer><strong>&#8211;Zig Ziglar</strong></footer></blockquote>


<p>Despite having read through the
<a href="http://www.amazon.com/The-Joy-Clojure-Thinking-Way/dp/1935182641/">several</a>
<a href="http://www.amazon.com/Clojure-Action-Amit-Rathore/dp/1935182595/">Clojure</a>
<a href="http://www.amazon.com/Programming-Clojure-Pragmatic-Programmers-Halloway/dp/1934356336/">books</a>
on my shelf and having played around with
<a href="http://www.4clojure.com/">4Clojure</a> (user “wislocki”, ranked 2309th
the last time I checked), I’m not feeling much confidence with the
language yet. We use embedded <a href="http://www.eclipse.org/jetty/">Jetty</a>
to run the website on personal development machines, and I figured it
would be great to be able to play with it interactively in a Clojure
<a href="http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a>
and get some more hands-on experience.</p>

<p>Practically speaking, since the website is a Java application, a Scala
REPL might make more sense, bridging the gap between the
object-oriented and functional styles more smoothly. But this is a
Clojure-learning exercise, after all.</p>

<p>We’re using
<a href="http://www.springsource.org/spring-framework">Spring 3.1.x</a>, and I
initially looked to see if there was any precedence for combining a
Clojure REPL with Spring. The links I found were mostly out of date or
not relevant.</p>

<p>I decided that I would use
<a href="https://github.com/clojure/tools.nrepl">nREPL</a>, since the
alternative, <a href="https://github.com/technomancy/swank-clojure">Swank</a>, is
no longer maintained. As we’re using
<a href="http://static.springsource.org/spring/docs/3.1.x/spring-framework-reference/html/beans.html#beans-java">Spring’s Java configuration system</a>,
it would have to be started up there. That’s actually pretty
straightforward. Using
<a href="https://github.com/clojure/tools.nrepl#embedding-nrepl-starting-a-server">code</a>
from the nREPL README to start the server, I created the following
Spring configuration class:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kn">package</span> <span class="n">com</span><span class="o">.</span><span class="na">theladders</span><span class="o">.</span><span class="na">lw</span><span class="o">.</span><span class="na">config</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="nn">java.io.StringReader</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="nn">javax.annotation.PostConstruct</span><span class="o">;</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">javax.inject.Inject</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="nn">org.springframework.context.ApplicationContext</span><span class="o">;</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">org.springframework.context.annotation.Configuration</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'><span class="kn">import</span> <span class="nn">clojure.lang.Compiler</span><span class="o">;</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">clojure.lang.RT</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'><span class="nd">@Configuration</span>
</span><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">NReplConfig</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="c1">// Clojure code to start the server. If you wanted to, you could probably move the port value</span>
</span><span class='line'>  <span class="c1">// into a Java properties file, and use the @Value annotation to provide it instead.</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">String</span> <span class="n">NREPL_INIT</span> <span class="o">=</span> <span class="s">&quot;(use &#39;[clojure.tools.nrepl.server :only (start-server stop-server)]) &quot;</span> <span class="o">+</span>
</span><span class='line'>                                           <span class="s">&quot;(start-server :port 7888)&quot;</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="nd">@Inject</span>
</span><span class='line'>  <span class="kd">private</span> <span class="n">ApplicationContext</span>  <span class="n">context</span><span class="o">;</span>
</span><span class='line'>
</span><span class='line'>  <span class="c1">// Annotated with @PostConstruct so that this method is run after the object is instantiated and </span>
</span><span class='line'>  <span class="c1">// any other dependency injection has taken place.</span>
</span><span class='line'>
</span><span class='line'>  <span class="nd">@PostConstruct</span>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">initializeRepl</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="c1">// Load the Clojure Runtime class so that the Compiler can properly use it.</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">Class</span><span class="o">.</span><span class="na">forName</span><span class="o">(</span><span class="s">&quot;clojure.lang.RT&quot;</span><span class="o">);</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// Start the nREPL server.</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">Compiler</span><span class="o">.</span><span class="na">load</span><span class="o">(</span><span class="k">new</span> <span class="n">StringReader</span><span class="o">(</span><span class="n">NREPL_INIT</span><span class="o">));</span>
</span><span class='line'>
</span><span class='line'>    <span class="c1">// Make the Spring context available in the &quot;lw&quot; namespace.</span>
</span><span class='line'>
</span><span class='line'>    <span class="n">RT</span><span class="o">.</span><span class="na">var</span><span class="o">(</span><span class="s">&quot;lw&quot;</span><span class="o">,</span> <span class="s">&quot;*context*&quot;</span><span class="o">,</span> <span class="n">context</span><span class="o">);</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>(Thanks to <a href="https://twitter.com/petitlaurent">Laurent Petit</a> of
<a href="https://code.google.com/p/counterclockwise/">counterclockwise</a>-fame
for recommending the above class-loading strategy, as well as the the
suggestion to replace the use of <code>(ns lw)</code> with <code>(in-ns 'lw)</code> in the
REPL example below.)</p>

<p>With this file in place, running <code>mvn jetty:run</code> starts up both the
web server and nREPL. While I could have used
<a href="https://github.com/technomancy/leiningen">Leiningen</a> to connect the
REPL, I decided to use <a href="http://www.gnu.org/software/emacs/">emacs</a>
instead. If you’re using <a href="http://batsov.com/">Bozhidar Batsov</a>’s
<a href="https://github.com/bbatsov/prelude">Prelude configuration</a>, you
already have support for nREPL installed, otherwise you can get it
through <a href="http://melpa.milkbox.net/">MELPA</a> or
<a href="http://marmalade-repo.org/">Marmalade</a> or directly
<a href="https://github.com/kingtim/nrepl.el">here</a>. After a quick <code>M-x
nrepl</code>, I can interact with the running server like so:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="o">;</span> <span class="n">nREPL</span> <span class="mf">0.1</span><span class="o">.</span><span class="mi">7</span><span class="o">-</span><span class="n">preview</span>
</span><span class='line'><span class="n">user</span><span class="o">&gt;</span> <span class="o">(</span><span class="n">in</span><span class="o">-</span><span class="n">ns</span> <span class="err">&#39;</span><span class="n">lw</span><span class="o">)</span>
</span><span class='line'><span class="err">#</span><span class="o">&lt;</span><span class="n">Namespace</span> <span class="n">lw</span><span class="o">&gt;</span>
</span><span class='line'><span class="n">lw</span><span class="o">&gt;</span> <span class="o">*</span><span class="n">context</span><span class="o">*</span>
</span><span class='line'><span class="err">#</span><span class="o">&lt;</span><span class="n">AnnotationConfigWebApplicationContext</span> <span class="n">Root</span> <span class="nl">WebApplicationContext:</span> <span class="n">startup</span> <span class="n">date</span> <span class="o">[</span><span class="n">Thu</span> <span class="n">Mar</span> <span class="mi">28</span> <span class="mi">16</span><span class="o">:</span><span class="mi">25</span><span class="o">:</span><span class="mi">12</span> <span class="n">EDT</span> <span class="mi">2013</span><span class="o">];</span> <span class="n">root</span> <span class="n">of</span> <span class="n">context</span> <span class="n">hierarchy</span><span class="o">&gt;</span>
</span><span class='line'><span class="n">lw</span><span class="o">&gt;</span> <span class="o">(</span><span class="n">def</span> <span class="n">jss</span> <span class="o">(.</span><span class="na">getBean</span> <span class="o">*</span><span class="n">context</span><span class="o">*</span> <span class="n">getBean</span> <span class="s">&quot;jobseekerService&quot;</span><span class="o">))</span>
</span><span class='line'><span class="err">#&#39;</span><span class="n">lw</span><span class="o">/</span><span class="n">jss</span>
</span><span class='line'><span class="n">lw</span><span class="o">&gt;</span> <span class="n">jss</span>
</span><span class='line'><span class="err">#</span><span class="o">&lt;</span><span class="n">JobseekerService</span> <span class="n">com</span><span class="o">.</span><span class="na">theladders</span><span class="o">.</span><span class="na">lw</span><span class="o">.</span><span class="na">jobseeker</span><span class="o">.</span><span class="na">service</span><span class="o">.</span><span class="na">JobseekerService</span><span class="err">@</span><span class="mi">1</span><span class="n">f4e3b7c</span><span class="o">&gt;</span>
</span><span class='line'><span class="n">lw</span><span class="o">&gt;</span> <span class="o">(</span><span class="n">def</span> <span class="n">jsid</span> <span class="o">(</span><span class="n">com</span><span class="o">.</span><span class="na">theladders</span><span class="o">.</span><span class="na">lw</span><span class="o">.</span><span class="na">jobseeker</span><span class="o">.</span><span class="na">model</span><span class="o">.</span><span class="na">JobseekerId</span><span class="o">.</span> <span class="mi">7906538</span><span class="o">))</span>
</span><span class='line'><span class="err">#&#39;</span><span class="n">lw</span><span class="o">/</span><span class="n">jsid</span>
</span><span class='line'><span class="n">lw</span><span class="o">&gt;</span> <span class="o">(.</span><span class="na">get</span> <span class="n">jss</span> <span class="n">jsid</span><span class="o">)</span>
</span><span class='line'><span class="err">#</span><span class="o">&lt;</span><span class="n">Jobseeker</span> <span class="n">com</span><span class="o">.</span><span class="na">theladders</span><span class="o">.</span><span class="na">lw</span><span class="o">.</span><span class="na">jobseeker</span><span class="o">.</span><span class="na">model</span><span class="o">.</span><span class="na">Jobseeker</span><span class="err">@</span><span class="mi">5</span><span class="n">dfcb9fd</span><span class="o">&gt;</span>
</span><span class='line'><span class="n">lw</span><span class="o">&gt;</span> <span class="o">(.</span><span class="na">getFirstName</span> <span class="o">(.</span><span class="na">get</span> <span class="n">jss</span> <span class="n">jsid</span><span class="o">))</span>
</span><span class='line'><span class="s">&quot;Daniel&quot;</span>
</span><span class='line'><span class="n">lw</span><span class="o">&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>Pretty straightforward!</p>

<p>The additions to the <code>pom.xml</code> file that enable nREPL support are below:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
</pre></td><td class='code'><pre><code class='xml'><span class='line'><span class="nt">&lt;repositories&gt;</span>
</span><span class='line'>...
</span><span class='line'>    <span class="c">&lt;!-- The following gives us access to Clojure libraries like clojure-complete. --&gt;</span>
</span><span class='line'>    <span class="nt">&lt;repository&gt;</span>
</span><span class='line'>        <span class="nt">&lt;id&gt;</span>clojars.org<span class="nt">&lt;/id&gt;</span>
</span><span class='line'>        <span class="nt">&lt;url&gt;</span>http://clojars.org/repo<span class="nt">&lt;/url&gt;</span>
</span><span class='line'>    <span class="nt">&lt;/repository&gt;</span>
</span><span class='line'>...
</span><span class='line'><span class="nt">&lt;/repositories&gt;</span>
</span><span class='line'><span class="nt">&lt;dependencies&gt;</span>
</span><span class='line'>...
</span><span class='line'>    <span class="c">&lt;!-- Clojure-complete enables tab-completion in the REPL. --&gt;</span>
</span><span class='line'>    <span class="nt">&lt;dependency&gt;</span>
</span><span class='line'>        <span class="nt">&lt;groupId&gt;</span>clojure-complete<span class="nt">&lt;/groupId&gt;</span>
</span><span class='line'>        <span class="nt">&lt;artifactId&gt;</span>clojure-complete<span class="nt">&lt;/artifactId&gt;</span>
</span><span class='line'>        <span class="nt">&lt;version&gt;</span>0.2.3<span class="nt">&lt;/version&gt;</span>
</span><span class='line'>    <span class="nt">&lt;/dependency&gt;</span>
</span><span class='line'>    <span class="nt">&lt;dependency&gt;</span>
</span><span class='line'>        <span class="nt">&lt;groupId&gt;</span>org.clojure<span class="nt">&lt;/groupId&gt;</span>
</span><span class='line'>        <span class="nt">&lt;artifactId&gt;</span>clojure<span class="nt">&lt;/artifactId&gt;</span>
</span><span class='line'>        <span class="nt">&lt;version&gt;</span>1.5.0<span class="nt">&lt;/version&gt;</span>
</span><span class='line'>    <span class="nt">&lt;/dependency&gt;</span>
</span><span class='line'>    <span class="nt">&lt;dependency&gt;</span>
</span><span class='line'>        <span class="nt">&lt;groupId&gt;</span>org.clojure<span class="nt">&lt;/groupId&gt;</span>
</span><span class='line'>        <span class="nt">&lt;artifactId&gt;</span>tools.nrepl<span class="nt">&lt;/artifactId&gt;</span>
</span><span class='line'>        <span class="nt">&lt;version&gt;</span>0.2.2<span class="nt">&lt;/version&gt;</span>
</span><span class='line'>    <span class="nt">&lt;/dependency&gt;</span>
</span><span class='line'>...
</span><span class='line'><span class="nt">&lt;/dependencies&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p>I hope you found this brief write-up useful. I plan on continuing my stumbling exploration, so look for other Clojure-related posts soon.</p>

<p>Join the discussion over on <a href="http://www.reddit.com/r/programming/comments/1bjh69/getting_some_clojure_nrepl_in_a_spring_app/">reddit</a>.</p>

<p><em>April 8th, 2013: Edited in response to suggestions from Laurent Petit.</em></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[TheLadders Onboarding: A Retrospective]]></title>
    <link href="http://dev.theladders.com/2013/03/theladders-on-boarding-a-retrospective/"/>
    <updated>2013-03-28T12:00:00-04:00</updated>
    <id>http://dev.theladders.com/2013/03/theladders-on-boarding-a-retrospective</id>
    <content type="html"><![CDATA[<blockquote><p>Whatever you do or dream you can do – begin it. Boldness has genius and power and magic in it.</p><footer><strong>&#8211;Johann Wolfgang von Goethe</strong></footer></blockquote>


<p><img class="center" src="http://dev.theladders.com/images/dilbert-onboarding.gif" title="'Onboarding'" ></p>

<p>Just before joining TheLadders, I read a post on their dev blog detailing a new onboarding process for incoming engineers. I was very interested in what I&rsquo;d soon learn about its execution. If you&rsquo;ve not had the chance to read through this post, you can find it <a href="http://dev.theladders.com/2013/02/onboarding/">here</a>.</p>

<p>The onboarding process could be broken into two main sections. The first started when we were given a few user stories and were asked to build a corresponding system while following some fairly restrictive <a href="https://github.com/TheLadders/object-calisthenics#the-rules">guidelines</a>. These restrictions challenged us to reevaluate our normal architecting techniques and develop less-than-obvious solutions to this problem. The rules allowed us to implement our solution in the language of our own choosing, and I chose Ruby. The chance to showcase my skills for the first time at a new company excited me.</p>

<p>The second section spanned the next eight days. We dove into Robert Martin’s SOLID principles video series. After watching, learning, and discussing each one of the the five principles, we proceeded to work on code with real-world <a href="https://github.com/TheLadders/solid-exercises">examples</a> of principle violations. Between these videos and exercises, we met with established engineers at TheLadders to discuss the subject matter and our own philosophies on software craftsmanship.</p>

<p>This was the first time I&rsquo;d ever had a formal onboarding. I&rsquo;ve worked for agencies and other product companies, both larger and smaller, before joining TheLadders. This was brand new. I&rsquo;d encourage every software company to seriously consider a process like this for their new engineers. Why? Here are a few reasons:</p>

<ul>
<li>I was allowed access to the top engineers in the company, learning not only about the way software was written at the TheLadders but also where and how the team was looking to grow.</li>
<li>I was able to get a strong feeling for the culture and technologies used as well as prevailing opinions on languages, practices and history. This was also a great opportunity for engineers at TheLadders to learn more about me and get a good picture of my experience and viewpoints.</li>
<li>It also served as a vacation from production code, deadlines and the stress of the everyday that we encounter wherever we go. We were encouraged to stretch our solutions and see where they took us.</li>
</ul>


<p>You may have heard of Rich Hickey’s <a href="http://blip.tv/clojure/hammock-driven-development-4475586">Hammock Driven Development</a>, the idea that spending time away from your code produces unexpected benefits. In the same vein, I&rsquo;ll call what we did for these two weeks Garbage Driven Development. What do I mean when I say garbage? The code written while practicing GDD is never going into production. It’ll never really do anything beyond providing a learning experience and proving a point you’d like to make. You don’t feel pressure to make sure the code fits within a particular context, that it’s easily readable by others, or that it’ll be maintainable going forward. In the end, the code may be quite eloquent or look like garbage, but it doesn&rsquo;t really matter. Try that idea you’ve always thought about but never had time to implement. Buck best practices to see if you can find better ones. When you’re done, it’s garbage. Developers would be well served to routinely make time to focus less on product and more on process.</p>

<p>Most of all, this process woke me up from the monotony of just pushing out code and inspired me to learn and pursue more of my own side projects. I&rsquo;ve wanted to read more, explore new languages, and try new ways of doing things. It was an excellent way to frame the start of a new chapter in my life as an engineer.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Trouble with Google's ImmutableSet]]></title>
    <link href="http://dev.theladders.com/2013/03/trouble-with-googles-immutableset/"/>
    <updated>2013-03-12T06:10:10-04:00</updated>
    <id>http://dev.theladders.com/2013/03/trouble-with-googles-immutableset</id>
    <content type="html"><![CDATA[<p><em>Note: this post was originally published March, 12, 2012.  We&rsquo;re bringing it back from the dead because we love zombies and happen to really like it.</em></p>

<blockquote><p>When the going gets weird, the weird turn pro.</p><footer><strong>&#8211;Hunter S. Thompson</strong></footer></blockquote>


<p>One feature we offer here at TheLadders is the ability for job seekers to &ldquo;follow&rdquo; recruiters, sort of how Twitter users can follow other Twitter users. Recruiters can then broadcast information, like announcements about new openings, to their followers.</p>

<p>At start-up, as well as at set times during the night, caches are loaded that contain mappings between followable recruiters and the job seekers who follow them.
Recently we noticed that the time to create one of these caches was gradually increasing. There&rsquo;s a cache that maps recruiters to job seekers, called the &ldquo;followers cache,&rdquo; which originally took a few minutes to build. We found that the time it was taking to build was steadily increasing over the course of just a few weeks to 30 minutes or more. This wasn&rsquo;t immediately apparent to us because the problem only manifested in production, and not in the day-to-day QA environment used by developers.</p>

<p>Below you can see the times taken to create the cache (in minutes) on four different nodes:</p>

<!-- more -->


<p><a href="http://dev.theladders.com/images/cache-creation-slowdown1.png"><img class="center" src="http://dev.theladders.com/images/cache-creation-slowdown1.png" width="300" height="171" title="Cache creation slowdown chart" ></a></p>

<p>You&rsquo;ll notice the jump on 1/16 from a gradual increase to a more rapid increase. That day a new release went out that included an increase in the size of a related cache- one that contains all job seekers who have permission to follow recruiters. Intersections of that cache with another are then used to build the followers cache.</p>

<p>At first we thought that the slowdown was a function of the increased cache size. Maybe pulling more data from the database to fill up the larger cache was the culprit. But from the logs we found that wasn&rsquo;t the case. In fact, we found that CPU usage was spiking to 100% while the cache was being built. This was in contrast to the QA machines, where CPU did not spike at all during the process. Further analysis determined the code was spending the vast majority of its time calculating the intersections of caches to produce the followers cache. The code is something along these lines:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">recruiterId</span> <span class="o">:</span> <span class="n">followersForRecruitersCache</span><span class="o">.</span><span class="na">keySet</span><span class="o">())</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="n">Set</span> <span class="n">s</span> <span class="o">=</span> <span class="n">Sets</span><span class="o">.</span><span class="na">intersection</span><span class="o">(</span><span class="n">followersForRecruitersCache</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">recruiterId</span><span class="o">),</span>
</span><span class='line'>  <span class="n">canFollowRecruitersCache</span><span class="o">);</span>
</span><span class='line'>  <span class="n">followersCache</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">recruiterId</span><span class="o">,</span> <span class="k">new</span> <span class="n">HashSet</span><span class="o">(</span><span class="n">s</span><span class="o">));</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>(The method Set.intersection() is from Google Collections.) So was the problem that we had so much more data in production than in our QA environment, and many more iterations were being spent calculating set intersections? We also found another interesting bit of information. The following code, used to build the canFollowRecruitersCache, a simple set of integer IDs, was also taking longer in production than in QA:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="n">ImmutableSet</span><span class="o">.</span><span class="na">Builder</span> <span class="n">builder</span> <span class="o">=</span> <span class="n">ImmutableSet</span><span class="o">.</span><span class="na">builder</span><span class="o">();</span>
</span><span class='line'><span class="k">for</span> <span class="o">(</span><span class="n">Integer</span> <span class="n">id</span> <span class="o">:</span> <span class="n">userPrivilegeSystem</span><span class="o">.</span><span class="na">getAllJobseekersWhoCanFollowRecruiters</span><span class="o">())</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="n">builder</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">id</span><span class="o">);</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'><span class="n">canFollowRecruitersCache</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">build</span><span class="o">();</span>
</span></code></pre></td></tr></table></div></figure>


<p>Actually, all the slowness was in the call to ImmutableSet.Builder.build(), also from Google Collections. In fact, we have a larger canFollowRecruitersCache in QA than in production due to the amount of test data in the QA database. So the problem must be in the data itself and its interaction with ImmutableSet. (For the source code for the ImmutableSet, see <a href="http://code.google.com/p/google-collections/source/browse/trunk/src/com/google/common/collect/ImmutableSet.java">http://code.google.com/p/google-collections/source/browse/trunk/src/com/google/common/collect/ImmutableSet.java</a>.)</p>

<p>Taking a closer look, the distribution of IDs in canFollowRecruitersCache in QA looks like this (X-axis is the array index, Y-axis is the ID):</p>

<p><a href="http://dev.theladders.com/images/canFollowRecruitersCache-ID-distribution-in-QA.png"><img class="center" src="http://dev.theladders.com/images/canFollowRecruitersCache-ID-distribution-in-QA.png" width="300" height="176" title="&#34;canFollowRecruitersCache ID Distribution in QA&#34;" alt="&#34;canFollowRecruitersCache ID Distribution in QA&#34;"></a></p>

<p>The standard deviation of this set of IDs is 12,266,077.</p>

<p>The distribution of IDs in canFollowRecruitersCache in production looks like this:</p>

<p><a href="http://dev.theladders.com/images/canFollowRecruitersCache-ID-distribution-in-production.png"><img class="center" src="http://dev.theladders.com/images/canFollowRecruitersCache-ID-distribution-in-production.png" width="300" height="173" title="&#34;canFollowRecruiterCache ID distribution in production&#34;" alt="&#34;canFollowRecruiterCache ID distribution in production&#34;"></a></p>

<p>The standard deviation of this set of IDs is 642,039.</p>

<p>Opening up the code in ImmutableSet, we find that it uses a linear array of buckets, and when a collision occurs it probes each bucket until it finds an empty one, and puts the element there. It uses a secondary hash function, called smear(), to index into the array. Presumably this is used to augment whatever hash is already supplied by the element to be inserted. Unfortunately this isn&rsquo;t enough to overcome the distribution of IDs in our production data. Since Java a Integer&rsquo;s hash code is just its value, the efficiency of the ImmutableSet ends up being subject to the distribution of the IDs. The number of collisions that occur with production data when building the set is a huge 3,217,789,124, but with our somewhat larger QA dataset, only 3,199,502. If we read in the same production data as Strings instead of Integers, however, the number of collisions drops dramatically, and production performance is equivalent to QA&rsquo;s.</p>

<p>Both our QA and production data cause the same size of array of buckets of 1,048,576 to be used internally by ImmutableSet. Rendering this array into a 1024 x 1024 square, and color-coding by number of collisions at that index, we can have a better idea of what&rsquo;s happening.</p>

<p>The colors in the graphics below have the following meaning:</p>

<table style="border:1px solid black;border-collapse:collapse;margin:auto;">
  <tbody>
    <tr>
      <th style="border:2px solid black;padding:10px;">collision count</th>
      <th style="border:2px solid black;padding:10px;text-align:right">color</th>
    </tr>
    <tr>  
      <td style="border:1px solid black;padding:10px;">0 (no element)</td>
      <td style="border:1px solid black;padding:10px;text-align:right">white</td>
    </tr>
    <tr>
      <td style="border:1px solid black;padding:10px;">0 (has element)</td>
      <td style="border:1px solid black;padding:10px;text-align:right">black</td>
    </tr>
    <tr>
      <td style="border:1px solid black;padding:10px;">&lt; 10</td>
      <td style="border:1px solid black;padding:10px;text-align:right">magenta</td>
    </tr>
    <tr>
      <td style="border:1px solid black;padding:10px;">&lt; 100</td>
      <td style="border:1px solid black;padding:10px;text-align:right">blue</td>
    </tr>
    <tr>
      <td style="border:1px solid black;padding:10px;">&lt; 1000</td>
      <td style="border:1px solid black;padding:10px;text-align:right">green</td>
    </tr>
    <tr>
      <td style="border:1px solid black;padding:10px;">&lt; 10000</td>
      <td style="border:1px solid black;padding:10px;text-align:right">green</td>
    </tr>
    <tr>
      <td style="border:1px solid black;padding:10px;">&lt; 25000</td>
      <td style="border:1px solid black;padding:10px;text-align:right">orange</td>
    </tr>
    <tr>
      <td style="border:1px solid black;padding:10px;">25000+</td>
      <td style="border:1px solid black;padding:10px;text-align:right">red</td>
    </tr>
  </tbody>
</table>


<p>Thus the array produced for QA data looks like this:</p>

<p><a href="http://dev.theladders.com/images/ImmutableSet-collisions-in-QA.png"><img class="center" src="http://dev.theladders.com/images/ImmutableSet-collisions-in-QA.png" width="300" height="300" title="&#34;ImmutableSet collisions in QA&#34;" alt="&#34;ImmutableSet collisions in QA&#34;"></a></p>

<p>Whereas the array produced for production data looks like this:</p>

<p><a href="http://dev.theladders.com/images/ImmutableSet-collisions-in-production.png"><img class="center" src="http://dev.theladders.com/images/ImmutableSet-collisions-in-production.png" width="300" height="300" title="&#34;ImmutableSet collisions in production&#34;" alt="&#34;ImmutableSet collisions in production&#34;"></a></p>

<p>The same algorithm is used to implement the set&#8217;s contains() method, so operations such as intersections are also extremely slow with this data. A simple change to Java&rsquo;s built-in HashSet fixes the problem.</p>

<p><a href="http://www.reddit.com/r/programming/comments/1a5by8/trouble_wi%20%20%20%20th_googles_immutableset/">Join the discussion on reddit</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Eric Evans Is Coming to TheLadders]]></title>
    <link href="http://dev.theladders.com/2013/03/eric-evans-is-coming-to-theladders/"/>
    <updated>2013-03-08T10:09:00-05:00</updated>
    <id>http://dev.theladders.com/2013/03/eric-evans-is-coming-to-theladders</id>
    <content type="html"><![CDATA[<blockquote><p>&#8230; when the Man comes around.</p><footer><strong>&#8211;Johnny Cash</strong></footer></blockquote>


<p>We are happy to announce that on April 10th, TheLadders will be hosting
<a href="http://domainlanguage.com/about/">Eric Evans</a>, the codifier of
<a href="http://domainlanguage.com/ddd/">Domain Driven Design</a>, for the New
York City DDD Meetup group. Engineers at TheLadders have long been
following developments in the DDD community. Some of
<a href="http://dev.theladders.com/ourteam/danielwislocki/">us</a> have already attended the DDD immersion course,
<a href="http://dev.theladders.com/ourteam/kylewinter/">more</a> of <a href="http://dev.theladders.com/ourteam/mattjankowski/">us</a> will be attending it shortly, and
our developers are regular participants in the <a href="http://www.dddnyc.org/">NYC DDD Meetup</a>.</p>

<p>For those of you who might be unfamiliar with DDD, its goal is the
design of software that creates business value. DDD guides technical
and domain experts to collaboratively create a mental model of the
central business concepts. This model is then used to drive
design. This sounds like common sense: developers work with people who
know about the business to make software that works to solve its
problems. What could be more straightforward?  And yet, the complexity
inherent in the business world and software engineering make it
difficult to create software that is supple and adaptable to changing
needs. DDD provides a framework for thinking about, creating, and
communicating mental models. And that requires the learning of new
skills and ideas. It isn’t an easy process, but we have found that it
is rewarding for both the business and developers.</p>

<h1>A Little of Our Experience with DDD</h1>

<p>While creating our new website, we’ve tried to remain focused on
“communicating mental models”. One way this can be done is through
DDD’s “Intention-Revealing Interfaces”: interfaces that clearly
communicate the contract between the domain model and its
users. Judging from the vast amount of code out there, this simple
idea is counterintuitive. Our own legacy codebase is no exception. For
example:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Subscription</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="n">AutoRenewFlags</span> <span class="nf">getAutoRenewFlag</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">autoRenewFlag</span><span class="o">;</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">setAutoRenewFlag</span><span class="o">(</span><span class="n">AutoRenewFlags</span> <span class="n">autoRenewFlag</span><span class="o">)</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">this</span><span class="o">.</span><span class="na">autoRenewFlag</span> <span class="o">=</span> <span class="n">autoRenewFlag</span><span class="o">;</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getUnsubscribeReason</span><span class="o">()</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">return</span> <span class="n">unsubscribeReason</span><span class="o">;</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'>
</span><span class='line'>  <span class="kd">public</span> <span class="kt">void</span> <span class="nf">setUnsubscribeReason</span><span class="o">(</span><span class="kt">int</span> <span class="n">unsubscribeReason</span><span class="o">)</span>
</span><span class='line'>  <span class="o">{</span>
</span><span class='line'>    <span class="k">this</span><span class="o">.</span><span class="na">unsubscribeReason</span> <span class="o">=</span> <span class="n">unsubscribeReason</span><span class="o">;</span>
</span><span class='line'>  <span class="o">}</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>This is an example of what in DDD is called an “anemic domain
model”. The Subscription class is a key part of the domain, yet not
only does it communicate nothing to the user about its usage, it in
fact does nothing. Values can be retrieved and set, but what then? How
do I actually unsubscribe someone? What prevents me from using any
arbitrary integer for the “unsubscribe reason”? What about changing
auto-renewal to null? There is no explicit contract here, and the user
of this object is given no guidance or constraints. By contrast, we’ve
worked hard in our newer code to avoid creating anemic domain models
and instead create objects that are not only useful, but clear in
their usage. Here is another example, this time from the new
Subscription class:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">Subscription</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>   <span class="o">...</span>
</span><span class='line'>
</span><span class='line'>   <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Subscription</span> <span class="nf">withAutoRenewOn</span><span class="o">();</span>
</span><span class='line'>
</span><span class='line'>   <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">Subscription</span> <span class="nf">withAutoRenewOff</span><span class="o">();</span>
</span><span class='line'>
</span><span class='line'>   <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">CanceledSubscription</span> <span class="nf">unsubscribe</span><span class="o">(</span><span class="n">UnsubscribeReason</span> <span class="n">unsubscribeReason</span><span class="o">);</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>In this case we’ve attempted to make the contract clear. Calling the
unsubscribe method actually unsubscribes the customer, returning a
CanceledSubscription. The “unsubscribe reason” is now strongly typed,
clearly communicating which values are allowed. Auto-renewal changes
yield a new immutable Subscription object, and can be chained together
with other &ldquo;with*&rdquo; methods to produce new subscriptions with appropriate
settings.</p>

<p>Another example of an anemic domain model, lacking an
intention-revealing interface, comes from the old Payment class in our
legacy codebase:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="nf">Payment</span><span class="o">(</span><span class="kt">int</span> <span class="n">paymentId</span><span class="o">,</span>
</span><span class='line'>               <span class="kt">int</span> <span class="n">paymentActionType</span><span class="o">,</span>
</span><span class='line'>               <span class="n">BigDecimal</span> <span class="n">amount</span><span class="o">,</span>
</span><span class='line'>               <span class="n">String</span> <span class="n">approvalCode</span><span class="o">,</span>
</span><span class='line'>               <span class="n">String</span> <span class="n">transactionReference</span><span class="o">,</span>
</span><span class='line'>               <span class="kt">int</span> <span class="n">creditCardId</span><span class="o">)</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="k">this</span><span class="o">.</span><span class="na">paymentId</span> <span class="o">=</span> <span class="n">paymentId</span><span class="o">;</span>
</span><span class='line'>  <span class="k">this</span><span class="o">.</span><span class="na">paymentActionType</span> <span class="o">=</span> <span class="n">paymentActionType</span><span class="o">;</span>
</span><span class='line'>  <span class="k">this</span><span class="o">.</span><span class="na">amount</span> <span class="o">=</span> <span class="n">amount</span><span class="o">;</span>
</span><span class='line'>  <span class="k">this</span><span class="o">.</span><span class="na">approvalCode</span> <span class="o">=</span> <span class="n">approvalCode</span><span class="o">;</span>
</span><span class='line'>  <span class="k">this</span><span class="o">.</span><span class="na">creditCardId</span> <span class="o">=</span> <span class="n">creditCardId</span><span class="o">;</span>
</span><span class='line'>  <span class="k">this</span><span class="o">.</span><span class="na">transactionReference</span> <span class="o">=</span> <span class="n">transactionReference</span><span class="o">;</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>This class contains only the constructor you see here, and getters for
each of the fields created. It turns out that it’s used as a parameter
object, and nothing more. There’s no guidance as to it’s valid
construction, and no hint about its usage. Yet obviously handling
subscription payments is an important part of the subscription domain,
and we want the contracts around the model of that domain clear and
easy to understand. Here’s our attempt at accomplishing those goals in
the new codebase:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="nf">Payment</span><span class="o">(</span><span class="n">PaymentAmount</span> <span class="n">paymentAmount</span><span class="o">,</span>
</span><span class='line'>               <span class="n">JobseekerId</span> <span class="n">jobseekerId</span><span class="o">,</span>
</span><span class='line'>               <span class="n">PaymentActionType</span> <span class="n">paymentActionType</span><span class="o">,</span>
</span><span class='line'>               <span class="n">ReportGroup</span> <span class="n">reportGroup</span><span class="o">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>The first difference is actually one that’s not visible from the code
itself. As we’ve developed the new site, we’ve agreed that all
arguments to methods or constructors are required; no passing of
nulls. If an argument is truly optional, a separate method or
constructor is added that excludes it. We want our code to be
confident, and this contract means that methods and constructors “say
what they mean and mean what they say”. In the legacy codebase, nulls
were passed liberally to and fro, and you could never be quite sure
which arguments were truly required.</p>

<p>Another change is the promotion of this class from a mere parameter
object to an active participant in the domain model:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='java'><span class='line'><span class="kd">public</span> <span class="n">Receipt</span> <span class="nf">makeWith</span><span class="o">(</span><span class="n">PaymentMethod</span> <span class="n">method</span><span class="o">)</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="n">method</span><span class="o">.</span><span class="na">make</span><span class="o">(</span><span class="k">this</span><span class="o">);</span>
</span><span class='line'>  <span class="k">return</span> <span class="k">new</span> <span class="nf">Receipt</span><span class="o">(</span><span class="n">paymentAmount</span><span class="o">.</span><span class="na">asAmountPaid</span><span class="o">());</span>
</span><span class='line'><span class="o">}</span>
</span><span class='line'>
</span><span class='line'><span class="kd">public</span> <span class="kt">void</span> <span class="nf">recordWith</span><span class="o">(</span><span class="n">PaymentRecorder</span> <span class="n">paymentRecorder</span><span class="o">)</span>
</span><span class='line'><span class="o">{</span>
</span><span class='line'>  <span class="n">paymentRecorder</span><span class="o">.</span><span class="na">record</span><span class="o">(</span><span class="n">paymentAmount</span><span class="o">);</span>
</span><span class='line'><span class="o">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>Instead of being passed around to “manager” objects and DAOs, a
Payment object can actually make a payment and create records of
itself. It seems only natural.</p>

<p>There’s much more qto Domain Driven Design than I’ve shown here &mdash;
this post barely scratches the surface. And as a team we know we still
have much farther to go in its mastery, but the time we’ve spent in
study and practice have been well worth the effort so far.</p>

<p><a href="http://www.dddnyc.org/events/80390502/">We hope you’ll join us April 10th</a> for an illuminating evening of
presentation and discussion.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Riders on the Storm: Take a long holiday, Let your children play]]></title>
    <link href="http://dev.theladders.com/2013/03/riders-on-the-storm-take-a-long-holiday-let-your-children-play/"/>
    <updated>2013-03-04T13:52:00-05:00</updated>
    <id>http://dev.theladders.com/2013/03/riders-on-the-storm-take-a-long-holiday-let-your-children-play</id>
    <content type="html"><![CDATA[<p><img class="center" src="http://dev.theladders.com/images/lightning_storm.gif" title="'Lightning Storm'" ></p>

<blockquote><p>It was the age of wisdom, it was the age of foolishness</p><footer><strong>&#8211;Charles Dickens</strong></footer></blockquote>


<hr />

<h1>Introduction</h1>

<p>I’ve decided to split this blog post up into three different sections as we have gone through three different phases with our usage of Storm.  The first describes how we used to use Storm at TheLadders.  This is followed by our “wake up call”, forcing us to the realization that how we had been using Storm was not sufficient.  The “wake up call” has led us to our current state of how we now use Storm at TheLadders. But first for those of you who aren’t familiar with Storm, a quick explanation straight from the horse’s mouth:</p>

<blockquote><p>Storm is a distributed real-time computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing real-time computation. Storm is simple, can be used with any programming language, is used by many companies, and is a lot of fun to use!</p><footer><strong>Nathan Marz</strong> <cite><a href='https://github.com/nathanmarz/storm'>Storm Readme</a></cite></footer></blockquote>


<hr />

<h1>The Past</h1>

<p>We were early users of Storm, starting out with the Storm 0.5.x releases and later upgrading to 0.6.2.  Our Storm cluster was very basic: Nimbus, Zookeeper with 2 Worker nodes;   5 topologies deployed, but only 3 of them really being exercised.  Many of these topologies were for non-critical portions of our application, as such we weren’t paying much attention to the cluster. The topologies we wrote had well-tested individual components; each of the Spouts and Bolts were written with testability in mind.  However we struggled when it came to end-to-end tests for an entire topology.</p>

<p>Other areas of our Storm related pain were:</p>

<ul>
<li>Very limited visibility into the overall health of the Storm cluster.  We lacked any monitoring and had very few metrics regarding our cluster.  We relied a lot on tailing the logs of the worker nodes to see how things were behaving.</li>
<li>We naively configured the topology resources, not really being aware of what resources were being used across the worker nodes.</li>
<li>A majority of our topologies used RabbitMQ as the entry-point and we had a very basic AMQP Spout implementation.  In fact, the initial AMQP Spout increased CPU usage on our RabbitMQ nodes from 4-10% to 40-45% with very little message throughput.</li>
<li>Guaranteed message processing was not always enforced (more due to lack of knowledge on the subject than anything).</li>
</ul>


<p>That list looks bad and one might wonder how we got along at all given those shortcomings.  To be honest, everything just “worked”, which was all we needed at that point.  The combination of Nimbus and Zookeeper did a great job of re-deploying topologies anytime “something” happened.  While we would occasionally open up the Storm web admin to see how things were doing, we really didn’t pay much attention to it; everything just worked.  Even the increase in RabbitMQ CPU usage was not considered overly serious because everything continued to work and was fairly stable.  This behavior continued for about a year or so.</p>

<hr />

<h1>The &ldquo;Wake Up Call&rdquo;</h1>

<p>Then came the day when we needed to deploy a new feature in one of our topologies.  We ran the standard release script to deploy a topology through Nimbus, and … nothing…  After some digging, we found that Nimbus had run out of disk space and our topologies had not been pulling messages off of RabbitMQ for an estimated 3 – 7 days.</p>

<p>In addition, shortly after this initial wake up call, we had some one-off topologies that needed to be run for a 24-hour period.  These topologies required a decent number of resources.  They quickly starved the existing topologies of resources and did a good job of bringing Storm to a screeching halt.  It was like watching a caged death match between all of our topologies that left everyone unconscious on the mat.</p>

<p>If something like the 7-10 day outage can go unnoticed for so long, and if we could starve topologies at the drop of a hat, how could we expect to successfully expand our usage of Storm to more critical portions of the application?</p>

<p>We needed to change, and fast!</p>

<hr />

<h1>The Present</h1>

<p>We immediately started figuring out what we didn’t know about Storm and which bits were the most important for immediate success.  Some members of the development team got together with our operations team and worked out monitoring of cluster components. While Operations beefed up what they could, the development team:</p>

<ul>
<li>Enforced guaranteed message processing in all of our topologies.  This has mainly been done through the use of the BaseBasicBolt, which provides emitting anchored tuples and acking for free. <a href="http://nathanmarz.github.com/storm/doc/backtype/storm/topology/base/BaseBasicBolt.html">(http://nathanmarz.github.com/storm/doc/backtype/storm/topology/base/BaseBasicBolt.html)</a></li>
<li>Refactored our AMQP Spout implementation to subscribe to a queue instead of using individual “gets”.  This has resulted in pre-Storm 0.6.2 levels of CPU usage on our RabbitMQ nodes (below 10% again).</li>
<li>Added an additional three worker nodes with more memory and CPU so we don’t have to constantly worry about resource starvation (although this should always be something to consider really).</li>
<li>Improved our unit testing of configured topologies using the simulated time cluster testing feature in Storm 0.8.1  <a href="https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/TestingApiDemo.java">(https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/TestingApiDemo.java)</a>:</li>
</ul>


<hr />

<h1>The Future</h1>

<p>Okay, so I lied.  There is a fourth phase: the future; where we plan to go with Storm.</p>

<p>We plan on upgrading to Storm 0.8.2 very soon.  It has a much-improved web admin that allows for deployment and rebalancing of topologies on the fly.  We hope this simplifies the process of deploying and rebalancing (if needed) of our topologies.</p>

<p>Upgrade to Storm 0.9.x as soon as possible once released.  We hear good things about this release; mainly the metric collecting which will be a huge win in terms of improving visibility into the streams and flow of tuples between Spouts and Bolts.</p>

<p>Finally, we are plan on expanding our topologies to be more than simple queue-to-spout designs.  We are experimenting with using Storm for scheduled batch processes, hoping to have something in production within the next week.</p>

<p>Hopefully this blog gave you a nice overview of how Storm can be used by a company.  I think one of the take-aways can be how easy Storm is to use.  Storm served us for over a year with very little intervention and minimal knowledge on our part; I believe this speaks volumes of Storm’s ease-of-use and reliability.</p>

<p>Stay tuned for some more detailed technical blogs on some of the things we did to improve our Storm usage.</p>

<p>Join the discussion over at <a href="http://www.reddit.com/r/programming/comments/19noko/how_we_use_twitters_storm_part_one/">reddit</a>.</p>
]]></content>
  </entry>
  
</feed>