Sneaking Around

4 04 2014

In recent times I had the pleasure of working with a particularly driven woman whose goal was to transition from scripter to ‘code-compent’ engineer. Together we structured a sprint so that she was accountable for delivering the automated functional tests for a story. Lo and behold the product feature was delivered to estimate with high quality, replete with functional BDD tests and Java bindings.

I was gobsmacked. Given the individual, I guess I shouldn’t have been all that shocked. She confided to me:

I’ve been sneaking around looking at the developers’ code

This wonderful engineer was the second I have witnessed who ‘crossed the quality chasm‘.

In a completely unexpected way, I recently witnessed a similar quality chasm crossing. Well designed code landed in my chat window for a project which uses BDD driven functional tests (using Selenium) with Java bindings. The project was written of course – soup to nuts – by a quality engineer looking to expand his scope into engineering and product.

Not only did this exemplary leader teach his quality team Java through Stanford online courses, he put the quality team, engineering and product in the same boat. He crossed the quality chasm and led the entire team to fully automated functional testing. He identified the collaboration and quality ‘true north’ and positioned his team to execute.

I’ve witnessed this halo effect multiple times. When you put your quality team in the code you create a conduit for mutual respect and collaboration between quality, engineering and product. You create a force multiplier for quality and ultimately product feature velocity.





Question Everything

28 06 2013

I personally love those moments of complete skepticism when a piece of code seems as though it should work and you know deep down inside that it won’t. My most recent find:

synchronized(Integer.valueOf(integer)) {
    // some code which should execute exclusively 
    // for two threads with equal integer values
}

It smacked of the author not fully understanding what’s happening behind the scenes. It’s also true that the code would not provide the mutual exclusion guarantees we were looking for.

Clearly the engineer was eager to move on, and didn’t really seek to understand what the Java Virtual Machine would do with his code. Martin Thompson recently mentioned a Jackie Stewart quote which has applicability here:

To get the best out of any car, you have to have a sympathy for how it actually works and then you can work in harmony with itJackie Stewart

Gory Details

The following post is an excellent read on why you should use Integer.valueOf(…). Further, there are some pretty interesting comments to Alex’s post. In particular the question about strong references generated by Integer.ValueOf(…) and the implications of use in WeakHashMaps was an interesting one.

The most interesting part about the original code block above, is that in some cases, it will actually provide the mutual exclusion guarantees the author was looking for. Interestingly the exclusion guarantees are non-deterministic and are based on the JVM’s implementation of Integer.valueOf(…). Most JVMs implement this as an Integer cache using an array of Integers. It’s important to note that the Integer cache is bounded by i >= -128 && i <= 127.

A simple test helped us prove that the code was non-deterministic. We proved that threads with integer values greater than -128 (and less than 127) did not behave in a mutually exclusive fashion. The solution we chose is a map of Integer keys to Lock values, like so:

Map o’ Locks

public class ConcurrentThingyManager implements ThingyManager {

    // Provides guarantees for access by multiple threads
    // in particular the 'putIfAbsent' atomic operation
    private final ConcurrentHashMap<Integer, Lock> 
                thingyIdLockMap = new ConcurrentHashMap();

    /**
     * Obtain a lock by thingy ID required for thingy access.
     *
     * @param thingyId The thingy ID for which concurrent 
     *                 access must be prevented
     * @return Lock    The lock associated with the access of 
     *                 the specified thingy ID
     */
    @Override
    public Lock getThingySchedulingLock(int thingyId) {
        thingyIdLockMap.putIfAbsent(Integer.valueOf(thingyId), 
                    new ReentrantLock(true));
        return thingyIdLockMap.get(Integer.valueOf(thingyId));
    }
}

Gorgeous Guava

Guava’s Striped also looks really interesting. After reading the documentation, it provides the requisite functionality while using fewer locks and less memory. It also means we have a nicer abstraction than a Map of locks to work with (snippets from the Jython console):

>>>> from com.google.common.util.concurrent import Striped
>>>> striped = Striped.lock(32)
>>>> lock1 = striped.get(13)
>>>> lock2 = striped.get(13)

>>>> lock1.isLocked()
False
>>>> lock2.isLocked()
False

>>>> lock1.lock()
>>>> lock1.isLocked()
True
>>>> lock2.isLocked()
True

Career Advice

Question everything. Turns out, the answers are right there in front of you.





Crossing the Quality Chasm

18 06 2013

Our traditional challenges with software quality assurance;

  • Culture – think intra or inter team hand-offs, eg: “it’s ready for QA”
  • Manual – think manual, costly, and inconsistent testing
  • Deployment – think manual gates in our software deployment pipeline

I propose that we can fix these problems in one fell swoop – quality engineering (as opposed to quality assurance). Is it even possible to make this pivot to quality engineering? Is it possible to cross the quality chasm?

Halo Effect

One thing I’ve observed which has been a huge enabler for quality teams looking to make the jump: Behavior Driven Development or BDD.

When we started out with BDD, we had some growing pains. Specifically, we selected different languages for our BDD tests and application code. We’ve recently transitioned our BDD tests to the same language as our application code, and it’s no secret that I believe speaking the same language is important.

But forget about the language – why is BDD a complete game changer?

  • Living Breathing Artifact – BDD scenarios live alongside the application code. That’s actually inside the application code for white-box tests, and close by for black-box tests. They’re committed, versioned and refactored, just like application code is.
  • Plain Old English – BDD scenarios are written in plain old English (or the written language of your choice). This means that scenarios can be distributed widely as a means of communicating current or planned system behavior.
  • Joint Ownership – BDD scenarios are a shared artifact. In fact, I’ve seen application engineers, quality and product owners committing to the same plain old English BDD feature files!
  • Reporting – Instead of the time consuming list of features compiled into a QA ‘acceptance’ email, if the build passes the code ships. Jenkins also happens to have great plugins for test results from cucumber-jvm which are again easily consumed by a wide audience. These test results metrics are archived with each build (just like metrics for code coverage or cyclomatic complexity) and tend to obviate the need for QA ‘acceptance’ in general.

If I had to put my finger on it, the two things which have changed the quality game completely are Shared Ownership and Plain Old English. And, I can truly say that in my teams, quality is now a shared responsibility.

And that halo effect? Once your quality engineers are poking around in the BDD feature files, if they don’t start writing the code for the BDD step definitions too, then maybe you just plain hired poorly. In reality, after a pivot to BDD, I’ve seen quality engineers blurring the lines between quality and application engineering by writing application code as well.

Lines Redrawn

Those traditional challenges of quality assurance – culture, lack of automation, lack of integration into the deployment pipeline – have typically meant that the role of quality architect is somewhat of a pipe dream. BDD has changed all of that. A quality engineer whom I have the pleasure of working with, recently redrew the lines for her role in a single email. It contained;

  • An architectural diagram for the black box testing components for a system we were building
  • Solicitation for code review, providing a link to the Git repository for the entire BDD code-base she had just written from scratch
  • Build pipeline integration details for how the black-box tests were added as a build success gate

Pivot

BDD isn’t just a way to automate your tests. It’s a way to pivot your team to a shared responsibility for quality and cross the quality chasm. It’s a way to allow your quality team to redraw the lines for their careers.





Intelligent Investment

14 06 2013

Recently our business team identified a huge opportunity in an emerging market. We had a working prototype, with some known shortcomings, and we wanted to scale. We thought there could be a lot of opportunity and we wanted to capitalize quickly.

So we finished up the design for the next iteration of our proof of concept, and started implementing. With the pressure to capitalize quickly, you wouldn’t believe the complexities we found;

  • Incomplete APIs – some part of a single operation available via the API, another part available via the user interface
  • Missing APIs – some operations completely missing from the API
  • Brand new product – very few requirements
  • Large volume of data

March to Knowledge

One of the strategies we used to improve our chance of success was pretty simple. Collaborate. Closely. On numerous occasions we marched across the office to collaborate directly with our end users. Some highly technical operators even helped with Python scripts (for low risk areas of the product). It got crazy.

So as we dug a level deeper, we discovered more and more requirements. We realized that we had to crawl tens of millions of web pages to get alignment data.

So with all of these marks against success, how could we possibly secure victory?

RAD

We released early and often. We released in really small increments, developed rapid prototypes which we deployed all the way to production. We automated all of our quality assurance. We didn’t sacrifice quality. We collaborated closely with our end users and product owners.

We got comfortable with not knowing everything. We knew we could evolve our design and architecture.

The graph below is a view of our revenue performance. The area covered by the ticks is about a 4-5 week period. The ticks in the graph represent our major releases to production. In reality there were many more minor releases too, which are omitted for clarity;

The impact of our rapid prototyping and release strategy on revenue

The impact of our rapid prototype and frequent release strategy on revenue

Where to Invest?

Our rapid application development approach delivered results early. It’s true we didn’t sacrifice on quality or delivery. It’s also true that we didn’t sacrifice on design or architecture either. We designed continuously, evolving services out of rapid prototypes as we went.

As our rapid prototypes drove more and more revenue, the business team weren’t guessing about where to invest. They knew.





What Language Do You Speak?

16 05 2013

Not so long ago, our quality assurance team began cutting their teeth on implementing automated tests. This was a great step forward – people with ‘quality’ in their titles automating our quality assurance critical path. We selected Python for the implementation language, though it’s tough to remember why. We probably discussed;

    • Accessibility of the language
    • Support in terms of libraries and community
    • Fit for the problem at hand

    For clarification, our application engineering team almost exclusively writes applications in Java. For further clarification, I love Python.

    So, a team of non-coders, starts writing automation code, in a language in which the entire team has zilch-zero-squat experience? Does that sound crazy to you? Crazy might be a stretch, but it was definitely far too complex;

    Decorators

    Decorators were just one nuance of the Python language which the quality engineering team were left to contemplate, research and identify how they might benefit their code design. The reality here is that the automation engineering team were inexperienced (they weren’t coders) and they had no support network of experienced python coders.

    Inefficient Tests

    The testing approach lacked the requisite amount of up-front design. Quality engineers were not producing detailed testing architecture blueprints. It’s probably not something large that was required. A small amount of up-front test design would’ve likely produced efficient automated test execution times. Instead, our entire test suite took somewhere in the neighborhood of 2 hours to execute.

    Critical Path

    Quality engineering development efforts were not considered part of the user story critical path. Stories were often accepted and promoted to production while automation efforts languished.

    Build Gate

    The point of the automated tests after all, was to automate the black box tests and fail the build. So after all of our efforts to automate the tests, when the continuous integration build for the application completed successfully, why did the automated tests sit mute, automating the testing of nothing?

    How not to Pivot

    So for a quality assurance team looking to pivot to quality engineering, was Python the right choice? Alongside an application engineering team who wrote >90% Java? In a word, no.

    We completely isolated the quality engineering team from the application engineering team. For an application engineer to contribute to the automated quality suite required a language context switch at a minimum. Granted, some other inefficiencies such as test design, critical path, and the build gate, were unrelated to language. Though with our quality engineering already on an language island, I’d argue that design, critical path and build gates were more challenging issues to fix.

    At the time, the industry had already started its pivot to embed quality assurance within engineering teams. Conversely, here’s my estimation of what we accomplished around the same time. Size of the blocks represents impact, speed, agility (bigger blocks are better). Integration of the blocks represents cohesion (more integrated is better):

    Impact and Cohesion of Quality Automation

    What language does your quality engineering team ‘speak’? Chances are, if it’s not the same language as your application engineering team, you could benefit from efficiencies.





Does Your Team Have Great Answers?

16 05 2013

It’s annual review time for my team. With a pivot to making a conversation out of the review session, some really interesting things happened. I’ve been asking a lot of questions and I’ve been getting a lot of really great answers. The results had me on the edge of my seat.

Did your team have good answers to the questions you had for them at review time? If they had great answers, then you’re doing a great job. If they didn’t, then you’re not.

Rands’ excellent post, ‘No Surprises‘ has been my informal guide for reviews this year. I took his, ‘A Review is a Conversation’ to heart, and debated with my team on where they’re headed, their areas of opportunity and the high leverage projects they should be working on.

Funniest thing is, if you’re making your people great, they should have great answers about their careers. And If you’re not asking questions of them at review time, perhaps you’re afraid the answers will fall short.

This year, my entire team had great answers.





Board Room to Back Office

5 11 2010

How quick are you from board room to back office?

Through the four stories I shared in my QCon presentation yesterday, the broad accomplishment of Shopzilla’s greenfield site redesign, was incredible business agility. It’s true that we’re very fast from board room to back office.

I’ll extrapolate a little here on how our continued investment in our engineering and deployment infrastructure has been the biggest enabler for our agility.

Archetypes

Perhaps our biggest enabler has been our Maven archetype. In fact, we no longer have only one archetype, but a myriad archetypes for different purposes; web-services, spring-batch, Java-based command-line jobs.

Our archetypes allow our engineering teams to focus more immediately on solving business problems. Engineers of all levels of seniority, focus less on boilerplate, project structure, and deployment concerns. Our Maven archetype encapsulates;

  • JMX MBean exposure using Spring
  • Metrics exposure, through JMX using JAMon
  • Sample Spring configuration and wiring
  • Patterns forcing separation of configuration from application binaries

Business Agility, Cross Pollination, DevOps

As a result project structures look mostly the same from team to team, making cross pollination easy. Importantly too, our applications are homogeneous from a deployment / DevOps perspective. Distilled, performance of applications deployed to our any environment, can be visualized on our internal dashboards (permanently) in a matter of minutes.

I’ve written about this topic previously on the Shopzilla Tech Blog. The hindsight here is particularly interesting, in seeing how much the archetype approach has allowed agility from a business perspective. The proof: beso.com and more recently, tada.com.








Follow

Get every new post delivered to your Inbox.