Virtual Lab Automation Blog

Dennis Stevenson blogs at Original Thinking and is a Director of Software as a Service (SaaS) for a software company in the healthcare automation space. He has extensive experience working on software engineering and in software labs. You can read his bio here.

He recently posted on “Getting Software Development Out of the Lab” on his blog. It was an interesting post and brought forth the relevance of involving the field in the software development/testing and lab processes. Clearly, Dennis is an expert in this domain and we caught up with him to get his perspective on the Changing Role and Expectations of Software Labs.

This blog has focused on virtual lab automation and lab management tools like VMLogix LabManager – we hope you find this detailed discussion on increasing the role/relevance of software labs in building great software useful.

In summary, Dennis made the following very important points:

• Traditionally, the software lab is a place where rapid prototyping and experimentation was done. In more recent times, the trend has shifted towards using the lab as an isolated sandbox where developers/testers can focus on building great software

• The three most important (technology) requirements of a lab today are – (a) Automation (b) Collaboration between multiple users and (c) Ability to mimic field like conditions in the lab

• The lab is treated as a black box in organizations today – this is an important problem for business – which demands the need for more transparency and communication between the lab and various other business/engineering functions

• The executive leadership in an organization needs to be a stakeholder in the lab – which will help transform the lab as a strategic asset

• Ultimately – the ability to ship great software (e.g., software being requirements compliant) – largely depends on the entire efficiency of the lab experience

[VMLogix] How have you been involved with software labs in your career?

[Dennis] In my experience, historically, the lab has been a very vague concept. Earlier in my career, we had engineering software labs where the real focus was the ability to try new things. And whenever I used the lab it was always in the prototyping and rapid deployment kind of a scenario. Labs for me were a place where the traditional production rules did not apply, where I did not have to follow the same methodologies and I did not have to worry about building production grade software, but where I could very rapidly deploy an idea and determine if that idea was going to make sense or not. I could run a pilot out of the lab very quickly and easily and make a decision about whether or not a given technology had relevance or applicability to the business problems that I was trying to solve or if it was something that I just wanted to let pass by. I had a very successful venture doing Wiki deployment where rather than developing it internally, we were trying different software products to decide which one we wanted to use as an enterprise solution. In that case the lab was the place where I could install software that I could not do in traditional data center environments, play with them and see in a “no-penalty” type of a way.

In other organizations, I’ve seen lab concepts exist though they did not recognize it as such. We had a series of engineers who were trying very hard to maintain an isolated environment where they could develop software according to requirements that were provided to them. In this case they were disconnected from the user experience. We attempted to simulate in the laboratory environment the end customer and the kinds of experience that they would bring to the software. The real driver for the lab was “here are the requirements” – just “write good code”, “be very efficient”, “be productive”. Labs gave us isolation, which meant focus and concentration and hence enabled great progress. We invested enormous amount of effort and time into building processes for creating, testing and certifying software. But none of those agile build, continuous integration or other processes could be used once systems went into a client or field environment. So more recently, I began to wrestle with – how do we use the lab? What is the proper role of a lab? Do we try to support the field experience out of the lab or do we have to do something else in order to be relevant to the field users?

[VMLogix] What are the top technical challenges for a software lab today?

[Dennis] Several things jump to mind, here are the key points:

• The lab environment must be automated. The concept of a lab is destroyed in my opinion if you don’t have automated testing, some sort of automated build, if you don’t have tools that look at the code produced and render some sort of verdict about its suitability, usability or its quality. And it is very important in a lab environment that the developers in a lab environment not be burdened down with a lot of administrative or repetitive tasks. The lab is more productive if it can make this process smoother and enable faster feedback to the developers about whether the code they’ve written works or doesn’t work.

• The other piece to this is the offshored/outsourced relationship in software engineering teams today – the lab absolutely needs to be portable. It is very easy to get in and build a heavy lab, where it is very site specific, where the processes are very heavily and locally customized and where the automation is unique to this particular place or circumstance. This ends up being a limitation and barrier over time, because we are working through the issue of how do we replicate our lab environment for somebody else to participate or collaborate with us. I think collaboration is an ongoing theme that we really have to wrestle with these days in terms of one team working on software, handing it off to another team or integrating things from another team and so on. To the extent that the lab environment were to operate the same, feel the same, and produce similar kinds of results, the stitching together process becomes so much easier. In my mind, the purpose of the lab is to get the efficiency and let people get focused on the real hard problems not the brute force problems.

• The lab cannot be entirely self focused. It is easy to think that the world is a sterile place that I can replicate or create in a lab environment. The reality is that when my software hits the real world and real users with completely different assumptions about what it should do and how it should work, interact with it, have different priorities than what the software developer or product manager think about – there needs to be fast communication loops with the field communicating back to the labs so that the lab can increasingly mimic what the field looks like and also so that software can be rapidly modified as it deals with this new scenario and is worked upon to work in a manner that it is designed to perform.

[VMLogix] What do you think business expects from the lab? And how can you have lab goals aligned with those priorities?

[Dennis] Business expects predictability and quality. By predictability, they need to know: How long software is going to be in the lab? When is it going to be released? And will it be what they expect when it comes out? The risk is that the lab becomes a tremendous black box environment to the rest of the business and that fosters this really difficult tension of “us” and “them” between the business users, software engineers and the lab. Penetrating this and being able to establish the relevancy and maintain cohesion is hard, especially if you are an ISV. It is easy for the business to move off in a direction — that maybe the lab does not understand and the lab is always playing catch up in that scenario… always trying to understand what the expectations are because the business seems to be running away. Software engineering is a discipline and someone does not understand that casually along the way. Attention needs to be given. Methodologies like Agile can help, but it is not a requirement for a lab. Lab as an organization needs to be able to communicate and give confidence to the business that they are getting what they expect / want or what they need. This has been the fundamental tension. Contrast this with the software development that happens in the field. In this case, business is very clear and comfortable with what’s happening – the problem, what fix is happening and the scope of the activity is relatively small. The challenge comes when software needs to be in the lab for 4-5 months, that is long time for the business to say “I don’t understand what you are doing”.

[VMLogix] Do you think labs can be a strategic asset?

[Dennis] Labs can be a black box mode, but at the same time be a strategic asset. Do we think the industry as a whole will migrate there? No, I don’t think so. However, players in the industry may adopt this. But as much as I know about people as software engineering, these tend to be more people kind of management problems, like communication management problems – and that is a lot harder to solve. Tools and technology make this problem easier and can mitigate some challenges that the lab faces in establishing its strategic value. I don’t know if this problem is entirely solvable with technology but is certainly made a lot less daunting with technology.

[VMLogix] Who are the lab stakeholders? And according to you, who should be the lab champion?

[Dennis] The technical leadership is clearly a stakeholder because the lab is the main vehicle by which the technical development happens. In my opinion the other big stakeholder is the product management organization who very often is the key driver of activity into the lab and validator of output out of the lab. Now that is really challenging because as a rule, product management tends to be outwardly focused. And so it is very difficult for a product manager to maintain that dual role (“we understand the market and what the market wants” AND “we understand how the lab is responding to the requirements and how they are creating software”). That is a difficult spot for one organization to be in. In my experience the other stakeholder is the parts of the organization that deal with the software once it goes out into the field. That may be a customer support or client services organization that has to face the slings and arrows of the user base when the software does not behave as it should in particular circumstances. To the extent that those organizations can be stakeholders and actually exercise influence, I think it is a powerful force for the delivery of market relevant functions.

At a higher level, the executive leadership of an organization needs to be considered as a stakeholder because they create the strategic direction that places the lab in a central role or requires the lab to move into a central role. And they need to understand what it takes to harvest value out of that or what it takes to create that value in the first place.

[VMLogix] What are your top wish list capabilities that you want to see labs enabled with?

[Dennis] The development process needs to be well thought out, well understood and embedded into the culture. The big piece that is difficult is the integration of the development and testing of software. And the better that can be done, the better the flow of the software and I think that ultimately the testing and the certification of the software as being requirements compliant – the extent to which you can do that and the speed with which you can do that is very closely hinging on the efficiency of the entire lab experience. You want an independent testing organization, but you don’t want a lot of organizational overhead or process overhead between the development and the testing of the software. Testing can go such a long way too – you can get into usability testing, feature function testing, integration testing, system testing, acceptance testing, performance testing, load testing etc. When you look at the number of available testing that can be performed on software (all of which are incredibly relevant as software starts getting deployed and people begin to depend upon it), having the ability in the lab to do all of those things quickly and easily is incredibly important. Honestly, it is so easy to say “we’re not setup to do load testing…we’re going to attempt to do it, but not do a good job at it, so we’re not going to know how the software is going to perform under a real load, because we never put it under the load because we did not have the means/mechanisms to do it”. Also, when we do find issues how do we feed that back to our development organization capturing ways in which software was unsatisfactory and what we would like to see development change, so that they can cycle back again. From requirements concept to installation – how do we get line of sight that what we are producing as output of this lab is conforming to the original requirements that came in. Historically, development hated requirements traceability. We need some means of determining what we are delivering meets what we were asked to deliver.