A greenfield project is a brand new project that is not subject to any legacy requirements and importantly does not deal with legacy code. The opportunity to work on a greenfield project does not come along very often and is generally viewed as a real opportunity by developers. In the excitement, it is tempting to focus on the new design and the new code.
Beyond the technology decisions of the new project, there are things to consider that will have an under appreciated impact on the project. These issues are the subject of this blog posting. If you ignore these issues or make the wrong decision, changing you mind could be very expensive.
Unit Testing and Code Coverage. You should decide before any code is written whether you are going to establish unit tests and track code coverage in your build. Anyone who has a legacy code base and wants to go back and unit test will tell you that there isn't a business in the world that will pay your developers to go back and do this - it is cost prohibitive. Not only that, the existing code base can affect your overall coverage numbers should you start trying to cover you new code and ignore your old code. (I'll actually discussion how to manage this in a forthcoming posting).
Coding Conventions. If your project is to adhere to coding conventions, they should be established early on. Depending on how detailed the conventions are and whether there is any tool support for transforming code, this could be expensive to retrofit once the project has a significant code base. The difficulties are compounded because code that is written to be testable is different (usually better designed) than code that is not, so you wouldn't be able to unit test legacy code without significant refactoring costs.
Language-specific Best Practices. Well beyond issues of formatting, some languages have best practices conventions that should be established early on. An example is "const correctness" in C++. Anyone who has tried to go back and retrofit "const correctness" in a code base knows how invasive this change can be. An effort like this can end up touching almost all of the classes in your project. Of particular insidiousness is that a casual code inspection often does not make the scope of such changes apparent.
Internationalization. On our projects, we always consider internationalization when we write our code, even if it is not an explicit requirement for the project. There are tools that can help sift through source code and refactor the string literals out into a separate module. This is helpful and better than nothing, but it is not as neat as doing this as you write your code. Additionally there are often considerations that make this one-size-fits-all approach suboptimal. For example, in our projects, we allow the logging (and debug information), which frequently is done on the serverside, to remain in English. We use the locale associated with the current session to determine what language the user interface shows to the end user.
Code Rules. There is a lot of tool support these days to analyze source code and determine its adherence to coding rules. A popular tool like this is FindBugs (part of Sonar). Use of this tool should be done very early on. Anyone who has turned this tool onto an existing code base knows the potentially thousands of issues/violations that can be raised. Although the rules used to generate violations can be tuned, it is far better to do this with a very small code base than a large. We turned this tool onto a large code base (say 500K LOC) and had to limit our efforts to address only critical and major issues because of the large volume. So do yourself a favor and activate this tool on all check-ins starting on day zero.
Database Content. This issue is subtle. Our application uses a backend database and to support our functional tests, we stand up test data. The data was captured as a dump. So over time, developers added test data, got their tests going, then took a new dump. The problem with this is that very quickly, nobody could explain why a particular piece of data was in the database. It had become a big tarball of opaque, but required data. Avoid this situation at all costs. You should manage your data just like you do your source code. Each addition to the test data should be identifiable and tracked. Ideally you could stand up a new test database based solely on these individual descriptions. (I'll discuss this more in a forthcoming blog entry).
Technical Debt. The urgency in establishing identification of technical debt is not as high as the other issues identified here. Technical debt is incremental and, for the most part, its cost is incremental. Delaying dealing with technical debt is a mistake for many reasons. Ideally it would be identified, tracked, and continually addressed from day zero. Doing so establishes the expectation on your development team that taking shortcuts is not free of cost and is not invisible. It will cost you in the future at some point.
Hopefully these issues will give you something to consider at your next greenfield opportunity. If you have any other ideas about issues that should be in almost all projects and are hard to retrofit, please leave comments.
Sunday, January 18, 2015
Saturday, November 22, 2014
Code Coverage Theater
From the title of this entry, you might think that this is going to be a screed that is going to rain down hard on the code coverage parade. Instead it will be some tempering and pragmatic advice taken from the experiences of a mature agile team.
If there is any bit of heresy, this is it - code coverage has no intrinsic value. Its value comes only from showing where there is untested code. All of the value in what we do in this aspect of development is in the quality of the unit tests. Maybe leaving code untested is a deliberate calculation, but maybe it was a mistake. As developers, we are pulled in multiple directions all the time. It can take some effort to get back into context and you can use your uncovered code to help you recover.
The single best way to use code coverage is to break the build. As Agilists, we all are practicing continuous builds, right? I mean a new build goes off with nearly every code commit, right? That frequency is exactly what you should use for running code coverage checks. When the build breaks due to a fall in coverage, you get the advantage of causality. The latest check-in probably broke the build. (Your thoughtful coworkers never check code into a broken build.) It is very important that all developers should be able to run the same coverage checks locally on their own machines that are done on the continuous build server. Thus a broken build due to coverage checks should never be a surprise to anyone.
So if we want to have minimum coverage limits that must be met or the build breaks, what is an appropriate limit? Unfortunately some misguided management mandate that a 100% line coverage be maintained. In my opinion, this is a very poor way to spend your limited resources.
First, not all code coverages are created equal. A 100% line coverage limit does not address the differences between line and branch coverage. Package and class coverage can tell you very coarse stories about the state of your coverage. But line and particularly branch coverage have a lot richer information. You might have 100% line coverage and have very poor branch coverage. Branch coverage can help you figure out if you are covering all the independent paths through the code? Do you even pay attention to cyclomatic complexity?
Second, code coverage tools typically run off the the Java byte code, not the actual source itself. This means that code that the compiler generates for you may not be mappable to any sensible line in your source code. (Ever wonder why the first curly brace after your class name is shown as uncovered?) You can find discussions of this issue on sites like stackoverflow.com. The practical significance of this is that you think you have all the code covered, but the coverage tool shows you odd "unreachable" corners in your source that have no coverage.
Finally, there will be some flows through the code, particularly obtuse error paths, that may be hard to test. Do you really want to spend 50% more effort to get that last 10% coverage? Is this a good use of your money?
There is no real best practice on code coverage limits. On my team, we use about an 80% limit for line and branch coverage. This mostly keeps new code from being checked in without coverage. And it leaves enough slack for the developers to exercise their judgement on whether it is worth chasing those extra few lines of coverage. In a future posting, I'll see if I can find statistics that show the cost of getting that higher coverage and the chances that the tests for the extra coverage actually find or prevent defects.
I work in a Java shop, and we use JaCoCo as our coverage tool. It is also used in Sonar. We used to use Cobertura but I believe at the time that the new Java 7 byte codes confused it. Coverage tools like JaCoCo do a pretty good job generating coverage reports that show you both coverage summaries and allowing you to drill into your packages, classes, and methods to see coverage for individual code lines.
While serviceable, these tools have a serious deficiency - the legacy code problem. You know that problem - the one where you start a code base completely ignoring unit tests and code coverage. Then somewhere along the line, one or more of you get a pang of guilt, see the light, and join modern software best practices. But now you have a problem. Nobody, and I mean nobody, is going to pay you to go back and write tests for coverage for that legacy code. One way to handle this is to just take a punch in the nose and watch your code coverage average be dragged down by a bunch of zeros. I'll cover how we handled this problem in a later blog.
Hopefully this has given you some things to consider in your own projects.
If there is any bit of heresy, this is it - code coverage has no intrinsic value. Its value comes only from showing where there is untested code. All of the value in what we do in this aspect of development is in the quality of the unit tests. Maybe leaving code untested is a deliberate calculation, but maybe it was a mistake. As developers, we are pulled in multiple directions all the time. It can take some effort to get back into context and you can use your uncovered code to help you recover.
The single best way to use code coverage is to break the build. As Agilists, we all are practicing continuous builds, right? I mean a new build goes off with nearly every code commit, right? That frequency is exactly what you should use for running code coverage checks. When the build breaks due to a fall in coverage, you get the advantage of causality. The latest check-in probably broke the build. (Your thoughtful coworkers never check code into a broken build.) It is very important that all developers should be able to run the same coverage checks locally on their own machines that are done on the continuous build server. Thus a broken build due to coverage checks should never be a surprise to anyone.
So if we want to have minimum coverage limits that must be met or the build breaks, what is an appropriate limit? Unfortunately some misguided management mandate that a 100% line coverage be maintained. In my opinion, this is a very poor way to spend your limited resources.
First, not all code coverages are created equal. A 100% line coverage limit does not address the differences between line and branch coverage. Package and class coverage can tell you very coarse stories about the state of your coverage. But line and particularly branch coverage have a lot richer information. You might have 100% line coverage and have very poor branch coverage. Branch coverage can help you figure out if you are covering all the independent paths through the code? Do you even pay attention to cyclomatic complexity?
Second, code coverage tools typically run off the the Java byte code, not the actual source itself. This means that code that the compiler generates for you may not be mappable to any sensible line in your source code. (Ever wonder why the first curly brace after your class name is shown as uncovered?) You can find discussions of this issue on sites like stackoverflow.com. The practical significance of this is that you think you have all the code covered, but the coverage tool shows you odd "unreachable" corners in your source that have no coverage.
Finally, there will be some flows through the code, particularly obtuse error paths, that may be hard to test. Do you really want to spend 50% more effort to get that last 10% coverage? Is this a good use of your money?
There is no real best practice on code coverage limits. On my team, we use about an 80% limit for line and branch coverage. This mostly keeps new code from being checked in without coverage. And it leaves enough slack for the developers to exercise their judgement on whether it is worth chasing those extra few lines of coverage. In a future posting, I'll see if I can find statistics that show the cost of getting that higher coverage and the chances that the tests for the extra coverage actually find or prevent defects.
I work in a Java shop, and we use JaCoCo as our coverage tool. It is also used in Sonar. We used to use Cobertura but I believe at the time that the new Java 7 byte codes confused it. Coverage tools like JaCoCo do a pretty good job generating coverage reports that show you both coverage summaries and allowing you to drill into your packages, classes, and methods to see coverage for individual code lines.
While serviceable, these tools have a serious deficiency - the legacy code problem. You know that problem - the one where you start a code base completely ignoring unit tests and code coverage. Then somewhere along the line, one or more of you get a pang of guilt, see the light, and join modern software best practices. But now you have a problem. Nobody, and I mean nobody, is going to pay you to go back and write tests for coverage for that legacy code. One way to handle this is to just take a punch in the nose and watch your code coverage average be dragged down by a bunch of zeros. I'll cover how we handled this problem in a later blog.
Hopefully this has given you some things to consider in your own projects.
Tuesday, June 9, 2009
Interviewing
Yep, its another blog/article on interviewing. I just searched on google for "interview programming" and got 21.8M hits. So what's so bad about one more? :)
I wanted to have a few words about interviewing for mid-level and senior developers. I do this for my company as one of my many functions (including software architect and lead developer). The reason I feel compelled to write is disappointment. I just cannot reconcile how good some resumes look and how poor the results are off the paper.
When I interview, I *never* *ever* go the puzzle/trick route. I think those have very little to do with our jobs at best and give misleading results at worst. To me, you should focus where the big cost items are in software - in its (lack of) structure and in its (lack of) flexibility. Does the code smell?
It is how easily the software can be changed over time that determines its real cost.
So when I interview, I first establish that they have good grasp of object orient design fundamentals. Then I focus on the key area of "best practices." Does the candidate understand when they are creating a mess? Do they recognize conceptual pollution when they see it?
To ascertain this, I give them a take home (java) "test." I want them to not feel a great deal of time pressure (when was the last time your boss dropped a programming task in your lap and said it needed to be done in 20 minutes?). I give them overnight to complete the test, but in reality it should only take them 20 minutes.
The actual test is short. It's a two page code sample for them to critique. I tell them that their boss is interested in hiring the developer of that code and boss wants their feedback. The test shows the following kinds of problems:
I don't consider this tricky questioning. It basically boils down to me having developers that don't keep me awake at night wondering what messes they are making. I feel that a competent developer ought to be able to get most of this almost by instinct.
So why do 90% of the candidates not make the bar?
I wanted to have a few words about interviewing for mid-level and senior developers. I do this for my company as one of my many functions (including software architect and lead developer). The reason I feel compelled to write is disappointment. I just cannot reconcile how good some resumes look and how poor the results are off the paper.
When I interview, I *never* *ever* go the puzzle/trick route. I think those have very little to do with our jobs at best and give misleading results at worst. To me, you should focus where the big cost items are in software - in its (lack of) structure and in its (lack of) flexibility. Does the code smell?
It is how easily the software can be changed over time that determines its real cost.
So when I interview, I first establish that they have good grasp of object orient design fundamentals. Then I focus on the key area of "best practices." Does the candidate understand when they are creating a mess? Do they recognize conceptual pollution when they see it?
To ascertain this, I give them a take home (java) "test." I want them to not feel a great deal of time pressure (when was the last time your boss dropped a programming task in your lap and said it needed to be done in 20 minutes?). I give them overnight to complete the test, but in reality it should only take them 20 minutes.
The actual test is short. It's a two page code sample for them to critique. I tell them that their boss is interested in hiring the developer of that code and boss wants their feedback. The test shows the following kinds of problems:
- embedding strings directly in code
- mixing of abstractions
- blocks of code commented out (with no other explanation as to why it was commented out)
- the use of public virtual functions called in a constructor (a questionable practice)
- complete lack of comments
- errors in relationships (coder assumed 1:1 when its really 1:N)
- missing abstractions (over use of string data)
- over-reliance on open-ended integers (instead of enums)
I don't consider this tricky questioning. It basically boils down to me having developers that don't keep me awake at night wondering what messes they are making. I feel that a competent developer ought to be able to get most of this almost by instinct.
So why do 90% of the candidates not make the bar?
Wednesday, April 8, 2009
Third Life, Fourth Life, ...
I was in a discussion today with a colleague of mine (I'll call him Jim). He was asking my opinion of an important message he put at the end of his application installer. I suggested some minor tweaks, which he appreciated. But I also asked him to emphasize things, possibly by an alert or by an extra dedicated panel (most installers are wizards). His comment was that doing so would only aid maybe 10% more of the users.
What sad but true observation.
I was taken back about a year earlier when I remember standing at the back of a training class watching one of our sales engineers using a product I designed. He was working his way through a guided work flow. When an error message came up (you know, the one with the big red X on it), he couldn't click it away fast enough. Breaking with best practices, I interrupted him asking him what the error message was and he didn't know.
This was truly an eye-opener for me. To him, an interrupting dialog/alert was a nuisance to be disposed of as quickly as possible. It was impeding his work flow. I try to put a lot of effort into the information content of error messages to be as helpful as possible to the end user. Was all that effort wasted? I liken careful error handling to putting guard rails on a winding mountainous road.
Back to my conversation with Jim.
We were discussing possible changes to user interfaces. I remarked that maybe with virtual reality becoming commonplace, we could actually be more effective in informing users of the severity of problems. Maybe a future application would have a user traveling down a path, say a winding mountain path. When a severe error occurs, rather than a dialog, we could actually push them off a virtual cliff. At least the experience would be hard to dismiss and might even be memorable.
Maybe instead of giving the user the Ok/Cancel button of today, such a future application might present a button with the label "Respawn?"
What sad but true observation.
I was taken back about a year earlier when I remember standing at the back of a training class watching one of our sales engineers using a product I designed. He was working his way through a guided work flow. When an error message came up (you know, the one with the big red X on it), he couldn't click it away fast enough. Breaking with best practices, I interrupted him asking him what the error message was and he didn't know.
This was truly an eye-opener for me. To him, an interrupting dialog/alert was a nuisance to be disposed of as quickly as possible. It was impeding his work flow. I try to put a lot of effort into the information content of error messages to be as helpful as possible to the end user. Was all that effort wasted? I liken careful error handling to putting guard rails on a winding mountainous road.
Back to my conversation with Jim.
We were discussing possible changes to user interfaces. I remarked that maybe with virtual reality becoming commonplace, we could actually be more effective in informing users of the severity of problems. Maybe a future application would have a user traveling down a path, say a winding mountain path. When a severe error occurs, rather than a dialog, we could actually push them off a virtual cliff. At least the experience would be hard to dismiss and might even be memorable.
Maybe instead of giving the user the Ok/Cancel button of today, such a future application might present a button with the label "Respawn?"
Thursday, January 15, 2009
Primitive Types and the Domain Model
I was at the No Fluff Just Stuff conference this year (5th year in a row for me), listening to Neal Ford talk about development concepts. He made a casual remark that caught and held my attention far after he had moved on to other topics. He said
Never use primitive types in your domain model
As you veteran developers know, the core of many well-built applications is a clear, concise domain model. This forms the basis for key abstractions and is the framework around which you typically hang your business logic. Most of my experience over the years has been with developing large, complex desktop applications where a good domain model is key. I assume that for medium-to-large enterprise web applications, a good domain model is also highly valued.
So what exactly does the quote above mean? I'll give you some examples. The first is from one of our early domain models where we represented network topology elements (computers, routers, etc). One of the data members of our network representation was bandwidth, as in set-the-bandwidth-of-this-network-to-X. We assumed that the number was megabits per second. Of course, we wrote some independent utility scaling routines to go to kilobits per second and gigabits per second.
Do you see the inferior design? Of course, we should never have used a primitive type (like double) to represent bandwidth. What we should have done was create a Bandwidth class and passed this to the "set" and "get" routines. This class would have had the concept of units (possibly a class like BitRate) and would have encapsulated all the conversion routines in one nice package. Maybe you would find lines of code like
double capacity = bw.inMegabitsPerSec();
And importantly constructed like this
Bandwidth bw = new Bandwidth (3.5, MegabitsPerSec);
That is so much easier to read and is far clearer than
double bw = 3.5;
A second example comes from some code that I trapsed across not a month ago. We have a switchboard-like, sockets-based server that handles communications between applications within a given suite. In sending a message through this switchboard to another application, you identify the target application by a name. The application names and versions that form a suite come from a configuration file. We allow aliases/monikers for the applications as well as trying not to be case sensitive. Part of this is convenience, and part of it is because the applications are written in diverse technologies (C++, Perl, Java, C#).
I was looking at this code for the first time, and I saw a lot of string manipulation going on. There was trimming (of spaces), folding of cases, and substring matching -a lot of it. In tracing through the code, a string that was already lower case, was set to lower case again. Clearly somewhere along the line, the programmer (or programmers) lost track of the flow of the data in the application, where messages were entering the system. The string hacking was so prolific that it made the code harder to read.
So what was missing here? Instead of using a string primitive type, how about a class called ApplicationName. It would encapsulate all the concepts of case insensitivity and of aliases. I even found code later on that emitted information to logs and to the end user, so it could even encapsulate the concept of "pretty name" (a standard, human-consumable name).
I know what a few of you are thinking. You are concerned about the performance cost of allocating all those instances. With modern optimizing compilers, you'll find that performance costs won't really be an issue. The cost that will eat your lunch is the maintenance cost for the software over time.
So the next time you are working in your domain model, and you feel the urge to use an int, double, string, or other primitive type, think again.
Never use primitive types in your domain model
As you veteran developers know, the core of many well-built applications is a clear, concise domain model. This forms the basis for key abstractions and is the framework around which you typically hang your business logic. Most of my experience over the years has been with developing large, complex desktop applications where a good domain model is key. I assume that for medium-to-large enterprise web applications, a good domain model is also highly valued.
So what exactly does the quote above mean? I'll give you some examples. The first is from one of our early domain models where we represented network topology elements (computers, routers, etc). One of the data members of our network representation was bandwidth, as in set-the-bandwidth-of-this-network-to-X. We assumed that the number was megabits per second. Of course, we wrote some independent utility scaling routines to go to kilobits per second and gigabits per second.
Do you see the inferior design? Of course, we should never have used a primitive type (like double) to represent bandwidth. What we should have done was create a Bandwidth class and passed this to the "set" and "get" routines. This class would have had the concept of units (possibly a class like BitRate) and would have encapsulated all the conversion routines in one nice package. Maybe you would find lines of code like
double capacity = bw.inMegabitsPerSec();
And importantly constructed like this
Bandwidth bw = new Bandwidth (3.5, MegabitsPerSec);
That is so much easier to read and is far clearer than
double bw = 3.5;
A second example comes from some code that I trapsed across not a month ago. We have a switchboard-like, sockets-based server that handles communications between applications within a given suite. In sending a message through this switchboard to another application, you identify the target application by a name. The application names and versions that form a suite come from a configuration file. We allow aliases/monikers for the applications as well as trying not to be case sensitive. Part of this is convenience, and part of it is because the applications are written in diverse technologies (C++, Perl, Java, C#).
I was looking at this code for the first time, and I saw a lot of string manipulation going on. There was trimming (of spaces), folding of cases, and substring matching -a lot of it. In tracing through the code, a string that was already lower case, was set to lower case again. Clearly somewhere along the line, the programmer (or programmers) lost track of the flow of the data in the application, where messages were entering the system. The string hacking was so prolific that it made the code harder to read.
So what was missing here? Instead of using a string primitive type, how about a class called ApplicationName. It would encapsulate all the concepts of case insensitivity and of aliases. I even found code later on that emitted information to logs and to the end user, so it could even encapsulate the concept of "pretty name" (a standard, human-consumable name).
I know what a few of you are thinking. You are concerned about the performance cost of allocating all those instances. With modern optimizing compilers, you'll find that performance costs won't really be an issue. The cost that will eat your lunch is the maintenance cost for the software over time.
So the next time you are working in your domain model, and you feel the urge to use an int, double, string, or other primitive type, think again.
Saturday, October 4, 2008
Usability of London's Tube Map
A colleague of mine just got back from London, where use of "the tubes" (subway) is highly encouraged. As a newbie to the system, she relied heavily on the tube map to get around.
She had good things to say about this map (it's aesthetically pleasing, it's easy to use). But her biggest complaint was that she could not tell that a certain destination was going to be twice as far as the last one.
Not being one to take a challenge lying down, I thought about it. The map crams a lot of information into a fairly small space (the image above is reduced a lot). It is fairly information rich. What could be added to that map that would still preserve its aesthetics and simplicity?
How about we give up our preconception that every map has only one scale? (A scale is a ratio of say map miles to real inches of map distance.) What if a map had multiple scales? What if it had a variable scale? Could we present a solution that would service both users who want to get real distances and those who want relative?
Here is what I came up with. Suppose that we put tick marks on the tracks. Each tick mark would represent a distance (say 2 km). This would be true for the whole map. This would allow users to quickly eyeball a source and a destination and either visually compare or actually count the distance. Notice that the actual map inches for each tick is variable. Check out a small section that I blew up and modified.
I think the idea is ready for usability testing! ;-)
She had good things to say about this map (it's aesthetically pleasing, it's easy to use). But her biggest complaint was that she could not tell that a certain destination was going to be twice as far as the last one.
Not being one to take a challenge lying down, I thought about it. The map crams a lot of information into a fairly small space (the image above is reduced a lot). It is fairly information rich. What could be added to that map that would still preserve its aesthetics and simplicity?
How about we give up our preconception that every map has only one scale? (A scale is a ratio of say map miles to real inches of map distance.) What if a map had multiple scales? What if it had a variable scale? Could we present a solution that would service both users who want to get real distances and those who want relative?
Here is what I came up with. Suppose that we put tick marks on the tracks. Each tick mark would represent a distance (say 2 km). This would be true for the whole map. This would allow users to quickly eyeball a source and a destination and either visually compare or actually count the distance. Notice that the actual map inches for each tick is variable. Check out a small section that I blew up and modified.
I think the idea is ready for usability testing! ;-)
Friday, August 15, 2008
The Problem with Thinking Additively
How does your company decide how to improve your products? If yours is like most, you have product managers who survey selected customers, soliciting feedback on improvements. Between their suggestions and your development staff, you come up with a list of new features for that next development cycle.
Maybe this works for your company, but it is a flawed process. By focusing on your existing product and thinking additively, you effectively warp your process into being tool-centric. The Next Great Release is only a few new features away. Developers think this way, product managers think this way, and existing customers think this way (potential customers may not).
So what exactly is wrong with this picture? Everyone is focused on the tool. It is a distraction from what is important, namely solving the user's problem. If you could really see how your tool is used by end users, you might notice that there is a lot of interaction between the user, the tool, and other things.
I'll call this workflow. It is the sequence of steps that the user is following in going from nothing to solution. It might involve using your tool multiple times in different ways. It might involve generating intermediate workproducts. It might involve interacting with others at certain points. Ultimately, it might be all this activity around your tool that determines its success, rat more than its feature set.
Workflow may not be prominent in all problem domains, but the more complex the domain, the more likely that workflow is significant. Consider my company's domain for example. We provide tools to allow technical people to work with network and server data, to build a models of their system, and to predict the behavior of the system under various loads.
Recipes
One day one of our consultants passed by and proudly flopped a meaty document on my desk and informed me that he had finally codified a methodology to solve one of our users' most common modeling problems. It was a masterpiece at 90 pages, was well organized, was valuable to our users, and made an impressive sound hitting my desk.
In a later meeting, while developers and product managers argued about getting SAN support into our tools, I was still thinking about that sound. If the problem domain was that complex, surely we should be selling guidance to our users, right? And we couldn't ship a consultant in every box, could we?
In a sense, the methodology was a recipe, and arguably was as valuable as the tool. If we could productize it, users would see real value in a productivity boost. The analogy to a cooking recipe is apt. It has ingredients (preconditions), a step by step description of what to do (a process), and a view of the expected result. Users might choose different recipes, depending on what they are trying to do.
One of the most valuable things a recipe can do is to establish context. By knowing context, any support you give the user can be made more relevant. Maybe your user has filtered a network traffic report and it is empty. Is that an error condition? Maybe. Is there sage advice to offer them - only if you know what they are trying to do. If you build tools without context, in a way, you can only offer abstract options and abstract advice. It becomes much harder to embed domain knowledge in your tool.
Steps
Steps are the building blocks of recipes. Our product is composed of more than one application, and some steps control these other applications, launching them with the appropriate inputs and outputs when appropriate. Some kinds of steps generate workproducts. Some steps encompass critique of data or state, sometimes encompassing expert advice. Some steps allow declaration of data (like a list of server names). A lot of steps have fixed or parameterized functionality, but one type of step is very highly customizable in terms of both its behavior and its designable user interface.
Flow and Dependency
There is a concept of information flow involved in recipes. Since recipes have multiple steps and steps might have inputs and outputs, there is a flow among the steps. The inputs and outputs of each step are typed and the flow matches up by type. This brings up one of the strengths of this approach, namely the managing of dependencies. If a user makes a change farther up in the recipe, all downstream steps that share that flow are now out-of-date, meaning that the steps need to be executed again to bring their outputs up-to-date. As long as this out-of-date state is visible to users, it is a simple matter of making sure that required changes propagate correctly.
Tracability
In our problem domain, users are making business decisions on modeling results, involving potentially millions of dollars. Confidence in the results is paramount, requiring an understanding of how those results were arrived at. If the results are unexpected, they need to work backwards in the flow to figure out where their expectations diverge from the results. Users might work backwards examining not only intermediate workproducts, but also look at assumptions along the way.
Consider this requirement in the context of a loop. Whether in a conventional programming language or a flow chart, a loop has a conditional, a loop body, and a loop terminator. There are undesirable characteristics of this representation. First, it assumes that the logic associated with the loop body is the same for each pass through the loop. When you consider that each pass might represent the analysis of a different set of data, and that the user might want to tweak each pass, the flow chart style logic is not flexible enough.
A way to represent this is to unroll the loop. That is, represent the loop body explicitly for each iteration of the loop. It takes more space to do this certainly, but it meets the traceability requirement and allows the user to modify each pass independently. There is also a dynamic reconfiguration aspect here. Maybe the loop is a pass for each test case in the collected data. So if the user adds a test case, a new series of steps might dynamically come into existence representing the new pass through the loop.
A final benefit in explicitly representing all steps required in a recipe, is that the user has a roadmap that tells them how far they need to go. They can tell how far they have come by a "done flag" that each step displays.
Business Politics
An easily overlooked issue with introducing workflow into your products involves business politics. You might have to champion a fairly sizable change to how you are going to spend your development dollars. If introducing workflow competes for the same resources that would be used to develop new features, you might meet resistance.
In my company, I started the sales pitch at the customer advisory council level. Being composed entirely of our biggest customers, the council readily bought into the concept. This should be expected, as users are the ones who ultimately pay the price for lack of workflow support through lost productivity.
Next up was an internal presentation that included developers, consultants, and product managers. The development staff was positive towards the idea. The product managers were uncertain about this; the workflow solution came out of some creative thinking in development, which is backwards from what they expected. The really interesting responses came from the consulting staff, who are effectively very advanced users. Most were favorable to the idea, but consultants have a history of being mavericks. As subject matter experts, each might have their own way of solving problems.
One consultant stood up in this meeting and gave the most negative response, saying something like "Tell me why I would ever use something like this. Why would we build this application?" I'm happy to say that the same consultant caught me in the hall about a year after the product was released and told me, "I don't know how I got along without this!"
I'd like to close this with an admission. To this day, I never have read every page of that original methodology. Just don't tell that consultant.
Maybe this works for your company, but it is a flawed process. By focusing on your existing product and thinking additively, you effectively warp your process into being tool-centric. The Next Great Release is only a few new features away. Developers think this way, product managers think this way, and existing customers think this way (potential customers may not).
So what exactly is wrong with this picture? Everyone is focused on the tool. It is a distraction from what is important, namely solving the user's problem. If you could really see how your tool is used by end users, you might notice that there is a lot of interaction between the user, the tool, and other things.
I'll call this workflow. It is the sequence of steps that the user is following in going from nothing to solution. It might involve using your tool multiple times in different ways. It might involve generating intermediate workproducts. It might involve interacting with others at certain points. Ultimately, it might be all this activity around your tool that determines its success, rat more than its feature set.
Workflow may not be prominent in all problem domains, but the more complex the domain, the more likely that workflow is significant. Consider my company's domain for example. We provide tools to allow technical people to work with network and server data, to build a models of their system, and to predict the behavior of the system under various loads.
Recipes
One day one of our consultants passed by and proudly flopped a meaty document on my desk and informed me that he had finally codified a methodology to solve one of our users' most common modeling problems. It was a masterpiece at 90 pages, was well organized, was valuable to our users, and made an impressive sound hitting my desk.
In a later meeting, while developers and product managers argued about getting SAN support into our tools, I was still thinking about that sound. If the problem domain was that complex, surely we should be selling guidance to our users, right? And we couldn't ship a consultant in every box, could we?
In a sense, the methodology was a recipe, and arguably was as valuable as the tool. If we could productize it, users would see real value in a productivity boost. The analogy to a cooking recipe is apt. It has ingredients (preconditions), a step by step description of what to do (a process), and a view of the expected result. Users might choose different recipes, depending on what they are trying to do.
One of the most valuable things a recipe can do is to establish context. By knowing context, any support you give the user can be made more relevant. Maybe your user has filtered a network traffic report and it is empty. Is that an error condition? Maybe. Is there sage advice to offer them - only if you know what they are trying to do. If you build tools without context, in a way, you can only offer abstract options and abstract advice. It becomes much harder to embed domain knowledge in your tool.
Steps
Steps are the building blocks of recipes. Our product is composed of more than one application, and some steps control these other applications, launching them with the appropriate inputs and outputs when appropriate. Some kinds of steps generate workproducts. Some steps encompass critique of data or state, sometimes encompassing expert advice. Some steps allow declaration of data (like a list of server names). A lot of steps have fixed or parameterized functionality, but one type of step is very highly customizable in terms of both its behavior and its designable user interface.
Flow and Dependency
There is a concept of information flow involved in recipes. Since recipes have multiple steps and steps might have inputs and outputs, there is a flow among the steps. The inputs and outputs of each step are typed and the flow matches up by type. This brings up one of the strengths of this approach, namely the managing of dependencies. If a user makes a change farther up in the recipe, all downstream steps that share that flow are now out-of-date, meaning that the steps need to be executed again to bring their outputs up-to-date. As long as this out-of-date state is visible to users, it is a simple matter of making sure that required changes propagate correctly.
Tracability
In our problem domain, users are making business decisions on modeling results, involving potentially millions of dollars. Confidence in the results is paramount, requiring an understanding of how those results were arrived at. If the results are unexpected, they need to work backwards in the flow to figure out where their expectations diverge from the results. Users might work backwards examining not only intermediate workproducts, but also look at assumptions along the way.
Consider this requirement in the context of a loop. Whether in a conventional programming language or a flow chart, a loop has a conditional, a loop body, and a loop terminator. There are undesirable characteristics of this representation. First, it assumes that the logic associated with the loop body is the same for each pass through the loop. When you consider that each pass might represent the analysis of a different set of data, and that the user might want to tweak each pass, the flow chart style logic is not flexible enough.
A way to represent this is to unroll the loop. That is, represent the loop body explicitly for each iteration of the loop. It takes more space to do this certainly, but it meets the traceability requirement and allows the user to modify each pass independently. There is also a dynamic reconfiguration aspect here. Maybe the loop is a pass for each test case in the collected data. So if the user adds a test case, a new series of steps might dynamically come into existence representing the new pass through the loop.
A final benefit in explicitly representing all steps required in a recipe, is that the user has a roadmap that tells them how far they need to go. They can tell how far they have come by a "done flag" that each step displays.
Business Politics
An easily overlooked issue with introducing workflow into your products involves business politics. You might have to champion a fairly sizable change to how you are going to spend your development dollars. If introducing workflow competes for the same resources that would be used to develop new features, you might meet resistance.
In my company, I started the sales pitch at the customer advisory council level. Being composed entirely of our biggest customers, the council readily bought into the concept. This should be expected, as users are the ones who ultimately pay the price for lack of workflow support through lost productivity.
Next up was an internal presentation that included developers, consultants, and product managers. The development staff was positive towards the idea. The product managers were uncertain about this; the workflow solution came out of some creative thinking in development, which is backwards from what they expected. The really interesting responses came from the consulting staff, who are effectively very advanced users. Most were favorable to the idea, but consultants have a history of being mavericks. As subject matter experts, each might have their own way of solving problems.
One consultant stood up in this meeting and gave the most negative response, saying something like "Tell me why I would ever use something like this. Why would we build this application?" I'm happy to say that the same consultant caught me in the hall about a year after the product was released and told me, "I don't know how I got along without this!"
I'd like to close this with an admission. To this day, I never have read every page of that original methodology. Just don't tell that consultant.
Subscribe to:
Posts (Atom)