The Easiest Developer Productivity Measure of All - Not!

It's 2019 and I can't believe I'm in this position again. I ran across an organization that was trying to find a way to measure the productivity of a team of developers. Their insistence on using lines-of-code (LOC) brought back some long-suppressed PTSD from years ago. Before diving into that subject, let's take a minute to get an orientation around the problems of using productivity performance metrics, as espoused in "Against metrics: how measuring performance by number backfires".

It's only a five-minute read. I'll wait...

https://aeon.co/ideas/against-metrics-how-measuring-performance-by-numbers-backfires?utm_source=hackernewsletter&utm_medium=email&utm_term=fav

Hopefully your unease has increased. So, let's get right into it.

LOC was probably the earliest attempt at measuring developer productivity. The concept was simple and was borrowed from operations research in the domain of a factory production. In such a setting, you might try to measure the productivity of an assembly line worker by counting how many widgets they processed - and that output widget pile just keeps on growing - go, team.

It's a simple hop to apply this in the software world. Developers produce value by writing code. The more the better. A developer who writes a lot of code is clearly better than a developer who writes less code.

If you are trying to measure a developer's or team's bottom line dollar impact to business, and you are looking covetously at LOC, then let me try to talk you down off that ledge by asking some questions.

* What do you want to use this productivity measure for?

* Are you trying to compare the delivered value among individual developers?

* Are you comparing delivered value of different teams?

* Are you deciding if a project delivered enough value to the business

You can see from the abstract reference to value in the above questions that LOC as a productivity is not a good fit. The LOC metric is very, very narrow and has the following false-isms (and with a few minutes more thought, this list can be increased).

* Non-coding tasks have no value

* Reducing risk via spikes has no value

* Design and architecture discussions have no value

* Concise coding has less value

* Taking more time and getting fewer bugs has less value

* Writing internal documentation has no value

* Performance testing has no value

As alluded to in the cited "backfire" article, your organization may morph under the pressure of productivity measures. Specifically, developers will gravitate to having "higher productivity" aka more LOC, so be careful what you incentivize. The absurdity of the situation is captured nicely in one of my all-time favorite Dilbert cartoons. (Yes, you can get this on a mug to carry your coffee around every workday!)

https://dilbert.com/strip/1995-11-13

So, you still insist on using LOC as a metric?

Let's look more closely at the code. What lines do we want to count? Configuration lines like Xml? What if those XML lines actually replace code as in the case of dependency injection? Data files like JSON? What if this is a declarative representation to replace ad hoc coding? Generated code? Should you be counting the sources of this generation instead? So you can see that the definition of what is code has moved beyond the traditional, simplistic definition.

Is a line-of-code a LOC?

Do comments have any value? Ha, ha, ya, it is a trick question. What about comparing across languages? A given bit of logic can be expressed concisely in some languages and more verbosely in others, so how does that affect things? What about lines of test code versus lines of production code? What about test code density? (Spock/Groovy is about 5 times denser than JUnit/Java)

How do you account for the lifespan of a LOC?

Refactoring can result in negative productivity. One of my long-time assertions is that one of the most valuable things a developer can do is to throw away code. An example of this would be when a developer reduces the damage of "copy-paste-itis" and coalesces to a function. That is going to reduce testing and maintenance costs. Also how do you handle splitting/merging of repositories? The effect of this can be very difficult to track against LOC metrics.

What about other measures?

In an agile environment, the team is refocused on the moving "business value" target at the start of each sprint. The sprint stories were vetted by the product owner, who should have marketing's estimated value of each feature. This is a very direct way to assess value. So what if you don't have access to that? Well, you could use the "T-Shirt Sizing" estimates as an abstract measure. If that isn't available, you might use stories/iteration or points/iteration, with the caution that the definition of a point will vary among teams.

Other Resources

You can find Martin Fowler's take on using LOC here... https://martinfowler.com/bliki/CannotMeasureProductivity.html

There is a good article on agile metrics for productivity here... https://www.business.com/articles/agile-team-productivity-metrics/

That's it for now - hopefully you are no longer on that ledge.

Hercules in the Age of Agility

Intro

I have been in the software development business for well over twenty five years and have seen many successful and unsuccessful projects. The past ten years have been on teams following Agile principles to varying degrees. The teams have always been composed of different levels of experience and expertise, but I have been very fortunate to have teams that are heavily populated with not only experienced developers, but with open-minded developers.

On these teams, I have seen patterns of behavior that, on the surface, seem beneficial to the team, the product, and the company, but in reality come with a lot of unrecognized costs. These costs are unmanaged; they are not planned for; they are not quantified. Oftentimes they are never historically reconciled.

There is one particular behavioral offender in this developer ecosystem and it might surprise you. I'm talking about the hero developer.

Hercules

Meet Hercules. He is a member of our mythical development team. Hercules fits pretty well into the team. He communicates reasonably. He is a competent developer. He gets his work done consistently. But where he really shines is when there are looming, seeming impossible deadlines. He is the one that stays up into the wee hours, throwing himself across the finish line just-in-time.

He is the hero. He is the one who gets the adoration of managers and that good ole back slap with an "at-a-boy". Honestly he really loves the attention and approval from his manager and the awe from the junior developers who hope to grow up to be like Hercules someday.

Managers see the deeds of Hercules and know that this is how successful software is created. If only they had a bunch developers just like Hercules, their software problems would be solved.

A Deeper Analysis

First, in the context of Hercules, let's acknowledge that being dedicated to your team and working hard are assets and that these assets are not found in all developers. But when we look at Herculean deeds from an Agile point-of-view, we start to see issues that can undo all of the gains that came from the providence of Hercules.

Hercules specializes in cranking out code in a short amount of time. Sometimes it seems the more pressure, the better. Hercules knows that code. Due to time pressure, he didn't get to do a lot of the unit or integration tests, but through sheer ability to focus, he delivered today's functionality, and that's what's important, right?

Hercules sure knows that code, but the other developers don't. The agile developer wants group code ownership. They want group input on design choices. They want the most well thought out, flexible code that the company and project can afford. The agile development team wants to de-silo the implementation, meaning that multiple developers are familiar with the code so future changes occur at a reasonable cost. But given the single source effort, the odds go against this goal.

A good hero sees heroics as an aberration - something to be avoided if possible. But I have seen cases where heroes use heroics as a path to job security. They hoard information by being the only custodians of said code, such that others recognize their area expertise. They want to make sure that there is a large painful hole in the development team should they ever leave. Naturally job security like this is good for them and really bad for the company.

Another issue involves scaling the team. If you have a Hercules, it is almost impossible to clone him/her. You should be able to find developers, but if you need one to fit into a chaotic, barely-managed environment, you need another Hercules, and they are both harder to find and cost more money.

The final drawback is a bit more subtle. Tom Demarco's book "Slack" (https://www.amazon.com/Slack-Getting-Burnout-Busywork-Efficiency/dp/0767907698) does a good job of discussing the issue. In a nutshell, if you have heroics as your normal means of delivering software, your heroes are >100% busy - and this means time for creativity and giving back (like mentoring) fall by the wayside. This cost is never recognized by management but can have a very large impact on the business over time.

Parting Advice

Hire as many heroes as you can and avoid heroics at all costs.

The Last Measurable Ounce of Quality Can Be Expensive

A Brief Lexicon of the Software Quality Landscape

The quality of a software product is a multi-dimensional measurement, spanning such things as functionality, correctness, performance, documentation, ease of use, flexibility, maintainability, and many others. Many of these qualities are difficult to measure, are difficult to see, and hence are difficult to manage. The result is that they are ignored by all but the most enlightened in management.

The topic of interest for this screed is going to be correctness. This is not going to concern correctness in the end-user sense, meaning the correct meeting of requirements. It is going to mean "is the code doing what we think it should be doing." Since we don't generally have provably correct programs, it is a matter of convincing ourselves, through lack of evidence, that our programs are working as we'd like. This precarious situation is perfectly portended by Edsger Dijkstra's observation that "absence of evidence is not evidence of absence."

So we have several levels of ways to convince ourselves of correctness. They are, from most detailed to most abstract, unit testing, integration testing, functional testing, and system testing. Note that there is not industry agreement concerning these exact terms, but the general concepts are recognized.

In typical object oriented designs, unit testing involves isolating a given class and driving its state and/or behavior and verifying that we see what we expect. Integration testing is a layer above that where we use multiple classes in the tests. Functional testing is yet above that, where we try to deploy our programs in the natural components that they would inhabit in production, like a server or a process. Finally system test covers testing in the full production-like environment.

Unit Testing

Our focus will be on the lowest level, namely unit testing. The expectation is that unit tests are both numerous (think many hundreds or even thousands) and are extremely fast (think milliseconds). To be effective, these tests should be run on every single compile on every developer's machine across the organization. The goal is that the unit tests precisely capture the design intent behind the implementation of the class code, and that any violation of those intents result in immediate feedback to the developer making code changes.

I'd like to tell you that every developer is doggedly focused on both the quality of the production logic and the thoroughness of the unit tests that back that logic. Through a combination of poor training, lack of emphasis at the management level, and just plain laziness, developers produce tests that span from greatness all the way down to downright destructive (more on that in another blog entry). One of the easiest ways to try to externally track this testing is through code coverage.

Code Coverage

Code coverage is a set of metrics that can give developers and other project stakeholders a sense of how much of the production logic has been tested by the unit tests. The simplest metric is the "covered lines of code" aka line coverage. This is usually a percentage and it means that if a class has 50 lines of code in it, and it has 60% code coverage, then 30 lines of that production logic is executed as part of the running of the unit tests for that class. There are other coverage metrics that can help you gauge the goodness of your tests, like branch coverage, class coverage, and method coverage. But here, we will focus on line coverage since that is most widely used.

The general, common sense assumption is that "more is better", so mis-guided management and deranged architects insist on 100% code coverage, thinking that would give the maximum confidence that the quality of the code is high. If we had an infinite amount of time and money to spend on projects, this conception could represent the optimum. Since this luxury has never been true in the last 4 billion years, we have to spend our money wisely. And this changes things drastically.

The truth is that it might cost M dollars+time to achieve say 80% line coverage, but it might take M *more* dollars+time to get that last 20%. In some cases, getting the last few percentage might be extremely expensive. The reason for this non-linear cost is complicated.

First, production logic should be tested through its public interface where possible rather than through a protected or private interface. It can be laborious to construct the conditions necessary to hit a line of code buried in try/catches and conditional logic behind public interfaces. This cost can be lowered by refactoring the code towards better testability, but this is a continuous struggle as new code is produced. There is a truism in the veteran developers that increasing the testability of the production logic improves its design.

Second, some code has high complexity also known as cyclomatic complexity. Arguably this code should be refactored, but projects do have a certain percentage of their code with high cyclomatic complexity that gets carried forward from sprint to sprint.

The third reason is a bit technical. Code like Java is compiled into byte code. The code coverage tools run off of an analysis of the byte code, not the source code. The Java compiler will consume the source code and emit byte code that may have extra logic in it, meaning code with extra branches. It might not be possible to control the conditions which would take one path or the other through this invisible branch. Further complicating this, is that the invisible logic can change from Java compiler release to release, putting a burden on the test logic to reverse engineer the conditions needed to cover this invisible logic.

Summary

Based on the above discussion, achieving 100% line coverage can be very expensive. On teams that I have worked on over the years, a reasonable line coverage would be 70% or more. But you should let the development team determine this limit. If you force your teams to get to 100% line coverage, you are spending money that might be better spent on automation tests. In addition, I have seen cases where developers will short-circuit the unit tests by writing tests only for the purpose of increasing the coverage. You can readily identify these test because they have no assertion or verification check in them - they just make a call and never check on the result.

In short, you should be careful what you ask for. Make sure you interact with the development team in making the decision about code coverage. Spending another 50% of scarce testing dollars on that last 10% coverage is unlikely to bring a return on investment.

Good Reasons to Dockerize Builds

The Problem

On my current team, we use a common tool stack of Jenkins, Maven, Robot Framework, and Java. Our build system is centralized with a couple of dozen projects. We have suffered build and developer friction, namely

• Our version of Jenkins is older than the Internet and cannot be easily upgraded
• Upgrading a plugin/tool risks incompatibility with other plugins/tools
• We have accumulated snippets of shell scripts embedded int POM files
• Developers sometimes use our continuous integration (CI) to do compiles and test code
• We sometimes have trouble with local builds passing and CI builds failing

These and more are the Usual Suspects on almost any build system that has grown over time. Having been on this team for around two years now, I have identified a direction we should take to modernize our build system.

A Solution

Docker and containerized builds have been around since at least 2013. There are some compelling reasons to consider using containerized builds. This is my attempt to clearly enumerate the advantages and to convince my chain-of-command to make the jump. Hopefully this will help any readers to do the same.

Moving to containers means we can deploy onto a cloud with minimal work. This can address scaling issues effectively. Note that some builds will still depend on lab access to local devices, and these dependencies will not scale.

Containerizing the build pipeline means easier upgrading. For example, running a component in its own container isolates it so other containers that depend on it are forced through a standard, explicit interface.

Containerizing the build means better partitioning. So instead of making environments that contain all possible tools and libraries, a container can only use those needed for its specific purpose. This has the side effect of reducing defects due to interacting 3rd party components.

Containerizing the build means a clean reset state. Instead of writing reset scripts, the container is discarded and resurrected from a fixed image. This is a phoenix (burn-it-down) immutable view of containers, and forces all build and configuration to be explicit (not accumulate in snow flake instances).

Containerizing means 100% consistency between local development and the build system, which should eliminate the but-it-works-on-my-machine problem.

Containerizing the build means effective postmortems for build failures, potentially leaving failed runs in the exact state of failure, rather than relying solely on extracted log files.

Containerizing the build means building and installing an intermediate, shared artifact onces, instead of 52 times, and potentially speeds up the build cycle.

Containerizing the build means that some tests can make temporary checkpoints via a temporary image/container and roll back to that state rather than tearing  down and rebuilding, affording a quicker build.

Judicious use of containers might help with diagnosing hard-to-reproduce issues in the field. I have seen instance of technical support sending/receiving VM images to/from customers. Containers would be both simpler and could be a lot smaller.

Containerizing the build is considered a modern best-practice and affords access to all kinds of build work flow and management tools.

That's it! Good luck convincing your management when the time comes to modernize your build systems.

On CheckStyle and Coding Standards

Recently I was given the opportunity to work on an open source project. My team had developed a new additional feature and we were eager to get is pushed upstream before a code freeze milestone. Our team was closed source and project was proprietary, so this effort was essentially to push a portion upstream as open source.

We decided to do three pushes due to various time and resource constraints: push the basic feature, then an enhancement, then some automated integration tests. Our team was not familiar with the process or requirements of the upstream project but two thirds of the code was written under contract to us by a former major upstream committer to that project.

The push of the basic feature was rougher than I thought, given the authorship of much of the code. Most of the issues were stylistic and our reliance on the contractor to "do the right thing" had been a bit shakey. After some corrective actions, the first merge happened. Based on code review issues raised on the first push, I revised the second batch of code, and submitted a pull request. As expected, there were a few minor issues raised which I corrected.

So this project has checkstyle enabled as warnings. But the build is so full of noise and warnings (that is deserving of another rant on its own) that developers tend to ignore warnings, especially those involving things like whitespace, which happens a lot with checkstyle.

So I submitted the pull request and sort of expected an approval and merge. This was highly desirable because the code freeze deadline was only a couple of days away. Instead what I got was a request to move the checkstyle setting from warning level (the project default) to error level.

This was a stunning request for several reasons.

* First, making the code freeze deadline was now unlikely.

* Second, this would be a new requirement on our code that was not met by any of the hundreds of thousands of lines of code that already existed. Plus our previous commit was accepted without that obligation.

* Third, the time to enable something like checkstyle is day zero of a project. Turn it on then and let issues be fixed as part of every single commit. If you have never turned on an analyzer like checkstyle on to an unsuspecting code base, it is dismal. Our commit was probably a few thousand lines of code, but checkstyle emitted about 250 "errors". This would be such gems as "there is a space between your method name and the parameter list". Manually fixing all of these would be tedious and expensive - probably several days of work.

There were a half-dozen "functional" issues that I fixed, like "unused import" and "catching naked exceptions" that were entirely reasonable to ask for a fix. Those functional fixes were done and the new pull request has been awaiting approval for three days now. I guess in open source, not everyone has a sense of urgency. Complicating this, your reviewer might actually work for a competing company, so politics might interlope.

Some observations...

I wanted to point out an issue of open versus closed source. In the past, on smaller closed source teams, we never enforced cosmetic coding standards. Everyone's slightly different coding styles acted as fingerprints so you could tell without looking who the developer was. But in an open source context, where there could be large numbers of developers, it makes a lot of sense to have strict coding standards and enforecment of those standards should be enabled by default.

Readability is subjective. The wars of bracket placement, and other whitespace issues, happen all the time and clearly have no real impact to the business. To me, a perfect solution would be that whitespace is always managed by the tools locally to each developer. For example, a developer's whitespace style is applied to the code base upon "get", does not show up as a "diff", and is removed as part of a "push".

The imperfect solution is the best we have today. You can get plug-ins into IntelliJ or Eclipse that will show you violations in "real time" so that you can take instant corrective actions. Unfortunately there is no comprehensive code reformatting tool to manage style violations. Many of them are amenable to automated fixes, but not all of them.

We have some ways to go before we can have our cake and eat it too. Maybe it will happen when we have spent enough expensive developer time chasing down that stray space in our code-under-review.

A New Gerrit User

Recently I had to commit work upstream to the public Open Daylight project. As part of that effort, I had to learn some new tools, particularly Gerrit. This is a code review/management tool that works with Git, which I was already familiar with.

Here are some ad hoc things I learned as part of committing work upstream...

Gerrit has a different workflow that you are likely using in downstream coding with other tools. There is a general tutorial and an ODL coding introduction to working with Gerrit.

We followed our own coding guidelines downstream and those did not match to the Google coding conventions used upstream. Differences we ran into were:

• Lower-case method names, even for JUnit test methods (I won't push back with Spock conventions) • Using the most restrictive visibility for access qualifiers (I won't push back the philosophy of library design)
• Use Java 8 lambdas everywhere possible

Gerrit (plugin) has some non-obvious controls, namely that you can kick off another build by putting "recheck" as your comment. Others are "rerun", "verify" as in here.

Upstream coders usually add the prefix "WIP: " to their bug report to let other developers know things are not ready for prime time reviews yet.

Mechanically you can review the issues by using "Diff against:" drop list to pick a revision to start at then go into the first file and then use the green upper right arrows to navigate between files. (It'd be nice if Gerrit had a "next feedback item" navigation).

The master branch on ODL can and does break, causing lots of down time with broken builds (days even).

Full builds take 10 hours and verifying-before-pushing is an evolving process.

If you are using "git review" and you forgot the "--amend" option on your commit, you will be prompted to squash and will end up with a new branch. You can recover using "git rebase -i HEAD~2" then using the force option and then abandoning the new branch.

Along with master breaking, master can also produce stale artifacts, so don't assume that clearing your m2 cache, creating a subproject repo and building will give you up-to-date code.

You can search jar files using find . -type f -name '*.jar' -print0 | xargs -0 -I '{}' sh -c 'jar tf {} | grep "YourClassHere" && echo {}' for helping verify your m2 has the latest.

The patch workflow is not very good at all for having multiple people working on an item; traditional git branches and management are superior in this respect so expect some pain.

If your patch won't build because master is broken, you can rebase it to pick up a fix temporarily. When you do a "git review" to push it, you will be prompted because there are two commits (your new patch revision and the fix). You do not want to merge that fix into your patch.

Hopefully, some of these tips will find their way to helping someone else along the way.

The Cost of Separation of Concerns

One of the things I enjoy doing is looking at other developers' code and dissecting it. You can find a lot of advice on how to refactor code (including this blog). One that caught my eye recently was this one. It shows a small C# example and how to refactor/improve it.

Go ahead and read it. It'll take 5 minutes.

I'll wait.

So I agree that those changes were probably improvements to the code. But when I see code, I see layers and coupling and dependencies. So I thought I'd post a refactoring here that is very different than you'll see in most places.

First, the original code...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

public class Invoice { public long Amount { get; set; } public DateTime InvoiceDate { get; set; } public void Add() { try { // Code for adding invoice // Once Invoice has been added , send mail MailMessage mailMessage = new MailMessage ("MailAddressFrom","MailAddressTo","MailSubject","MailBody"); this.SendEmail(mailMessage); } catch (Exception ex) { System.IO.File.WriteAllText(@"c:\Error.txt", ex.ToString()); } } public void Delete() { try { // Code for Delete invoice } catch (Exception ex) { System.IO.File.WriteAllText(@"c:\Error.txt", ex.ToString()); } } public void SendEmail(MailMessage mailMessage) { try { // Code for getting Email setting and send invoice mail } catch (Exception ex) { System.IO.File.WriteAllText(@"c:\Error.txt", ex.ToString()); } } }

And now the original refactored code...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

public class Invoice { public long Amount { get; set; } public DateTime InvoiceDate { get; set; } private FileLogger fileLogger; private MailSender mailSender; public Invoice() { fileLogger = new FileLogger(); mailSender = new MailSender(); } public void Add() { try { fileLogger.Info("Add method Start"); // Code for adding invoice // Once Invoice has been added , send mail mailSender.From = "rakesh.girase@thedigitalgroup.net"; mailSender.To = "customers@digitalgroup.com"; mailSender.Subject = "TestMail"; mailSender.Body = "This is a text mail"; mailSender.SendEmail(); } catch (Exception ex) { fileLogger.Error("Error while Adding Invoice", ex); } } public void Delete() { try { fileLogger.Info("Add Delete Start"); // Code for Delete invoice } catch (Exception ex) { fileLogger.Error("Error while Deleting Invoice", ex); } } } public interface ILogger { void Info(string info); void Debug(string info); void Error(string message, Exception ex); } public class FileLogger : ILogger { public FileLogger() { // Code for initialization i.e. Creating Log file with specified // details } public void Info(string info) { // Code for writing details into text file } public void Debug(string info) { // Code for writing debug information into text file } public void Error(string message, Exception ex) { // Code for writing Error with message and exception detail } } public class MailSender { public string From { get; set; } public string To { get; set; } public string Subject { get; set; } public string Body { get; set; } public void SendEmail() { // Code for sending mail } }

So here is my refactoring. You will see that it is very, very different than the original improved suggested version.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105

interface Invoice { void Add(); void Delete(); } interface BetterInvoice extends Invoice { void Error(String msg, Exception e); } class InvoiceBasics implements Invoice { public void Add() { // code for adding invoice } public void Delete() { // code for deleting invoice } } class InvoiceLogging implements BetterInvoice { InvoiceLogging(Invoice inner) { this.inner = inner; } public void Add() { fileLogger.Info("Add method Start"); inner.Add(); } public void Delete() { fileLogger.Info("Add Delete Start"); inner.Delete(); } public void Error(String msg, Exception e) { fileLogger.Error(msg, e); } } class InvoiceErrorFlow implements Invoice { BetterInvoice inner; InvoiceErrorFlow(BetterInvoice inner) { this.inner = inner; } public void Add() { try { inner.Add(); } catch (Exception ex) { inner.Error("Error while Adding Invoice", ex); } } public void Delete() { try { inner.Delete(); } catch (Exception ex) { inner.Error("Error while Deleting Invoice", ex); } } } class InvoiceMailer implements Invoice { InvoiceMailer(Invoice inner) { this.inner = inner; } public void Add() { inner.Add(); sendMail(); } private void sendMail() { mailSender.From = "rakesh.girase@thedigitalgroup.net"; mailSender.To = "customers@digitalgroup.com"; mailSender.Subject = "TestMail"; mailSender.Body = "This is a text mail"; mailSender.SendEmail(); } public void Delete() { inner.Delete(); } } class InvoiceBuilder { public Invoice create() { return new InvoiceMailer( new InvoiceErrFlow( new InvoiceLogging(new( new InvoiceBasics()))); } }

I bet you weren't expecting that. I have been a developer for over 30 years and it has taken some time to see software the way I do. So lets evaluate this. Is the code better? It depends.

• The code above has separation of logging, error flow, and notification and emphasizes composition instead of inheritance.
• The main business logic (found in InvoiceBasics) is very simple and clean.
• More functionality can be added without disturbing everything else.
• It is far more testable than the original.
• It conforms more to the Single Responsibility Principle than the original.

But there is a cost. There are more classes (that do less). It is probably easier to understand what happens in the original code, where you'll have to think about it with the newly refactored code. It took more time to do than the refactored version, but you gain in testability, flexibility, etc. So you have to take all this into account. You, as the designer, have to decide how much money to spend on this code and how long you expect it to live. It is not always easy to decide.

Becoming a Top Notch Programmer

So you want to be the best developer you can be? Warning - your career needs resemble those of a living plant more than that inert diploma sitting on your wall. Uncle Bob of Clean Coders, said that developers should spend upwards of 20 hours per week in “sharpening the saw”. I’ll let that sink in for a minute – that is half of your normal work week – on top of your real work. Now you know what you’re up against.

So here are my recommendations to anyone who wants to get better…

On Software Craftsmanship

You are paid to make decisions. One of the critical ones is how much money to spend on developing a chunk of logic. Be thoughtful about taking shortcuts because you are just pushing even bigger costs into the future. Those costs will be born by you, your colleagues, and your company. The trick is to "do the right thing" without over-engineering (over spending). Here are some great technical resources you should consider:

Clean Code by Robert Martin (also website has good video series)

Design Patterns by Gamma and Beck

Refactoring by Fowler and Beck

Working Effectively with Legacy Code by Michael Feathers

Look on the Pragmatic Bookshelf for others

On Polyglot Programming

Make it your goal to learn a new programming language each year, even if you aren't going to use it in your job. It will change the way you think about problems.

Really great ones to learn are: Groovy, Javascript, Scala, and Go

On Agile Practices

If you want to find a way to efficiently deal with the unknown and constantly changing demands, you need to fully understand agile.

Phoenix Project by Kim et. al. (you will have a hard time putting this book down)

Pragmatic Programmer by Hunt and Thomas

Practices of an Agile Developer by Venkat Subramaniam

On Career Planning

Don't ignore your career. Don't drift along. Even if you don't have specific goals right now, you should read up on career planning - you'll get lots of ideas. A good place to start is

Passionate Programmer by Chad Fowler

On Learning via Blogs

Spend 20 minutes each day trying to keep up with the flood of great insights and ideas by reading blogs. Here are some of my favorites.

Martin Fowler Bliki (design and abstractions)

Robert Martin’s 8^th Light (software process and practices)

Jeff Atwood’s Coding Horror (hard core rubber meets the road advice)

Petri Kainulainen (software testing)

Yegor Bugayenko (software rebel and contrarian – this will get you out of your rut)

Java at DZone (great for keeping up with the latest)

Scot Hanselman’s Computer Zen (ignore the .NET stuff but lap up all the rest)

Ted Neward’s Interoperability Happens (general software, product, practices)

On Recharging Your Creative Batteries at Conferences

Nothing like mingling with your peers to recharge your batteries and to get exposed to new ideas.

Go to all the conferences that you can fit/afford.

No Fluff Just Stuff in various cities across the country including San Diego (one of my favorite and very affordable)

That's the five minute recommendation. Good luck. Grow. Prosper.

Avoiding Code Coverage Legacy Dilution

How do you keep your weakly tested legacy code from dragging down your coverage numbers for your new code?

If you have the luxury of excluding all the legacy code, you can have your coverage numbers reflect only the new code. You might try running two coverage calculations - one for the legacy and one for the new code. This is doable but cumbersome.

If you want to have a single coverage calculation in your build, dealing with legacy code might mean diluting your coverage numbers. Nobody is going to pay your team to go back and retrofit unit tests to match the standards of the new code.

What to do, what to do?

On my previous team, we ran into this situation. We were using JaCoCo to do coverage calculations and some of the code in our code base was user interface logic and was hard/expensive to test, so it had low coverage. All the server-side logic and shared logic had really good code coverage. We really wanted to treat them differently but did not want to open the door for developers to get lazy and go to the "lowest common denominator".

So what I did was build a tool that allowed us to express flexible coverage limits based on individual or groups of packages and/or classes per coverage metric. The tool took a set of rules as input, the current JaCoCo coverage report, and applied the rules to the metrics, signalling an error if any of the rules were violated.

The simplest rules file contents is shown below. Since this is CSV-based, the first row shows the static column headers. For these examples, we will only talk about LINE coverage, but obviously other coverage metrics are possible.

This single rule says that all packages in-aggregate, with all classes in each package in-aggregate, should have a minimum 70% line coverage.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
*, *, LINE_MISSED, LINE_COVERED, 0.7

The following rules show how you can special-case a single class for its own coverage limit of 40%. The important thing here is that the special class follows one rule and all the rest of the code follows the other rule.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
com.acme, FooBar, LINE_MISSED, LINE_COVERED, 0.4
*, *, LINE_MISSED, LINE_COVERED, 0.7

As a final example, we will specify that all the classes in the special apex package should in-aggregate, have a 50% coverage.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
com.acme, FooBar, LINE_MISSED, LINE_COVERED, 0.4
com.apex, *, LINE_MISSED, LINE_COVERED, 0.5
*, *, LINE_MISSED, LINE_COVERED, 0.7

I will be working on and open sourcing a coverage rules tool very soon that will follow rules like those above (and a few more).

The Greenest of Fields

A greenfield project is a brand new project that is not subject to any legacy requirements and importantly does not deal with legacy code. The opportunity to work on a greenfield project does not come along very often and is generally viewed as a real opportunity by developers. In the excitement, it is tempting to focus on the new design and the new code.

Beyond the technology decisions of the new project, there are things to consider that will have an under appreciated impact on the project. These issues are the subject of this blog posting. If you ignore these issues or make the wrong decision, changing you mind could be very expensive.

Unit Testing and Code Coverage. You should decide before any code is written whether you are going to establish unit tests and track code coverage in your build. Anyone who has a legacy code base and wants to go back and unit test will tell you that there isn't a business in the world that will pay your developers to go back and do this - it is cost prohibitive. Not only that, the existing code base can affect your overall coverage numbers should you start trying to cover you new code and ignore your old code. (I'll actually discussion how to manage this in a forthcoming posting).

Coding Conventions. If your project is to adhere to coding conventions, they should be established early on. Depending on how detailed the conventions are and whether there is any tool support for transforming code, this could be expensive to retrofit once the project has a significant code base. The difficulties are compounded because code that is written to be testable is different (usually better designed) than code that is not, so you wouldn't be able to unit test legacy code without significant refactoring costs.

Language-specific Best Practices. Well beyond issues of formatting, some languages have best practices conventions that should be established early on. An example is "const correctness" in C++. Anyone who has tried to go back and retrofit "const correctness" in a code base knows how invasive this change can be. An effort like this can end up touching almost all of the classes in your project. Of particular insidiousness is that a casual code inspection often does not make the scope of such changes apparent.

Internationalization. On our projects, we always consider internationalization when we write our code, even if it is not an explicit requirement for the project. There are tools that can help sift through source code and refactor the string literals out into a separate module. This is helpful and better than nothing, but it is not as neat as doing this as you write your code. Additionally there are often considerations that make this one-size-fits-all approach suboptimal. For example, in our projects, we allow the logging (and debug information), which frequently is done on the serverside, to remain in English. We use the locale associated with the current session to determine what language the user interface shows to the end user.

Code Rules. There is a lot of tool support these days to analyze source code and determine its adherence to coding rules. A popular tool like this is FindBugs (part of Sonar). Use of this tool should be done very early on. Anyone who has turned this tool onto an existing code base knows the potentially thousands of issues/violations that can be raised. Although the rules used to generate violations can be tuned, it is far better to do this with a very small code base than a large. We turned this tool onto a large code base (say 500K LOC) and had to limit our efforts to address only critical and major issues because of the large volume. So do yourself a favor and activate this tool on all check-ins starting on day zero.

Database Content. This issue is subtle. Our application uses a backend database and to support our functional tests, we stand up test data. The data was captured as a dump. So over time, developers added test data, got their tests going, then took a new dump. The problem with this is that very quickly, nobody could explain why a particular piece of data was in the database. It had become a big tarball of opaque, but required data. Avoid this situation at all costs. You should manage your data just like you do your source code. Each addition to the test data should be identifiable and tracked. Ideally you could stand up a new test database based solely on these individual descriptions. (I'll discuss this more in a forthcoming blog entry).

Technical Debt. The urgency in establishing identification of technical debt is not as high as the other issues identified here. Technical debt is incremental and, for the most part, its cost is incremental. Delaying dealing with technical debt is a mistake for many reasons. Ideally it would be identified, tracked, and continually addressed from day zero. Doing so establishes the expectation on your development team that taking shortcuts is not free of cost and is not invisible. It will cost you in the future at some point.

Hopefully these issues will give you something to consider at your next greenfield opportunity. If you have any other ideas about issues that should be in almost all projects and are hard to retrofit, please leave comments.

Code Coverage Theater

From the title of this entry, you might think that this is going to be a screed that is going to rain down hard on the code coverage parade. Instead it will be some tempering and pragmatic advice taken from the experiences of a mature agile team.

If there is any bit of heresy, this is it - code coverage has no intrinsic value. Its value comes only from showing where there is untested code. All of the value in what we do in this aspect of development is in the quality of the unit tests. Maybe leaving code untested is a deliberate calculation, but maybe it was a mistake. As developers, we are pulled in multiple directions all the time. It can take some effort to get back into context and you can use your uncovered code to help you recover.

The single best way to use code coverage is to break the build. As Agilists, we all are practicing continuous builds, right? I mean a new build goes off with nearly every code commit, right? That frequency is exactly what you should use for running code coverage checks. When the build breaks due to a fall in coverage, you get the advantage of causality. The latest check-in probably broke the build. (Your thoughtful coworkers never check code into a broken build.) It is very important that all developers should be able to run the same coverage checks locally on their own machines that are done on the continuous build server. Thus a broken build due to coverage checks should never be a surprise to anyone.

So if we want to have minimum coverage limits that must be met or the build breaks, what is an appropriate limit? Unfortunately some misguided management mandate that a 100% line coverage be maintained. In my opinion, this is a very poor way to spend your limited resources.

First, not all code coverages are created equal. A 100% line coverage limit does not address the differences between line and branch coverage. Package and class coverage can tell you very coarse stories about the state of your coverage. But line and particularly branch coverage have a lot richer information. You might have 100% line coverage and have very poor branch coverage. Branch coverage can help you figure out if you are covering all the independent paths through the code? Do you even pay attention to cyclomatic complexity?

Second, code coverage tools typically run off the the Java byte code, not the actual source itself. This means that code that the compiler generates for you may not be mappable to any sensible line in your source code. (Ever wonder why the first curly brace after your class name is shown as uncovered?) You can find discussions of this issue on sites like stackoverflow.com. The practical significance of this is that you think you have all the code covered, but the coverage tool shows you odd "unreachable" corners in your source that have no coverage.

Finally, there will be some flows through the code, particularly obtuse error paths, that may be hard to test. Do you really want to spend 50% more effort to get that last 10% coverage? Is this a good use of your money?

There is no real best practice on code coverage limits. On my team, we use about an 80% limit for line and branch coverage. This mostly keeps new code from being checked in without coverage. And it leaves enough slack for the developers to exercise their judgement on whether it is worth chasing those extra few lines of coverage. In a future posting, I'll see if I can find statistics that show the cost of getting that higher coverage and the chances that the tests for the extra coverage actually find or prevent defects.

I work in a Java shop, and we use JaCoCo as our coverage tool. It is also used in Sonar. We used to use Cobertura but I believe at the time that the new Java 7 byte codes confused it. Coverage tools like JaCoCo do a pretty good job generating coverage reports that show you both coverage summaries and allowing you to drill into your packages, classes, and methods to see coverage for individual code lines.

While serviceable, these tools have a serious deficiency - the legacy code problem. You know that problem - the one where you start a code base completely ignoring unit tests and code coverage. Then somewhere along the line, one or more of you get a pang of guilt, see the light, and join modern software best practices. But now you have a problem. Nobody, and I mean nobody, is going to pay you to go back and write tests for coverage for that legacy code. One way to handle this is to just take a punch in the nose and watch your code coverage average be dragged down by a bunch of zeros. I'll cover how we handled this problem in a later blog.

Hopefully this has given you some things to consider in your own projects.

Interviewing

Yep, its another blog/article on interviewing. I just searched on google for "interview programming" and got 21.8M hits. So what's so bad about one more? :)

I wanted to have a few words about interviewing for mid-level and senior developers. I do this for my company as one of my many functions (including software architect and lead developer). The reason I feel compelled to write is disappointment. I just cannot reconcile how good some resumes look and how poor the results are off the paper.

When I interview, I *never* *ever* go the puzzle/trick route. I think those have very little to do with our jobs at best and give misleading results at worst. To me, you should focus where the big cost items are in software - in its (lack of) structure and in its (lack of) flexibility. Does the code smell?

It is how easily the software can be changed over time that determines its real cost.

So when I interview, I first establish that they have good grasp of object orient design fundamentals. Then I focus on the key area of "best practices." Does the candidate understand when they are creating a mess? Do they recognize conceptual pollution when they see it?

To ascertain this, I give them a take home (java) "test." I want them to not feel a great deal of time pressure (when was the last time your boss dropped a programming task in your lap and said it needed to be done in 20 minutes?). I give them overnight to complete the test, but in reality it should only take them 20 minutes.

The actual test is short. It's a two page code sample for them to critique. I tell them that their boss is interested in hiring the developer of that code and boss wants their feedback. The test shows the following kinds of problems:

• embedding strings directly in code
• mixing of abstractions
• blocks of code commented out (with no other explanation as to why it was commented out)
• the use of public virtual functions called in a constructor (a questionable practice)
• complete lack of comments
• errors in relationships (coder assumed 1:1 when its really 1:N)
• missing abstractions (over use of string data)
• over-reliance on open-ended integers (instead of enums)

In all, there are about a dozen issues that could be flagged. I have had several candidates tell me that everything looked good to them. My average score is probably 5 issues. I tend to get interested in a candidate if they score 7 or more. The best I have seen was from a really, really good developer who did it sight-unseen in about 10 minutes and got 14 issues (I give credit if they raise and can defend most technical concerns).

I don't consider this tricky questioning. It basically boils down to me having developers that don't keep me awake at night wondering what messes they are making. I feel that a competent developer ought to be able to get most of this almost by instinct.

So why do 90% of the candidates not make the bar?

Third Life, Fourth Life, ...

I was in a discussion today with a colleague of mine (I'll call him Jim). He was asking my opinion of an important message he put at the end of his application installer. I suggested some minor tweaks, which he appreciated. But I also asked him to emphasize things, possibly by an alert or by an extra dedicated panel (most installers are wizards). His comment was that doing so would only aid maybe 10% more of the users.

What sad but true observation.

I was taken back about a year earlier when I remember standing at the back of a training class watching one of our sales engineers using a product I designed. He was working his way through a guided work flow. When an error message came up (you know, the one with the big red X on it), he couldn't click it away fast enough. Breaking with best practices, I interrupted him asking him what the error message was and he didn't know.

This was truly an eye-opener for me. To him, an interrupting dialog/alert was a nuisance to be disposed of as quickly as possible. It was impeding his work flow. I try to put a lot of effort into the information content of error messages to be as helpful as possible to the end user. Was all that effort wasted? I liken careful error handling to putting guard rails on a winding mountainous road.

Back to my conversation with Jim.

We were discussing possible changes to user interfaces. I remarked that maybe with virtual reality becoming commonplace, we could actually be more effective in informing users of the severity of problems. Maybe a future application would have a user traveling down a path, say a winding mountain path. When a severe error occurs, rather than a dialog, we could actually push them off a virtual cliff. At least the experience would be hard to dismiss and might even be memorable.

Maybe instead of giving the user the Ok/Cancel button of today, such a future application might present a button with the label "Respawn?"

Primitive Types and the Domain Model

I was at the No Fluff Just Stuff conference this year (5th year in a row for me), listening to Neal Ford talk about development concepts. He made a casual remark that caught and held my attention far after he had moved on to other topics. He said

Never use primitive types in your domain model

As you veteran developers know, the core of many well-built applications is a clear, concise domain model. This forms the basis for key abstractions and is the framework around which you typically hang your business logic. Most of my experience over the years has been with developing large, complex desktop applications where a good domain model is key. I assume that for medium-to-large enterprise web applications, a good domain model is also highly valued.

So what exactly does the quote above mean? I'll give you some examples. The first is from one of our early domain models where we represented network topology elements (computers, routers, etc). One of the data members of our network representation was bandwidth, as in set-the-bandwidth-of-this-network-to-X. We assumed that the number was megabits per second. Of course, we wrote some independent utility scaling routines to go to kilobits per second and gigabits per second.

Do you see the inferior design? Of course, we should never have used a primitive type (like double) to represent bandwidth. What we should have done was create a Bandwidth class and passed this to the "set" and "get" routines. This class would have had the concept of units (possibly a class like BitRate) and would have encapsulated all the conversion routines in one nice package. Maybe you would find lines of code like

double capacity = bw.inMegabitsPerSec();

And importantly constructed like this

Bandwidth bw = new Bandwidth (3.5, MegabitsPerSec);

That is so much easier to read and is far clearer than

double bw = 3.5;

A second example comes from some code that I trapsed across not a month ago. We have a switchboard-like, sockets-based server that handles communications between applications within a given suite. In sending a message through this switchboard to another application, you identify the target application by a name. The application names and versions that form a suite come from a configuration file. We allow aliases/monikers for the applications as well as trying not to be case sensitive. Part of this is convenience, and part of it is because the applications are written in diverse technologies (C++, Perl, Java, C#).

I was looking at this code for the first time, and I saw a lot of string manipulation going on. There was trimming (of spaces), folding of cases, and substring matching -a lot of it. In tracing through the code, a string that was already lower case, was set to lower case again. Clearly somewhere along the line, the programmer (or programmers) lost track of the flow of the data in the application, where messages were entering the system. The string hacking was so prolific that it made the code harder to read.

So what was missing here? Instead of using a string primitive type, how about a class called ApplicationName. It would encapsulate all the concepts of case insensitivity and of aliases. I even found code later on that emitted information to logs and to the end user, so it could even encapsulate the concept of "pretty name" (a standard, human-consumable name).

I know what a few of you are thinking. You are concerned about the performance cost of allocating all those instances. With modern optimizing compilers, you'll find that performance costs won't really be an issue. The cost that will eat your lunch is the maintenance cost for the software over time.

So the next time you are working in your domain model, and you feel the urge to use an int, double, string, or other primitive type, think again.