Becoming a Top Notch Programmer

So you want to be the best developer you can be? Warning - your career needs resemble those of a living plant more than that inert diploma sitting on your wall. Uncle Bob of Clean Coders, said that developers should spend upwards of 20 hours per week in “sharpening the saw”. I’ll let that sink in for a minute – that is half of your normal work week – on top of your real work. Now you know what you’re up against.

So here are my recommendations to anyone who wants to get better…

On Software Craftsmanship

You are paid to make decisions. One of the critical ones is how much money to spend on developing a chunk of logic. Be thoughtful about taking shortcuts because you are just pushing even bigger costs into the future. Those costs will be born by you, your colleagues, and your company. The trick is to "do the right thing" without over-engineering (over spending). Here are some great technical resources you should consider:

Clean Code by Robert Martin (also website has good video series)

Design Patterns by Gamma and Beck

Refactoring by Fowler and Beck

Working Effectively with Legacy Code by Michael Feathers

Look on the Pragmatic Bookshelf for others

On Polyglot Programming

Make it your goal to learn a new programming language each year, even if you aren't going to use it in your job. It will change the way you think about problems.

Really great ones to learn are: Groovy, Javascript, Scala, and Go

On Agile Practices

If you want to find a way to efficiently deal with the unknown and constantly changing demands, you need to fully understand agile.

Phoenix Project by Kim et. al. (you will have a hard time putting this book down)

Pragmatic Programmer by Hunt and Thomas

Practices of an Agile Developer by Venkat Subramaniam

On Career Planning

Don't ignore your career. Don't drift along. Even if you don't have specific goals right now, you should read up on career planning - you'll get lots of ideas. A good place to start is

Passionate Programmer by Chad Fowler

On Learning via Blogs

Spend 20 minutes each day trying to keep up with the flood of great insights and ideas by reading blogs. Here are some of my favorites.

Martin Fowler Bliki (design and abstractions)

Robert Martin’s 8^th Light (software process and practices)

Jeff Atwood’s Coding Horror (hard core rubber meets the road advice)

Petri Kainulainen (software testing)

Yegor Bugayenko (software rebel and contrarian – this will get you out of your rut)

Java at DZone (great for keeping up with the latest)

Scot Hanselman’s Computer Zen (ignore the .NET stuff but lap up all the rest)

Ted Neward’s Interoperability Happens (general software, product, practices)

On Recharging Your Creative Batteries at Conferences

Nothing like mingling with your peers to recharge your batteries and to get exposed to new ideas.

Go to all the conferences that you can fit/afford.

No Fluff Just Stuff in various cities across the country including San Diego (one of my favorite and very affordable)

That's the five minute recommendation. Good luck. Grow. Prosper.

Avoiding Code Coverage Legacy Dilution

How do you keep your weakly tested legacy code from dragging down your coverage numbers for your new code?

If you have the luxury of excluding all the legacy code, you can have your coverage numbers reflect only the new code. You might try running two coverage calculations - one for the legacy and one for the new code. This is doable but cumbersome.

If you want to have a single coverage calculation in your build, dealing with legacy code might mean diluting your coverage numbers. Nobody is going to pay your team to go back and retrofit unit tests to match the standards of the new code.

What to do, what to do?

On my previous team, we ran into this situation. We were using JaCoCo to do coverage calculations and some of the code in our code base was user interface logic and was hard/expensive to test, so it had low coverage. All the server-side logic and shared logic had really good code coverage. We really wanted to treat them differently but did not want to open the door for developers to get lazy and go to the "lowest common denominator".

So what I did was build a tool that allowed us to express flexible coverage limits based on individual or groups of packages and/or classes per coverage metric. The tool took a set of rules as input, the current JaCoCo coverage report, and applied the rules to the metrics, signalling an error if any of the rules were violated.

The simplest rules file contents is shown below. Since this is CSV-based, the first row shows the static column headers. For these examples, we will only talk about LINE coverage, but obviously other coverage metrics are possible.

This single rule says that all packages in-aggregate, with all classes in each package in-aggregate, should have a minimum 70% line coverage.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
*, *, LINE_MISSED, LINE_COVERED, 0.7

The following rules show how you can special-case a single class for its own coverage limit of 40%. The important thing here is that the special class follows one rule and all the rest of the code follows the other rule.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
com.acme, FooBar, LINE_MISSED, LINE_COVERED, 0.4
*, *, LINE_MISSED, LINE_COVERED, 0.7

As a final example, we will specify that all the classes in the special apex package should in-aggregate, have a 50% coverage.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
com.acme, FooBar, LINE_MISSED, LINE_COVERED, 0.4
com.apex, *, LINE_MISSED, LINE_COVERED, 0.5
*, *, LINE_MISSED, LINE_COVERED, 0.7

I will be working on and open sourcing a coverage rules tool very soon that will follow rules like those above (and a few more).

The Greenest of Fields

A greenfield project is a brand new project that is not subject to any legacy requirements and importantly does not deal with legacy code. The opportunity to work on a greenfield project does not come along very often and is generally viewed as a real opportunity by developers. In the excitement, it is tempting to focus on the new design and the new code.

Beyond the technology decisions of the new project, there are things to consider that will have an under appreciated impact on the project. These issues are the subject of this blog posting. If you ignore these issues or make the wrong decision, changing you mind could be very expensive.

Unit Testing and Code Coverage. You should decide before any code is written whether you are going to establish unit tests and track code coverage in your build. Anyone who has a legacy code base and wants to go back and unit test will tell you that there isn't a business in the world that will pay your developers to go back and do this - it is cost prohibitive. Not only that, the existing code base can affect your overall coverage numbers should you start trying to cover you new code and ignore your old code. (I'll actually discussion how to manage this in a forthcoming posting).

Coding Conventions. If your project is to adhere to coding conventions, they should be established early on. Depending on how detailed the conventions are and whether there is any tool support for transforming code, this could be expensive to retrofit once the project has a significant code base. The difficulties are compounded because code that is written to be testable is different (usually better designed) than code that is not, so you wouldn't be able to unit test legacy code without significant refactoring costs.

Language-specific Best Practices. Well beyond issues of formatting, some languages have best practices conventions that should be established early on. An example is "const correctness" in C++. Anyone who has tried to go back and retrofit "const correctness" in a code base knows how invasive this change can be. An effort like this can end up touching almost all of the classes in your project. Of particular insidiousness is that a casual code inspection often does not make the scope of such changes apparent.

Internationalization. On our projects, we always consider internationalization when we write our code, even if it is not an explicit requirement for the project. There are tools that can help sift through source code and refactor the string literals out into a separate module. This is helpful and better than nothing, but it is not as neat as doing this as you write your code. Additionally there are often considerations that make this one-size-fits-all approach suboptimal. For example, in our projects, we allow the logging (and debug information), which frequently is done on the serverside, to remain in English. We use the locale associated with the current session to determine what language the user interface shows to the end user.

Code Rules. There is a lot of tool support these days to analyze source code and determine its adherence to coding rules. A popular tool like this is FindBugs (part of Sonar). Use of this tool should be done very early on. Anyone who has turned this tool onto an existing code base knows the potentially thousands of issues/violations that can be raised. Although the rules used to generate violations can be tuned, it is far better to do this with a very small code base than a large. We turned this tool onto a large code base (say 500K LOC) and had to limit our efforts to address only critical and major issues because of the large volume. So do yourself a favor and activate this tool on all check-ins starting on day zero.

Database Content. This issue is subtle. Our application uses a backend database and to support our functional tests, we stand up test data. The data was captured as a dump. So over time, developers added test data, got their tests going, then took a new dump. The problem with this is that very quickly, nobody could explain why a particular piece of data was in the database. It had become a big tarball of opaque, but required data. Avoid this situation at all costs. You should manage your data just like you do your source code. Each addition to the test data should be identifiable and tracked. Ideally you could stand up a new test database based solely on these individual descriptions. (I'll discuss this more in a forthcoming blog entry).

Technical Debt. The urgency in establishing identification of technical debt is not as high as the other issues identified here. Technical debt is incremental and, for the most part, its cost is incremental. Delaying dealing with technical debt is a mistake for many reasons. Ideally it would be identified, tracked, and continually addressed from day zero. Doing so establishes the expectation on your development team that taking shortcuts is not free of cost and is not invisible. It will cost you in the future at some point.

Hopefully these issues will give you something to consider at your next greenfield opportunity. If you have any other ideas about issues that should be in almost all projects and are hard to retrofit, please leave comments.

Code Coverage Theater

From the title of this entry, you might think that this is going to be a screed that is going to rain down hard on the code coverage parade. Instead it will be some tempering and pragmatic advice taken from the experiences of a mature agile team.

If there is any bit of heresy, this is it - code coverage has no intrinsic value. Its value comes only from showing where there is untested code. All of the value in what we do in this aspect of development is in the quality of the unit tests. Maybe leaving code untested is a deliberate calculation, but maybe it was a mistake. As developers, we are pulled in multiple directions all the time. It can take some effort to get back into context and you can use your uncovered code to help you recover.

The single best way to use code coverage is to break the build. As Agilists, we all are practicing continuous builds, right? I mean a new build goes off with nearly every code commit, right? That frequency is exactly what you should use for running code coverage checks. When the build breaks due to a fall in coverage, you get the advantage of causality. The latest check-in probably broke the build. (Your thoughtful coworkers never check code into a broken build.) It is very important that all developers should be able to run the same coverage checks locally on their own machines that are done on the continuous build server. Thus a broken build due to coverage checks should never be a surprise to anyone.

So if we want to have minimum coverage limits that must be met or the build breaks, what is an appropriate limit? Unfortunately some misguided management mandate that a 100% line coverage be maintained. In my opinion, this is a very poor way to spend your limited resources.

First, not all code coverages are created equal. A 100% line coverage limit does not address the differences between line and branch coverage. Package and class coverage can tell you very coarse stories about the state of your coverage. But line and particularly branch coverage have a lot richer information. You might have 100% line coverage and have very poor branch coverage. Branch coverage can help you figure out if you are covering all the independent paths through the code? Do you even pay attention to cyclomatic complexity?

Second, code coverage tools typically run off the the Java byte code, not the actual source itself. This means that code that the compiler generates for you may not be mappable to any sensible line in your source code. (Ever wonder why the first curly brace after your class name is shown as uncovered?) You can find discussions of this issue on sites like stackoverflow.com. The practical significance of this is that you think you have all the code covered, but the coverage tool shows you odd "unreachable" corners in your source that have no coverage.

Finally, there will be some flows through the code, particularly obtuse error paths, that may be hard to test. Do you really want to spend 50% more effort to get that last 10% coverage? Is this a good use of your money?

There is no real best practice on code coverage limits. On my team, we use about an 80% limit for line and branch coverage. This mostly keeps new code from being checked in without coverage. And it leaves enough slack for the developers to exercise their judgement on whether it is worth chasing those extra few lines of coverage. In a future posting, I'll see if I can find statistics that show the cost of getting that higher coverage and the chances that the tests for the extra coverage actually find or prevent defects.

I work in a Java shop, and we use JaCoCo as our coverage tool. It is also used in Sonar. We used to use Cobertura but I believe at the time that the new Java 7 byte codes confused it. Coverage tools like JaCoCo do a pretty good job generating coverage reports that show you both coverage summaries and allowing you to drill into your packages, classes, and methods to see coverage for individual code lines.

While serviceable, these tools have a serious deficiency - the legacy code problem. You know that problem - the one where you start a code base completely ignoring unit tests and code coverage. Then somewhere along the line, one or more of you get a pang of guilt, see the light, and join modern software best practices. But now you have a problem. Nobody, and I mean nobody, is going to pay you to go back and write tests for coverage for that legacy code. One way to handle this is to just take a punch in the nose and watch your code coverage average be dragged down by a bunch of zeros. I'll cover how we handled this problem in a later blog.

Hopefully this has given you some things to consider in your own projects.

Interviewing

Yep, its another blog/article on interviewing. I just searched on google for "interview programming" and got 21.8M hits. So what's so bad about one more? :)

I wanted to have a few words about interviewing for mid-level and senior developers. I do this for my company as one of my many functions (including software architect and lead developer). The reason I feel compelled to write is disappointment. I just cannot reconcile how good some resumes look and how poor the results are off the paper.

When I interview, I *never* *ever* go the puzzle/trick route. I think those have very little to do with our jobs at best and give misleading results at worst. To me, you should focus where the big cost items are in software - in its (lack of) structure and in its (lack of) flexibility. Does the code smell?

It is how easily the software can be changed over time that determines its real cost.

So when I interview, I first establish that they have good grasp of object orient design fundamentals. Then I focus on the key area of "best practices." Does the candidate understand when they are creating a mess? Do they recognize conceptual pollution when they see it?

To ascertain this, I give them a take home (java) "test." I want them to not feel a great deal of time pressure (when was the last time your boss dropped a programming task in your lap and said it needed to be done in 20 minutes?). I give them overnight to complete the test, but in reality it should only take them 20 minutes.

The actual test is short. It's a two page code sample for them to critique. I tell them that their boss is interested in hiring the developer of that code and boss wants their feedback. The test shows the following kinds of problems:

• embedding strings directly in code
• mixing of abstractions
• blocks of code commented out (with no other explanation as to why it was commented out)
• the use of public virtual functions called in a constructor (a questionable practice)
• complete lack of comments
• errors in relationships (coder assumed 1:1 when its really 1:N)
• missing abstractions (over use of string data)
• over-reliance on open-ended integers (instead of enums)

In all, there are about a dozen issues that could be flagged. I have had several candidates tell me that everything looked good to them. My average score is probably 5 issues. I tend to get interested in a candidate if they score 7 or more. The best I have seen was from a really, really good developer who did it sight-unseen in about 10 minutes and got 14 issues (I give credit if they raise and can defend most technical concerns).

I don't consider this tricky questioning. It basically boils down to me having developers that don't keep me awake at night wondering what messes they are making. I feel that a competent developer ought to be able to get most of this almost by instinct.

So why do 90% of the candidates not make the bar?

Third Life, Fourth Life, ...

I was in a discussion today with a colleague of mine (I'll call him Jim). He was asking my opinion of an important message he put at the end of his application installer. I suggested some minor tweaks, which he appreciated. But I also asked him to emphasize things, possibly by an alert or by an extra dedicated panel (most installers are wizards). His comment was that doing so would only aid maybe 10% more of the users.

What sad but true observation.

I was taken back about a year earlier when I remember standing at the back of a training class watching one of our sales engineers using a product I designed. He was working his way through a guided work flow. When an error message came up (you know, the one with the big red X on it), he couldn't click it away fast enough. Breaking with best practices, I interrupted him asking him what the error message was and he didn't know.

This was truly an eye-opener for me. To him, an interrupting dialog/alert was a nuisance to be disposed of as quickly as possible. It was impeding his work flow. I try to put a lot of effort into the information content of error messages to be as helpful as possible to the end user. Was all that effort wasted? I liken careful error handling to putting guard rails on a winding mountainous road.

Back to my conversation with Jim.

We were discussing possible changes to user interfaces. I remarked that maybe with virtual reality becoming commonplace, we could actually be more effective in informing users of the severity of problems. Maybe a future application would have a user traveling down a path, say a winding mountain path. When a severe error occurs, rather than a dialog, we could actually push them off a virtual cliff. At least the experience would be hard to dismiss and might even be memorable.

Maybe instead of giving the user the Ok/Cancel button of today, such a future application might present a button with the label "Respawn?"

Primitive Types and the Domain Model

I was at the No Fluff Just Stuff conference this year (5th year in a row for me), listening to Neal Ford talk about development concepts. He made a casual remark that caught and held my attention far after he had moved on to other topics. He said

Never use primitive types in your domain model

As you veteran developers know, the core of many well-built applications is a clear, concise domain model. This forms the basis for key abstractions and is the framework around which you typically hang your business logic. Most of my experience over the years has been with developing large, complex desktop applications where a good domain model is key. I assume that for medium-to-large enterprise web applications, a good domain model is also highly valued.

So what exactly does the quote above mean? I'll give you some examples. The first is from one of our early domain models where we represented network topology elements (computers, routers, etc). One of the data members of our network representation was bandwidth, as in set-the-bandwidth-of-this-network-to-X. We assumed that the number was megabits per second. Of course, we wrote some independent utility scaling routines to go to kilobits per second and gigabits per second.

Do you see the inferior design? Of course, we should never have used a primitive type (like double) to represent bandwidth. What we should have done was create a Bandwidth class and passed this to the "set" and "get" routines. This class would have had the concept of units (possibly a class like BitRate) and would have encapsulated all the conversion routines in one nice package. Maybe you would find lines of code like

double capacity = bw.inMegabitsPerSec();

And importantly constructed like this

Bandwidth bw = new Bandwidth (3.5, MegabitsPerSec);

That is so much easier to read and is far clearer than

double bw = 3.5;

A second example comes from some code that I trapsed across not a month ago. We have a switchboard-like, sockets-based server that handles communications between applications within a given suite. In sending a message through this switchboard to another application, you identify the target application by a name. The application names and versions that form a suite come from a configuration file. We allow aliases/monikers for the applications as well as trying not to be case sensitive. Part of this is convenience, and part of it is because the applications are written in diverse technologies (C++, Perl, Java, C#).

I was looking at this code for the first time, and I saw a lot of string manipulation going on. There was trimming (of spaces), folding of cases, and substring matching -a lot of it. In tracing through the code, a string that was already lower case, was set to lower case again. Clearly somewhere along the line, the programmer (or programmers) lost track of the flow of the data in the application, where messages were entering the system. The string hacking was so prolific that it made the code harder to read.

So what was missing here? Instead of using a string primitive type, how about a class called ApplicationName. It would encapsulate all the concepts of case insensitivity and of aliases. I even found code later on that emitted information to logs and to the end user, so it could even encapsulate the concept of "pretty name" (a standard, human-consumable name).

I know what a few of you are thinking. You are concerned about the performance cost of allocating all those instances. With modern optimizing compilers, you'll find that performance costs won't really be an issue. The cost that will eat your lunch is the maintenance cost for the software over time.

So the next time you are working in your domain model, and you feel the urge to use an int, double, string, or other primitive type, think again.