The Cost of Separation of Concerns

One of the things I enjoy doing is looking at other developers' code and dissecting it. You can find a lot of advice on how to refactor code (including this blog). One that caught my eye recently was this one. It shows a small C# example and how to refactor/improve it.

Go ahead and read it. It'll take 5 minutes.

I'll wait.

So I agree that those changes were probably improvements to the code. But when I see code, I see layers and coupling and dependencies. So I thought I'd post a refactoring here that is very different than you'll see in most places.

First, the original code...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

public class Invoice { public long Amount { get; set; } public DateTime InvoiceDate { get; set; } public void Add() { try { // Code for adding invoice // Once Invoice has been added , send mail MailMessage mailMessage = new MailMessage ("MailAddressFrom","MailAddressTo","MailSubject","MailBody"); this.SendEmail(mailMessage); } catch (Exception ex) { System.IO.File.WriteAllText(@"c:\Error.txt", ex.ToString()); } } public void Delete() { try { // Code for Delete invoice } catch (Exception ex) { System.IO.File.WriteAllText(@"c:\Error.txt", ex.ToString()); } } public void SendEmail(MailMessage mailMessage) { try { // Code for getting Email setting and send invoice mail } catch (Exception ex) { System.IO.File.WriteAllText(@"c:\Error.txt", ex.ToString()); } } }

And now the original refactored code...

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

public class Invoice { public long Amount { get; set; } public DateTime InvoiceDate { get; set; } private FileLogger fileLogger; private MailSender mailSender; public Invoice() { fileLogger = new FileLogger(); mailSender = new MailSender(); } public void Add() { try { fileLogger.Info("Add method Start"); // Code for adding invoice // Once Invoice has been added , send mail mailSender.From = "rakesh.girase@thedigitalgroup.net"; mailSender.To = "customers@digitalgroup.com"; mailSender.Subject = "TestMail"; mailSender.Body = "This is a text mail"; mailSender.SendEmail(); } catch (Exception ex) { fileLogger.Error("Error while Adding Invoice", ex); } } public void Delete() { try { fileLogger.Info("Add Delete Start"); // Code for Delete invoice } catch (Exception ex) { fileLogger.Error("Error while Deleting Invoice", ex); } } } public interface ILogger { void Info(string info); void Debug(string info); void Error(string message, Exception ex); } public class FileLogger : ILogger { public FileLogger() { // Code for initialization i.e. Creating Log file with specified // details } public void Info(string info) { // Code for writing details into text file } public void Debug(string info) { // Code for writing debug information into text file } public void Error(string message, Exception ex) { // Code for writing Error with message and exception detail } } public class MailSender { public string From { get; set; } public string To { get; set; } public string Subject { get; set; } public string Body { get; set; } public void SendEmail() { // Code for sending mail } }

So here is my refactoring. You will see that it is very, very different than the original improved suggested version.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105

interface Invoice { void Add(); void Delete(); } interface BetterInvoice extends Invoice { void Error(String msg, Exception e); } class InvoiceBasics implements Invoice { public void Add() { // code for adding invoice } public void Delete() { // code for deleting invoice } } class InvoiceLogging implements BetterInvoice { InvoiceLogging(Invoice inner) { this.inner = inner; } public void Add() { fileLogger.Info("Add method Start"); inner.Add(); } public void Delete() { fileLogger.Info("Add Delete Start"); inner.Delete(); } public void Error(String msg, Exception e) { fileLogger.Error(msg, e); } } class InvoiceErrorFlow implements Invoice { BetterInvoice inner; InvoiceErrorFlow(BetterInvoice inner) { this.inner = inner; } public void Add() { try { inner.Add(); } catch (Exception ex) { inner.Error("Error while Adding Invoice", ex); } } public void Delete() { try { inner.Delete(); } catch (Exception ex) { inner.Error("Error while Deleting Invoice", ex); } } } class InvoiceMailer implements Invoice { InvoiceMailer(Invoice inner) { this.inner = inner; } public void Add() { inner.Add(); sendMail(); } private void sendMail() { mailSender.From = "rakesh.girase@thedigitalgroup.net"; mailSender.To = "customers@digitalgroup.com"; mailSender.Subject = "TestMail"; mailSender.Body = "This is a text mail"; mailSender.SendEmail(); } public void Delete() { inner.Delete(); } } class InvoiceBuilder { public Invoice create() { return new InvoiceMailer( new InvoiceErrFlow( new InvoiceLogging(new( new InvoiceBasics()))); } }

I bet you weren't expecting that. I have been a developer for over 30 years and it has taken some time to see software the way I do. So lets evaluate this. Is the code better? It depends.

• The code above has separation of logging, error flow, and notification and emphasizes composition instead of inheritance.
• The main business logic (found in InvoiceBasics) is very simple and clean.
• More functionality can be added without disturbing everything else.
• It is far more testable than the original.
• It conforms more to the Single Responsibility Principle than the original.

But there is a cost. There are more classes (that do less). It is probably easier to understand what happens in the original code, where you'll have to think about it with the newly refactored code. It took more time to do than the refactored version, but you gain in testability, flexibility, etc. So you have to take all this into account. You, as the designer, have to decide how much money to spend on this code and how long you expect it to live. It is not always easy to decide.

Becoming a Top Notch Programmer

So you want to be the best developer you can be? Warning - your career needs resemble those of a living plant more than that inert diploma sitting on your wall. Uncle Bob of Clean Coders, said that developers should spend upwards of 20 hours per week in “sharpening the saw”. I’ll let that sink in for a minute – that is half of your normal work week – on top of your real work. Now you know what you’re up against.

So here are my recommendations to anyone who wants to get better…

On Software Craftsmanship

You are paid to make decisions. One of the critical ones is how much money to spend on developing a chunk of logic. Be thoughtful about taking shortcuts because you are just pushing even bigger costs into the future. Those costs will be born by you, your colleagues, and your company. The trick is to "do the right thing" without over-engineering (over spending). Here are some great technical resources you should consider:

Clean Code by Robert Martin (also website has good video series)

Design Patterns by Gamma and Beck

Refactoring by Fowler and Beck

Working Effectively with Legacy Code by Michael Feathers

Look on the Pragmatic Bookshelf for others

On Polyglot Programming

Make it your goal to learn a new programming language each year, even if you aren't going to use it in your job. It will change the way you think about problems.

Really great ones to learn are: Groovy, Javascript, Scala, and Go

On Agile Practices

If you want to find a way to efficiently deal with the unknown and constantly changing demands, you need to fully understand agile.

Phoenix Project by Kim et. al. (you will have a hard time putting this book down)

Pragmatic Programmer by Hunt and Thomas

Practices of an Agile Developer by Venkat Subramaniam

On Career Planning

Don't ignore your career. Don't drift along. Even if you don't have specific goals right now, you should read up on career planning - you'll get lots of ideas. A good place to start is

Passionate Programmer by Chad Fowler

On Learning via Blogs

Spend 20 minutes each day trying to keep up with the flood of great insights and ideas by reading blogs. Here are some of my favorites.

Martin Fowler Bliki (design and abstractions)

Robert Martin’s 8^th Light (software process and practices)

Jeff Atwood’s Coding Horror (hard core rubber meets the road advice)

Petri Kainulainen (software testing)

Yegor Bugayenko (software rebel and contrarian – this will get you out of your rut)

Java at DZone (great for keeping up with the latest)

Scot Hanselman’s Computer Zen (ignore the .NET stuff but lap up all the rest)

Ted Neward’s Interoperability Happens (general software, product, practices)

On Recharging Your Creative Batteries at Conferences

Nothing like mingling with your peers to recharge your batteries and to get exposed to new ideas.

Go to all the conferences that you can fit/afford.

No Fluff Just Stuff in various cities across the country including San Diego (one of my favorite and very affordable)

That's the five minute recommendation. Good luck. Grow. Prosper.

Avoiding Code Coverage Legacy Dilution

How do you keep your weakly tested legacy code from dragging down your coverage numbers for your new code?

If you have the luxury of excluding all the legacy code, you can have your coverage numbers reflect only the new code. You might try running two coverage calculations - one for the legacy and one for the new code. This is doable but cumbersome.

If you want to have a single coverage calculation in your build, dealing with legacy code might mean diluting your coverage numbers. Nobody is going to pay your team to go back and retrofit unit tests to match the standards of the new code.

What to do, what to do?

On my previous team, we ran into this situation. We were using JaCoCo to do coverage calculations and some of the code in our code base was user interface logic and was hard/expensive to test, so it had low coverage. All the server-side logic and shared logic had really good code coverage. We really wanted to treat them differently but did not want to open the door for developers to get lazy and go to the "lowest common denominator".

So what I did was build a tool that allowed us to express flexible coverage limits based on individual or groups of packages and/or classes per coverage metric. The tool took a set of rules as input, the current JaCoCo coverage report, and applied the rules to the metrics, signalling an error if any of the rules were violated.

The simplest rules file contents is shown below. Since this is CSV-based, the first row shows the static column headers. For these examples, we will only talk about LINE coverage, but obviously other coverage metrics are possible.

This single rule says that all packages in-aggregate, with all classes in each package in-aggregate, should have a minimum 70% line coverage.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
*, *, LINE_MISSED, LINE_COVERED, 0.7

The following rules show how you can special-case a single class for its own coverage limit of 40%. The important thing here is that the special class follows one rule and all the rest of the code follows the other rule.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
com.acme, FooBar, LINE_MISSED, LINE_COVERED, 0.4
*, *, LINE_MISSED, LINE_COVERED, 0.7

As a final example, we will specify that all the classes in the special apex package should in-aggregate, have a 50% coverage.

PACKAGE, CLASS, ABC_MISSED, ABC_COVERED, LIMIT
com.acme, FooBar, LINE_MISSED, LINE_COVERED, 0.4
com.apex, *, LINE_MISSED, LINE_COVERED, 0.5
*, *, LINE_MISSED, LINE_COVERED, 0.7

I will be working on and open sourcing a coverage rules tool very soon that will follow rules like those above (and a few more).

The Greenest of Fields

A greenfield project is a brand new project that is not subject to any legacy requirements and importantly does not deal with legacy code. The opportunity to work on a greenfield project does not come along very often and is generally viewed as a real opportunity by developers. In the excitement, it is tempting to focus on the new design and the new code.

Beyond the technology decisions of the new project, there are things to consider that will have an under appreciated impact on the project. These issues are the subject of this blog posting. If you ignore these issues or make the wrong decision, changing you mind could be very expensive.

Unit Testing and Code Coverage. You should decide before any code is written whether you are going to establish unit tests and track code coverage in your build. Anyone who has a legacy code base and wants to go back and unit test will tell you that there isn't a business in the world that will pay your developers to go back and do this - it is cost prohibitive. Not only that, the existing code base can affect your overall coverage numbers should you start trying to cover you new code and ignore your old code. (I'll actually discussion how to manage this in a forthcoming posting).

Coding Conventions. If your project is to adhere to coding conventions, they should be established early on. Depending on how detailed the conventions are and whether there is any tool support for transforming code, this could be expensive to retrofit once the project has a significant code base. The difficulties are compounded because code that is written to be testable is different (usually better designed) than code that is not, so you wouldn't be able to unit test legacy code without significant refactoring costs.

Language-specific Best Practices. Well beyond issues of formatting, some languages have best practices conventions that should be established early on. An example is "const correctness" in C++. Anyone who has tried to go back and retrofit "const correctness" in a code base knows how invasive this change can be. An effort like this can end up touching almost all of the classes in your project. Of particular insidiousness is that a casual code inspection often does not make the scope of such changes apparent.

Internationalization. On our projects, we always consider internationalization when we write our code, even if it is not an explicit requirement for the project. There are tools that can help sift through source code and refactor the string literals out into a separate module. This is helpful and better than nothing, but it is not as neat as doing this as you write your code. Additionally there are often considerations that make this one-size-fits-all approach suboptimal. For example, in our projects, we allow the logging (and debug information), which frequently is done on the serverside, to remain in English. We use the locale associated with the current session to determine what language the user interface shows to the end user.

Code Rules. There is a lot of tool support these days to analyze source code and determine its adherence to coding rules. A popular tool like this is FindBugs (part of Sonar). Use of this tool should be done very early on. Anyone who has turned this tool onto an existing code base knows the potentially thousands of issues/violations that can be raised. Although the rules used to generate violations can be tuned, it is far better to do this with a very small code base than a large. We turned this tool onto a large code base (say 500K LOC) and had to limit our efforts to address only critical and major issues because of the large volume. So do yourself a favor and activate this tool on all check-ins starting on day zero.

Database Content. This issue is subtle. Our application uses a backend database and to support our functional tests, we stand up test data. The data was captured as a dump. So over time, developers added test data, got their tests going, then took a new dump. The problem with this is that very quickly, nobody could explain why a particular piece of data was in the database. It had become a big tarball of opaque, but required data. Avoid this situation at all costs. You should manage your data just like you do your source code. Each addition to the test data should be identifiable and tracked. Ideally you could stand up a new test database based solely on these individual descriptions. (I'll discuss this more in a forthcoming blog entry).

Technical Debt. The urgency in establishing identification of technical debt is not as high as the other issues identified here. Technical debt is incremental and, for the most part, its cost is incremental. Delaying dealing with technical debt is a mistake for many reasons. Ideally it would be identified, tracked, and continually addressed from day zero. Doing so establishes the expectation on your development team that taking shortcuts is not free of cost and is not invisible. It will cost you in the future at some point.

Hopefully these issues will give you something to consider at your next greenfield opportunity. If you have any other ideas about issues that should be in almost all projects and are hard to retrofit, please leave comments.

Code Coverage Theater

From the title of this entry, you might think that this is going to be a screed that is going to rain down hard on the code coverage parade. Instead it will be some tempering and pragmatic advice taken from the experiences of a mature agile team.

If there is any bit of heresy, this is it - code coverage has no intrinsic value. Its value comes only from showing where there is untested code. All of the value in what we do in this aspect of development is in the quality of the unit tests. Maybe leaving code untested is a deliberate calculation, but maybe it was a mistake. As developers, we are pulled in multiple directions all the time. It can take some effort to get back into context and you can use your uncovered code to help you recover.

The single best way to use code coverage is to break the build. As Agilists, we all are practicing continuous builds, right? I mean a new build goes off with nearly every code commit, right? That frequency is exactly what you should use for running code coverage checks. When the build breaks due to a fall in coverage, you get the advantage of causality. The latest check-in probably broke the build. (Your thoughtful coworkers never check code into a broken build.) It is very important that all developers should be able to run the same coverage checks locally on their own machines that are done on the continuous build server. Thus a broken build due to coverage checks should never be a surprise to anyone.

So if we want to have minimum coverage limits that must be met or the build breaks, what is an appropriate limit? Unfortunately some misguided management mandate that a 100% line coverage be maintained. In my opinion, this is a very poor way to spend your limited resources.

First, not all code coverages are created equal. A 100% line coverage limit does not address the differences between line and branch coverage. Package and class coverage can tell you very coarse stories about the state of your coverage. But line and particularly branch coverage have a lot richer information. You might have 100% line coverage and have very poor branch coverage. Branch coverage can help you figure out if you are covering all the independent paths through the code? Do you even pay attention to cyclomatic complexity?

Second, code coverage tools typically run off the the Java byte code, not the actual source itself. This means that code that the compiler generates for you may not be mappable to any sensible line in your source code. (Ever wonder why the first curly brace after your class name is shown as uncovered?) You can find discussions of this issue on sites like stackoverflow.com. The practical significance of this is that you think you have all the code covered, but the coverage tool shows you odd "unreachable" corners in your source that have no coverage.

Finally, there will be some flows through the code, particularly obtuse error paths, that may be hard to test. Do you really want to spend 50% more effort to get that last 10% coverage? Is this a good use of your money?

There is no real best practice on code coverage limits. On my team, we use about an 80% limit for line and branch coverage. This mostly keeps new code from being checked in without coverage. And it leaves enough slack for the developers to exercise their judgement on whether it is worth chasing those extra few lines of coverage. In a future posting, I'll see if I can find statistics that show the cost of getting that higher coverage and the chances that the tests for the extra coverage actually find or prevent defects.

I work in a Java shop, and we use JaCoCo as our coverage tool. It is also used in Sonar. We used to use Cobertura but I believe at the time that the new Java 7 byte codes confused it. Coverage tools like JaCoCo do a pretty good job generating coverage reports that show you both coverage summaries and allowing you to drill into your packages, classes, and methods to see coverage for individual code lines.

While serviceable, these tools have a serious deficiency - the legacy code problem. You know that problem - the one where you start a code base completely ignoring unit tests and code coverage. Then somewhere along the line, one or more of you get a pang of guilt, see the light, and join modern software best practices. But now you have a problem. Nobody, and I mean nobody, is going to pay you to go back and write tests for coverage for that legacy code. One way to handle this is to just take a punch in the nose and watch your code coverage average be dragged down by a bunch of zeros. I'll cover how we handled this problem in a later blog.

Hopefully this has given you some things to consider in your own projects.

Interviewing

Yep, its another blog/article on interviewing. I just searched on google for "interview programming" and got 21.8M hits. So what's so bad about one more? :)

I wanted to have a few words about interviewing for mid-level and senior developers. I do this for my company as one of my many functions (including software architect and lead developer). The reason I feel compelled to write is disappointment. I just cannot reconcile how good some resumes look and how poor the results are off the paper.

When I interview, I *never* *ever* go the puzzle/trick route. I think those have very little to do with our jobs at best and give misleading results at worst. To me, you should focus where the big cost items are in software - in its (lack of) structure and in its (lack of) flexibility. Does the code smell?

It is how easily the software can be changed over time that determines its real cost.

So when I interview, I first establish that they have good grasp of object orient design fundamentals. Then I focus on the key area of "best practices." Does the candidate understand when they are creating a mess? Do they recognize conceptual pollution when they see it?

To ascertain this, I give them a take home (java) "test." I want them to not feel a great deal of time pressure (when was the last time your boss dropped a programming task in your lap and said it needed to be done in 20 minutes?). I give them overnight to complete the test, but in reality it should only take them 20 minutes.

The actual test is short. It's a two page code sample for them to critique. I tell them that their boss is interested in hiring the developer of that code and boss wants their feedback. The test shows the following kinds of problems:

• embedding strings directly in code
• mixing of abstractions
• blocks of code commented out (with no other explanation as to why it was commented out)
• the use of public virtual functions called in a constructor (a questionable practice)
• complete lack of comments
• errors in relationships (coder assumed 1:1 when its really 1:N)
• missing abstractions (over use of string data)
• over-reliance on open-ended integers (instead of enums)

In all, there are about a dozen issues that could be flagged. I have had several candidates tell me that everything looked good to them. My average score is probably 5 issues. I tend to get interested in a candidate if they score 7 or more. The best I have seen was from a really, really good developer who did it sight-unseen in about 10 minutes and got 14 issues (I give credit if they raise and can defend most technical concerns).

I don't consider this tricky questioning. It basically boils down to me having developers that don't keep me awake at night wondering what messes they are making. I feel that a competent developer ought to be able to get most of this almost by instinct.

So why do 90% of the candidates not make the bar?

Third Life, Fourth Life, ...

I was in a discussion today with a colleague of mine (I'll call him Jim). He was asking my opinion of an important message he put at the end of his application installer. I suggested some minor tweaks, which he appreciated. But I also asked him to emphasize things, possibly by an alert or by an extra dedicated panel (most installers are wizards). His comment was that doing so would only aid maybe 10% more of the users.

What sad but true observation.

I was taken back about a year earlier when I remember standing at the back of a training class watching one of our sales engineers using a product I designed. He was working his way through a guided work flow. When an error message came up (you know, the one with the big red X on it), he couldn't click it away fast enough. Breaking with best practices, I interrupted him asking him what the error message was and he didn't know.

This was truly an eye-opener for me. To him, an interrupting dialog/alert was a nuisance to be disposed of as quickly as possible. It was impeding his work flow. I try to put a lot of effort into the information content of error messages to be as helpful as possible to the end user. Was all that effort wasted? I liken careful error handling to putting guard rails on a winding mountainous road.

Back to my conversation with Jim.

We were discussing possible changes to user interfaces. I remarked that maybe with virtual reality becoming commonplace, we could actually be more effective in informing users of the severity of problems. Maybe a future application would have a user traveling down a path, say a winding mountain path. When a severe error occurs, rather than a dialog, we could actually push them off a virtual cliff. At least the experience would be hard to dismiss and might even be memorable.

Maybe instead of giving the user the Ok/Cancel button of today, such a future application might present a button with the label "Respawn?"