Agile Enterprise Upgrades

Scope:

This article covers an Agile approach to the Enterprise Upgrades of software.

Problem:

Enterprise especially within the technology industry, see upgrades as an evil that introduces risk:

"Upgrades of software introduce the risk that the new version (or patch) will contain a bug, causing the program to malfunction in some way or not to function at all." [1]

The common approach to this is always to stay one version behind. The view is that the new version will be field tested by "other" people/customers. And then when these are fixed the laggards can then on board as it will be safer.

Examples:

“Industry best practice for releases like this is to wait one year, or for the first service pack (presumably Windows 10 SP1 in this case). This allows time for all those bugs to be discovered and fixed, or at least have a knowledge base developed on how to work around those pesky bugs.”

“For most users, the wisest path is to wait to upgrade until after the initial bugs are discovered and resolved before taking the plunge.” [2]

And ....

“It's one of the great truisms of IT: Wait for Service Pack 1. Through the years, that's been consistently excellent advice. It's not so much dogma as common sense. Unless there's a compelling business advantage to deploying a brand-new release, you'll suffer less downtime and require fewer antacids if you wait and watch as other people discover the places where it crashes or falls short.” [3]

Except who are these mysterious 'other people'?

Alternatively this article from the same website says hold off on the Service Pack itself for the same reasons as the others above say you should wait until it is out:

“Any pessimism needs to be weighed against the improvements offered by the new bug fixes and/or new features. From what I've read, the improvements in Windows 7 SP1 are trivial and/or irrelevant to most users. Thus pessimism wins hands down.”

“give SP1 about five months of exposure to everything the world has to offer.” [4]

So, one article claims to wait for the Service Pack to fix all the bugs in the new version of the software (an upgrade) and the other one says wait some more.

Always some reason never to upgrade and stay on latest. And so, it goes on.

I could explain why you should always upgrade with some nice reasons about Security, existing known bugs, Compliance and Legal reasons, Business reasons, performance improvements. But actually all that is irrelevant.

There are just two real reasons why you should always stay on latest.

1. Valid Support:

If you are not in support, you are dead with the vendor. To be clear here, even if you have an SLA, expensive License, piece of legal paper, that claims your covered, chances are if you not on the latest version your not covered.

You see vendors can always get out of any legal agreement. There will be buried in the small print that their software only works on the latest X hardware, only in the correct set-up which is subject to constant change and the variable use-case usage. In fact there are so, many get outs that they can always get out of any situation if your not on the latest version. And sometimes even if you are.

Customers never discover this of course until it goes badly wrong.

Here is how the realisation often occurs:

Bring Bring ...  red-hot alarm phone is answered

"Hallo CEO of Vendor Company here" - CEO

"My business is going bust and its your software's fault and no one in your company will help me, you should legally help me?" - CIO

"Yes I have spoken to my people and they advise me that you have not upgraded to the latest version which solves the problem" - CEO

"But I have coverage with <insert whatever coverage has been paid for>" - CIO

"Yes, but if you read/notice our <insert get outs that are legally covered here>, then you should have upgraded, thus just like an insurance policy we have no liability to cover you now" - CEO

"Whhhatttt?" - CIO

"Look we can help you, but of course for an emergency situation like this it will cost you...big time" - CEO

At this point the CIO is done. Actually it is worse than this, but we shall cover that later. Thus having any comeback is often nil for the CIO who has not stayed on latest.

But it get's worse the more you stay off latest as we shall see in 2. Hidden Fixes.

2. Hidden Fixes:

Everyone on this topic talks about the "hidden bugs" and buyer beware. But what about the "hidden fixes" within the latest version? Here is how this goes down.

While developing the latest version an Engineer discovers that they created a major horror bug in a previous version. Lucky for their job it was not discovered by any major client (the small ones who got burned are often ignored by big vendors). So, he quickly fixes it on the sly without telling anyone. The fix gets unrecorded.

Same as above, but this time the Engineer discovers the horror within someone else’s work (maybe they have left the company), but escalates to the Team Manager. The Manager realises they are the responsible person for letting this happen. So, to save their job they tell the Engineer, that the customers will complain and could sue. Best it gets resolved but also covered up.
The fix gets unrecorded.

Now the same story unfolds from team lead to Program Manager. That manager covers it up the same way and on it goes up the tree. The difference is now we have a compounded set of hidden fixes across, multiple team members, multiple teams, multiple products. Or in other words all of the software, all of the time.
The fix gets unrecorded.

And the CEO? They do not want to know. Plausible Deniablility of course. We see this in corporations all the time.

Where is the proof? I can supply none as its always covered up! But how many times have you had software that was intermittent and then you upgraded and it went away?

Although if the naysayers can claim "watch out for those unknown bugs" then surely the converse is true, "watch out for not obtaining those fixes in your current out of date version". We could play this game forever.

However it happens, you will become more exposed to risk the longer you stray from being on any version other than latest. Those bugs in the latest version may have been in their in your current version too! Or they may have been fixed, unrecorded.

The only way to get out of these two situations which almost all CIO's will experience, is to stay on latest. So, why do they not do it? Why is the industry in denial?

"But old habits die hard. Amazingly, 21 percent of the AMR survey respondents sold their enterprise software upgrades to their business based on vendors’ announcing desupport dates for the software. That’s not planning. That’s desperation." [5]

The underlying problem is that they never test until it is too late. The familiar Waterfall failure of leaving testing till last. The real reason why people do not like upgrading is the cost and involved testing.

What would the Agile way be?

Solution:

Agile can solve the problem of Upgrades that are needed all the time, staying on the latest version limiting the exposure in a safe to fail environment. As the first principles of the manifesto state:

"Our highest priority is to satisfy the customer through early and continuous delivery of valuable software."  [6]

This is entirely possible when you look at the approaches of CI/CD. Continuous Integration reduces the cost of testing through automation, but this in turn is only possible if you "integrate continuously". To perform this one cannot be waiting, Agile approaches of short delivery and auto-tested iterations must be applied.

"CIOs know that putting an upgrade through its paces in a test environment is critical" [7]

This can also be done with the Beta versions of product before it goes live. This is important as even if you do get support on an older version, chances are most of the Vendor company is working on the next version so, support will not be provided by the premium resources of the vendor.

"Even if the vendor continues to support old software versions, it will shift the bulk of its people and resources to new versions" [7]

If you test the Beta early you can via the CAB (Customer Advisory Board) feed all issues back into the top resources of the vendor to get these fixes before the latest version becomes GA. An Inspect and Adapt feedback loop.

This is evidenced once again in my Agile Development Poster where testing is performed in a Continuous and Automated manner across multiple activities. [9]

Always stay on latest.

References:

[1] Upgrades: https://en.wikipedia.org/wiki/Upgrade
[2] Upgrade to Windows 10: http://www.recordonline.com/article/20150719/NEWS/150719356
[3] Computer World article 1: http://www.techproresearch.com/article/how-long-should-you-wait-before-deploying-windows-10/
[4] Computer World article 2: http://www.computerworld.com/article/2470790/microsoft-windows/windows-7-service-pack-1---don-t-install-it--yet.html
[5] CIO article page 2: http://www.cio.com/article/2440405/enterprise-resource-planning/enterprise-software-upgrades--less-pain--more-gain.html?page=2
[6] Agile Manifesto Principles: http://www.agilemanifesto.org/principles.html 
[7] CIO article page 1: http://www.cio.com/article/2440405/enterprise-resource-planning/enterprise-software-upgrades--less-pain--more-gain.html?page=1
[8] Agile Development Poster: https://agiledogma.blogspot.co.uk/2014/04/agile-development-poster-there-are.html