Where Are Your Blind Spots?

Orig­i­nally posted August 28th, 2007 at Progress.

I have this fan­tasy that I’ll quit this labor of love here at Progress, and start a real career sell­ing t-shirts with street-signs-that-would-be-funny-as-t-shirt-sayings. Always on the look­out for that break­out sign, today I found one to share.

I drove by this big rig with a sign on the side, behind the driver’s door, that said “you are in my blind spot.” I thought, how funny would it be if I wore that on the front of a t-shirt when I did my next prod­uct pre­sen­ta­tion? (Answer: Very funny!)

In the car with time to think, I thought about that devel­oper at a major bank I was work­ing with who knew his blind spot, but seemed help­less to avoid the result­ing pain.

Fact is, he had two blind spots… one he knew about and one he didn’t.

He knew that there were devel­op­ers using his ser­vices as his project wrapped up. On project com­ple­tion, his hard­ware was recon­fig­ured and reused. He knew those devel­op­ers were there but didn’t know exactly who they were. He warned those he could that his ser­vices were going away, and gave them the new pro­duc­tion loca­tions. They were “too busy” and either didn’t read their email, or chose to ignore the warn­ings. My client bud­geted a whole week to deal with the expected disruption.

A cou­ple of days before shut-off, my client fever­ishly tracked down ser­vice con­sumers, vis­it­ing them in per­son to help with the migra­tion. Even so, it still took three days post shut­down for things to return to normal.

All told, a week lost for a prob­lem he saw a mile away. It was just that the root cause of his prob­lem was in his blind spot. For my client, find­ing out who was using his ser­vices involved lots of “grep­ping log files,” and beg­ging IT to help him track down IP addresses.

Sim­i­lar prob­lems occur on a larger scale, but larger scale prob­lems don’t have as con­ve­nient work-arounds. Alis­tair Croll over at Cora­di­ant points out that the recent Skype fail­ure is indica­tive of, “how com­plex sys­tems can fail in unex­pected ways,” as a result of inter­twin­gling of the Inter­net. I fully agree with Alis­tair, and point out that spread­sheets and data­bases can be used to man­age things on a small scale, but large scale sys­tems, like enter­prise SOA, require auto­matic intertwingle-awareness.

SOA What? Well, a lot of time could have been saved if my client didn’t have his sec­ond blind spot. The blind spot that whis­pered in his ear, “you don’t need run­time gov­er­nance until you’re in production.”

See, if he had deployed the Actional soft­ware (yes, a shame­less plug) he had already pur­chased into his devel­op­ment envi­ron­ment to auto­mat­i­cally and dynam­i­cally dis­cover his ser­vice con­sumers and SOA-wide depen­den­cies, he would have known exactly who was doing what. Even bet­ter, he could have trans­par­ently routed them over to the new envi­ron­ment with them none the wiser.

With­out this sec­ond blind spot, he would have saved over a week of time, and, from what I saw, an awful lot of Tums.