On promises made by marketing

In a large software organization, like in all large and complicated mechanism, friction always occurs at the interfaces.To overcome it, multiple communication lines are often erect between different functions. Take the army for example - while there's communication up-top, between the generals, often there is a lot of direct communication in the middle ranks and even more in the lower ranks. Doing otherwise would cause havoc on the battlefield - the air-force might bomb a location, but the message from the ground forces scout (that the target is long gone) is still en-route through the "official" channels.
In a software organization however, there's very little communication between the R&D department and the marketing department at the low ranks. Only the top managers - product level and upwards usually - meet with the people that actually meet the customers. This is usually excused by the need to "free the low-ranking R&D people (read-  the developers) from business concerns".
However, this lack of communication causes strange scenarios, where the marketing people don't really know what the product can and can't do and, even worse, don't know the future directions that can be traversed. Sometime only a tiny change in the code is required to enable a huge benefit (and a large increase in sales) for the customers. However, this request for change never comes to R&D because it's filtered by the marketing devision.
The same issue, only in reverse, occurs in the R&D department. Sometime we, developers, labor for days and months to enable a complex feature - that ultimately is used only by a small fraction the customers. Even more often, we over-develop - create a feature too wide in scope and too sophisticated - when the customer really just wants the simple thing done (usually over and over - and could we please provide some automation for that?).
Probably the worst sin of the marketing devision is committing to something that can't be delivered - or can be, but with great costs (development, architectural or performance). This is not done out of spite, though the complete effect does sometimes seem like the something done by an evil gremlin. The marketing devision truly wants to make the customers happy - but they lack the specialized knowledge the R&D personnel have - how exactly the product works. The worst sin R&D can make is barring (or even removing) features without really understanding the impact on the customer base. Again - not out of spite, but simply because no one has any idea.

The emerging pattern is of two disparate organizations - one dedicated for the "beauty and pureness of code", the other for "placating the customers" - while none is really dedicated for "create a product the customers want to use". This pattern especially visible the more "technological" the product is - the harder it is to actually understand the underlying logic.

On perspective (or "concentrate on what's important")

"We already have this great framework we use during the main data processing, why not for user management?"

Or

"There's already a bulk of code responsible for loading and saving millions of records - why should I write something new to load and save configuration records (all 10 of them)?"

I hear those (and similar) often. Or worse, I don't hear them, just see the commits in the version control. I think this is a price we pay for modularizing our code. The more functional and setup-free, the easier it is to reuse a piece of code in many contexts.

When a developer write a piece of code, he usually has a specific use-case in mind. This drives performance requirements, defaults, existing fixtures that need to be in place and other "non-functional" requirements. Innocently, that developer checks-in the code, usually along with it's dependent "client" code (and hopefully, some tests), and says to himself "Good job. Well done, me - time for some coffee".

Then, another developer appears. Let's call the first one "Fred", for being a Framework developer, and the newcomer "Cliff", for developing a new Client code.

So, anyway, Cliff comes in with a new requirement - create a client to change a specific configuration. Since Cliff is "constructively lazy", he asks around for some similar pieces of code to what he needs. Cliff finds Fred, still drinking his coffee and basking in the warmth that comes from a job well done. Fred tells Cliff - "Oh yes, I've just checked-in a very useful piece of code for the highly complex, extremely sensitive, super-important project from the high-lord boss. It's a beauty.". Cliff is happy now - all he needs to do for his tiny client part that, truly, is not that important to the main business, is link it with the shiny new framework Fred has written. He does so, and shortly joins Fred for some black tea (anything with caffeine goes...).

The result is that now both the "super important" code (main data processing, critical engine part, analysis engine...) and the relatively "less important" code (system configuration load and save) use the same underlying code - the framework code. However, all Cliff needed was a simple SQL statement - one for loading, one for writing.

Think about it like this - it's reasonable to hire a chef to cater for out important formal Dinner. However, you probably don't want the same chef, his sous-chefs, the kitchen boy and the delivery girl all running in your kitchen, just because you're peckish and want to grab a cold snack.

The smarter that framework is, the more sophisticated its analyzers are, the bigger the cost is when using it on a non-main path. Easy tasks should have easy solutions. The fact that the engine is there, ready to be used, doesn't mean it should be used - there are usually many hidden costs ("non functional" - remember?). Not to mention the extra coincidental dependency that now exists. Think about it the other way around - you wouldn't use the simple code for your main flow. If you try that, you'll hit problems very soon - performance, compatibility, corner cases. However, no one thinks twice about using the complex code in use cases analogous to opening the fridge and getting a chocolate bar.

To finish, a very known quote from an old Chinese master (paraphrased): "Don't use a cannon to kill a mosquito".

On marketing your ideas - the developer's side of the story

When I was younger, about 13, I went to my dad's place of work (a software company - what else?) where I witnessed an argument between two developers over the way their product should move forward. After a while their manager, who was sitting and listening, made a decision and the argument was over. I didn't really understand what it was about. However, what really amazed me was that the decision was made at all - how could the manager know which way is better? Neither was tried and both required quite a lot of effort to even be tested. When I asked my dad about it, he said "that guy (that won) sold his idea better".
At that tender age this struck me as totally unfair. What do you mean "sold his idea better"?! They were both developers! What's selling your idea has to do with it? Either your idea is right or wrong. - and if it's wrong, no amount of talking would help you.
Moral of the story: You've got to be able to sell your ideas, otherwise no one will listen, no matter how smart they are.

When I was drafted into the army (mandatory service in Israel) at the age of 18, I thought it'd be like the books I've read. "The person in command would notice I'm doing good / the others are slacking off" (hey, I'm a born nerd). To my surprise, they didn't. Not only that - they didn't even care they didn't. Being shy, I didn't make a peep and, as you can guess, nothing good happened.
Moral of the story: You've got to be able to sell yourself, otherwise even if your actions are good, you won't be promoted while other people, who are better at self-marketing, would (not to mention there'd be no personal gain).

When I got into work I got into an argument with my team leader. I said "we should do A, which is technically complex but correct". My team leader went to another person in the team which said "we should do B - or nothing at all". Since I was the new guy, I was overruled. About a year later the old team leader was replaced with a new one. Since I was already a bit of a veteran (and my new team leader was much more technically oriented). my idea was heard and implemented.
Moral of the story: It's better to be born a year earlier :). Seriously though - it's better to be well connected to people who have the power to decide.

I'm seeing the same patterns today, at my work place. Developers and team leaders who have a loud voice, who can convince people they're right, who can "sell" their ideas (and who are not marked as "annoying" by the lower-mid management tier) - get heard. The others who can't, are ordered to follow the initiatives of the ones who do. More than once I witnessed a good idea get buried because the initiator didn't push hard enough or wasn't liked enough.
I'm not "dissing" my organization. My managers aren't stupid. You (probably) couldn't bluff them into agreeing to ludicrous ideas. They'd demand proof of concept, demos, numbers and figures. But - they're people, not gods. At the end of the day, they'd listen the most to whomever is talking the most. The people they like, those they trust, have a more time to sell their ideas.
All (or most) ideas brought up are smart. Most of them need doing. Almost all would make the product better. However, time in a release is limited - so all of the ideas are competing with each other. The person that can adapt his ideas to show how they match the overall plan for the product, the one who talks with passion and manages to infect his audience, the one who's adapt at gaining support - or - in short, the person that can "sell" better than his peers - is the one that'll end with the winning idea.
Not always. But often.
 I used to think "management" - now I think "managers". And they are just human. Like me.

On the old and the new

"Out with the old, in with the new" - this statement doesn't really work in software. We cherish the old. We like the familiar. We cling to what we know, and label everything else as "not mature" or even just plain "risky".
And we should. The new is often wrong. In the "old days", there were a lot of "new" ways of doing what we now need. Evolution (mutation and natural selection) culled most of these and left us with a golden way of "how it should be done". But this kind of process takes time - so that shiny "new thing" is now the "old and familiar".
However, once in a while we get to hear the big news - "500M users due to an innovative design", "Millions of requests per second" (but using out-of-this world architecture), "Terrabytes of memory available for use" (but re-implementing everything we thought we knew about VMs). While the growth is usually gradual, the change in thinking that enabled these is instantaneous. You either try to migrate to a new database or not. You either try to build your data centers with second grade equipment, or not. You either rewrite stuff from scratch - or fight the same battle everyone else is.
I envy those with both talent and guts to go the distance. I pity those who fail. I glorify those who succeed. And sometimes - I try to push a little on my own little corner. Maybe shake a few old cobbles loose and let some new light shine in
I think a statement that fits the software world much better is "no pain, no gain".

On solving client problem - do you _really_ need to solve them?

[Note: Apparently my blog got reset. Not a big deal, since there wasn't very much there to begin with. Oh well.] [Note2: Apparently I got mixed up and opened another blog. Oh well - this one looks nice anyway :). The old one (with only 3 posts) is available here: http://www.jroller.com/objectpool]

[Disclaimer: this is a purely hypothetical situation]

T-17 days

"The customer is having problems with a specific flow. The support could not help - so it's escalated to R&D. Please advise".

What's more fun than starting your morning with such an email. Customer support. No - it's OK, we didn't have anything else planned for today - just some back-log of bugs, plans for the next version and maybe, just maybe, catching up on some unread emails. Oh well. Let's dig in.

T-14 days

"After investigating, it appears a sub-flow is not behaving correctly, suspecting performance issues, moving to team X".

We loafed around for 2 days (well - did some other stuff, not less urgent), met with the customer (online of course - no budget for flying), then whipped out the magic wand - it's a performance issue. Hooray for us! Let's make our customer prove their database, network and PCs are working as they should.

T-13 days

"Customer reports heavy disk activity during the problematic flow".

Oops. Someone smart is sitting there. Who's smart idea was it to log disk events?

T-10 days

"Investigation comes up with a problem in a specific sub-flow Z - takes too long, causing timeouts."

Yep - we finally found the problem. Apparently Z is causing a resource bloat. But why? (oh, the pun. I should be shot :) ). We have lot's of Z's in the system, why is this an issue?

T-7 days: "To team Y: Maybe you should execute Z'?"
T-6 days: "To team X: Maybe you should fix the Z?"
T-5 days: "To team Y: What's the big deal? It's not a big change in the code."
Weekend, no work
T-3 days: "To team X: We don't want to create a private patch for a single customer."
T-1 days: "To all teams: We have to fix this - what are the alternatives?"

T-0:

"We can change the Z just so... and it should now take about 0.1%  (yes 1/1000) of what it used to take (resources / CPU cycles)".

It's the first time someone from the technical side actually discussed the whole flow with someone from the knowledge side. And - surprise, surprise! The bloated result doesn't make sense. So they drill down a little further - "Ah - we should only ask what we REALLY need to." - "Oh - this reduces Z and let us preform some special optimizations on it - since we now know what should really occur".

---------------------------------

This sad story is fictional (I've changed team parts, mingled the problem and merged 3 different cases).
However, it depicts a sad truth in the industry.
We (software developers) are very technical people. We cherish our finely sharpened problem solving skills, our domain-specific knowledge and our advantage over "mere mortals". We rule the bits.

But we lose the big picture.

More often than not, what the customer really wants is not what we provide. It's close, and the customer can work with it - but he'd really rather get something else, simpler and easier to implement.

However, we're so caught up with our own technical wizardy and ability to "create a generic solution that'll apply to ALL customers from now until forever!" - oh, aren't we the greatest of all possible great things that can happen?

But we don't talk to each other. We don't really know what's wrong. We only know what's broken in our own part, and try to optimize our part, not considering it's only one link in a great chain - and often not the weakest link. It takes a great mental effort to take the holistic approach and actually see the issue for what it really is.

On your next support issue, I dare you to think twice. To look at what's really wrong and not at your own part of it. To truly understand what the user wants to do, and fix the root of the issue rather than the symptom

Then add some documentation - because after all, the user should've done "that" from the beginning - no?