Tuesday, December 19, 2006

Product Success Metrics

This is a general checklist about what to measure in order for us to say a product, as in a 'software product' succeeds. There could be various assumptions across the industry which finally boils down to these metrics:

Customers new and old

How many new customers are you acquiring per month? How many customers have canceled maintenance or subscriptions per month? This is not a direct metric of the product itself since it has other parameters such as Sales efficiency, Time to market etc. However, measuring this metric is key to product success. Even if customers are still 'in' and do belong to the client list, the degree of engagement is important to be measured.

Quality!

Yes :-) this is an often heard term but product quality means so many things to so many people.

Product quality should be monitored on an ongoing basis to make sure it doesn't become a serious problem. Set up a tracking mechanism with either weekly or monthly statistics with not just the number of defects but quality of the quality process itself. I have seen product heads just track the traceability from requirements to defects and nothing more. A well tracked quality system produces just the right metrics which would exactly show the quality health of the product.

For released products, technical support typically keeps track of open defects. As soon as a product is released, it is common for the number of defects to go up as more people use the product. If the quality is high, it should be a manageable number and should settle down after awhile. For unreleased products, QA and release management should maintain a strict vigilance on the numbers. If the number of open defects is not going down as the release date approaches, the release date is likely to slip. Obviously we want the Severity 1 and Severity 2 defects approaching zero as the release date approaches. Severity four "defects" are typically enhancement requests and are often ignored in terms of product quality. Yet they indicate areas where the product fails to satisfy the customer. Product quality is the best indicator of internal health of the product;

Technical Support

Product Management and Technical Support should work at tandem. Care should be taken that metrics are not tweaked for a lesser benefit. I have seen Customer Support increase no. of calls they attended by logging the same problem in different ways and product management trying to respond to multiple problems with a single statement of direction or FAQ that would portray them as having anticipated that as a known problem.

Technical (or Customer) Support already measures the number of calls per product and the nature of the calls. Product management should analyze these numbers to identify areas of that product that can be improved for a better customer experience. Are there problems with installation of the product? Is the documentation too hard to understand? Is the product too hard to use? By streamlining a process, will you cut down the number of technical calls? Review the number and type of calls for your product to uncover hidden profit leaks.

Product Schedule, scope and slippage

How long does it take from the time you define requirements until you have a finished product in the market? Keep track of the percentage of original requirement specs that get delivered as the final product. How many "out of scope" requirements were included? Do your developers have a clear idea of what problem to solve and for whom? The problem may be poor requirements definition, which you can control, or it may be scope creep which you need to communicate to your management. Report the facts and let management manage.
Pre-Sales and Sales' support

How many non-sales people does it take to support the sales cycle? It would be a great metric to know how much time is being spent by non-sales people on direct sales efforts such as demo support, conference calls with prospective customers, onsite presentations, requirements assessment, and other sales calls. Early in a company's life, people wear many hats and this is a common thing but as a company grows, product selling needs to be 'empowered' with the right sales kits and training. Sales personnel need to continously update themselves as opposed to just having client meetings and playing golf.

When non-sales people participate heavily in the sales process, they are not doing their day job. And the cost is rarely attributed to the sales activity; another hidden profit leak.

Ultimately, we want to build and sell products profitably.

Product revenue/profit

How do you know which products are worth investing in and which should be retired?

Sales should be able to tell you how much revenue is generated per product. In order to calculate profit, each discrete product (not product line) has a profit and loss statement tracked in accounting. Finance usually has these numbers but it is Sales' responsibility to provide the profitablity numbers to the top management.

Market share

What market share you have. By tracking this metric over time (year over year), you can discover whether the market is saturated or ripe for expansion. If the market is growing and yet your market share is shrinking, it indicates that another competitor is growing at a rate faster than you are. Getting this metric is the hardest among all metrics and not all survey agencies who do this provide reliable data for Sales. Many standard business analysts provide reports for the vertical industry itself such as Forrester, Gartner etc but again, each company has to investigate somethng that works best for them. And management must have the guts to accept the facts.
ROI on Marketing

There is a need for a closed loop lead tracking system that allows you to track activity from the marketing program that generated the lead all the way through to a closed deal. Are the right market segments being targeted? Do the positioning and market messages get a buy in with the buyers? Do you have a compelling solution? Is that packaged as a compelling solution?


And finally....

It's not possible to track all metrics at once, but determine the key performance indicators you need to start tracking. Report the baseline, begin tracking on a periodic basis, graph the trends, and drill down to find out what is going on.

Start with the metrics for customers and product quality. These are the first ones to tackle.
Calculate the profitability of your product. If you have a loss leader, can you justify further investments? Do win/loss analysis to find out whether your product is driving business, even though it isn't profitable. If it should be retired, show some leadership and present the facts about why it should be retired and how you would redeploy the resources in a more profitable way. There's noplace for personal attachments here! And no place for vested interests if you know what I mean.

Tuesday, November 28, 2006

Ensuring product excellence

By 'product' I mean a typical software engineering product within the IT industry. If we look at purely from the experience point of view it is not rocket science to achieve product excellence. However there are many ifs, buts and loose ends to take care of. Where do the loose ends lie?

Usually, there is a 'Vision' team and there is an 'Execution' team. Care should be taken that people with the right skills should be fitted within the right teams. For example, someone with very high qualifications from world's leading technical institutes may not necessarily be part of the vision team. It might be a worthwhile exercise to conduct an aptitude fitment test such as MBTI test. See who qualifies to part of the Vision/Strategy team. If we follow the balanced score card method of evaluation, the score table will point to those people who understand the industry dynamics, who have continuously tried out other products in the same league, understand the strengths and weaknesses of the competitors' products ('know thine enemy') as much as his/her own products and most importantly can provide an unbiased, devils-advocate kind of opinion when it comes to product strategies.

I have seen in well established product companies that the product marketing team is so soaked into their own products that they blindly evangelize it without working on the gaps.

The strategy team should also evaluate positioning, packaging, pricing the product innovatively and this should be backed by a lot, lot of research. Essentially this team is what will make the share holder value increase and if the guard from the hill top does not see the enemy marching through the terrains, the soldiers in the plains will see no attack coming.

Okay, what about the 'execution' lot? It's no inferior position to be in. Since the visionary team is not perfect, the execution team should have people who are pure managers as well as half-leaders and half-managers. The execution team will need pure managers to see if the product design is well done, if reviews are injected at every stage, if the risk is well managed at every stage, if ample automation is planned for building the software as well as for quality assurance. We need a balanced score card here too. Checklists will not work if they only track if a task was completed or not, if a review was completed or not instead they should track if a task or review was 'rightly' done or not.

And if there are hidden strategists within the execution team, they should be allowed free expression of opinions and should not be suppressed under multiple managerial levels. How often have we seen a brilliant idea or prototype by a junior executive, not well understood by his dinosaur manager.

So, essentially, the visionary team and execution team go hand in hand. They both are equally important and no one reserves to be within one team or the other purely by paper qualifications. It should be based upon the fire in the belly and on natural wiring of the individual.

Product excellence is a function of right vision and right execution.

Tuesday, October 24, 2006

I can't change my hair style! My biometrics will fail!

Alright, this statement is slightly exaggerated but only slightly. We might arrive at this soon.

If you are an avid internet surfer and a member of many web portals, just think about how many times you asked for 'Forgot Password'. It's hard on us. With all due respect to SSO, I need to memorize about 10-15 passwords and use a password manager (it's easier to crack the password manager utility than than password itself) and inspite of all this, I request for the forgotten password. Recently I read that 60-75% of the HelpDesk requests to famous portals are for the forgotten passwords.

Biometrics has made great inroads. Not that all laptops have that facility and it's specially not there with all the sales personnel in the world who travel with unsecured laptops. But if biometrics does become common place and wins the catch up game with the hackers, we might be in for some reprieve.

I'm particularly interested in how the face, finger and voice based biometrics work. Some of them have the multi layered verification system where there's registered finger print, the voice based pass phrase and then the password itself. However, as far as I have seen, you are again tied in or hard coded to the finger touch recognition software's provider.

With the face, it's trickier. There are supposedly technologies that will recognize the face even if the face dimensions change with a smile (don't smile ear to ear), frowning , blinking etc the risks are as much as that associated with regular image mapping techniques.

What if the face is posed at an angle of 15-20 degrees? What if the light is insufficient.

The most important thing to remember is the title of this blog - yes, face biometrics is yet to transcend the changes made by your hair dresser. It's quite possible that the world's leading hair dressers might register themselves in a face-provider registry and discovered by the web service out of face biometrics and your latest face will be sent in for verification.

How about using a string of regular passwords as in older times ;-)

Sunday, September 10, 2006

Can 'My PC' be presented to me same everywhere? Trends for Mobile Computing!

It's been funny - the whole change in my gadget setup from the last few months. I took a fancy to those laptop ads on a TV where a funky haired guy sits in a beach and laptops his way to work. I was obsessed about buying a light laptop, the lightest in fact, which I would connect through wireless connectivity from anywhere at home and work from that. It could be from the terrace staring at the moon and stars, or lying down on a heap of pillows with the laptop on me. Yeah, all this worked and how! Until I found this new piece of software (mobipocket) that gets installed on my mobile and does practically everything. Now my whole paradigm of working has changed. I use only my mobile to read news in the morning (downloaded as RSS feeds from the news site)- my family is very happy about this because we don't fight for the early morning newspaper anymore, read the latest technology, business and Scientific American latest posts on it. I also religiously download the technical documents I need to review as part of work - functional specs, technical specs, SOW, contracts ....what not in a mobile optimized format right on my mobile. I review these documents at night with reading lamps off, without disturing anyone....man, what a change! As you have guessed it, I also download fiction, non-fiction works for reading and I can read a whole book at night without switching on reading lamps and being a nuisance to others.

Mobile-ing of documents, mobile-ing of news, mobile-ing RSS aggregation has changed my life.

My wish list would be like this:
- Single Sign On from my mobile to a range of sites, to VPN of my office network
- A more reliable VoIP calling service that I can use for conference calls across the globe (my OS is Symbian and Skype yet doesn't work as expected although Google's gdskype? has something similar)
- Email profiles automatically maintained from my email provider. For.e.g. I want only certain emails to be delivered or I want all emails that meet a criteria to be made color=RED on server side, desktop-client side and mobile
- User profiling and 'My Documents' implemented for 'Me' across any device. For e.g. if a global SAN is maintained for me as an user with my documents (just like email - could be presonal, work, music etc) then I can access 'My Documents' from anywhere via any protocol.
- LDAP - Global address book with appropriate rules for categories and grouping rules
- Imaging or something like imaging. For e.g., if I'm at SFO airport trying to access my mails and documents, can my 'image' of my folder structure, my documents, my profile be automatically configured on the go so that I can work on my environment even from airport terminal.
- Hack proof methods

WDYT?

Saturday, August 05, 2006

Data versus Information

I was recently searching for foreign exchange rates on a particular date for the purpose of filing my income tax returns. I searched Google and other search engines for 'currency rate', 'exchange rate', 'currency converter','foreign exchange rate' etc and tried to use all the usual terminologies associated with it. I could not get the result I wanted. I was a dissatisfied customer of search engines. Then, I did the linear search - going through the forex sites, Reserve bank site etc one by one and bingo! I found Reserve Bank's site with exchange rates for every day for the last few years. And I noticed that it was tagged as 'Reference Rate'!

Do you get my problem, rather a search enthusiast's problem? While search is based on the keyword, the keyword itself may vary depending on the searcher's profile such as regional parlance, educational levels, professional levels etc.

While a layman would search for 'salary difference', a HR pro might search for 'salary gap analysis'. Layman- 'How to get modem working', IT pro -'IP configuration'.

So, jst like there's a difference beween what you seek and what yu want, there is a difference between Data versus Information. Data is raw while Information is contextual (apart from being analytical).

While providing information from raw data, it is a good idea to look at following thumb rules:

- User profile, what kind of user is asking for this.
- User Topology and demographic information
- Automatic learning by the information classification system. e.g. collections or business rules should be dynamically changed as the system learns more and more from the information
- The learning itself is automated by a good business intelligence and analytics tool
- User experience management - this is a big discipline. In short, monitor how promotions were effective, how good merchandise interfaces are being used, how relevant offers are based on user identity etc.
- User context is always maintained and continously evolved. One example for this could be that if a user is found to search for certain terms in the legal profession and the same user is searching for certain terms in Advanced Robotics, then the system can 'learn' that the user is a well read person. Similarly if a user is always typing with spelling errors and searching for typical teenage non-intellectual subjects, the user can be put into a collection of casual not very well informed users.
- Similarly Reports that contain information should be based according to context.

It's probably the next 'in' thing within Search and other information retrieval systems for laymen.

Saturday, May 13, 2006

The insured testing professional

Have you ever interviewed 'Testing professionals' whose resume has everything under the sun under testing? While there are exceptions, I have most often seen people who appear for a testing job unprepared for a technical interview. They could answer any number of 'process' or theoritical problems such as what is a black box testing, or integration testing, regression testing etc but when you drill deeper into solving a day to day problem they cannot respond very well.

I'm wondering if it is a chicken and egg problem. The process of quality assurance is not the first in the software development cycle. A test spec often always follows a requirements or functional spec and at that very stage, the tester transforms to someone who is supposed to follow or validate what the developer thought of. Is the test spec always limited by the functional spec's scope? Are we creating/developing testers who just need to validate and need not really invest in understanding the latest technology trends, the latest product vulnerabilities?

Let me give couple of examples.
Ask a question about how would the interviewee test a simple web application that submits a form with two simple fields 'name' and 'address' to a repository at server. More often than not, 90% and above, you will get the first response as UI testing. 'Validate the input fields', 'check for input length' 'check for special characters'.... well, any tester is supposed to have already known the kinder garten stuff of testing. Okay, dwelve a little deeper, ask him/her that you want the interviewee to provide functional examples and bang will come the reply 'check the database and see if the input entered via form is updated properly'. Ask him/her 'do you think data can be entered by other means and you should ensure testing for those conditions', again bang comes the reply 'well, the form should have strict validation and from the server side the database administrator should take precautions'.

I'm yet to see testers talk about vulnerabilities in authentication, input filtering, SQL injection, transport security, error handling etc.

I hope I'm proved wrong. I think 90% of testers are people who do what they are asked to do and the functional spec is an insurance for them against customer defects or hacker attacks.

Check it out - ask an interviewee next time 'How can my Forgot Password feature be exploited'?

Saturday, April 08, 2006

False Positive Emails

Today I dug out a very important email from my mailbox from the SPAM folder. As I had guessed it, it was 'False Positive Filtering'.

In the simplest terms, False +ve emails are those which should have hit your inbox but the spammer algo with your email provider thought it was a spam. So, how does this work? Usually most sites such as Google, Yahoo etc keep their spammer rules very secretive for security reasons.

Common possibilities are these:
- When the email's HTML contains links to images with names remotely suggestive of porn stuff
- Many spam emails ask user to explicitly click on link 'remove me from this list' only to trap their email and domain names to send more spam. So, sometimes, even genuine emails with 'remove me....' can be considered spam
- If the sender's email domain address does not match the 'From' string. For e.g. if the mail says From: Income Tax Dept' and domain is 'gxbvghh.com' . Sometimes genuine emails, which are sent as mass mails using a 3rd party provider can get flagged as positive.
- Suggestive attachment names

Interestingly there is a metric to measure email deliverability and one of them is to reduce false positive emails. I believe Google has it highest at around 99%.

So, look for those missing emails in your bulk/spam folder.

Friday, March 24, 2006

Software as a service (SaaS)

This is the newest coinage of the term which is making a new wave now with big companies like IBM hosting exclusive seminars for it. Idea-wise and technically there is nothing new here. It is simply the culmination of the software evolving from a provider providing one-size-fits-all product to a more collaborative connective world where each provider provides a solution and everything falls nicely into place with each other. 'Live and let live'.

The sops offered are for CTOs and CIOs. Lesser maintenance headaches, no need to go up on the application upgrade treadmill, not to bother about compatibility and adaptability and in some ways, the best of breeds put together.

Athough ASPs and On-demand hosting are cousins to SaaS, theer are subtle differences. SaaS is loosely based on SOA where you can have services (a la products) talk to each other at different levels. It could be at raw API level, architecture level or application level. All the usual demons of integration has to be planned in advance and addressed such as security, service policies (which by itself can be a service), accounting and autherization, platform dependencies...etc.

SaaS modelled application can be hosted or could be on-premise. The licensing and pricing models and rules would be different from what we have today.
Web developers today are still not trained on technologies such as BPEL to readily develop SaaS based services. Most developers would be happy to integrate services/apps using plain old XML and HTTP. And again, most old software cannot be thrown away for want of SaaS. So, the trend would be to develop wrappers around them and 'somehow' make them a service that others can discover, register and use.

I saw a cynic's post calling SaaS as 'Same old software as service'. For those of us who know Hindi, SaaS means mom-in-law and I guess this is closest to it's meaning. 'Put up with it'!Software as a service (SaaS)

Wednesday, March 01, 2006

8 terabyte desktop? For whom?

I just read this piece of news: 8 Terabytes Desktop

It is like you can never have too much of the goodies! With such a large disk space on my desktop, I can have 'anything' I want stored on that. One of the first obvious advantages is that I can store videos, movies, arhcive literally everything and will 'never need to delete but just sort' - a la gmail.

But, think about it. This might be great for a student or a graphic designer who would need to store loads of data who works from one single place. But what about the regular IT pro? An IT pro's profile is like this. He/She travels from home to office and back everyday and sometimes to different offices within the country or outside his/her home countey. And sometimes to other destinations on travel. So, how would he/she efficiently use the TBs of info which is situated on his desktop at one office? Will this massive data storage be augmented by massive network bandwidths, by massive backup and recovery strategies?

To me, an ideal environment would be like this. I would stash away the laptop and any dependency with the hard disk out there. I would log on to the internet - from home, office, airport, from my car (car can have a light weight network computer), from my hotel and probably from general travel lounges across the world without being bothered about disk space, backup, RAID and security.

To me, internet bandwidth and availability are much more useful than terabytes of storage on a PC.

Thursday, February 16, 2006

Economies of Data Integration

What are the real issues or 'real time' issues of data integration? While we see everyday ads or
claims from companies about how their software can integrate data, there are still a lot of nitty
gritties that have to carefully considered while making a 'one strategy for all' decision. One of
my favourite quotes is this: 100% automation leads to ineffectiveness.
All organizations, from the smallest of SMBs to the largest enterprise have silos of data, silos
of information across finance, IT Operations, various customer or client data repositories,
internal repositories, internal workflow systems etc. Unless an enterprise is ideally designed,
most of them would have parallel streams of workflow. One stream would be the integration between
internal operations, finance, costing, auditing, compliance divisions. These are largely
Microsoft Excel based, Tally based or based on similar tools. How many of the currently available
Accounting softwares, Network monitoring softwares or Compliance checkers are interoperable with
each other and with enterprise data flow management? The internal Ops data is also highly
sensitive and protected and the typical finance manager does not know or does not believe in SSO
kind of things and he only knows authentication established locally. In 80% of the companies
according to a research site on macro economics, the most sensitive financial data is still
resting on the hard drive of the chief finance person's laptop which is ofcourse well backed up.
The second stream that can be integrated is the inhouse knowledge and information management,
including competence building, training, market gap analysis well tuned with product analysis,
gap based skills acquisition. This stream data is not as hard to integrate as the first stream
but nevertheless difficult. If each division such as Corporate Training, Knowledge Management, IT
Management, Program Management etc have different products controlling them (which is usually the
case) as varied as Microsoft spreadsheets, Siebel based systems, SAP based systems, Tivoli based
data - well, I can hear SOAists screaming, but we still have a long way to go on that.
The least difficult of the data integration streams is the enterprise level data and information
integration. We all know that all major enterprise players have developed various adapters and
stand alone products which will recognize and discover services, understand business rules,
transform business data according to that and 'talk to each other'.

I'm trying to gather information about how well information integration itself is working out. Readers, please pass on any web sites that publish information about the economics of
service-to-service adoption and how do their returns measure up against cost in clear
quantifiable terms.

Friday, January 06, 2006

My dear Biz portal- get noticed!

After crossing all the hurdles of creating a business portal and effectively creating it's functional aspects, comes the real hurdle - which is to ensure that THIS particular portal is what customer's will find when they look for one. Easier said than seen!

Broadly, this means the customer may be directed to the business portal via advertisement banners, reference links, pay-per-click programmes etc. But the largest possibility is for the customer to reach via a search engine. Again, not all search engines. According to a study, customers find a business site 35% of the time using Google, 30% of the time using Yahoo, about 15% of the time using AOL or MSN search engines and occasionally using the likes of Lycos or AskJeeves etc. While I'm not professing these claims, these figures do indicate what we know commonly about. So, it boiles down to essentially being spotted by Google, Yahoo and perhaps MSN.

So, the second requirement is that it is not enough to be just listed (or discovered) by these search engines. Most users behave like this(again, a study based data), 75% of the time the user quits or changes search criteria once he/she doesn't find relevant information in the first page. This translates to about ten to 15 results at the maximum assuming user is using a full sized window.

So then how do you get to the top of the search results page? It's not a staright forward formula. The parameters are varied and the web site constantly needs to optimize itself inorder to be found by these kingpin search engines.

Basic optimization is to fine tune primary key words, phrases and descriptions.

The next step is to look for the search engine's ranking and it's directory positioning. This will help the web site designer to optimize it for the most suited search engine.

The next step is to select the best pages or right pages and mark the content for theme-based indexing.

Look for those keywords, phrases etc. that may be potentially marked by search cops (spiders) as spams although it may be unintentional.

Make keyword optimized directory submissions so that webservices can discover the right services in a jiffy.

Once these basic steps are done, the 'get noticed' result is achieved. However this is by no means the end of web designing because a lot goes into Traffic planning, the site mapping and navigation architecture, image structure, site structure etc.

Site hit report and analysis is a key market intelligence area.