Evaluation FAQS

Why can't we just purchase the biggest name-brand software package and be done with it?

There might be some mid-tier solution that is a spot-on match or an up-and-comer who doesn’t have much market share but is hungry and willing to do some tailoring to give you exactly what you want (which is a tactic a number of smaller software companies have used to become bigger software companies at which point they stop tailoring.)  If an organization chooses to simply select the best-known solution, they run the risk of spending a gob of money on a package, implementing it, and being no better off than they were in the beginning. It happens.


Can you define our requirements for us?

Sorry, no.  The actual need, the business requirement, must come from within the organization.  However, outsiders can be an invaluable resource in eliciting responses and outlining the requirements for consistency.  Our clients change to a new software platform for a reason - perhaps the existing platform is no longer fulfilling their needs or allowing the organization to compete in the market place.  The substantive requirements must come from within the organization.

Consider using outsiders to interview key personnel and draft an initial version of the requirements.  The approval and the priority of the requirements should come from within.


How many people should be on the software evaluation team?

We prefer 5-7 people.  That usually covers all the functional areas of most organizations, and allows for meetings where everybody can be heard without being too long.  As meeting participants grow, so does the length of time to conduct the meeting (particularly if they are poorly run).


Who should be the project team leader?

The answer to this is usually obvious.  There always seems to be somebody who has the right temperament and experience to navigate the myriad of issues and personalities.  Some key characteristics of a good project manager are:

  • Calm
  • Organized
  • Handles difficult people and situations well
  • Slightly reluctant to have the role
  • Runs a tight meeting
  • Is both tough and tender hearted
  • Respected by the majority of the people to get the job done

This should not necessarily be assigned by job function.  The IT manager may not be the best person to serve as the foreman of this effort.  Base the decision on the characteristics I listed above.  An IT manager or Controller who runs a lousy meeting and is obnoxious will make this effort more difficult than necessary.  Instead, use the senior, savvy operations manager (for example), though they may have no software experience.

Also, the project manager should not play a ‘rah-rah’ role.  That is for the project sponsor.  The project manager is there to execute and facilitate the project, not to be a cheerleader.


What part does the software price play in the overall evaluation?

There are two schools of thought regarding the evaluation of price.  One approach is to take it into account as part of the product.  Price gets a ‘weight’ in the overall evaluation.  For example, price could get a 40% weighting and the overall functionality of the software could get a 60% weighting.  In other words, price is evaluated at the same time as the product.

The other method is to evaluate price after everything else has been done.  The team does all of its work, and then the project manager passes out the pricing matrix.  This may simply decide it.  Members who were on the fence may be swayed considerably when all of the costs related to acquiring, customizing, and implementing the software are summarized.  This is our preferred method because it is less mathematical, more intuitive, and gives more room to ask questions and interpret.   It’s more human and gives rise to terrific discussions and team interplay.

Additionally, price carries a huge bias.  Getting the price information in advance can needlessly close an evaluator’s mind to functionality, particularly when they aren’t contemplating the fact that price is one of the few things that is negotiable in this whole process.   There is little upside to giving out the price information in advance.

Along these same lines, be extremely wary of large price discrepancies and discounts.  If one vendor is coming in significantly under what the other vendors quoted, be suspicious.  This may be an immature company who is looking to get market share (a foot in the door) at any cost.  When a vendor is chosen, you then become motivated for them to stay in business.  You don’t want a company that is going to give steep discounts to win software engagements unless there is a good reason for it.


Is there a ‘Best Practice’ with regard to software evaluation?

No.  That is a large consulting firm’s way of scaring clients into thinking that they are doing something stupid or something that is not in their best interests.  When it comes down to it, consulting firms are essentially doing the same thing, which is some version of the following:

  • Organization and Infrastructure
  • Project Planning
  • Requirements Definition
  • Preliminary Vendor Pool
  • RFP Creation
  • Evaluation Preparation
  • Creation of Scoring Templates
  • Product Evaluation
  • Vendor Evaluation
  • Final Selection

The ‘best’ part of ‘best practices’ is not determined by the market, but by the client - the purchasing organization. What works ‘best’ for them?


When casting the final ‘vote’ on which software to acquire, should the vote be unanimous or will a consensus do?

Obviously, unanimous is best – everybody is on board and there are no issues or politics to deal with going into the accompanying software implementation.  However, sometimes there is a split decision, which is okay, too.   Therefore, it is important to determine the voting methodology well in advance of the final evaluation and selection meeting.   If done correctly, everybody will have had ample time to give input, the evaluation criteria will be intact, and a natural solution will flow.


What does ‘Best of Breed’ mean?

‘Best of Breed’ means using a specific (‘best’) software program package for each particular application or requirement.  To share information between the applications, the information is either printed out from one package and manually input to the next, or the packages are linked either by the vendor or with a third-party ‘middleware’ package to provide varying degrees of integration.  These middleware systems generally require a considerable amount of IT time to set up and maintain.

There is quite a bit of literature written about the benefits of a fully integrated system versus those of a best of breed solution.   Two benefits of a best of breed solution are:  it is a narrow, best-fit solution (thus the term ‘best of breed’); and, these off-the-shelf products may be easier to use.

What is the downside of a best of breed approach?  There are several:  1) There will be multiple vendors and implementations to deal with.  2)  Interfaces will have to be built between the various software products.   These ‘bridges’ could possibly affect the integrity of an organization’s data.  3)  It is more difficult to measure the total software costs with a best of breed solution.  There are just more moving parts to calculate.

The project team needs to talk about these concepts regarding the possible use of a best of breed solution.  There is no right or wrong here:  what works for one organization may not work for another one.  However, it would be a mistake to not discuss all the options.


What is the difference between a software demonstration and a software examination?

A software demonstration is where the software vendor sales representative is controlling the process.   They set the agenda, show off the ‘cool’ features in the software and, in essence, do everything in their power to sell their product.

Be aware that many software sales professionals are very, very good at what they do.  Many times it’s an uneven playing field, as it is easy to be seduced by the ‘wow’ factor inherent in many software demonstrations.  Unfortunately, the ‘wow’ almost always has little to do with solving an organization’s problems.

A software examination occurs when the organization has taken control of the vendor visit.  They set the agenda, create a scoring methodology, prioritize requirements, train the software evaluation team on how they should act, determine ‘X’ factors, and create a ‘show me’ script for the vendors to follow.  The organization should also determine which scripts they are going to provide to the salespeople in advance.  The vendors hate this, because it takes a large portion of the salesmanship out of their hands.  Companies must be in control of what they want to see.  Do not defer to a software team that is trying to sell their wares.


What are some good questions to ask software vendors?

Q.  What is your company’s vision statement?

This is different than a mission statement.  It’s important to know where the company is going from a technological standpoint:   What is their vision?  What is the direction of their product?  Are they a trendsetter?  Do they even have a vision statement?  Who sets the vision?

These questions are very hard to answer:  some vendors can state their vision effortlessly, while others are completely stymied.

Q.  What is your unique selling proposition?

A unique selling proposition is the essence of what distinguishes and differentiates a company from its competition.  It should answer:  Why should we buy your product or service?  This is important!

Q.  What is your current number of employees and how many are dedicated to our business line vertical?

It is important to get a feel for the size of the organization.  It’s more important to know how many employees are allocated directly to your line of business.  For example, a vendor may tout their base of 900 employees, but their allocation for your particular business line may only be 15.  The 15 number is much more relevant than the 900.

Q.  What software products are you not in, but plan on introducing?

Essentially, the question asks what is on the horizon.  This may or may not be relevant, but it’s worth asking.

Q.  What products do you currently have in the market but will no longer support or plan on discontinuing?

This may be hugely relevant, particularly if this existing functionality is referenced when they answer the functional portion of the Request for Proposal (RFP).

Q.  Who are your major competitors?

The better firms will always know and will be able to discuss this confidently.  Do not ask why the vendor loses deals to the competition.  This will open up a can of worms and the answers are not relevant to the end decision.  Interesting?   Sure.  Relevant?  Probably not.

Q.  What is your market share?

The answer is usually skewed to the vendor’s benefit.  The follow up questions should be where they get their information and how they calculate market share.  A vendor with the most market share is not necessarily the best thing for an organization.   However, it does mean there are others who have taken this leap before, and their satisfaction level will be discovered during the reference checks.

Q.  Describe your percentage of revenue in terms of new sales versus annual recurring revenue?

This is a good question.  An organization that has some sort of stable annuity coming in through hosting services, license fees, or maintenance revenues is ideal.  A good recurring-sales-to-new-sales ratio should be around 70:30.

Q.  Please list all current lawsuits that your company is involved in.

Do not ask about just major lawsuits, as this gives them control of the question.  It is possible that the vendor may not want to give this information.  Stand firm.  This is most relevant to the significant investment of funds.   Clients sign a non-disclosure agreement that contractually binds their organization to non-disclosure.  It should be taken very seriously.  If supplied with litigation information, clients have an ethical and legal responsibility to keep it to themselves.   It is not water-cooler talk.

Q.  Do you have plans for any major changes within your organization?

Essentially what this asks is whether the vendor is 1)  downsizing, 2)  acquiring any other companies, or 3)  in negotiations to be acquired.  All are relevant to the evaluation process.

Q.  How many salespeople do you have?

It is healthy to have a good number of sales personnel at a software company.  It is an indicator of overall organizational health.


Who are the best software vendors?

While there is no ‘best’ software vendor, there is a best-fit software vendor for every company.   The evaluation project’s purpose is to find out whose functionality best fits a company’s detailed business and technical requirements at a price that is within its capital investment budget.


What type of scoring mechanism do you use during the software evaluation process?

Lupine Partners has developed some moderately complex scoring mechanisms.  They are difficult to describe in a Q&A format.  But, what is more important than the type of mechanism is how much of the final decision is based on ‘hard’ scoring and how much is based on impressions and intuition?

It should be approximately 50/50.  There are elements of both art and science in software selection projects.   Generally speaking, the “art” relates to working and dealing with people and with using intuition and experience to navigate issues.   “Science” relates more to the mathematical compilation of a functional score for a particular vendor.  The selection of software should involve both.  Despite what some say, the highest scoring software vendor shouldn’t automatically win.   Similarly, vibe can’t be the only factor.  Both art and science are needed.


How is choosing software different than purchasing other products and services for my organization?

Software is unique in that it is one of the few major capital expenditures (maybe the only one) that is critical to a business yet is always changing and improving - thereby making the initial product obsolete at some point.  A competitor may gain a competitive advantage by switching to a new software solution with some new/improved functionality.  

This may force an organization to make a change themselves to avoid losing market share.

The smart move is to stay abreast of the software market and to think at least 18 months out.   Be proactive, not reactive.


What is the hardest, most time-consuming aspect of the software selection process?

The hardest part will be defining the business and technical requirements.  In order to get a requirements-to-functionality match, the requirements definition process is critical - and difficult.  Some companies hire it out to consultants, but the consultants still have to interview key personnel and have their work reviewed by someone in the company.   The second most difficult thing of a selection project is convincing clients that the sheer cost of the acquisition will necessitate them going through the process in a mature and methodical manner.


What should we be looking for when hiring a consulting firm?

Companies should focus on a couple of key areas:  1)  Has the firm done this before, and do they have testimonials and references?  2)  Do they have a proven methodology, and can they explain it?  3)  Are they willing to work solely on the software evaluation and not on the accompanying implementation?   4)  How many people are they going to put on the engagement?  For most software evaluations, one person should suffice.


If I hire a consulting firm, will a team of consultants descend upon our workplace?   Can’t this be a one-person job?

It is appropriate to be concerned about this.  This is called ‘piling on’.  One consultant should suffice for most projects.


What is the best way to conduct software product demonstrations?

A scripted demonstration is where the vendors are given a script, in advance, of exactly what the client wants to see in their software.  What are the benefits to using this technique?  There are a number of them, and the implications are huge.

  • The client maintains control of the demonstration. By determining what they do and don’t want to see, they keep the agenda tight and focused on their needs.
  • It takes some salesmanship out of the equation. By taking most of the demonstration down to “boring” functional items (which tie back to business objectives), it takes away from some of the discussion of the newest features that may or may not be relevant to the business.
  • It will reveal if the software vendor salespersons are winging it or really want the business. Are they willing to work for the business by taking the time to create a terrific scripted demonstration? Are they willing to listen? Will they listen to you in the future?
  • It will clarify whether the client’s business objectives will be handled by the software.
  • It makes the software vendor salesperson “show you.” It provides a forum where all statements by the salesperson can be followed by “show me.” It is impossible for the salesperson to hide in this environment. If he is representing an inferior product, it will come out in this process.


Is it absolutely necessary to send out a Request for Proposal (RFP)?

In general, an RFP should be created when any of these conditions exist: 1. The price of the software can be negotiated. 2. There are multiple vendors that can provide essentially the same solutions. 3. There are multiple solutions available for the client’s business and system needs. 4. The lowest price does not necessarily win.

The RFP is the intermediary between a business and the software vendor. This document allows for consistency of answers among software vendors and allows each party to work from the same group of rules and understandings. It is an ongoing interpreter between the project team and the software vendor and will continue to be used in this regard through the final implementation. The document sets the framework for everything that happens going forward.


What is a vendor long list and how do I obtain one?

A vendor long list can’t be obtained - it must be created.  It is the list of vendors who survived the initial round of “tire kicking” that occurs when an organization begins talking about a new software solution.

Determining which vendors will be on the long list is more of a process of exclusion rather than inclusion.  Exclude “no chance” vendors based on a high-level review of cost, functionality, technological direction, and vendor maturity.  The number of available vendors in the initial pool can drive some of this as well.  It’s easy to start striking out vendors if this initial pool consists of 100 vendors as opposed to 15.


Do vendor reference checks really work?

In general, the answer is no.  Many companies are unwilling to give direct, unbiased recommendations and are also unwilling to say what their role may have been if things didn’t go as well as planned.  Also, software vendors are only going to give their Cadillac implementations as references.  A better tactic is to find references that are not on the vendor-provided list.


Is there any value to visiting software vendors’ offices, or is it a waste of time?

We believe this is not the greatest use of time or money.  Clients can be swayed both positively and negatively by the reactions they get from visiting the offices.  This is more of a box-checking exercise.