How am I supposed to evaluate this? The simple answer is probably – you will not be able to evaluate it and the problem is twofold: 1) You may not get the product or services you need; and 2) The fairness of the RFP process may suffer.

The Problem

Your ability to evaluate the RFP responses are largely depending on the instructions you gave the respondent as well as the questions you asked or HOW you asked them to be accurate.

A company we consulted asked “Please provide an overview of your company’s history” – the purpose of the question was to evaluate the ability to support the product the seller was offering based on the assumption that a company that had been around for years would be in a better position to offer the desired support. The company’s history can, in most cases, be found on their homepage and it will in most cases not provide any information usable information about their ability to support their products – neither will the age of the company by the way.

In another example a company asked “Does product ABC interface with system XYZ?” The answers was unusable because most vendors said “Yes” but the fact was that the majority of them needed to  develop the interface specifically and the price for that work was NOT included in the RFP. One vendor did however have a standard interface and the price WAS included, but that was not clear based on the answers received.
Asking the right questions

Let’s take the interface question from above as our case story and examine what good questions are. The company wanted to know if a standard interface was available to ensure that they would not have additional costs of integration afterwards. What they got was exactly that – extra costs.

The question is in itself not bad, but it should have been more granular to ensure that all of the relevant information was provided. A better structure of the question could be:

1.    Does product ABC interface with system XYZ?
2.    Provide a list of all standard interfaces provided with your solution and indicate how they apply to our specific solution/design/architecture
3.    Is any development, integration or configuration of the interface needed which is not included in this proposal? If so, please describe:

a.    what has to be done
b.    whom does it
c.    what cost is associated with it
d.    why it isn’t integrated in your proposal

4.    List the two most recent customers to implement alternative interface solutions and their motivation to do so
5.    List the two most recent customers to discontinue the use of your interface solution and their motivation to do so

More detailed and well thought through questions will allow an evaluation process which highlights potential vendor omissions, hidden costs and other problems with a proposal.

Are you wondering about question 4 and 5? The quick explanation is that a vendor will have to tell you about known problems with their product in either or both of these questions. If they do not explain but list a few companies you know something will need investigation before moving forward with that vendor. If no companies are listed you must ask them during the vendor presentation if the question was answered truthfully because if a known problem turns up after the contract is signed you can cancel the contract “for cause”

Where do the bad questions come from?

In most cases the problem comes from the requirements. They are not well understood by the team preparing the RFP, not well defined or not defined at all. The total absence of requirements is more common than one would expect and the result is often chaotic RFP processes, especially when it comes to the evaluation phase.

In some cases the bad questions also come from a lack of business understanding. A company asked “How old is your company?” The purpose of the question in this case was to evaluate the stability of the company. Good question but it does not provide the expected information about the stability. Let’s say that the vendor responds that they are 20 years old and premium partner of this company and gold partner of that company. What they don’t tell you is that they lost all of their top five customers in the last 12 month to a startup company with less than 2 years on the market?

You need to know why the question is important to you and what you want to find out with it. Only like that can you ask questions that will help you evaluate the vendor.

What you need to do

First you need to get the right people around the table and explain their roles in developing the questions.  You may want to read our document on RFP evaluation to learn how the questions and the rating of the RFP are linked.

You must understand that it is an iterative process to write an RFP. Make sure that you drill the questions with the team members. Why are we asking that? How do we evaluate the responses? What if a vendor answers X and not Y?

As you go through the process you will soon realize if parts of the requirements are not well defined or if the stakeholder definitions are not complete.
Many companies are thinking of this part of the process and cumbersome but it is in fact the phase that determine the success of the RFP Process.

Repeated questions

In most cases the development of the questions are broken down into functional team. Finance ask their questions, IT will ask theirs and maybe Sales will have their pitch also. The potential for the same question being asked multiple times is obvious.

There is a number of ways to mitigate that through either using software or even web based applications offered by a range of companies. But you should also consider including key questions in multiple locations as a kind of “nonsense checks”. If you include a similar question in both the technical section and also in the financial section you can check if the answers are consistent.