SupplierSelect enables buyers to define a set of evaluation criteria, structure these into sections and sub-sections, and then assign weightings to any level of the resulting criteria questionnaire. Previously, it was possible for buyers using SupplierSelect to show criteria weightings to respondents by downloading the weightings into a spreadsheet, and then uploading them as a project attachment (which can be downloaded by respondents). We noticed that this was a common need amongst our users, and have therefore added a simple configuration option to support this process.

The new setting “Expose Weightings to Respondents” can be set either when creating a project with the New Project Wizard, or by editing “Rules” under “Advanced Options” in the Project tree menu.

When this setting is selected, respondents will be able to view weightings when answering the RFP. A new link, “Weightings”, appears in the Respondents navigation menu. Clicking this shows weightings for all sections and questions in the questionnaire.

A strong RFP should ask enough questions to permit the buyer to identify the best vendor for their requirements. How many questions this is depends on the complexity of the good or service being sourced. Many of SupplierSelect’s customers issue RFPs with over 500 questions. Answers from all vendors are scored. Each question is assigned a weighting which determines how much it contributes to the total score for each vendor.

For a typical RFP this means 8 vendors answer 500 questions. That’s 4,000 scored questions. As answers are scored by the buyer SupplierSelect calculates a total weighted scores for each vendor. You can drill down to see total scores by section, subsection and question.

However, even given this ability to drill down and the automatic calculation of weighted scores, it can still be difficult to identify the issues that most clearly differentiate the bids from the various vendors.

To assist with this, SupplierSelect has developed a new sensitivity analysis report which highlights these key differentiators. The report makes the following assumptions:

1. The questions given the highest weighting are the most important and contribute the most the final decision
2. Questions where the all vendors were awarded a similar score don’t help to differentiate between vendors

So the report plots all questions as points on a chart with two axes. The Y axis shows the weighting for the question. The X axis plots the Standard Deviation for scores awarded to answers to that question. This allows us to divide the scatter chart into four quadrants:

The chart allows you to identify which question is represented by each scatter point by hovering over or clicking that point. Clicking a point drills down to the specific question. Clicking on different sections in the RFP Questionnaire menu filters the search to show only questions in that section.

Document vs Data Perspectives

SupplierSelect questionnaires are designed as a document which organizes questions into a hierarchy of sections and sub sections. Questions can be richly structured with multiple fields - text, multiple choice, grids etc. This perspective is convenient for working with the resulting RFP as a single cohesive document - delivered in web, MS Word or PDF formats. However, RFP responses often contain data which is useful outside of the document structure, e.g.:

1. Price information
2. Performance or service values
3. Product data

Such information can be useful for detailed vendor comparisons or for use in secondary documents such as Service Level Agreement or contract documents.

To enable users to make use of the information locked inside Vendor Responses we have developed a new feature:  “Answer Reports”.

Answer Reports

Answers reports collate vendor responses from different RFP questions into a single report. The report can be viewed online (see screenshot) or downloaded as a spreadsheet.

Managing Answer Reports

Answer Reports can be managed via a link in the Project tree menu:

Creating Answer Reports

Reports can be defined via an intuitive point and click interface:

1. Click “Add New Data Point”
2. Select the question containing the relevant data in the tree menu
3. Click cells to select the data point and the label value. The label can be edited afterwards

This article describes using Excel’s Pivot Table functionality to analyze data thus exported. The screenshot below shows how the  sample dataset appears when it is first loaded in Excel:

Note that to follow the steps in this article you need a questionnaire structured into Sections and Subsections. If this is not your case, then this article should give a fair overview in any case.

One row corresponds to one respondent’s answer to one question. Note that the same data is duplicated on many rows. For example, the Section Titles are repeated. This is correct for exporting to an OLAP style analysis tool. The data is said to be “denormalized” - ie data taken from a normalized SQL database is denormalized by dumping it all, with duplications, into a single file. Pivot Tables use this duplicated data to isolate values for grouping and aggregating data.

The next step is to create a pivot table in Excel. Select a cell somewhere in the recently imported data, click the “Data” menu, and select “Pivot Table and Pivot Chart Report”. This brings up the following Wizard:

Select the default options for Step 1, as shown above. For Step 2, Excel should, by default, choose a range of values including all the data you have downloaded from SupplierSelect. If not, you will need to select the data yourself.

For the final Step 3, the default option is to create the Pivot Table in a new Worksheet. This is the easiest and best option. Clicking “Finish” should bring to you a screen looking something like this:

Note that the Field List corresponds to the Columns in the original data list. These Fields can now be dragged into the Pivot Table as either:

• Page Fields - to filter the data processed in the table
• Column Fields - grouping the data into columns
• Row Fields - grouping data by rows
• Data Item - this is the data that you want to analyze - generally either Score or Weighted Score.

To get started, try the following:

1. Click on “Section” in the Field List and drag it into “Drop Page Fields Here”
2. Drag “Subsection” from the Field List into “Row Fields”
3. Drag “Respondent” into Column Fields
4. Drag “Weighted Score” into “Drop Data Items Here”

Then click on the Drop Down arrow next to “Section” and select a Section. The data is now filtered.

N.B. By default, Pivot Tables deal with Data Items by counting values - NOT by summing numbers. To change this behaviour, right click on a Data Item and select “Field Settings”:

To see the Sum of Scores by Section, choose “Sum of Weighted Score”. The Pivot is now Calculating total SubSection scores, by Respondent, for the Selected Section.

The great thing about Pivot Tables is that it is incredibly easy to play around with your data, and to create new reports or views of it. Try dragging the Fields (Section, Score) etc around the Pivot Table, and see how it responds. Pivot Tables can also produce Charts in a similar way - just click the Chart icon in the Pivot Table menu, drag around Fields and produce charts like the following, which allows you to pick a respondent and then view the distribution of their scores across sections as a Pie Chart:

Excel’s Pivot Tables are a very powerful yet convenient way of analyzing large datasets. We hope this article will help users unfamiliar with Pivot Tables to benefit from them.

1. The “best” vendor is the one most suited to your requirements
2. The only “Expert” is yourself - the buyer. Stay focused on your own requirements
3. Don’t be afraid to let the vendors educate you about their products or services
4. Closed questions (multiple choice) are better than open questions
5. Fewer good questions are better than many questions of variable quality

Objective Bid Comparison

An RFP (Request For Proposal) doesn’t have to involve questionnaires and surveys. At its most simple, an RFP
can be just that, a letter or email to potential suppliers saying “We want to buy a new product/service, please send me your proposal”.

The problem with this approach is that the proposals returned will look very different from
one another. This makes accurate comparison difficult and time consuming, and it will frequently be necessary to request further information from some or all of the bidders.

A well structured RFP should request specific information from bidders, and require that
this information is presented in a format that permits side by side comparison. Questions should be framed so as to elicit only information pertinent to the evaluation, and which lends itself to objective comparison and ranking of vendors.

Who is the Expert?

A major concern when starting an RFP project is whether or not the RFP author is fully qualified to assess the vendors. Big RFP projects often involve comparing very sophisticated products or services. The buyer is rarely a expert in the category. This can seem a weakness to be exposed by sharp talking salesmen. A common reaction is to draft a huge RFP that asks lots of complicated questions about the product or service being procured - the assumption being that this will put the vendor salesmen on the back foot. It doesn’t work. Sale teams receiving RFPs like this conclude:

1. The buyer doesn’t know what they want
2. They will be lost in the deluge of information returned by the RFP responses
3. The RFP is going to turn into a beauty contest, and the final decision is going to go to the safest bet (nobody gets fired for buying IBM…)

The best defence is for the buyer to focus on their own organization’s needs. And the only expert in that respect is the buyer, not the vendor. An RFP is about putting power and control into the hands of the buyer, not the salesman. To ask lots of technical questions is to take the battle into the vendor’s territory. Here are two example questions for buying a customer software system:

• Does your product/service include complex notifications features xzy, abc and foobar2? - BAD
• ABC corporation needs to keep clients informed of the progress of their support tickets. Please describe how your system supports this objective - GOOD

RFP Structure & Criteria Granularity

Generally, the greater the degree to which information requested by an RFP is broken down into sections and individual questions, the better. Each section and question can be given a weighting to determine how much it contributes to the total score for a supplier. However, if questions become too fine grained then it can be impossible to assign a meaningful weight.

For example, suppose we start with a broad question about an IT vendors support services:

This question will invite a very general answer, covering product documentation, website support,
phone support, 24/7 coverage and so on. A buyer might view some of these things as important,
others as not. But with only one question there is no way to reflect these priorities. A better structure is:

Inspired by this improvement, we might decide we want yet more control and elaborate further
on question 3.4:

We’ve achieved a greater level of detail, but by further breaking down the criteria it becomes more difficult to assign useful weights or scores. Do we really care whether forum comments are moderated? If so, do we care more about that or anonymous comments? In reality, we probably don’t care much either way, and neither should we.

So good rfp questions should be fine grained enough to allow us to fully express our priorities, but not so detailed that we get lost in irrelevant detail. In cases where we wish to assign a single weighting to a question, but where we need to be sure to capture specific information within the question, we need to design questions with sub-elements. This is a screen shot of such a question designed in SupplierSelect:

This question collects very detailed information, but is treated as one unit for scoring and weighting purposes. One of the more flexible characteristics of SupplierSelect is the ability to create questionnaires with any combination of sections and subsections to any depth, and with questions containing any number and type of input elements.

Closed vs. Open Questions

Open questions allow bidders to give a free form answer to a question. in some cases this may be difficult to avoid:

“Please describe your company history”

Answers to open questions can be difficult to compare. To take the above example, what might it be about a company’s history that will affect their suitability as a supplier? One obvious factor is age. A young company might be viewed as being less stable or reliable. So we might replace the open question above with a new question:

“Please indicate the age of your company in years”

Now we’ve got some tangible data for side by side comparison. If we assume older is better, then we could score comparatively, so that the oldest bidder gets the best score for this question. However, on reflection we might find that this gives rather unhelpful results.

Does Vendor C really deserve a higher score than Vendor B? Not really. So we need to think about our requirements. Let’s say that we’re very worried by companies less than 3 years, concerned about companies less than 7 years old, and totally relaxed about the rest. So we can improve our question again:

Please indicate your company’s age:

3 years or less 3-7 years More than 7 years

Now we can assign a score to each option and have a fairly rational basis for comparison.

Questions can be too closed

We reflect further on our question about company age. Are there circumstances under which this rigorous, closed questioning might be unreasonable? For example, what happens if the company was formed as a subsidiary of a larger, older company, is staffed by a long established team from the parent company, and is well funded by the parent? We cannot foresee all the possible circumstances that might affect our assessment of this point. So we decide to allow the bidder to qualify their answer with a comments box:

Please indicate your companies age:

3 years or less 3-7 years More than 7 years

Comments:

With this question we can now apply auto scores to each multiple choice question so that scores are added automatically when the bidder submits their response. But the comment field allows us to moderate these scores for special cases.

Well Understood Requirements; Well Framed RFP Questions

Thinking through what the sourcing exercise seeks to achieve, and breaking this down into detailed service or product requirements is the critical aspect of an RFP. When the requirements have been clearly enunciated the process of drafting good RFP questions is relatively straightforward. Good questions are those that elicit the information required in a way that encourages accurate evaluation and comparison of vendors.

Drafting an RFP should be an iterative process. As the RFP author endeavors to frame questions it soon becomes apparent which aspects of the project definition and requirements are vague or poorly defined. Struggling to write good questions should not be thought of as a tiresome chore but as an exercise in better understanding what problem is being addressed, and what service or product characteristics are required to solve it.

Web Based RFP Software & Collaborative RFP Design

Using a web based RFP software system can help the buyer to create good RFP questions because the questions are accessible, and editable, online. Thus a team of RFP participants can collaborate on the same questionnaire, reviewing, editing and approving questions. Furthermore, many of the same questions tend to crop up across different RFP projects. Fully featured RFP management software enables the user to build a library of reusable questionnaire elements (sections and individual questions) that promote reuse between projects.

Further Reading

Effective RFP Development, by Bud Porter-Roth

Electronic Medical Records system adoption has been much slower than was anticipated 10 years ago. Whilst the benefits of industry wide adoption of standards compliant systems are obvious, technical and cultural barriers have hindered growth in the sector. One reflection of this is the wide variety of vendors in the market. These range from solutions produced by international software houses to niche products originally developed for in house use. Finding a way to navigate through this array of alternatives is daunting - this is the challenge that EMR Match addresses.  The process for anyone wishing to run a EMR RFP via EMR Match is as follows:

1. Sign up at www.emr-match.com
2. Create a new project, starting with the EMR evaluation template, and then customize as required.
3. Use the EMR Screen service to narrow down your search to the 6 or so vendors that best match your needs.
4. Invite the short list of vendors to respond to your RFP

From then on the RFP process benefits from all the standard features of SupplierSelect -  structured workflow, in process messaging, detailed audit event log, multiple evaluator scorers, flexible weighting and reporting options.

Student Information Systems (SIS) are crucial to the running of a modern university. Such systems provide a single view of a students data - billing, housing, academic performance etc. In some ways they are the analog of ERP systems for manufacturing or distribution companies.

Auckland University’s RFP project faced a number of challenges. Firstly, a Student Information System is intensively used by many university staff in different functions, with different priorities. Therefore these diverse and sometimes divergent and priorities must be coordinated in the RFP. Secondly, New Zealand’s public sector procurement guidelines are some of the most rigorous in the world. This can greatly increase the administrative burden for running an RFP project.

SupplierSelect is designed to meet precisely these challenges. Multiple evaluators can collaborate on defining requirements and on assessing vendor offerings. Strict procurements guidelines for vendor communication, process transparency, and fine grained auditing can be enforced by clicking one or two options when setting up an RFP project.

For this project Auckland University of Technology registered 34 evaluator users, and defined 430 questions, broken up into 7 sections with additional subsections. The questionnaire was begun by importing an Excel spreadsheet , and was then customised and extended online, using SupplierSelect’s RFP question editing functions. Many questions were closed (multiple choice), but with comments available, as described in a recent article about creating good RFP questions.

For the RFP evaluation, scores from different evaluators where collated using SupplierSelect’s multiple score set functionality, with many users choosing to use evaluate RFP responses offline before importing the scores back into SupplierSelect for aggregation and analysis.  4 different weighting sets were configured, each reflecting a different scoring methodology, rather than the typical approach of having different weighting sets to reflect different people’s or departments priorities.

It’s always been possible to associate an attachment with a question in SupplierSelect. For example, a typical question is:

“2.4.2. Please download, complete, and upload the attached spreadsheet ‘pricingMatrix.xls’”.

It’s also been possible to attach images which are then referred to in the question:

“3.5.2. The attached file ’specification.gif’ is a technical drawing of the solution we require. Please indicate timings and costs for the implementation of this specification”

It is now possible to embed an image inline with the question:

To embed an image in a question, add a “Attachment Element” to a question. In the field editing panel at the top of the edit page there is form upload field. Use this to pick the image file from your local computer and then upload it. You will then be prompted with options for whether to display the image inline and whether to crop its size.

When a vendor comes to answer this question, it looks like this:

• Price information, and you wish to compare prices between respondents
• Financial background information
• Technical details about the product or service being sourced

Previously SupplierSelect text boxes were entirely free from - the user could input any information. A new feature recently added is Input Validation. When applied to a text input field, this checks the data input by a respondent conforms to a given “validation mask”. The following masks are predefined:

• Positive Integer or zero - e.g “23″, “0″, “342″
• Integer - the data must be a whole number, positive or negative e.g. “89″, “-5″
• Float - a positive or negative number, optionally with decimal places eg. “34.2334″, “34″, “0.342

These masks can be set when designing a question.

There is drop down menu allowing one of the predefined masks to be chosen:

If the existing existing masks don’t fit your requirements, it is possible to design your own by choosing “User Defined” from the menu. This is an advanced topic, introduced below, but you should always feel to contact us , in which case we’ll often be able to provide the mask you need by return of email.

Advanced - User Defined Validation

For the technically inclined, it is also possible to define your own validation mask using Regular Expressions. Regular Expressions are a very powerful programming tool for matching text patterns.They are not at all simple to use in more complex cases, but it’s a good thing to ask your IT guys to help out with.

For example, if you were asking a bank to take the time they take to settle transactions, you might wish to allow answer in the form:

“3 days” or “36 hours”

In this case, you want a rule saying:

“Allow numbers (optionally decimal) followed by either ‘day’,'days’,'hour’ or ‘hours’. “.

A regular expression for this would look like:

/^\d+(\.*\d*) (day|hour)s*$/

For example, if you wished to restrict an answer to one containing “yes”, “no” or “maybe”, you would write the following regular expression:

/yes|no|maybe/i

The pipe character “|” is translated as “or”.  The forward slashes indicate the beginning and end of the expression. The trailing “i” character indicates that the search is case insensitive.

This would allow answers such as “yes, in all cases” or “no, never”. To allow only “Yes”,”No” or “maybe” answers the regular expression must define the beginning and end of the input:

/^(yes|no|maybe)$/

“^” means the match starts from the beginning of the input, “$” means the end of the input. The allowed values are in brackets to indicate “allow any one of this group”.

To make the match case insensitive you can add the “i” flag:

/yes|no|maybe/i

Problems with the RFx Process

The traditional RFx cycle involves the buyer requesting a bunch of information from vendors, and providing a deadline by which they must submit the information.This workflow is ideal for projects where the buyer will only request information from the vendor once - such as awarding a specific contract. But in many cases it will be necessary to request the same information repeatedly, e.g.

• A periodic review of vendors, such a due diligence exercise. In many sectors it is necessary to periodically request new information from the vendor for regulatory and compliance purposes.
• A procurement consultant might issue similar RFx to the same group of vendors. Much of the information in these questionnaires is essentially reference data, and does not change on a client by client basis. This information needs to be timely and validated.
• An industry body such as a trade association or a publication might wish to collate a set of data about all industry participants to help buyers to form a short list before entering into a full blow tender.

In such cases, following the traditional RFx workflow results in two problem:

1. Duplication. Much of the information requested will be the same each time the RFx is issued, duplicating work for both vendor and supplier. SupplierSelect has always had some tools to help deal with this duplication but the process still generates a significant volume of duplicated data.
2. Timeliness. A periodic process (such as an annual review)  can only take a snapshot of a given vendor at one time.

Updateable Information

To address these issues, and based on detailed feedback from our customers, SupplierSelect has implemented a new feature - “Reference Projects” ¹. A questionnaire is created by the evaluator, and vendors are invited ² to respond as for a normal project.

From here on the workflow diverges. With a reference project, the evaluator is able to view the respondents’ answers as soon as they are entered. Scores can be maintained by the evaluator (these are not visible to the Respondent). This process can continue indefinitely.

The option to create a Reference Project is presented on stage 4, “Rules”, in SupplierSelect’s new Project Wizard.

When a Vendor views their list of invitations, a Reference Project’s invitation is marked as being a status “Updateable”. Vendors can also use Reference Projects to maintain a set of of “Standard Answers” which can then be used for standard bid development.

Scoring Efficiencies

Reference Projects provide a further area of efficiencies, especially for industry niche consultants.

Specialist consultants are often engaged in many RFPs within the same set of vendors, and often using questionnaires that share a good deal of the same questions. Such case involve a great deal of duplication. The vendors receive and must answer the same questions. The consultants must re-examine and re-score what are often the same answers as they have previously analysed.

SupplierSelect has always provided a solution to this - answer importing. This is a feature that allows vendors to import their answers from a previous project with one click. Scores to answers which are unchanged since the last project are also imported. This enables both sides of the transaction to cut down on duplication a good deal.

The Problem with Answer Importing

The solution outlined above has been successful and is vendor popular with big sales teams using SupplierSelect to submit tens of thousands of answers per year. However, it has drawbacks for both evaluators and RFx respondents.

For evaluators, having scores silently copy across with imported answers can save a lot of time wasted re-analyzing the same information. But it can lead to scoring inaccuracies creeping in. One problem is that there is often a great deal of subjectivity involved in scoring, and different RFxs can reflect different strengths and weakness amongst the vendors. When scores are copied blindly between projects it can  be tempted to accept them without sufficient scrutiny. Another issue is that while a score from three years may have been based on a site visit and detailed due diligence this view may no longer be sufficiently timely. Linked to this is the problem that the evaluator has no control over where the respondent chooses to import their answers from. In a big practise there may be dozens of RFPs to choose from. It’s unlikely that the quality of analysis and scoring was equally high in all projects, thus the the robustness of the scores can become corroded.

For vendors the big problem is that they have to cope with RFPs both from within SupplierSelect and being issued through Word documents or spreadsheets. Additionally, many sales teams use 3rd party proposal management software such as PMAPS (www.proposalsoftware.com).  Vendors must then decide where to maintain their canonical set of up to date answers.

Proposal Management with Reference Projects

Both evaluators and respondents can benefit from using Reference Projects to help with answering and scoring RFxs.

With Reference Projects an evaluator working on a new RFx can tell SupplierSelect to use “Reference Project A” as the template for the new RFx. SupplierSelect will then perform a full text match against all the answers in both project. Where matches are found, answers can imported. Because the Reference Projects hold the most timely and accurate information (the validity of which having been assessed on and ongoing basis) this ensure that each new engagement is being referenced against the best possible data.

Similar benefits accrue to Respondents. By managing their “authorized” set of answers in a Reference Project, a sales team can be confident of where to find the best and most timely answers to their questions.

Forthcoming releases of SupplierSelect are going to extend this functionality for respondents:

1) Providing an MS Word plugin for transferring information between SupplierSelect and desktop applications.

2) The ability to create and manage their own Project and thus define a database of precisely the answers they want to save.

¹ “Projects”, in SupplierSelect, provide a way to group together all information related a specific evaluation exercise such as a tender or RFP.

² It is also possible to mark a Project as “Public” - this means that vendors can submit information without being explicitly invited by the evaluator.