Score a single respondent’s answer to an RFx question.

The Questionnaire Tree

The fundamental control for designing questionnaires is the questionnaire tree in edit mode, as seen when the Project is in Draft status. Links at the top of the tree provide functions to create, edit, move and delete questions or sections. Whilst being very useful for editing an existing questionnaire, working directly with the Tree can be cumbersome to use when creating a big new questionnaire from scratch.

Creating Sections from a Text Format

When creating a new questionnaire with many sections and nested subsections SupplierSelect provides a way to write out the structure in plain text, and then simply create the entire structure in one go. This page can be accessed either by clicking “Import Options” ->”Bulk Create Sections” after clicking a Section in the Tree, or by clicking directly on “Bulk Create Sections” when first defining a new questionnaire.

This form uses dashes (hyphens) before each section title to determine the nested structure of sections and sub-sections. Of course this structure can be first drafted in an email or any other text file, and pasted once the structure is confirmed. Clicking “Submit” produces the following questionnaire structure:

Creating Questions from Excel

SupplierSelect supports any type of question element - multiple choice, text area, check boxes, file uploads - and these can be structured in complex grid layout questions. However, this very flexibility means that creating many questions can be time consuming. To help in such cases, SupplierSelect allows questions to be defined in an Excel spreadsheet format and then imported into SupplierSelect.

A very common situation is where a company new to SupplierSelect has previously managed their RFP / RFX questionnaires with spreadsheets. Thus many of the questions are already written in Excel format. Even in cases where the questions do not already exist in Excel, it can be more convenient to draft 100 new questions in a spreadsheet whilst working offline.

The option to import questions from a spreadsheet is found under “Import Options” link accessed by clicking on a Section title in the Tree. Questions can be imported Section by Section. This leads to a page which explains the format for the spreadsheet which must be used to create the new questions. This is a very basic format, with one column for the question title, and one for the question weight.

Custom Excel Formats

Many organisations have their own unique and sophisticated formats for RFPs written as Excel spreadsheets. Due this variety it is not possible to support all formats directly. However, SupplierSelect has developed a number of custom import scripts for specific customers, mapping different spreadsheet columns to sections or question elements. Please contact enquiries@supplierselect.com if you would like to discuss this option.

Any Other Ideas?

We are well aware that creating complex questionnaires is a difficult and time consuming process, so we are constantly on the lookout for ideas to make the process easier. Please add a comment to this article if you have any ideas.

In SupplierSelect, all questions are scored on the same scale (1-10 by default). Each question is then given a weighting which can be any positive whole number. By default, the weighting for a question is 1. Most questionnaires are organised into a structure of sections and sub sections, so SupplierSelect allows further weightings to be allocated to sections.

When a survey consists of sections and subsections (nested to any depth), with weights applied at both question and section level, the mathematics for deriving a total score for each vendor becomes surprisingly complex and error prone when calculated in a spreadsheet. Two alternative scoring formulas are supported by SupplierSelect.

For example, consider the following example RFP questionnaire for evaluating Hotels:

1 Location  (1)
    1.1 Distances From Key Locations (2)
    1.2 Mini bus collection (1)
 2 Services  (1)
    2.1 Swimming Pool? (1)
    2.2 How many bars? (1)
    2.3 Health Club? (1)
  3 Pricing  (2)
    3.1 Standard Room Rate (1)
    3.2 Deluxe Room Rates (1)

The figures in brackets show the weighting allocated to each section and question. These weightings are applied when calculating total scores for each Supplier’s answered response. SupplierSelect supports two different formulas for calculating Total weighted scores:

1. Arithmetic
2. Normalised

The difference between these formulas is that, in Normalised scoring, changing the weight of a question will only change the contribution of that question to the total score for the parent section, not to the Total Score for the entire RFx questionnaire. By contrast, if you increase the question weighting in Arithmetic scoring this will have the effect of increasing the parent section’s contribution to the Total Questionnaire Score.

Arithmetic Scoring Formula

This is the simplest formula. The total score for a respondent is found as follows:

1. Calculate the question score X question weight for each answered question
2. Sum the weighted question scores for each section, then multiply by the section weight
3. Sum the weighted section scores to find the total score.

This approach is illustrated in the following screenshot.

This is a simple way to calculate the totals. In summary, questions scores are multiplied by question weights and section weights, and then summed to produce a total score.

As described above, the crucial thing about Arithmetic scoring is that increasing the weighting for a question has the effect of increasing the contribution that the parent section makes to the total questionnaire score. In other words, the Section weighting does not alone describe the contribution of that section to the questionnaire as a whole. In order to separate the effect of question and section weighting we need to normalise the scores for each section and subsection

Normalised Scoring Formula

Normalising the scores converts the weighted score for each answer in a section to the percentage of the total possible score for that section. So the total possible score for a section is 100%. Therefore, with Normalised totals, Section weightings define and cap the contribution that a given section can make to the total questionnaire score.

Switching Scoring Formulas

Which scoring formula to use is determined on a per Project basis. There is an option to switch between formulas under the “Manage Weightings” link in the questionnaire tree.

Alternative Scoring Algorithms

For organisations that wish to follow their own scoring calculation formula SupplierSelect provides the ability to download all scoring data into spreadsheets for offline calculation.

Previously, it was possible for respondents to import answers from previous projects into a full questionnaire, but not to import section by section.

New functionality released today allows answers to be imported section by section. The Screenshot below shows a Response with Section 1.1 selected:

Following the “Import Answers” link takes you to the standard Import page where you can select a previous project to import answers from.

This article describes using Excel’s Pivot Table functionality to analyse data thus exported. The screenshot below shows how the a sample dataset appears when it is first loaded in Excel:

Note that to follow the steps in this article you need a questionnaire structured into Sections and Subsections. If this is not your case, then this article should give a fair overview in any case.

One row corresponds to one respondent’s answer to one question. Note that the same data is duplicated on many rows. For example, the Section Titles are repeated. This is correct for exporting to an OLAP style analysis tool. The data is said to be “denormalised” - ie data taken from a normalised SQL database is denormalised by dumping it all, with duplications, into a single file. Pivot Tables use this duplicated data to isolate values for grouping and aggregating data.

The next step is to create a pivot table in Excel. Select a cell somewhere in the recently imported data, click the “Data” menu, and select “Pivot Table and Pivot Chart Report”. This brings up the following Wizard:

Select the default options for Step 1, as shown above. For Step 2, Excel should, by default, choose a range of values including all the data you have downloaded from SupplierSelect. If not, you will need to select the data yourself.

For the final Step 3, the default option is to create the Pivot Table in a new Worksheet. This is the easiest and best option. Clicking “Finish” should bring to you a screen looking something like this:

Note that the Field List corresponds to the Columns in the original data list. These Fields can now be dragged into the Pivot Table as either:

• Page Fields - to filter the data processed in the table
• Column Fields - grouping the data into columns
• Row Fields - grouping data by rows
• Data Item - this is the data that you want to analyse - generally either Score or Weighted Score.

To get started, try the following:

1. Click on “Section” in the Field List and drag it into “Drop Page Fields Here”
2. Drag “Subsection” from the Field List into “Row Fields”
3. Drag “Respondent” into Column Fields
4. Drag “Weighted Score” into “Drop Data Items Here”

Then click on the Drop Down arrow next to “Section” and select a Section. The data is now filtered.

N.B. By default, Pivot Tables deal with Data Items by counting values - NOT by summing numbers. To change this behaviour, right click on a Data Item and select “Field Settings”:

To see the Sum of Scores by Section, choose “Sum of Weighted Score”. The Pivot is now Calculating total SubSection scores, by Respondent, for the Selected Section.

The great thing about Pivot Tables is that it is incredibly easy to play around with your data, and to create new reports or views of it. Try dragging the Fields (Section, Score) etc around the Pivot Table, and see how it responds. Pivot Tables can also produce Charts in a similar way - just click the Chart icon in the Pivot Table menu, drag around Fields and produce charts like the following, which allows you to pick a respondent and then view the distribution of their scores across sections as a Pie Chart:

Excel’s Pivot Tables are a very powerful yet convenient way of analysing large datasets. We hope this article will help users unfamiliar with Pivot Tables to benefit from them.

These charts can be viewed by selected the “Analysis” tab from the questionnaire browsing pages (reached by selected the questionnaire tree on the right after loading a project).

• Shipping fees by country
• Delivery times by country
• Insurance and indemnification by country
• Number of deliveries currently sent by country

To take the first example (fees), you could create one question for each country:

This is an acceptable approach when only a few countries are involved, but when there are 80 countries to consider, it involves a great deal of duplication. Using Dynamic Questions can hugely reduce the chore of building questions to collect the same information by topic (in this case countries).

To build a Dynamic Question, just create one row of input fields, and mark them as “Dynamic”.

When the question is displayed additional rows will be generated for each Topic:

Before you begin

Before you start using Dynamic Questions you must complete two steps:

1. Create the master list of Topics (e.g. countries). This is done from the Dashboard->Settings->Topics page. Any combination of these Topics can subsequently be assigned to a Project.
2. Open the project you are working on, then go to Advanced Options -> Project Topics, and assign the required Topics to the project.

A new tab has been added to the questionnaire pages inside Project - “Weighting”.

Like other questionnaire pages, this displays information relevant to the position currently selected in the questionnaire tree menu. If you click a section containing question, it lists all the questions in that section, and enables you to either edit the values directly, or to use plus/minus buttons to change their relative values.

Scale Up

The plus/minus buttons allow you to increase or decrease the relative importance of a given section or section. We aim to make these increases in increments close to 1%. However, because the weighting for individual questions must be a whole number, it is not always possible to do this. If you have 10 questions, all with a weighting of one, then it is not possible to increase the weighting of a single question by just one percent. In such situations, SupplierSelect will suggest that you “Scale Up”. This means increasing the weightings of all questions by a factor of 10, so that more fine grained adjustments are possible.

Benchmark Issues allow the evaluator to associate a “benchmark” response with any previous response from that respondent organisation. This means that scores are imported from the benchmark, and answers to the current response are compared with the benchmark. The benefit of this is that the benchmark may belong to a project that featured an unusually high degree of research and due diligence - you know these scores are good, so it makes sense to compare new responses to that benchmark.

For example, Acme Consultants are specialists in the Fashion industry. They evaluate clothing manufacturers for fashion brands. They issue very similar RFP questionnaires to subsets of the same 25-30 suppliers.

In June 2005 they undertook a sourcing project for a very big fashion label - F.U.B.A. It was a high value project, so they flew to each supplier and conducted numerous interviews to back up the information gained from the RFP questionnaire. They then scored and re-scored each response.

Later, in May 2006, Acme are working on a smaller project. This time, it’s a smaller project, and there is no budget to fly consultants around the world. However, they issue a very similar RFP to the 80% the same suppliers. Birmingham Textile group is one of the respondents. They have answered the same RFP questionnaire 3 times previously, and they need to decide which one to import their answers from. One of the previous responses was for F.U.B.A project, but they didn’t do very well on this (they lost the contract), so they don’t think their responses were good enough. So they use the response set from an earlier, smaller project where they did get shortlisted.

When ACME consultants come to score the responses to the current project, they would usually compare imported answers to the answers/scores from the source project. Where the answers are the same, it’s assumed there is no need to re-score them. But in this case ACME know they have one set of answers and scores in which they have great confidence - the F.U.B.A. project. So what they’d like to compare is to compare the current response answers with the F.U.B.A project.

Benchmark Issues allow them to do exactly that. By defining the “Benchmark Issue” for a current response, SupplierSelect will compare answers and scores to that project, not the project from where the answers were imported.

To set a Benchmark Issue, go to Projects > Project Title > Issue Title, and click the link to “Select Benchmark Response”.

Once Bencmark Issues are set, you can use the links under Analysis->Scoring differences to see a table showing the differences between scores awarded to Benchmark answers, and scores awarded in the current project. TODO - elaborate and screenschot.

This screenshots show the format of the scoring spreadsheet which can be downloaded from SupplierSelect for offline completion.

This functionality is found under the “Score Offline” link under “Project” in the tree menu on the left hand side of the page after you load a Project. The first step is to download the current scores in Excel format. This delivers a spreadsheet with one question per row, with a column for each respondent. You can now edit the scores directly in this spreadsheet. When finished, save the edited spreadsheet on your PC. Then return to the same “Score Offline” page in SupplierSelect, and click the “Browse” button to locate the spreadsheet and then the “Upload” button to import these edited scores back into SupplierSelect.

If you download a spreadsheet which already has some scores populated, and then just edit a few of these before re-uploading, SupplierSelect will only overwrite the scores that have changed.