Copilot Questions

Feature Overview

The Questions feature allows Analysts to manage typical questions their Copilot and Skills are expected to answer. This can server as a hub for your copilot from before the first skill is even written.  Start by identifying the questions that a copilot will be expected to answer.  You can then keep a watchful eye as the copilot is built out to handle more of the questions originally envisioned. 

Key Benefits and Purpose

  • Organize questions into Collections.
  • Run regression tests on demand for an entire collection.
  • Use questions as Starter Questions that are offered to users as they begin a chat.
  • Questions can be used as Previews to help exercise your code in Skill Studio.
  • Questions can also be used as Hints to guide the LLM when similar questions are asked by users.

Accessing the Feature

Navigation Path

  1. Navigate to "Skill Studio".
  2. Choose the "Copilots" area using the left navigation panel.  Copilots are selected by default upon first navigating to Skill Studio.
  3. Select the "Questions" tab at the top of the page.

Step-by-Step Instructions

Creating a Question

  1. Navigate to the Questions tab within "Copilots".
  2. Click on "Add New Question."
  3. Enter the question in the provided field.
  4. Add a Hint to guide the LLM when the question is asked.
    • Tip: You can use the keyword {skill} when adding a hint.  This will automatically use the Skill Name chosen in the dropdown for the question.  This can be helpful if your skill names change.
  5. Select the Skill best suited to answer the question.
  6. Define the Skill Preview Parameters.
  7. Check the "Start Question" box if it should appear as a Starter Question for users.
  8. Choose or create a Collection to store the question.

Managing Collections

  1. Navigate to the Collections section within the Questions tab.
  2. Create new Collections by clicking "Add Collection."
  3. Organize existing questions into appropriate Collections.

Previewing Questions

  1. In the Questions tab, select a question.
  2. Click on "Preview" to see the answer generated by the Copilot.
  3. Review and manually adjust if necessary.

Running Diagnostics

  1. After a question is run, go to the Diagnostics section.
  2. Review the diagnostics data provided.
  3. Troubleshoot any issues by analyzing the output and adjusting hints or Skills as needed.

Use Cases

Regression Testing

Description

Using a Collection to run regression tests ensures that the Copilot and Skills perform as expected over time.

Steps to Implement

  1. Organize questions into a Collection.
  2. Navigate to the Tests tab within Skill Studio.
  3. Select the Collection and run the regression test.

Benefits

  • Validates Copilot and Skill performance.
  • Identifies issues proactively.

Example

A collection of frequently asked customer questions is tested to ensure the Copilot provides consistent and accurate responses.

Starter Questions

Description

Presenting Starter Questions to users when they start a new thread helps guide them and improve their initial interaction with the Copilot.

Steps to Implement

  1. Create questions and check the "Start Question" box.
  2. Save the questions in the appropriate Collection.
  3. When a user starts a new thread, the Starter Questions are displayed.

Benefits

  • Enhances user experience.
  • Provides immediate guidance to users.

Example

New users are presented with common troubleshooting questions, streamlining their interaction with the support Copilot.

Best Practices

Collections

A Collection should be used like folders for organizing your questions.  For example put all of your Starter Questions within one collection.  Similarly put all of your hints within another collection.  Then organize each of your regression tests into their own collections.

Hints

Hints are inserted into the System Prompt during runtime. Only the most relevant semantic matches are inserted.  On performing parameter selection on the relevant semantic matches for the already selected skil are inserted.  Be sure your hints stay current with the configuration and data of the overall Copilot.  For example if a hint suggests a certain Skill should be run then make sure that Skill name stays current or use the handy {skill} to have it automatically stay current. Allowing your hints to diverge from the actual system configuration can result in unintended outcomes.

A suggested format to help with both Skill selection and parameter selection is a hint in this format:

[{Skill}](multiparameter1=[value1], parameter2=value2, etc...)

Previews

Previews use JSON to execute them.  The JSON takes the format of each parameter and the expected values for that parameter.  When running that preview in Skill Studio while developing Skills it executes the current Skill with the exact parameters specified in the JSON.  One can find sample JSON formats on the diagnostic panel under the Grounding section.

{ "brand": ["brand X"], "metric": "value", "periods": ["q4 2023"]}

Updated

Was this article helpful?