Test the assistant and find out what it costs
In System → AI Centre → Assistant tests a set of questions is run through the real assistant, at the basic level and with your company’s data. When it finishes it says which questions it answered well, which ones it gets wrong and why, and what a month of use would cost compared with the credits a plan includes. It is a technical support screen.
What questions does the set contain?
From four families, about thirty in total:
| Family | What it checks |
|---|---|
| Self-knowledge | Help with the program itself (‘where are favourites set?’) and questions about the assistant (‘which model are you?’) |
| Tasks in sequence | Requests with several chained steps (‘create the customer and then make them a quote’) |
| Model escalation | That it offers the professional analysis when the question needs judgement, and doesn’t offer it for a figure |
| Complex queries | Data questions that require combining several lookups (‘how much have I invoiced, how much am I owed and by whom’) |
Each question opens a new conversation. Some have several messages in a row, to check that the assistant follows the context (‘and how much do they owe me now?’).
Can it change anything in my company?
No. The assistant prepares its proposed actions as usual, and the test run cancels them all when each question ends: it never confirms anything. The only thing it uses is credits, just as if a person had asked the questions.
While it runs you can go to other screens: the test run carries on by itself and saves each question as soon as it ends. What you shouldn’t do is use the assistant panel in the same tab, because the test run uses that same assistant; when it finishes, the panel starts a new conversation. If the tab is closed, the test run stops, but what was done up to then is saved as an incomplete run.
How is it decided whether an answer is right?
In two ways, and both can be seen in the Case by case table:
- With fixed rules. Each question says what has to appear in the answer (‘My menu’), what must not appear (the name of an artificial intelligence model, never), whether it has to offer the professional analysis, whether it has to leave an action ready or whether it has to give figures.
- With a judge. If the Judge box is ticked, another economical model reads the conversation and says in one sentence whether it does what was asked. What it said can be seen by opening See the answer.
Each question comes out Right, Wrong or Could not be assessed. The last one is for what does not depend on the assistant: if your company’s professional analysis is used up this month, the questions that should offer it cannot do so, and they don’t count as a failure.
What do the cost columns mean?
All of them start from what each question really cost —what was deducted from the credits— and take it to a month of use with the Queries per month figure (200 to start with: about ten per working day).
| Column | What it says |
|---|---|
| Credits per query | What a query like this costs at the basic level |
| Monthly cost | What all the month’s queries would cost if they were like this one |
| Shortfall | What is missing over the credits included in the plan chosen in Compare with the plan; ‘covered’ if they are enough |
| Advanced monthly cost | The same, adding the professional analysis every time it was offered |
| Advanced shortfall | What is missing in that case |
The professional analysis is not accepted unless you tick Accept the professional analysis when it is offered: without that box, the advanced cost uses what the offer itself estimated.
Are previous runs kept?
Yes. Each run is saved with its date, and the Run drop-down lets you go back to any of the earlier ones. That way you can compare how the assistant answers before and after a change.
If something goes wrong
| What you see | What has happened | What to do |
|---|---|---|
| The escalation questions come out ‘Could not be assessed’ | The professional analysis for the month is used up | Increase the month’s budget and run the set again |
| A question comes out ‘The turn failed’ | The assistant didn’t answer in time or gave an error | Run it again; if it happens again, look at the incident in the AI Centre |
| The run shows as ‘incomplete’ | The tab was closed, Stop was pressed or the server restarted halfway | What you see is valid; for the rest, run the set again |
| The cost of a question looks too high | Someone in the company used the assistant at the same time | Run the set at a quiet time |
See also: See what the assistant uses · Ask for a professional analysis · See what level it answered you at
More about Assistant
- Can I attach a CSV or a spreadsheet for the assistant?
- Can I show the assistant an image or a PDF?
- How do I teach the assistant about my company?
- How do I add several items at once with the assistant?
- How do I choose how much effort the assistant makes?
- Working with the assistant
- How do I ask for a professional analysis and what does it cost me?
- How do I ask the assistant to do something in the ERP?
- How do I put the assistant on my website?
- How do I ask the assistant something?
- Can I tell the assistant to stop asking me about the professional analysis?
- What does the assistant know and what doesn’t it?