Skip down to the research record

AI call scorecard checks: 5 rules before the forecast

Quota Forecast Editorial Team · Published · 9 min read

Verify an AI call scorecard before a forecast by checking transcript length, who scores each question, the score scale, timestamps, and Yes bias.

Key takeaways

  • Verify transcript length before trusting AI output.
  • Distinguish automatic from manual scoring.
  • Understand the score scale.
  • Use timestamps for verification.
  • Account for AI bias.

Why unvalidated AI scores skew your forecast

Using AI-generated call evaluations without validation risks distorting your pipeline health predictions. The core issue is a systematic tendency in the AI model to favor affirmative responses. As noted in the Kaia Coach Card Writing Best Practices documentation, the system exhibits a bias toward "Yes," which necessitates setting a high bar for questions such as "Did the rep identify the decision-making process?" reported by Outreach. This bias means that unverified scores may overstate deal progress if the evaluation criteria are not strictly calibrated.

Before you mix the score into a forecast category, read Pipeline, Best Case, Commit: Forecast Categories Explained.

Transcript and language requirements for valid processing

Before you trust any AI-generated evaluation, confirm the underlying transcript meets the minimum processing threshold. Gong states that AI answers are not available for calls answered by voice mail, or for calls with transcripts that have fewer than 100 words Gong. This limit applies specifically to the Gong platform; other tools may have different minimums, so do not assume this 100-word floor is universal. If your call falls below this count, the system will not generate an AI response, meaning you cannot rely on automated scoring for that interaction.

Language accuracy is equally critical for valid processing. If the initial transcription used the wrong language, the resulting AI content may be inaccurate or unusable. Gong provides a specific remediation path: retranscribe the call first using the Transcribe function in the correct language, accessed from the More actions icon in the top right of the page, then select Regenerate AI Content to rebuild the brief Gong. This two-step process ensures the AI engine analyzes the correct linguistic data before generating any insights.

To verify these requirements in your workflow, check the word count of your most recent call transcript directly. If the count is under 100, the AI scorecard will be empty or unavailable. If the language was misidentified, do not attempt to edit the existing brief; instead, follow the retranscription steps to force a fresh analysis. These checks prevent you from entering a forecast call with missing or corrupted data points that could skew your pipeline assessment.

Recommendation: Before your next weekly review, open three recent calls and verify their transcript word counts. If any are below 100 words, exclude them from AI-based deal health predictions and rely on manual notes instead. If you suspect a language error, use the retranscription feature to regenerate the content before referencing it in your forecast.

Rules for automatic and manual question scoring

Distinguishing between automated and manual scoring is critical for data integrity. According to Aircall, only Aircall predefined questions are evaluated automatically by AI. For teams using the Standard plan, Aircall states that all custom questions must be answered manually. If you rely on these custom fields for deal health predictions, you must account for the human effort required to populate them. Failure to do so leaves gaps in your data that can skew forecast accuracy.

In contrast, other platforms offer broader automated capabilities. Avoma describes its AI Coaching Assistant as a tool that helps save time on listening to meetings and calls by auto-scoring them based on parameters defined in default Avoma AI scorecards. This automation reduces the manual burden for standard evaluations. The Avoma documentation notes that the AI Coaching Assistant will automatically score the member with the scorecard and display the score. This direct display of the score allows for immediate review without additional calculation steps.

When building your validation checklist, note the difference in scope. Aircall restricts automatic evaluation to its predefined set, while Avoma uses default scorecard parameters for its auto-scoring function. You should verify which questions in your current workflow fall into the "predefined" or "default" category or the "custom" category. For any question that requires manual entry, assign a specific owner to ensure it is completed before the weekly forecast call. This prevents missing data from entering your pipeline reports. Check your specific plan details to confirm which questions require manual input.

Score normalization and missing value handling

Scores in AI call evaluations are not always on the same scale, so verifying how each tool normalizes values is a critical step before using them in your forecast. Gong’s documentation explains that values are normalized, which means they are adjusted to a number within a certain scale; in this case, between 0 and 1, as reported by Gong. This normalization process ensures that different question types contribute proportionally to the overall score. For instance, Gong provides a specific calculation example: 7-1 = 6; (3-1)/6 = 1/3; normalized value is 0.33, according to Gong. Understanding this mathematical adjustment helps you interpret why a raw score of 3 on a 1-to-7 scale translates to a specific decimal in the final total.

Handling skipped questions varies by the type of question involved. Gong states that if the skipped question is a range question from 1 to 5, the value given to it in the overall score is 1, as published by Gong. Conversely, if it’s a yes/no question, the value given is 0, according to Gong.

Avoma uses a different approach for non-binary questions. For non-binary evaluation type of questions, the score will be prorated based on the number of options, as reported by Avoma.

Gong also notes a customizable range question, from 0 to 100, that can be scored manually or calculated based on weighted questions, according to Gong.

PublisherNormalization ScaleSkipped Value RuleQuestion Type
GongBetween 0 and 11 for range (1-5), 0 for yes/noRange, Yes/No
Gong0 to 11 for range (1-5)Range

Manual verification steps for AI-generated call briefs

Before you use any AI-generated evaluation to predict deal health, verify the underlying data points against the source recording. Each tool handles re-evaluation and timestamp verification differently, so check the specific mechanics for your platform.

Re-evaluate scores when needed If an initial AI score seems off, you can delete an evaluation and re-evaluate with the same or a different scorecard if required, according to Aircall. This allows you to adjust the criteria applied to the conversation without starting the entire analysis process from scratch.

Trigger manual scoring In some environments, you can also manually trigger a request to the AI Coaching Assistant to score a conversation, reported by Avoma. This is useful when automatic processing fails or when you want to apply a specific coaching framework to a particular call.

Verify timestamps against the recording To ensure the AI is evaluating the correct part of the conversation, click on the timestamp to jump to the point of the meeting that the AI used to score that particular question, as noted by Avoma. This step confirms that the score reflects the actual dialogue rather than a misaligned segment.

Check for inherent bias Be aware that AI has a bias toward "Yes," so set a high bar when writing your evaluation questions, such as "Did the rep identify the decision-making process?", according to Outreach. This bias means that vague questions may receive positive scores even when the rep’s performance was weak, so specific, high-bar questions are necessary for accurate data.

Review the brief structure Call briefs provide an AI-generated summary of a recorded call, including a recap, key points, and next steps, published by Gong. When verifying these briefs, check that the recap accurately summarizes the conversation, that the key points reflect the main takeaways, and that the next steps match the actions identified from the call, as described by Gong.

Action step On Gong, open the call brief and check the recap, key points, and next steps. On Avoma, click the timestamp to jump to the point of the meeting that the AI used to score that particular question.

Illustrative example of one score before the call

Picture one recorded call with a transcript of 100 words. You keep that call because the transcript is not under 100 words, and you set aside a call answered by voice mail. The scorecard has 4 questions. Question 1 is a range from 1 to 5 and was skipped, so you enter 1. Question 2 is yes/no and was skipped, so you enter 0. Question 3 was answered with 3. Question 4 stays a manual note you finish before the meeting. You then read the recap, the key points, and the next steps, and you open the timestamp for question 3. You compare the entered 1 and the entered 0 with the answered 3 before you speak. Those checks are the whole worked pass for this one call.

Next step: check your transcript word count

Open your most recent call evaluation and check the transcript word count to ensure it meets the minimum threshold for AI processing. If the transcript falls below this limit, the evaluation will not generate AI answers. Verify the word count before relying on the scorecard data for your weekly forecast call.

For a close date that changed, see Close date change log: why the date moved. Before the forecast call, also read Quote checks before the forecast call: 4 tools.

FAQ: how to verify ai call scorecard accuracy before forecasting

What happens to the overall score if a range question is skipped?

For example, if the skipped question is a range question from 1 to 5, the value given to it in the overall score is 1, according to Gong.

Can AI automatically answer custom questions in all plans?

No, automatic evaluation depends on the plan type. As reported by Aircall, all custom questions in Standard must be answered manually. Predefined questions are auto-evaluated, but custom entries require manual input on that specific plan.

How does the system handle calls with fewer than 100 words?

Gong states that Gong AI answers are not available for calls with transcripts that have fewer than 100 words. This limitation also applies to calls answered by voice mail, meaning no AI evaluation occurs for these short or automated interactions.

Is it possible to manually trigger a re-score for a specific conversation?

Yes, users can initiate scoring outside of automatic triggers. Avoma notes that you can also manually trigger a request to Avoma AI Coaching Assistant to score a conversation. This allows for on-demand evaluation of specific calls.

How are non-binary evaluation questions prorated in the final score?

For non-binary evaluation type of questions, the score will be prorated based on the number of options, according to Avoma.

Sources