Einstein Audit and Feedback setup
Key Takeaway
Audit and Feedback stores each generative AI request and how users reacted in Data 360. The data can contain PII, so restrict who can query it, including System Admin users under the default "Allow All" policy.
Einstein generative AI Audit and Feedback records each generative AI request and how users react to the response. It stores that data in Data 360. You turn it on to monitor generative AI usage, to check that the Einstein Trust Layer is masking sensitive data and catching toxic responses, and to report on response quality.
What Audit and Feedback collects
Audit and Feedback collects two types of data. Audit data, also called the audit trail, comes from the LLM Gateway. Feedback data comes from the Feedback API and records how users reacted to each response.
Audit data includes these items:
- The prompt template ID
- The hydrated prompt
- The PII-masked prompt
- The retrieved data
- The LLM response
- Safety and toxicity scores
- Model details, such as temperature
Audit data also tracks Einstein Trust Layer functionality, such as data masking and toxicity detection. Einstein Trust Layer detector results are stored by detector type. Detector types include PII, TOXICITY, PROMPT_DEFENSE, InstructionAdherence, and TaskResolution (Beta), so the data covers more than toxicity alone.
Feedback data includes these items:
- Thumbs up and thumbs down reactions with reason text
- Whether the user accepted, regenerated, modified, declined, or ignored a response
- The modified final response that was used