Event-Driven Model Scans

Event-driven model scans automatically run in response to specific events. Use these to react to changes in real time. You can create either Custom or Managed event-driven model scans. Custom scheduled model scan enables you to define your own automation logic. With managed scheduled model scan you can use predefined out of the box integrations.

Event driven custom model scan

1. In the side menu, go to AI, then select Model Scanning.
2. Click Create Model Scan. Choose the AI Event Driven Model Scan.
3. Fill out the Model Scan form:
   • Name - Enter a name for your model scan.
   • Project - Select the project that will be associated with the model scan. Or click Create to create a new project.
   • Description - Provide a brief description of the model scan.
   • Events - Choose one or more events that will trigger this model scan.
   • Integration Type - Choose Custom.
   • Integration - Choose an existing Lambda or Webhook integration, or click Create to set up a new one.
   • (Optional) Context - Click Add key to add key/value pairs for model scan parameters.
4. Click Submit.

Event driven managed model scan

1. In the side menu, go to AI, then select Model Scanning.
2. Click Create Model Scan. Choose AI Event Driven Model Scan.
3. Fill out the Model Scan form:
   • Name - Enter a name for your model scan.
   • Project - Select the project that will be associated with the model scan. Or click Create to create a new project.
   • Description - Provide a brief description of the model scan.
   • Events - Choose one or more events that will trigger this model scan.
   • Integration Type - Choose Managed.
     • AWS Bedrock Foundational Security Testing - This managed model scan will fuzz your endpoint and generate observations.
   • Project - Select the project that will be associated with the model scan. Or click Create to create a new project.
   • LLM probe - Select the LLM probe for your model scan. These probes test large language model (LLM) interactions and ensure they work as expected. Learn more about the probes and their uses here.
   • Bedrock integration - Select a previously created AWS Bedrock Invoke Integration. Or click Create to set up a new integration.
   • AWS Region - Specify the AWS region where the Bedrock model will run.
   • AWS Model - Select the AWS model that will be used in the testing process.
   • Token budget - Specify the maximum number of tokens to use for running this model scans within the defined budget interval. This helps you manage your token consumption effectively, ensuring you stay within your allocated budget for each model scans execution.
   • Token budget interval - Define the interval for the token budget (e.g., minutes, hours, days). This allows for better tracking and optimization of your token usage across scheduled runs.
   • Max tokens - Specify the maximum number of tokens to use for the model's response. Some models support up to 8192 tokens. If not set, the model will use its default value for token usage.
   • Temperature - (Optional) Sets the randomness of the model's responses.
     • Higher values (e.g., 0.7 to 1.0) gives more creative and varied responses.
     • Lower values (e.g., 0.1 to 0.3) makes the model more predictable and focused.
   • Top-p sampling - (Optional)Top-p sampling controls how the model selects words based on probability:(Optional) The model selects tokens until the cumulative probability exceeds p.
     • The model picks tokens until the cumulative probability exceeds p (e.g., p = 0.9 means the model will select words that make up 90% of the likelihood).
     • Lower p values result in more focused and safe outputs, while higher values allow for more diverse and creative responses.
   • Top k sampling - (Optional) The model picks from the k most likely words (note: not all models support this).
     • For example, if k = 50, it selects from the 50 most likely words.
     • Lower k values gives a more focused output while higher values introduce more variability.
4. Click Submit.