loading…

Dashboard

Ops

▶AI_Agent_Security_Redteam / Jailbreak_Via_Tool_Result_Injection0/20 · 0%cap $12
▶DLP_Data_Exfiltration_Response / Confidential Document Printing Review Investigates printing of highly confidential documents outside approved hours to identify possible unauthorized data handling.0/20 · 0%cap $25
▶DLP_Data_Exfiltration_Response / Outbound SFTP PII Detection Correlates outbound file transfers with DLP matches for personally identifiable information to identify suspicious data exfiltration.0/20 · 0%cap $25
▶EDR Endpoint Containment / Living-off-the-Land Binary Hunt Detects abuse of legitimate system tools, such as Certutil and Rundll32, to identify malicious execution without traditional malware files.0/20 · 0%cap $25
▶Vulnerability Management & CVE Triage / Scanner False-Positive Triage Reviews vulnerability scanner findings against compensating controls to determine whether a reported vulnerability is genuinely exploitable.0/20 · 0%cap $25
▶EDR Endpoint Containment / Ransomware Canary Investigation Investigates modifications to ransomware canary files to identify the accessing account, process, and possible encryption activity.0/20 · 0%cap $25
▶Vulnerability Management & CVE Triage / Internet-Facing CVE Scan Scans public-facing assets for a specified vulnerability and identifies affected hosts that require patching.7/20 · 35%cap $25
▶EDR Endpoint Containment / Suspicious Process Tree Investigation Investigates suspicious process execution chains, terminates malicious processes, and collects forensic evidence from the endpoint.20/20 · 100%cap $25
▶DLP_Data_Exfiltration_Response / USB Mass Copy Detection Detects large transfers of confidential files to USB devices and supports blocking the device to prevent unauthorized data removal.31/20 · 155%cap $25
▶Vulnerability Management & CVE Triage / KEV Exploit Prioritization Identifies known exploited vulnerabilities on internet-facing assets, prioritizes them using exploitability information, and supports emergency remediation.30/20 · 150%cap $25
▶Credential_Exfiltration_Guardrails / Token_Exposure_In_Error_Messages21/20 · 105%cap $12
▶Credential_Exfiltration_Guardrails / Secret_Leak_Via_Tool_Output25/20 · 125%cap $12
▶AI_Agent_Security_Redteam / Jailbreak_Via_Tool_Result_Injection33/20 · 165%cap $12
▶Email_Security / Inspect sender, headers, URLs, attachments, domain reputation, recipients, mailbox actions, and remove/quarantine matching messages.41/20 · 205%cap $25
▶Privileged_Access_Security / Find admin sessions, commands/actions, elevation events, source devices, approvals, and determine whether activity was authorized.31/20 · 155%cap $25
▶SOC Operations / Review incoming alerts and correlate users, hosts, IPs, processes, and related events. Determine severity, collect evidence, and escalate confirmed incidents.0/20 · 0%cap $25
▶Malware_Analysis / Inspect suspicious files, hashes, processes, domains, IPs, and behavioral indicators. Extract IOCs, correlate related activity, and determine whether containment is required.20/20 · 100%cap $25
▶Incident_Response / Gather alerts and evidence, identify affected assets, determine scope, and reconstruct the attack timeline. Contain confirmed threats, track remediation, and update incident status.20/20 · 100%cap $25
▶Detection_Engineering / Analyze security telemetry and existing detection coverage to identify detection gaps. Create, test, and tune rules while minimizing false positives.28/20 · 140%cap $25
▶Secrets_Security / Search repositories, workloads, configurations, and logs for exposed credentials, tokens, keys, and certificates. Validate exposure and rotate or revoke compromised secrets safely28/20 · 140%cap $25
▶Env Report / Help me to track the ip of google24/20 · 120%cap $25

pass@1 36% · pass@3 44% · 416 traces to review · total spend $51.28

Task completeness

task	tools	vars	MCP	scen	gen	judged	appr	rev	exp
System_Prompt_And_Tool_Schema_Extraction	—	—	—	—	—	—	—	—	—
Cross_Tenant_Data_Leakage	—	—	—	—	—	—	—	—	—
Resource_Exhaustion_Via_Tool_Loops	—

Dataset coverage (approved 519)

patternssingle_call 18% · 96/773 pass (target 30 -12)parallel 11% · 57/310 pass (target 15 -4)chain 18% · 94/421 passno_call 15% · 77/283 passerror_recovery 38% · 195/283 pass (target 20 +18)

variantscamel 26%snake 24%dotted 24%verbfirst 26%·tools 225/530 covered

turnssingle-turn 100% · multi-turn 0% · avg 2.6 tool calls/conv

vocabularyAnalyzeRulePerformance · AssignItem · CreateCorrelationRule · GetDetection · GetItem · GetModelAnalysis · GetNotableEvent · HuntThreats · ListDetections · ListItems · ListNotableEvents · ListSuspiciousRepos · SearchEvents · TriageThreat · TuneCorrelationRule · TuneDetection · UpdateNotableStatus · aggregate_access_metrics · aggregate_alerts_by_rule · aggregate_portfolio_metrics · analyze_bypass · analyze_exfiltration · analyze_file_integrity · analyze_group_membership_changes · analyze_kerberos_anomalies · analyze_user_risk · approve_access_request · assess_model_robustness · assign_item · block_exfiltration · certify_access · checkout_account · configure_policy · create_case · create_custom_rule · create_data_incident · create_engagement · create_exception · create_incident · create_investigation · create_safeguard · create_security_exception · create_security_incident · create_watchlist · deploy_active_response · detect_kerberos_attacks · enrich_artifacts · enumerate_attack_paths · enumerate_privileged_accounts · env.assignItem · env.getItem · env.listItems · env.searchItems · envAssignItem · envExportItems · envGetItem · envListItems · envSearchItems · env_assign_item · env_export_items · env_get_item · env_list_items · env_notify_owner · escalate_risk_incident · execute_adversarial_test · execute_response_playbook · export_forensics · export_incidents · generate_attack_payload · generate_compliance_report · generate_forensic_report · generate_iocs · generate_security_report · generate_threat_report · get_access_request · get_access_review · get_access_session · get_agent_details · get_alert · get_alert_details · get_asset_context · get_case · get_compliance_report · get_detection · get_exploit_by_id · get_identity_risk_profile · get_incident · get_item · get_policy · get_policy_coverage · get_posture_assessment · get_review_items · get_risk_assessment · get_risk_profile · get_rule_details · get_security_incident · get_security_posture · get_session · get_session_recording · get_supplier_profile · get_threat_intelligence · get_user_risk_profile · get_violation · get_vulnerability · hugging.listSuspiciousRepos · hugging.triageThreat · huggingCaseThreat · huggingGetModelAnalysis · huggingListSuspiciousRepos · hugging_case_threat · hugging_get_model_analysis · hugging_list_suspicious_repos · hunt_anomalous_access · hunt_file_integrity · hunt_identities · hunt_indicators · hunt_model_anomalies · hunt_threats · initiate_supplier_audit · investigate_anomaly · investigate_authentication_anomaly · investigate_lateral_movement · investigate_sca_compliance · list_accounts · list_active_agents · list_applications · list_assets · list_attack_scenarios · list_authenticators · list_cases · list_contained_endpoints · list_detections · list_entitlements · list_hunts · list_incidents · list_indicators · list_investigations · list_items · list_notable_events · list_pending_certifications · list_policies · list_policy_matches · list_policy_violations · list_privilege_escalations · list_privileged_accounts · list_rbac_risks · list_risk_assessments · list_security_alerts · list_security_posture · list_threat_detections · list_threats · list_vault_accounts · list_violations · list_vulnerabilities · manage_agent_groups · monitor_privileged_operations · monitor_traffic · quarantine_content · quarantine_endpoints · quarantine_session · quarantine_workload · query_access_events · query_alerts · query_events · query_findings · query_security_logs · query_sessions · query_suppliers · query_vulnerabilities · remediate_threat · remediate_violation · remediate_vulnerability · rotate_vault_password · run_compliance_scan · scan_application · scan_content · scan_endpoints · scan_input · scan_sensitive_data · scan_workloads · search_alerts · search_content_matches · search_events · search_incidents · search_indicators · search_items · search_security_findings · search_sessions · search_violations · search_vulnerability_detections · siem.analyzeRulePerformance · siem.getDetection · siem.getNotableEvent · siem.list_detections · siem.list_notable_events · siem.tune_detection · siem.updateNotableStatus · siemListDetections · siem_analyze_rule_performance · siem_create_correlation_rule · siem_get_detection · siem_get_notable_event · siem_hunt_threats · siem_list_detections · siem_list_notable_events · siem_search_events · siem_tune_correlation_rule · siem_tune_detection · siem_update_notable_status · terminate_session · test_authentication · track_shipment_integrity · triage_alert · triage_finding · triage_incident · triage_security_incident · triage_vulnerability · tune_policy_thresholds · update_alert_status · update_case_status · update_incident_status · update_risk_score · update_rule · update_supplier_status · update_violation_status

LLM roles

role	calls	tokens	usd	err	avg ms
assistant	5,996	13.1M	$43.57	7.7%	2,284
judge	2,420	1.7M	$6.03	7.8%	1,941
tool_gen	84	137.0k	$1.64	17.9%	13,161
user_sim	749	224.8k	$0.02	99.7%	446
variant_gen	77	32.8k	$0.01	0.0%	2,029
playground	6	5.4k	$0.00	0.0%	1,073