BlackIce – A Container Based Red Teaming Toolkit for AI Security Testing

In Cybersecurity News - Original News Source is cybersecuritynews.com by Blog Writer

Databricks has officially announced the release of BlackIce, an open-source, containerized toolkit designed to streamline AI security testing and Red Teaming.

Originally introduced at CAMLIS Red 2025, BlackIce addresses the fragmentation and configuration challenges that security researchers often face when evaluating Large Language Models (LLMs) and Machine Learning (ML) systems.

By bundling 14 widely used open-source security tools into a single, reproducible environment, Databricks aims to provide a solution analogous to “Kali Linux,” but specifically tailored for the AI threat landscape.

The motivation behind BlackIce stems from significant practical hurdles in the current AI security ecosystem. Red teamers frequently encounter “dependency hell,” where different evaluation tools require conflicting libraries or Python versions.

Furthermore, managed notebooks often restrict users to a single Python interpreter, making it difficult to orchestrate complex, multi-tool testing workflows.

BlackIce mitigates these issues by delivering a version-pinned Docker image. The architecture divides tools into two categories to ensure stability.

Static tools, which are evaluated via command-line interfaces, are installed in isolated Python virtual environments or Node.js projects to maintain independent dependencies.

Dynamic tools, which allow for advanced Python-based customization and attack code development, are installed in a global Python environment with carefully managed requirement files.

This structure allows researchers to bypass setup hassles and focus immediately on vulnerability assessment.

Integrated Toolset and Capabilities

The toolkit consolidates a diverse array of tools spanning Responsible AI, security testing, and adversarial ML. These tools are exposed through a unified command-line interface and can run from a shell or within a Databricks notebook.

The initial release includes high-profile tools such as Microsoft’s PyRIT, NVIDIA’s Garak, and Meta’s CyberSecEval.

Table 1: BlackIce Integrated Tool Inventory

Tool Organization Category GitHub Stars (Approx)
LM Eval Harness Eleuther AI Evaluation 10.3K
Promptfoo Promptfoo LLM Testing 8.6K
CleverHans CleverHans Lab Adversarial ML 6.4K
Garak NVIDIA Vulnerability Scanning 6.1K
ART IBM Adversarial Robustness 5.6K
Giskard Giskard AI Testing 4.9K
CyberSecEval Meta Safety Evaluation 3.8K
PyRIT Microsoft Red Teaming 2.9K
EasyEdit ZJUNLP Model Editing 2.6K
Promptmap N/A Prompt Injection 1K
Fuzzy AI CyberArk Fuzzing 800
Fickling Trail of Bits Pickle Security 560
Rigging Dreadnode LLM Interaction 380
Judges Quotient AI Evaluation 290

To ensure the toolkit meets enterprise security standards, Databricks has mapped the capabilities of BlackIce to established risk frameworks, specifically MITRE ATLAS and the Databricks AI Security Framework (DASF).

This mapping confirms that the toolkit covers critical threat vectors ranging from prompt injection to supply chain vulnerabilities.

Table 2: Risk Framework Mapping

Capability MITRE ATLAS Reference DASF Reference
Prompt Injection / Jailbreak AML.T0051 (Prompt Injection), AML.T0054 (Jailbreak) 9.1 Prompt inject, 9.12 LLM jailbreak
Indirect Prompt Injection AML.T0051 (Indirect Injection) 9.9 Input resource control
LLM Data Leakage AML.T0057 (Data Leakage) 10.6 Sensitive data output
Hallucination Detection AML.T0062 (Discover Hallucinations) 9.8 LLM hallucinations
Adversarial Evasion (CV/ML) AML.T0015 (Evade Model), AML.T0043 (Craft Data) 10.5 Black box attacks
Supply Chain Safety AML.T0010 (Supply Chain Compromise) 7.3 ML supply chain vulnerabilities

Databricks has made the BlackIce image available publicly on Docker Hub. The toolkit includes custom patches to ensure seamless interaction with Databricks Model Serving endpoints out of the box.

Security professionals can pull the current Long Term Support (LTS) version using the tag databricksruntime/blackice:17.3-LTS.

For integration into Databricks workspaces, users can configure their compute clusters using Databricks Container Services to point to this image URL, enabling immediate orchestration of AI security assessments.

Follow us on Google News, LinkedIn, and X for daily cybersecurity updates. Contact us to feature your stories.

The post BlackIce – A Container Based Red Teaming Toolkit for AI Security Testing appeared first on Cyber Security News.