StackRox MCP E2E Testing

End-to-end tests for the StackRox MCP server using mcpchecker.

Quick Start

Smoke Test (No Agent Required)

Validate configuration and build without running actual agents:

cd e2e-tests
./scripts/smoke-test.sh

This is useful for CI and quickly checking that everything compiles.

Prerequisites

Go 1.26.10+
Google Cloud Project with Vertex AI enabled (for Claude agent)
OpenAI API Key (for LLM judge)

Setup

1. Build mcpchecker

cd e2e-tests
./scripts/build-mcpchecker.sh

2. Configure Environment

Create .env file:

# Required: OpenAI API Key (for the agent and LLM judge)
OPENAI_API_KEY=<OpenAI API Key>

# Optional: OpenAI-compatible endpoint shared by the agent and judge
OPENAI_BASE_URL=https://api.openai.com/v1

Configure the judge model in mcpchecker/eval.yaml under config.llmJudge.ref.model. The deprecated MODEL_BASE_URL and JUDGE_* settings are no longer used by the wrapper.

Note: No StackRox API token required - tests use WireMock mock service.

Running Tests

Run tests against the WireMock mock service:

./scripts/run-tests.sh

The test suite:

Starts WireMock automatically on localhost:8081
Uses deterministic test fixtures
Requires no StackRox API tokens
Fast and reliable for development and CI

Results are saved to mcpchecker/mcpchecker-stackrox-mcp-e2e-out.json.

View Results

# Summary
jq '.results[] | {taskName, taskPassed}' mcpchecker/mcpchecker-stackrox-mcp-e2e-out.json

# Tool calls
jq '[.results[] | .callHistory.ToolCalls[]? | {name: .request.Params.name, arguments: .request.Params.arguments}]' mcpchecker/mcpchecker-stackrox-mcp-e2e-out.json

Test Cases

Test	Description	Tool
`list-clusters`	List all clusters	`list_secured_clusters`
`cve-detected-workloads`	CVE detected in deployments	`get_deployments_for_cve`
`cve-detected-clusters`	CVE detected in clusters	`get_clusters_with_orchestrator_cve`
`cve-nonexistent`	Handle non-existent CVE	`get_clusters_with_orchestrator_cve`
`cve-cluster-does-exist`	CVE with cluster filter	`get_clusters_with_orchestrator_cve`
`cve-cluster-does-not-exist`	CVE with non-existent cluster	`list_secured_clusters`
`cve-clusters-general`	General CVE query	`get_clusters_with_orchestrator_cve`
`cve-cluster-list`	CVE across clusters	`get_clusters_with_orchestrator_cve`
`cve-log4shell`	Well-known CVE (log4shell)	`get_deployments_for_cve`
`cve-multiple`	Multiple CVEs in one prompt	`get_deployments_for_cve`
`rhsa-not-supported`	RHSA detection (should fail)	None

Configuration

mcpchecker/eval.yaml: Test configuration, agent settings, assertions
mcpchecker/mcp-config-mock.yaml: MCP server configuration for WireMock
mcpchecker/tasks/*.yaml: Individual test task definitions

How It Works

mcpchecker uses a proxy architecture to intercept MCP tool calls:

AI agent receives task prompt
Agent calls MCP tool
mcpchecker proxy intercepts and records the call
Call forwarded to StackRox MCP server
Server executes and returns result
mcpchecker validates assertions and response quality

Troubleshooting

Tests fail - no tools called

Verify WireMock is running: make mock-status
Check WireMock logs: make mock-logs

Build errors

go mod tidy
./scripts/build-mcpchecker.sh

GeezerVison

Expand file tree

e2e-tests

Directory actions

More options