How to Evaluate MCP Servers Before You Connect Them to Enterprise Workflows
Evaluate MCP servers across eight dimensions and a 30-day approval workflow to meet EU AI Act and DORA enterprise governance needs.
TL;DR: Evaluate MCP servers across eight dimensions and a 30-day approval workflow to meet EU AI Act and DORA enterprise governance needs.
MCP servers are not harmless developer plugins; they are privileged workflow infrastructure that exposes tools, data, and identity to AI agents. Connecting an MCP server to your enterprise workflow without rigorous evaluation is a compliance and security gamble. Why this matters: For European scale-ups, the EU AI Act sandbox milestone of 2 August 2026 (S6) and DORA (S9) transform MCP server evaluation from optional hardening into a mandatory compliance artifact. CTOs, platform engineering leads, AI transformation leads, security leads, and operations leaders must adopt a structured evaluation framework to avoid exposing their organizations to tool-execution abuse, data leakage, and regulatory penalties.
The short answer
Evaluate MCP servers across eight dimensions: data access, tool permissions, identity, auditability, sandboxing, prompt-injection exposure, vendor maturity, and rollback. Use a 30-day phased workflow that includes initial security review, sandbox pilot, and production rollout with rollback drill. For European scale-ups, this evaluation is a compliance artifact under EU AI Act (S6) and DORA (S9).
Why this matters for European scale-ups
European scale-ups face a unique regulatory landscape. The EU AI Act (S6) requires conformity assessment for high-risk AI systems; an MCP server that executes tool calls on business resources qualifies as part of the system. DORA (S9) Article 28 demands third-party risk management for ICT services; MCP servers are third-party components that must be evaluated. Scale-ups often have limited security resources, but skipping due diligence invites operational disruption and regulatory penalties. A structured evaluation program protects your organization and creates evidence for compliance audits.
The audit-facing version of this argument is even sharper. EU AI Act Article 16 obligations and DORA Article 28 third-party risk requirements are not satisfied by a vendor questionnaire response that says "we use OAuth and we log calls." They are satisfied by repeatable evidence: a per-server scorecard, a sandbox-pilot run log, a rollback-drill timestamp, an audit-trail sample. Pre-MCP, you could argue the AI integration was opaque. With MCP servers exposing a structured tool list, the regulator can ask: which tools does this server expose, which agents can call them, what data do they touch, and when was the last rollback drill? Those four questions have crisp answers when the eight-dimension matrix has been run; without it, the answers are anecdotal. For a 20-person to 50-person engineering team, the difference between a half-day audit response and a two-week audit response is whether the MCP server evaluation was structured up front. Founder-led companies and growing software teams should adopt the evaluation discipline before they have many MCP servers, because the cost scales with the number of servers in production.
What an MCP server actually is, in operational terms
The Model Context Protocol (MCP) is an open standard for connecting AI models to external tools and data sources (S1). An MCP server is a service that receives tool call requests from an AI client, executes those calls against underlying systems (APIs, databases, file systems), and returns results (S2). The protocol is published as an open specification (S1) and is documented in vendor reference materials including the Anthropic agents-and-tools docs (S14), which give engineering leaders a stable shape to evaluate against rather than a moving vendor-specific surface. In operational terms, the MCP server is a gateway: it translates an AI model's intent into real actions on your infrastructure. It exposes a list of tools, each with a name, description, input schema, and execution logic. When a connected AI agent invokes a tool, the server runs the associated code with whatever credentials it holds. This trust boundary is the server's tool list (S1, S2).
The eight evaluation dimensions
Data access. What data can the MCP server read or write? Evaluate the scope of resources (files, databases, APIs) the server interacts with. Verify that the server requests only the minimum data necessary for its function. Read-only access is preferable unless write operations are explicitly required and justified.
Tool permissions. Which tools does the server expose, and what authorization model governs their invocation? Check whether any AI agent may call any tool or if there are client-side or server-side restrictions. Tool permissions must be granular and configurable.
Identity. How does the MCP server authenticate to downstream services? Ideally, it uses scoped delegation via OAuth 2.0 (S5) or short-lived service accounts. Avoid servers that reuse a single static credential for all tool calls without scoping.
Auditability. Does the server log every tool call with structured, append-only records? Logs should include the prompt that triggered the call, the tool name, arguments, result, and latency (S2, S10). Tamper-proof logging is critical for incident response and compliance.
Sandboxing. Is the MCP server runtime isolated from production infrastructure? Code-execution servers must run in a sandboxed environment with no production credentials (S7, S8). Network egress should be restricted to required endpoints.
Prompt-injection exposure. How does the server handle instructions that might be adversarial? Prompt injection (S3, S4) is the dominant attack vector against tool-calling agents. The server should validate tool arguments, enforce input constraints, and implement rate limiting to reduce blast radius.
Vendor maturity. Is the server actively maintained? Check for signed releases (S8), versioning policies, a security policy, and an incident response process. The OpenSSF Scorecard (S7) provides a quantitative signal of project health.
Rollback. What is the plan if the MCP server causes problems? Every adoption decision must include a documented rollback path covering credential rotation, log retention, and tool-list removal (S9, S10). Rollback drills should be tested before production deployment.
The MCP server evaluation matrix
| Dimension | What you measure | Where to find evidence | Suggested gate | Red flag |
| Data access | List of resources the server reads/writes; whether access is read-only or write; data classification level | Server source code, documentation, or an SBOM (S10) | Read-only for production data unless explicitly justified | Broad wildcard access patterns (e.g., SELECT FROM ) |
| Tool permissions | Tool names, parameters, and authorization model; whether tools are callable by any client or require scoped tokens | Server specification (S2) and configuration files | Client-side tool filters and server-side authorization | Tools that allow arbitrary SQL or shell execution |
| Identity | Authentication mechanism used for downstream services; delegation approach (OAuth, API keys) | Server configuration and integration tests | Scoped OAuth 2.0 tokens (S5) or short-lived API keys | Use of the same credential for multiple servers without restriction |
| Auditability | Existence of structured, append-only logs for each tool call; log retention period; log output format | Server logs, documentation, and source (S2, S10) | Logs include prompt, tool name, arguments, result, latency; stored in immutable storage | No logging or logs that can be easily tampered with |
| Sandboxing | Isolation level (container, VM, serverless); network egress controls; filesystem access | Deployment artifacts, Dockerfile, cluster manifest | Code-execution servers run in isolated network sandbox with no production secrets | Server can access internal production resources directly |
| Prompt-injection exposure | Input validation, rate limiting, anomaly detection on tool call patterns | OWASP LLM Top 10 guidelines (S3, S4) | Server validates tool arguments and implements rate limits per client | No validation; any string from the model can become a tool argument |
| Vendor maturity | Release cadence, signed releases (S8), OpenSSF Scorecard (S7), incident response documentation | GitHub repository, OpenSSF Scorecard, SLSA attestations | Maintained in last 90 days, Scorecard >= 7, signed releases | No releases in past year, no security policy, low Scorecard |
| Rollback | Documented rollback procedure, credential rotation process, log retention during rollback | Runbook, security policy, deployment scripts | Rollback plan tested at least quarterly; rollback completes within 1 hour | No rollback plan or plan assumes manual intervention without verification |
A worked example: evaluating three illustrative MCP server categories
Read-only data-warehouse MCP server. This server executes read-only SQL queries against a data warehouse. The evaluator checks data access: which tables are exposed, and are queries read-only? Tool permissions: only SELECT statements are allowed. Identity uses a read-only service account. Auditability logs every query with the full SQL and timestamp. Sandboxing is minimal but the server runs in a restricted network. Prompt-injection exposure: the server uses parameterized queries to prevent SQL injection. Vendor maturity: if from a known vendor, verify signed releases; if community-maintained, check OpenSSF Scorecard. Suggested gate: low risk after initial validation. Pilot in sandbox tenant is optional.
Write-capable issue-tracker MCP server. This server can create, update, and delete issues. Data access includes read and write to the issue database. Tool permissions distinguish read-only and write operations. Identity uses scoped OAuth tokens (S5). Auditability logs every mutation with user context. Sandboxing is required to prevent access to other systems. Prompt-injection exposure: validate that issue titles and descriptions cannot contain script injection. Vendor maturity: prefer servers with signed releases and documented rollback. Suggested gate: moderate risk; require a 2-week sandbox pilot and audit validation.
Experimental community-maintained code-execution MCP server. This server allows the AI agent to run arbitrary shell commands. Data access is unrestricted unless sandboxed; tool permissions include code execution. Identity should be scoped but often is not. Auditability may be absent. Sandboxing is critical: must run in isolated container with no production secrets. Prompt-injection exposure is extreme: any prompt could produce a malicious command. Vendor maturity likely low; rely on OpenSSF Scorecard and source inspection. Suggested gate: high risk; require full security review, sandboxed pilot, and explicit CTO approval before any production use.
A 30-day approval workflow
Days 1 to 7: Intake + initial security review + tier classification. The AI transformation lead or CTO submits an intake form describing the intended workflow and data sensitivity. The platform engineering lead performs an initial scan using OpenSSF Scorecard (S7) and dependency vulnerability check (S11). The security lead classifies the MCP server into a tier (low, moderate, high) based on data access and tool permission scope. The procurement-aware engineering manager checks vendor maturity documentation.
Days 8 to 21: Pilot in a sandbox tenant + log capture + audit-trail validation. The operations leader deploys the MCP server in a sandbox environment with no production data. The AI transformation lead designs test scenarios that include normal use and adversarial prompts. The security lead validates that audit logs contain all required fields (S2, S10). The platform engineering lead monitors resource usage and network calls. The procurement-aware engineering manager reviews the outcome and updates the risk register for DORA (S9) compliance.
Days 22 to 30: Production rollout + rollback drill + procurement sign-off. The CTO reviews the final risk assessment and approves production deployment. The operations leader executes a rollback drill: stops the server, rotates any shared credentials, and verifies that logs are preserved. The security lead confirms that audit trails are append-only and immutable. The procurement-aware engineering manager signs off on the usage agreement or contract. All evaluation artefacts are archived for EU AI Act and DORA evidence. The rollback drill itself is the most underrated checkpoint in this workflow: most teams document a rollback plan and never test it; the first time they execute it is during an incident, when the cost of an unfamiliar procedure is highest. A scheduled drill on day 28 to 30 turns the rollback plan from a document into operational muscle memory, and the drill log becomes part of the evidence package for the next conformity assessment review.
What you can automate safely today
You can automate OpenSSF Scorecard scans for every new MCP server candidate using a CI pipeline (S13). Automate dependency vulnerability scanning with Dependabot (S11) against the server's repository, and cross-check against the GitHub Advisory Database (S12) for published CVEs against the server's runtime, transport library, or transitive dependencies. Automate sandbox deployment via Infrastructure as Code and log collection into a SIEM. Automated checks catch common issues early, but they cannot assess the risk of a novel prompt-injection vector or the trustworthiness of a maintainer's incident response. Treat the automated scan as the cheap pre-filter that lets the human review focus on the dimensions automation cannot see: data classification, scope decisions, and rollback rehearsal.
What must remain human-reviewed (and what not to automate yet)
- Do not let an MCP server's automated evaluation become a substitute for security review of the underlying tool surfaces, secrets handling, or supply-chain integrity of the server's runtime.
- Do not fully automate the first connection to a production data source. A human must verify the data classification and confirm that only necessary resources are accessed.
- Do not skip manual inspection of tool permissions for write-capable servers. Automated scans may miss context-specific authorization gaps.
- Do not rely solely on vendor documentation for identity delegation. Test the actual OAuth flow or API key scoping in a sandbox.
- Do not approve a server without a verified rollback plan. Rollback must be tested, not just documented.
- Do not assume a server with many GitHub stars is secure. Star count is not a security metric; evaluate actual maintainer health.
How MCP evaluation maps to EU AI Act and DORA
Under the EU AI Act (S6), providers of high-risk AI systems must demonstrate risk management and transparency. MCP servers with tool-execution authority are integral to the system's behaviour; their evaluation artefacts (audit logs, permission matrices, security test results) directly support the technical documentation required for conformity assessment. For DORA (S9), Article 28 mandates ICT third-party risk management. MCP server evaluation outputs feed into the risk register, contractual clauses, and reporting obligations. Without a structured evaluation, your compliance documentation is incomplete.
For professional guidance on aligning your MCP server evaluation with EU AI Act and DORA, see our AI Readiness Assessment page (https://radar.firstaimovers.com/page/ai-readiness-assessment) and AI Consulting page (https://radar.firstaimovers.com/page/ai-consulting).
Limits and failure modes
No evaluation is foolproof. An MCP server can be updated with new tool behaviours after approval. Prompt injection research evolves, and new attack vectors may bypass existing controls. Vendor maturity can decline over time. Internal misuse by authorised agents remains a risk. Mitigate these by (a) requiring signed releases (S8) and version pinning, (b) implementing continuous monitoring of tool call patterns for anomalies, (c) scheduling quarterly reassessments of all production MCP servers, and (d) maintaining a security incident response plan that includes MCP-specific scenarios.
A second class of failure deserves explicit naming because it bites scale-ups harder than enterprises. Tool-list drift: the evaluation captures the server's tool list at time of approval, but the maintainer adds a new tool in a later release. If the consumer pins the version and refuses upgrades, the security posture is stable but the team loses bug fixes. If the consumer auto-upgrades, the new tool may bypass the original evaluation. The right middle ground is a tool-list diff check on every release: re-evaluate any new tool against the matrix before allowing the upgrade to land in production. This is a small CI script that compares the latest release's tool list to the approved baseline and fails the deploy if the diff is non-empty without an updated evaluation artefact.
A third class is human. Engineers under shipping pressure will request approval to bypass the sandbox-pilot phase for a server they personally trust. This is the easiest failure to miss because the request is reasonable on its face. Mitigations: require a written exception with a documented rollback plan and a one-week post-deployment review; surface bypass requests to the CTO and security lead in a monthly review; track the bypass rate as a process metric. If the bypass rate exceeds 20% of new MCP server adoptions in any quarter, the workflow itself needs revision, not the bypass policy.
Frequently Asked Questions
Q: How long does the full evaluation take for a new MCP server? A: The 30-day approval workflow is typical, but low-risk read-only servers from trusted vendors may be expedited to 7 days after initial scan.
Q: Should we trust a community-maintained MCP server in production? A: Only with heightened scrutiny: require signed releases (S8), OpenSSF Scorecard >=7, and sandboxing for any code-execution server. Even then, treat as high-risk pending pilot.
Q: Does the evaluation matrix replace a security review? A: No, it structures the review but does not replace expert judgment. Use it as a checklist within a broader security assessment.
Q: How does MCP server evaluation interact with EU AI Act and DORA? A: It provides evidence for conformity assessment (EU AI Act) and third-party risk management (DORA). Without evaluation, your compliance documentation is incomplete.
Q: Can the same MCP server be approved for one workflow and rejected for another? A: Yes. Approval is context-dependent: a read-only server for a non-sensitive dataset may be approved, while the same server for a critical production database may be rejected due to data access concerns. Record the approval-context boundary explicitly in the evaluation artefact so that a future engineer who tries to reuse the approved server for a different workflow triggers the matrix again rather than coasting on the prior decision.
