Skip to content

tests: add MCP-spec regression coverage#371

Merged
mrexodia merged 2 commits into
mrexodia:mainfrom
NeKroFR:tests/mcp-spec-regression-coverage
Apr 22, 2026
Merged

tests: add MCP-spec regression coverage#371
mrexodia merged 2 commits into
mrexodia:mainfrom
NeKroFR:tests/mcp-spec-regression-coverage

Conversation

@NeKroFR

@NeKroFR NeKroFR commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

Adds a test suite that catches the class of bug that broke tools/list for MCP clients like LM Studio.

New dep

  • jsonschema>=4.0 (dev only). The existing _matches_schema helper in tests/test_output_schema.py was too permissive and missed the missing outputSchema.type. A real JSON-Schema validator matches what strict Zod clients do on the wire, so regressions fail here instead of in production.

However if you don't want to add the dependency, I can drop it and hand-roll a minimal validator covering the subset we actually use (type, properties, required, anyOf, oneOf, const, items, additionalProperties, pattern).

What's covered

6 files, 93 tests, all IDA-free.

I made sure to be implementation-independent for the tests so we can change the internal attributes with no problems as long as the MCP contract holds, and all tests pass obviously :)

@mrexodia

Copy link
Copy Markdown
Owner

I was thinking about the following approach (possibly as a replacement, possible as an addition to what you have now):

This way we do not need to create specialized tests, 'just' some glue code and run all the tests twice. What do you think?

@NeKroFR

NeKroFR commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

I think the two approaches test different layers and complement each other.

The current tests target schema generation, so every Python return-type shape (Union[TypedDict], NotRequired, nested, Optional, etc.) _generate_tool_schema must produce a spec-compliant schema. Production tools don't exercise every shape, so dropping the synthetic fixtures would create blind spots.

And the E2E-through-MCP approach targets runtime conformance, so every production tool's real response must validate its advertised schema. That's a class of bug the current PR doesn't catch. I will add the E2E-through-MCP layer alongside my tests.

Just one question first, what do you want me to use to do that? Would the official mcp Python SDK is fine? The goal as I understand it is just to route test calls through MCP and validate schemas, I think it is enough.

@mrexodia

Copy link
Copy Markdown
Owner

Yeah agreed! I think any SDK is fine, as long as the client actually validates the schemas from the outputs (not sure what the official one does)

@mrexodia

Copy link
Copy Markdown
Owner

Nice! I think we also need to enable this option for at least one IDA version (9.3, maybe all?)

@NeKroFR

NeKroFR commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

no wait before merging, shouldn't have pushed yet mybad. I need to check a few things first

@NeKroFR NeKroFR force-pushed the tests/mcp-spec-regression-coverage branch from a754423 to cc0801f Compare April 16, 2026 14:24
@NeKroFR

NeKroFR commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

Tested with 9.3, all good now. I think the better option would be to enable it on 9.3 only for now. MCP-mode catches schema and wire-format regressions, which aren't IDA-version dependent, so running on every version would roughly double CI time for little added signal. And as it is it will be easy to extend to 9.4+ later.

@mrexodia mrexodia merged commit 95a0f47 into mrexodia:main Apr 22, 2026
5 checks passed
@NeKroFR NeKroFR deleted the tests/mcp-spec-regression-coverage branch April 22, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants