tests: add MCP-spec regression coverage#371
Conversation
|
I was thinking about the following approach (possibly as a replacement, possible as an addition to what you have now):
This way we do not need to create specialized tests, 'just' some glue code and run all the tests twice. What do you think? |
|
I think the two approaches test different layers and complement each other. The current tests target schema generation, so every Python return-type shape ( And the E2E-through-MCP approach targets runtime conformance, so every production tool's real response must validate its advertised schema. That's a class of bug the current PR doesn't catch. I will add the E2E-through-MCP layer alongside my tests. Just one question first, what do you want me to use to do that? Would the official |
|
Yeah agreed! I think any SDK is fine, as long as the client actually validates the schemas from the outputs (not sure what the official one does) |
|
Nice! I think we also need to enable this option for at least one IDA version (9.3, maybe all?) |
|
no wait before merging, shouldn't have pushed yet mybad. I need to check a few things first |
a754423 to
cc0801f
Compare
|
Tested with 9.3, all good now. I think the better option would be to enable it on 9.3 only for now. MCP-mode catches schema and wire-format regressions, which aren't IDA-version dependent, so running on every version would roughly double CI time for little added signal. And as it is it will be easy to extend to 9.4+ later. |
Adds a test suite that catches the class of bug that broke
tools/listfor MCP clients like LM Studio.New dep
jsonschema>=4.0(dev only). The existing_matches_schemahelper intests/test_output_schema.pywas too permissive and missed the missingoutputSchema.type. A real JSON-Schema validator matches what strict Zod clients do on the wire, so regressions fail here instead of in production.However if you don't want to add the dependency, I can drop it and hand-roll a minimal validator covering the subset we actually use (
type,properties,required,anyOf,oneOf,const,items,additionalProperties,pattern).What's covered
6 files, 93 tests, all IDA-free.
test_mcp_spec_tools_list.py: tool schema invariants over a broad fixture (name regex, description,inputSchema.type,outputSchema.type, no dupes). Direct regression guard for Fix MCP output schema wrapping for union object results #357.test_mcp_spec_schema_generation.py: every return-type shape (int,list,Optional,Union[TypedDict], nested,NotRequired,dict[str, TypedDict]) produces a roottype: "object"schema, and runtime returns validate against it.test_mcp_spec_envelope.py: JSON-RPC 2.0 envelope over real HTTP (id echo, method-not-found code, error-code range, notifications return no body).test_mcp_spec_endpoints.py:initialize,ping,prompts/list,resources/list,resources/templates/list.test_mcp_spec_truncation.py: regression guard for Error during analysis: Invalid structured content returned by tool decompile: Additional properties are not allowed ('_download_hint', '_download_url', '_output_id', '_output_truncated', '_total_chars #361 and e802b32. TruncatedstructuredContentstill validates the advertisedoutputSchema, no_-prefixed leaks, no{"_truncated": ...}sentinels.test_mcp_spec_http_e2e.py: real localhostThreadingHTTPServer,urllibclient, full envelope validation.I made sure to be implementation-independent for the tests so we can change the internal attributes with no problems as long as the MCP contract holds, and all tests pass obviously :)