fix(core): sniff MCP image MIME types

[Dasoam]

Fix MCP Image MIME Type Detection

Fixes #27731

Problem

WebP images returned by Figma MCP integration were incorrectly labeled as image/png, causing HTTP 400 errors when sent to the Gemini API.

Solution

Implements local image signature sniffing:

• Inspects base64 binary data magic bytes
• Detects PNG, WebP, JPEG, GIF formats
• Corrects mismatched MIME types before sending to API
• Only applies to images (audio types unchanged)
• Logs mismatches for debugging

Changes

• New: packages/core/src/utils/imageMimeDetection.ts - Image MIME detection utility
• New: packages/core/src/utils/imageMimeDetection.test.ts - Tests for MIME detection utility
• Modified: packages/core/src/tools/mcp-tool.ts - Integrated MIME sniffing
• Modified: packages/core/src/tools/mcp-tool.test.ts - Added test cases for MIME correction

Testing

• ✅ Unit tests for WebP/PNG mismatch scenarios
• ✅ Tests for embedded resource blocks
• ✅ Type-checking (npm run typecheck) and linting (npm run lint) pass with 0 errors/warnings
• ✅ All existing tests still pass

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[github-actions]

📊 PR Size: size/L

• Lines changed: 331
• Additions: +320
• Deletions: -11
• Files changed: 4

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where WebP images from Figma MCP integrations were incorrectly identified as PNGs, leading to failures in the Gemini API. By implementing a robust signature-based detection mechanism, the system can now verify and correct MIME types dynamically, ensuring reliable image processing and better error logging for mismatches.

Highlights

• MIME Type Detection Utility: Introduced a new utility, imageMimeDetection.ts, which inspects base64 image magic bytes to identify the true format (PNG, WebP, JPEG, or GIF).
• MCP Tool Integration: Updated mcp-tool.ts to automatically validate and correct MIME types for image and resource blocks before they are processed by the LLM, preventing HTTP 400 errors.
• Enhanced Testing: Added comprehensive unit tests for the new detection utility and updated existing MCP tool tests to cover scenarios where mismatched MIME types are corrected.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

🛑 Action Required: Evaluation Approval

Steering changes have been detected in this PR. To prevent regressions, a maintainer must approve the evaluation run before this PR can be merged.

Maintainers:

1. Go to the Workflow Run Summary.
2. Click the yellow 'Review deployments' button.
3. Select the 'eval-gate' environment and click 'Approve'.

Once approved, the evaluation results will be posted here automatically.

[gemini-code-assist]

Code Review

This pull request introduces image MIME type detection and correction utilities to validate and correct mismatched image and resource block MIME types based on their magic bytes. The review feedback highlights two important improvements: first, restricting MIME type correction to cases where the declared type is already an image or a generic binary type to avoid false positives on non-image resources (such as text files starting with 'GIF'); second, applying trimStart() to the base64 data before slicing to ensure leading whitespace does not interfere with magic byte detection.

[gemini-code-assist]

Running image MIME type detection on all resource blobs (including text files, PDFs, etc.) can lead to false positives. For example, a plain text file starting with the word "GIF" (e.g., "GIF is a format...") will have the magic bytes 0x47 0x49 0x46 and be incorrectly corrected to image/gif. This will cause the Gemini API to fail when it tries to decode the text as an image.

To prevent this, we should restrict MIME type correction to cases where the declared MIME type is already an image type, or is a generic/unknown binary type (like application/octet-stream or unknown type).

export function validateAndCorrectMimeType(
  declaredMimeType: string,
  base64Data: string,
): string {
  const lowerDeclared = declaredMimeType.toLowerCase();
  const isImageOrGeneric =
    lowerDeclared.startsWith('image/') ||
    lowerDeclared === 'application/octet-stream' ||
    lowerDeclared === 'unknown type';

  if (!isImageOrGeneric) {
    return declaredMimeType;
  }

  const detectedType = detectImageMimeType(base64Data);
  if (detectedType && detectedType !== declaredMimeType) {
    debugLogger.warn(
      "Image MIME type mismatch: declared as " + declaredMimeType + " but detected as " + detectedType,
    );
    return detectedType;
  }
  return declaredMimeType;
}

[gemini-code-assist]

If the base64Data string contains leading whitespace or newlines, slice(0, 48) might capture only whitespace, resulting in an empty or too-short cleanPrefix after stripping whitespace.

Using trimStart() before slicing ensures we robustly capture the actual base64 characters while maintaining high performance.

Suggested change

	const cleanPrefix = base64Data.slice(0, 48).replace(/\s+/g, '');
	const cleanPrefix = base64Data.trimStart().slice(0, 48).replace(/\s+/g, '');