A Guide to Turning Architectural Images into JSON with the Gemini API

What will you learn in this guide?

You can extract structured notes about a space, objects, or visible features from an architectural image, then receive those notes in JSON format. This guide walks through sending text and an image together to the Gemini API, writing a clear prompt, and defining the response structure with a schema. The examples use an interior image and also show how to account for uncertainty in prompts, so the model does not guess measurements or materials that are not visible.

Google’s Gemini API documentation describes image understanding for use cases such as image captioning, classification, and visual question answering. Images can be provided via URL, inline data, or the Files API. The model name used in this example, gemini-3.8-flash, appears in the documentation examples.

Requirements

• A client or REST request that can work with the Gemini API. The documentation provides Python and JavaScript examples, as well as examples for Java and Go.

• A way to send the image to the API: base64-encoded inline data for small images, or the Files API for large files or images you plan to reuse.

• If you want JSON output, a JSON Schema defining the fields in the response. Google GenAI SDKs let you define schemas with Pydantic for Python and Zod for JavaScript.

• If using REST, an API key in the request. This guide makes no assumptions about account creation, fees, or usage quotas that are not covered by the sources.

As an example, suppose we want a structured list of visible objects and location descriptions in an interior image. The schema below shows a simple structure for this purpose. You can change the fields to suit your actual workflow; what matters is that the fields in the schema match the output requested in the prompt.

{
  "type": "object",
  "properties": {
    "gozlemler": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "unsur": { "type": "string" },
          "konum": { "type": "string" },
          "gorunur_ozellik": { "type": "string" }
        },
        "required": ["unsur", "konum", "gorunur_ozellik"]
      }
    }
  },
  "required": ["gozlemler"]
}

Implementation steps

1. Define the image and objective

First, be clear about what question the image should answer. For example, the goal might be to list the furniture visible in a perspective view; in another use case, you might describe visible surface features. Do not ask for measurements, structural systems, or product details that are not shown in the image. If such information cannot be determined conclusively from the image, ask the model to say so explicitly.

2. Write a clear, well-scoped prompt

Google’s prompt design guide recommends clearly stating the task and constraints, and specifying the desired response format. The example below applies this approach to an interior image:

Görseli bir iç mekân sunum paftasını inceler gibi değerlendir. Yalnızca görüntüde açıkça seçilebilen unsurları listele. Her unsur için kısa bir konum tarifi ve görülebilir özelliğini yaz. Ölçü, marka veya malzeme türünü görüntüden kesin biçimde belirleyemiyorsan tahmin etme; bunu belirt. Yanıtı, tanımladığım JSON şemasındaki alanlara uygun biçimde ver.

The prompt defines the task and scope; the schema defines the fields in the response. Simply saying “analyze the image” leaves the model to decide which details matter. Specifying the required fields and what not to guess up front makes the output easier to use in project notes. Even so, the result is not a verified measurement or technical assessment of the image.

3. Add the image to the API request

You can base64-encode a small image and send it in the data field. The documentation uses image data and a MIME type alongside the text prompt for this method. Inline image data is subject to a total request size limit of 20 MB, including the prompt text and system instructions. If the request might exceed this limit, upload the file to the Files API and include the returned URI in the request.

The documentation recommends the Files API for large files or when you plan to use the same image in multiple requests. Include the text and image inputs together in the request, and specify the image’s MIME type as appropriate. One of the image types shown in the documentation is image/jpeg.

4. Configure the JSON response format

With structured outputs, you set the response format to JSON and add the schema to the request. In the REST example in the documentation, mime_type is set to application/json and the schema is provided in the schema field. The Python example also generates a JSON Schema from a Pydantic model and validates the response against that model; the JavaScript example uses a Zod schema.

Avoid making the schema unnecessarily large for an architectural workflow. For example, if you only need the object, location, and visible feature, adding extra fields your workflow does not use can create maintenance and review overhead. Also make sure the required fields in the schema align with the information requested in the prompt.

5. Review and use the response

Check the JSON returned by the model to see whether the specified fields are populated and whether the content matches the image. Structured outputs make it easier to produce responses that conform to a schema; that does not mean the image interpretation is always correct. In particular, verify conclusions about materials, measurements, or design decisions against drawings, project documents, or expert review.

Using the same schema for multiple images can make results easier to compare. In each request, clearly state what aspect of the image to examine, and assess the responses by trying small prompt variations for the same task. Prompt design is iterative: rather than treating the first attempt as final, identify incorrect or unnecessary fields and clarify the instructions.

Common mistakes

• Requesting certainty about what is not visible: Perspective, resolution, or framing can make some details unclear. Ask the model to report uncertainty rather than guess.

• Misaligning the schema and prompt: If a requested field is missing from the schema, or the schema requires a field that the task does not explain, it becomes harder to use the results. Review both together.

• Sending a large file as inline data: Inline image data is subject to a total request size limit of 20 MB. Use the Files API for large files.

• Treating JSON formatting as a guarantee of accuracy: A structured response does not prove that the information accurately reflects the image. Independently verify important design and technical decisions.

• Treating the first prompt as the final version: However clearly the task and constraints are stated, you still need to review the output on real examples and refine the prompt.

Impact on architecture and archviz workflows

This method can be useful for architecture firms, interior designers, and students who want to describe interior images in a visual archive, create structured observation lists from presentation visuals, or compare different images using shared fields. JSON keeps data more organized in evaluation and classification tasks where the same fields are used repeatedly. However, results should not be transferred directly to project boards or technical documents as verified information.

For compatibility, the client and API request you use must support the image input and structured output formats described in the documentation. Choose between sending an image inline and using the Files API based on file size and whether you need to reuse it. Since the sources for this guide do not specify API access or cost terms, check the current access and licensing terms separately before adopting the method in a real office workflow.

Next steps

Start by testing a small schema on a few representative images. Identify fields that recur in the responses and are genuinely useful in your workflow; flag uncertain observations separately. Then review the prompt, schema, and image submission method together. If you need more extensive JSON structures, consult the SDK examples in the Gemini API structured output documentation, and the Image understanding guide for image input.

Sources and licensing

This content has been adapted into Turkish using Google AI for Developers’ “Structured outputs,” “Image understanding,” and “Prompt design strategies” guides. All three sources are available under the CC BY 4.0 license.