Course contentsShow
← AI Engineering
Lesson 1723 of 1,88642. Multimodal SystemsFree lesson

Using OpenAI's GPT-4 Vision API

Making API calls with images, handling base64 encoding, and managing multimodal prompts.

Using OpenAI's GPT-4 Vision API

You'll learn how to make practical API calls to GPT-4 Vision, handle image encoding, and structure multimodal prompts that combine text and images.

Understanding Multimodal API Calls

GPT-4 Vision (also called GPT-4V) extends the standard text-based API to accept images alongside text prompts. Think of it like having a conversation with an assistant who can both read your questions and look at photos you show them. The API accepts images in two formats: publicly accessible URLs or base64-encoded strings embedded directly in the request.

Base64 Encoding for Images

When working with local images or private data, you'll encode them as base64 strings. This converts binary image data into text that can be safely transmitted in JSON:

import base64

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

base64_image = encode_image("product_photo.jpg")

Structuring Multimodal Prompts

The API uses a messages structure where content can be an array mixing text and image objects:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}"
                    }
                }
            ]
        }
    ],
    max_tokens=300
)

print(response.choices[0].message.content)

You can include multiple images in a single prompt, mix text and images in any order, and leverage the model's ability to reason across both modalities simultaneously.

Key Takeaway: GPT-4 Vision API calls combine text and images through structured message content, using either URLs or base64-encoded strings to pass visual data alongside your prompts.