Using OpenAI's GPT-4 Vision API
You'll learn how to make practical API calls to GPT-4 Vision, handle image encoding, and structure multimodal prompts that combine text and images.
Understanding Multimodal API Calls
GPT-4 Vision (also called GPT-4V) extends the standard text-based API to accept images alongside text prompts. Think of it like having a conversation with an assistant who can both read your questions and look at photos you show them. The API accepts images in two formats: publicly accessible URLs or base64-encoded strings embedded directly in the request.
Base64 Encoding for Images
When working with local images or private data, you'll encode them as base64 strings. This converts binary image data into text that can be safely transmitted in JSON:
import base64
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode('utf-8')
base64_image = encode_image("product_photo.jpg")
Structuring Multimodal Prompts
The API uses a messages structure where content can be an array mixing text and image objects:
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4-vision-preview",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
}
}
]
}
],
max_tokens=300
)
print(response.choices[0].message.content)
You can include multiple images in a single prompt, mix text and images in any order, and leverage the model's ability to reason across both modalities simultaneously.
Key Takeaway: GPT-4 Vision API calls combine text and images through structured message content, using either URLs or base64-encoded strings to pass visual data alongside your prompts.