Multimodal

Optional extras:

pip install --extra-index-url https://elsai-agents.elsai.ai/root/ "elsai-agents-tools[diagram]==0.3.0"
pip install --extra-index-url https://elsai-agents.elsai.ai/root/ "elsai-agents-tools[twelvelabs]==0.3.0"

Tools

generate_image_stability

Generate images from text prompts using Stability AI models with configurable size and style

Environment variables

Variable	Required	Default	Description
STABILITY_API_KEY	yes	—	Stability AI API key
STABILITY_MODEL_ID	optional	stability.stable-image-core-v1:1	Stability model to use
STABILITY_OUTPUT_DIR	optional	unset	If set, save generated images to this folder

image_reader

Load local image files and return descriptions or extracted text for vision-style agent reasoning

Environment variables

No dedicated environment variables.

generate_image

Generate images from prompts using Amazon Bedrock image models (e.g. Titan, Stable Diffusion)

Environment variables

Variable	Required	Default	Description
AWS_REGION	yes	—	AWS region for Bedrock / Polly when not already set in your environment

nova_reels

Create short videos from text prompts with Amazon Nova Reels on Bedrock

Environment variables

Variable	Required	Default	Description
AWS_REGION	yes	—	AWS region for Bedrock / Polly when not already set in your environment
NOVA_REEL_DEFAULT_SEED	optional	0	Default random seed for video generation
NOVA_REEL_DEFAULT_FPS	optional	24	Default frames per second
NOVA_REEL_DEFAULT_DIMENSION	optional	1280x720	Default video size
NOVA_REEL_DEFAULT_MAX_RESULTS	optional	10	Default max results when listing jobs

speak

Convert text to speech using macOS say locally or Amazon Polly in the cloud

Environment variables

Variable	Required	Default	Description
AWS_REGION	yes	—	AWS region for Bedrock / Polly when not already set in your environment
SPEAK_DEFAULT_VOICE_ID	optional	Joanna	Default Polly / macOS voice
SPEAK_DEFAULT_OUTPUT_PATH	optional	speech_output.mp3	Default path for saved audio
SPEAK_DEFAULT_MODE	optional	fast	Default speak mode
SPEAK_DEFAULT_STYLE	optional	green	Default console style
SPEAK_DEFAULT_PLAY_AUDIO	optional	True	Play audio automatically after generation

diagram

Render cloud architecture and UML diagrams from agent descriptions (AWS, Azure, GCP, and generic)

Extra: diagram

Environment variables

No dedicated environment variables.

chat_video

Ask questions about uploaded video content using TwelveLabs multimodal understanding

Extra: twelvelabs

Environment variables

Variable	Required	Default	Description
TWELVELABS_API_KEY	yes	—	TwelveLabs API key
TWELVELABS_PEGASUS_INDEX_ID	yes	—	Default index for chat_video (or pass it as a tool argument)

search_video

Semantic search across indexed video libraries to find clips matching a natural-language query

Extra: twelvelabs

Environment variables

Variable	Required	Default	Description
TWELVELABS_API_KEY	yes	—	TwelveLabs API key
TWELVELABS_MARENGO_INDEX_ID	yes	—	Default index for search_video (or pass it as a tool argument)

Examples

Requires OPENAI_API_KEY and AWS credentials (Bedrock image models / Polly). Set AWS_DEFAULT_REGION=us-west-2 if not already configured.

pip install --extra-index-url https://elsai-agents.elsai.ai/root/ elsai-agents==0.3.5
pip install --extra-index-url https://core-packages.elsai.ai/root/elsai-model/ "elsai-model[openai]==2.1.1"
pip install --extra-index-url https://elsai-agents.elsai.ai/root/ elsai-agents-tools==0.3.0

import os

from elsai import Agent
from elsai.agent import AgentConfig
from elsai_model.openai import OpenAIModel
from elsai_tools.generate_image import generate_image
from elsai_tools.speak import speak

agent = Agent(
    model=OpenAIModel(
        model_id="gpt-4o-mini",
        client_args={"api_key": os.environ["OPENAI_API_KEY"]},
    ),
    tools=[generate_image, speak],
    system_prompt=(
        "Use generate_image and speak when the user asks for images or speech. "
        "After using tools, answer the user clearly in plain text."
    ),
    config=AgentConfig(name="multimodal_assistant"),
)

result = agent(
    "Generate a simple image of a steampunk robot playing chess, "
    "then speak a one-sentence confirmation that the image was created."
)
print(result)

Parameter reference: Multimodal API.

See also

• Prebuilt Tools overview