llama.cpp

Run a local GGUF model via a llama.cpp HTTP server with LlamaCppModel. No cloud API key required — the model runs on your machine.

For standalone invoke / stream usage, see LLM Models — llama.cpp.

Install

pip install --extra-index-url https://elsai-agents.elsai.ai/root/ elsai-agents==0.3.1
pip install --extra-index-url https://core-packages.elsai.ai/root/ "elsai-model[llama-cpp]==2.1.0"

Setup

1. Download a GGUF model (e.g. from Hugging Face).
2. Start the llama.cpp server with the model loaded locally:

llama-server -m /path/to/model.gguf --host 0.0.0.0 --port 8080

3. Point the client at your server:

export LLAMACPP_BASE_URL=http://localhost:8080
export LLAMACPP_MODEL_ID=default

Agent — basic

import os
from elsai import Agent
from elsai_model import LLM, Provider

model = LLM(
    provider=Provider.LLAMA_CPP,
    model=os.getenv("LLAMACPP_MODEL_ID", "default"),
    base_url=os.getenv("LLAMACPP_BASE_URL", "http://localhost:8080"),
    params={"temperature": 0.2},
)
agent = Agent(
    model=model,
    system_prompt="You are a concise assistant.",
)
result = agent("Summarize what an AI agent is in two sentences.")
print(result)