LlamaCppModel

Run a local GGUF model with llama.cpp's HTTP server. Download a .gguf weights file, start the server, then connect with LlamaCppModel.

Local setup

# Download a GGUF model (example — pick a model suited to your hardware)
# https://huggingface.co/models?library=gguf

# Start the llama.cpp server (model must be loaded locally)
llama-server -m /path/to/model.gguf --host 0.0.0.0 --port 8080

Environment variables: LLAMACPP_BASE_URL (default http://localhost:8080), LLAMACPP_MODEL_ID

import os
from elsai_model.llama_cpp import LlamaCppModel

model = LlamaCppModel(
    base_url=os.getenv("LLAMACPP_BASE_URL", "http://localhost:8080"),
    model_id=os.getenv("LLAMACPP_MODEL_ID", "default"),
    params={"temperature": 0.2},
)
messages = [{"role": "user", "content": "Say hello in one short sentence."}]

response = model.invoke(messages)
print(response["choices"][0]["message"]["content"])

for chunk in model.stream_text(messages):
    print(chunk, end="", flush=True)

Or via the factory (base_url; no api_key):

from elsai_model import LLM, Provider

model = LLM(
    provider=Provider.LLAMA_CPP,
    model=os.getenv("LLAMACPP_MODEL_ID", "default"),
    base_url=os.getenv("LLAMACPP_BASE_URL", "http://localhost:8080"),
    params={"temperature": 0.2},
)

---

See also

• LLM Models overview — install, extras, factory
• LLM factory