Caching

There are three distinct caches: graph-node results, complete model answers and provider prompt prefixes. The first two can skip work; provider prefix caching still makes a model request. Choose keys and invalidation based on what affects the result.

Complete Installation for the OpenAI examples below.

Run this page’s examples in order in one IEx session; later blocks reuse the modules and values defined earlier.

Cache a graph node

defmodule MyApp.CachedClassifier do
  use BeamWeaver.Graph
  alias BeamWeaver.Core.{ChatModel, Message}

  graph do
    node :classify, &__MODULE__.classify/1, cache: [ttl: 60_000]
    edge start(), :classify
    edge :classify, finish()
  end

  def classify(%{subject: subject}) do
    model = BeamWeaver.Models.init_chat_model!("openai:gpt-5.4-mini", max_tokens: 128, timeout: 60_000)

    with {:ok, reply} <-
           ChatModel.invoke(model, "Classify as billing or shipping. Reply only with that word. Ticket: #{subject}") do
      %{category: reply |> Message.text() |> String.trim() |> String.downcase()}
    end
  end
end

cache = BeamWeaver.Cache.ETS.new()
input = %{subject: "I was charged twice for order A-1001."}
{:ok, first} = MyApp.CachedClassifier.invoke(input, cache: cache)
"billing" = first.category
{:ok, stream} = MyApp.CachedClassifier.stream_events(input, cache: cache, live: true)
events = Enum.to_list(stream)
true = Enum.any?(events, &match?(%{event: %BeamWeaver.Stream.Events.Task{kind: :cache_hit, node: "classify"}}, &1))
false = Enum.any?(events, &match?(%{event: %BeamWeaver.Stream.Events.Error{}}, &1))

The first call uses the provider; the second reports an actual node cache hit. Only successful normal node results are cached. Failure, interruption and error-handler recovery do not become normal cached node results.

Choose the cache key

The default key includes graph, node and input state. A custom key: receives {graph, node, state}; namespace: separates versions of your policy or prompt. ttl: is milliseconds and a hit does not extend it. Metadata can accompany an entry.

Include every dependency that affects the answer: tenant, source revision, model/prompt version and relevant configuration. If the node reads mutable external state absent from the key, a cache can return stale or cross-tenant data. Do not cache side-effecting actions as a substitute for idempotency.

This key varies with the tenant and source revision, while ignoring an unrelated UI field:

alias BeamWeaver.Graph
alias BeamWeaver.Graph.Compiled
cache = BeamWeaver.Cache.ETS.new()

graph =
  Graph.new()
  |> Graph.add_node(:quote, fn %{tenant: tenant} -> %{quote: "Quote for #{tenant}"} end,
    cache: [namespace: "quote-v1", key: fn {_graph, _node, state} -> {state.tenant, state.revision} end]
  )
  |> Graph.add_edge(Graph.start(), :quote)
  |> Graph.add_edge(:quote, Graph.end_node())
  |> Graph.compile!(cache: cache)

{:ok, %{quote: "Quote for acme"}} = Compiled.invoke(graph, %{tenant: "acme", revision: 1, ui: "compact"})
{:ok, events} = Compiled.stream_events(graph, %{tenant: "acme", revision: 1, ui: "wide"})
true = Enum.any?(events, &match?(%{event: %BeamWeaver.Stream.Events.Task{kind: :cache_hit}}, &1))

Where the cache lives

Provide an explicit cache adapter per module invocation or when compiling a graph. Cached nodes without one fail with :explicit_cache_required. ETS belongs to its creating process; Ecto uses PostgreSQL and the cache migration in Storage . Call options can override declared adapters.

Cache model answers

Models.cached/3 reuses a full answer for the same model inputs/options:

alias BeamWeaver.Core.{ChatModel, Message}
model = BeamWeaver.Models.init_chat_model!("openai:gpt-5.4-mini", max_tokens: 128, timeout: 60_000)
cached = BeamWeaver.Models.cached(model, BeamWeaver.Cache.ETS.new(), ttl: 60_000)
{:ok, original} = ChatModel.invoke(cached, "Define a parcel briefly.")
{:ok, replayed} = ChatModel.invoke(cached, "Define a parcel briefly.")
true = byte_size(Message.text(original)) > 0
true = Message.text(replayed) == Message.text(original)
0 = replayed.usage_metadata.total_cost

A cached response reports zero new cost; its retained token counts describe the stored answer, not fresh provider consumption. Cache replay streaming is not a second live provider stream. Use this wrapper only when reuse is correct for the request's freshness and ownership requirements.

Expiry and size

Cache ttl: is in milliseconds. The bulk Cache.set_many/3 convenience accepts {value, ttl_seconds} pairs and converts those seconds to milliseconds. Expired entries are missed and can be swept with Cache.sweep_expired/2. ETS max_entries: bounds growth; cleanup and pruning capabilities depend on the adapter. lookup, put, delete and clear also support direct application caching. Check hit/miss telemetry when diagnosing unexpected work or stale results.

Exercise TTL and capacity operations directly:

orders = BeamWeaver.Cache.ETS.new()
:ok = BeamWeaver.Cache.put(orders, :status, "A-1001", "shipped", ttl: 10)
:ok = BeamWeaver.Cache.put(orders, :status, "A-1002", "shipped", ttl: 10)
Process.sleep(20)

BeamWeaver.Cache.sweep_expired(orders)
#=> {:ok, 2}

Prompt caching

Provider caching reuses eligible prefixes while generating a new response. The prompt_caching declaration configures supported provider hints:

Google also exposes explicit cached-content resources; its provider guide shows creation, use and deletion. Caching requirements and account behavior remain provider-specific. A declaration does not guarantee a hit.

Keys include scope, model and prompt; scope: and version: customize invalidation. Keep stable content before changing per-request content. Avoid moving trusted instructions into untrusted user text merely to improve a hit rate.

Check prompt cache hits

Read usage_metadata.input_token_details.cache_read from real replies. A zero/missing value is not a verified hit. Do not test caching by inventing usage metadata or by initializing a model without calling it.

Hits can depend on prefix length, expiry and provider routing.

Make two real calls with the same substantial prefix and report the provider’s measured cache reads. Zero means no hit was observed in that call:

alias BeamWeaver.Core.{ChatModel, Message}

model =
  BeamWeaver.Models.init_chat_model!("openai:gpt-5.4-mini",
    max_tokens: 128,
    timeout: 60_000,
    prompt_cache_key: "beamweaver-refund-guide-v1"
  )

prefix =
  "Answer briefly from the following policy.\n" <>
    String.duplicate("Refunds take five business days. Do not promise same-day refunds.\n", 160)

usage =
  for _ <- 1..2 do
    {:ok, reply} = ChatModel.invoke(model, [Message.system(prefix), Message.user("How long do refunds take?")])
    true = byte_size(Message.text(reply)) > 0
    get_in(reply.usage_metadata, [:input_token_details, :cache_read]) || 0
  end

IO.inspect(usage, label: "Observed cached input tokens per call")