Building Local AI Agents with Next.js 16, Ollama, and Vercel AI SDK

Why Local AI

Every AI feature in a web app today routes through OpenAI, Anthropic, or Google. That means API latency (500ms–3s per call), per-token pricing, data privacy exposure, and a hard dependency on internet connectivity.

Local inference changes that calculation. Running a model on the same machine as your dev server cuts response latency to 50–150ms per token batch. No data leaves your network. No API key is needed. And for prototyping — where you iterate on prompts, tool definitions, and response parsing — the feedback loop drops from minutes to seconds.

Ollama makes local models trivial to run. It wraps llama.cpp, exposes an OpenAI-compatible HTTP API, and supports models from 1B to 70B parameters. For a frontend agent that calls tools and returns structured data, a 7B model like llama3.2 or mistral is sufficient — and runs comfortably on a MacBook or standard Linux workstation.

This tutorial builds a local AI agent that:

• Runs entirely on your machine with no cloud dependencies
• Calls tools (functions the model decides to execute)
• Streams responses to the browser in real time
• Handles cold starts, timeouts, and context limits

---

Prerequisites

Before writing code, you need:

• Ollama installed — curl -fsSL https://ollama.com/install.sh | sh (Linux/macOS) or download from ollama.com
• A model pulled — ollama pull llama3.2 (2.0GB, runs on 8GB RAM)
• Node.js 20+ and a Next.js 16 project (npx create-next-app@latest)
• @ai-sdk/openai and ai packages

npm install ai @ai-sdk/openai zod

@ai-sdk/openai is the OpenAI-compatible provider. Since Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1, you can point this provider at your local instance. No actual OpenAI account needed.

Verify Ollama is running:

curl http://localhost:11434/api/tags
# Should return JSON with a "models" array

---

Setting Up the Route Handler

The Vercel AI SDK uses a Route Handler pattern: the frontend calls a /api/chat endpoint, which streams tokens from the model to the client.

Create app/api/chat/route.ts:

// app/api/chat/route.ts
import { openai } from '@ai-sdk/openai'
import { streamText, tool } from 'ai'
import { z } from 'zod'

// Point the provider at Ollama's local endpoint
const localModel = openai('llama3.2', {
  baseURL: 'http://localhost:11434/v1',
})

export async function POST(req: Request) {
  const { messages } = await req.json()

  const result = streamText({
    model: localModel,
    system: `You are a helpful assistant with access to tools.
Only call a tool when the user asks for information you cannot provide from your training data.
If you call a tool, wait for the result before responding.`,
    messages,
    tools: {
      getWeather: tool({
        description: 'Get the current weather for a city',
        parameters: z.object({
          city: z.string().describe('City name (e.g., London, Tokyo)'),
        }),
        execute: async ({ city }) => {
          // Simulated weather fetch — replace with real API call
          const conditions = ['sunny', 'cloudy', 'rainy', 'windy']
          const temp = Math.round(10 + Math.random() * 25)
          return {
            city,
            temperature: `${temp}°C`,
            condition: conditions[Math.floor(Math.random() * conditions.length)],
          }
        },
      }),
    },
  })

  return result.toDataStreamResponse()
}

Key details:

• baseURL: 'http://localhost:11434/v1' redirects the OpenAI SDK to Ollama. No API key needed.
• streamText returns tokens as a Server-Sent Events stream. The client reads them incrementally.
• Tools are defined inline with Zod schemas. The model decides when to call them based on the user's request.
• The system prompt constrains tool usage — without it, small models may call tools unnecessarily.

---

Adding Tool Calling

Tool calling is what transforms a chat completion into an agent. The model generates a structured JSON request for a tool, the runtime executes it, and the result is fed back into the next model turn.

Weather Tool

The getWeather tool above returns a static object. In production, you'd call an actual weather API:

const getWeather = tool({
  description: 'Get the current weather for a city using Open-Meteo API (free, no key)',
  parameters: z.object({
    latitude: z.number(),
    longitude: z.number(),
  }),
  execute: async ({ latitude, longitude }) => {
    const res = await fetch(
      `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&current_weather=true`
    )
    const data = await res.json()
    return data.current_weather
  },
})

Database Query Tool

For a more realistic agent, add a tool that queries a local SQLite database:

import Database from 'better-sqlite3'

const queryDatabase = tool({
  description: 'Run a read-only SQL query against the local products database',
  parameters: z.object({
    sql: z.string().describe('SELECT query to execute'),
  }),
  execute: async ({ sql }) => {
    // Read-only guard prevents accidental mutations
    if (!/^select\b/i.test(sql.trim())) {
      return { error: 'Only SELECT queries are allowed' }
    }
    const db = new Database('./data/products.db')
    const rows = db.prepare(sql).all()
    db.close()
    return rows
  },
})

Now add queryDatabase to the tools object in the route handler. The model will call it when the user asks questions like "Which products are out of stock?" or "Show me orders over $500."

How Tool Calling Works Under the Hood

1. The model receives the user message plus tool definitions.
2. If the model decides a tool is needed, it returns a tool_call with JSON arguments instead of a text response.
3. The AI SDK intercepts the tool_call, runs execute(), and sends the result back to the model as a new message.
4. The model generates a text response incorporating the tool result.

This loop repeats until the model decides it has enough information to respond. In practice, most agent interactions resolve in 1–2 tool calls.

Tool Choice and Multi-Tool Patterns

By default, the model decides which tool to call and when. You can override this with toolChoice:

// Force the model to always call a specific tool first
const result = streamText({
  model: localModel,
  messages,
  tools: { getWeather, queryDatabase },
  toolChoice: 'auto',     // default — model decides
  // toolChoice: 'required', // forces a tool call every turn
  // toolChoice: { type: 'tool', toolName: 'getWeather' }, // force specific tool
})

toolChoice: 'required' is useful for classification pipelines where every user message should first be categorized (via a tool) before generating a response. For a search agent, you might force a database query tool on every turn and only generate text from the results.

Small models struggle with multi-tool decisions. If your agent has five tools, llama3.2 may call the wrong one or call a tool with malformed JSON arguments. Limit the tool set to 2-3 per agent and compose multiple specialized agents if you need more capabilities.

Structured Data Extraction with Tools

Tools aren't only for side effects. You can use them to extract structured data from the model's output:

const extractMeetingInfo = tool({
  description: 'Extract date, time, and attendees from a meeting request',
  parameters: z.object({
    date: z.string().describe('Meeting date in YYYY-MM-DD format'),
    time: z.string().describe('Meeting time in HH:MM format'),
    attendees: z.array(z.string()).describe('List of attendee email addresses'),
    title: z.string().max(100).describe('Meeting title'),
  }),
  execute: async (data) => data, // Pass through — model generates the structure
})

This pattern converts natural language into typed data without writing a parser. The Zod schema acts as the interface contract. If the model's output doesn't match the schema, the AI SDK returns a validation error and the model retries with corrected arguments.

---

Building the React Frontend

The frontend uses the useChat hook from the AI SDK. It handles the streaming connection, manages the message array, and exposes append for sending new messages.

// app/page.tsx
'use client'

import { useChat } from 'ai/react'
import { useRef, useEffect } from 'react'

export default function ChatPage() {
  const { messages, input, handleInputChange, handleSubmit, isLoading, error } =
    useChat({
      api: '/api/chat',
      // Keep stream alive for local models that generate slowly
      onError: (err) => console.error('Chat error:', err),
    })

  const bottomRef = useRef<HTMLDivElement>(null)

  // Auto-scroll as tokens arrive
  useEffect(() => {
    bottomRef.current?.scrollIntoView({ behavior: 'smooth' })
  }, [messages])

  return (
    <div className="max-w-2xl mx-auto p-4">
      <div className="space-y-4 mb-4">
        {messages.map((m) => (
          <div
            key={m.id}
            className={`p-3 rounded-lg ${
              m.role === 'user'
                ? 'bg-blue-100 ml-12'
                : 'bg-gray-100 mr-12'
            }`}
          >
            <div className="text-xs font-medium mb-1 text-gray-500">
              {m.role === 'user' ? 'You' : 'Assistant'}
            </div>
            {/* Render tool invocations as expandable details */}
            {m.parts?.map((part, i) => {
              if (part.type === 'tool-invocation') {
                return (
                  <details key={i} className="text-sm border rounded p-2 mb-2">
                    <summary className="cursor-pointer font-mono text-xs">
                      🛠 {part.toolInvocation.toolName}
                    </summary>
                    <pre className="text-xs mt-1 overflow-auto">
                      {JSON.stringify(part.toolInvocation.args, null, 2)}
                    </pre>
                    {part.toolInvocation.state === 'result' && (
                      <pre className="text-xs mt-1 bg-green-50 p-1 rounded">
                        → {JSON.stringify(part.toolInvocation.result, null, 2)}
                      </pre>
                    )}
                  </details>
                )
              }
              if (part.type === 'text') {
                return <div key={i} className="whitespace-pre-wrap">{part.text}</div>
              }
              return null
            })}
          </div>
        ))}

        {isLoading && (
          <div className="text-sm text-gray-400 animate-pulse">
            Thinking...
          </div>
        )}

        {error && (
          <div className="text-sm text-red-600 bg-red-50 p-3 rounded border border-red-200">
            {error.message}
          </div>
        )}

        <div ref={bottomRef} />
      </div>

      <form onSubmit={handleSubmit} className="flex gap-2">
        <input
          value={input}
          onChange={handleInputChange}
          placeholder="Ask the local AI agent..."
          className="flex-1 p-2 border rounded-lg"
          disabled={isLoading}
        />
        <button
          type="submit"
          disabled={isLoading || !input.trim()}
          className="px-4 py-2 bg-blue-600 text-white rounded-lg disabled:opacity-50"
        >
          Send
        </button>
      </form>
    </div>
  )
}

This renders the full conversation. Each message displays text parts inline and tool invocations as collapsible <details> elements so the user can see the raw tool calls and responses — useful for debugging what the model is doing.

Rendering Streaming Text with Markdown

Raw token output is hard to read. Parse markdown in the stream and render it progressively:

import ReactMarkdown from 'react-markdown'

// Inside the message render loop:
{m.role === 'assistant' && m.parts?.map((part, i) => {
  if (part.type === 'text') {
    return (
      <ReactMarkdown
        key={i}
        components={{
          code: ({ className, children, ...props }) => {
            const isInline = !className
            if (isInline) {
              return <code className="bg-gray-200 px-1 rounded text-sm">{children}</code>
            }
            return (
              <pre className="bg-gray-900 text-gray-100 p-3 rounded-lg overflow-auto text-sm my-2">
                <code className={className}>{children}</code>
              </pre>
            )
          },
        }}
      >
        {part.text}
      </ReactMarkdown>
    )
  }
  // ...tool invocation rendering from earlier
})}

The stream feeds partial markdown to the renderer. react-markdown handles incomplete tokens gracefully — a heading ## that hasn't received its text yet simply doesn't render until the tokens arrive. This is critical for local models where generation speed is slow enough that users watch text appear word by word.

Streaming Behavior

useChat connects to the Route Handler via fetch and reads the response as a stream. Each token from the model arrives as an SSE event and is appended to the last assistant message. The UI updates in real time without waiting for the full response.

For local models, this is important: llama3.2 generates at roughly 20–40 tokens/second on a modern laptop. Streaming means the user sees output as it's generated rather than staring at a spinner for 10 seconds.

---

Performance & Error Handling

Local inference has failure modes that cloud APIs don't. Handle them explicitly.

Model Cold Start

Ollama loads models on first request. The first call after ollama pull or after a period of inactivity takes 2–5 seconds while the model loads into memory. Subsequent calls are near-instant.

Mitigation: Warm the model on server startup or during the first request.

// app/api/chat/route.ts — warmup helper
let warmed = false

async function warmModel() {
  if (warmed) return
  await fetch('http://localhost:11434/api/generate', {
    method: 'POST',
    body: JSON.stringify({ model: 'llama3.2', prompt: 'hello', stream: false }),
  })
  warmed = true
}

// Call during app initialization
warmModel()

Streaming Timeout

Local models can stall on complex prompts, especially with tool calling. Set a timeout on the stream:

const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), 30_000) // 30s max

const result = streamText({
  model: localModel,
  messages,
  tools: { ... },
})

// Clean up timeout when stream completes
result.finally(() => clearTimeout(timeout))

return result.toDataStreamResponse({ signal: controller.signal })

Context Window Limits

llama3.2 has an 8K token context window. Tool definitions consume tokens — each tool's description, parameter schema, and invocation result counts against this limit. After ~5 tool calls with verbose responses, the model may begin to "forget" earlier conversation context.

Solutions:

• Keep tool descriptions under 100 characters.
• Use concise parameter names and descriptions.
• Truncate tool results that return large datasets. Return a summary instead of raw rows.
• If you exceed the window, Ollama silently truncates the oldest messages. Test your agent with worst-case conversation length.

Error Recovery and Retry Logic

Local models fail differently than cloud APIs. Instead of rate limits, you get OOM crashes. Instead of timeout errors, you get stalled token generation. Build a retry layer into the Route Handler:

export async function POST(req: Request) {
  const { messages } = await req.json()
  const maxRetries = 2
  let attempt = 0

  while (attempt <= maxRetries) {
    try {
      const result = streamText({
        model: localModel,
        messages,
        tools: { getWeather, queryDatabase },
        maxTokens: 2048,
        // Stop generation if no tokens produced for 15s
        experimental_telemetry: { isEnabled: true },
      })

      return result.toDataStreamResponse()
    } catch (err) {
      attempt++
      if (attempt > maxRetries) {
        return Response.json(
          { error: 'Model unavailable after retries. Reload Ollama and try again.' },
          { status: 503 }
        )
      }
      // Wait before retry — Ollama may be loading the model
      await new Promise((r) => setTimeout(r, 2000 * attempt))
    }
  }
}

On the frontend, the onError callback in useChat should offer a retry button instead of just displaying the error:

{error && (
  <div className="text-sm text-red-600 bg-red-50 p-3 rounded border border-red-200">
    <p>{error.message}</p>
    <button
      onClick={() => reload()}
      className="mt-2 px-3 py-1 bg-red-600 text-white text-xs rounded"
    >
      Retry
    </button>
  </div>
)}

useChat exposes a reload function that re-sends the last message. This lets the user retry without retyping their prompt — essential for local models that may fail on the first cold start.

Memory Pressure

Each loaded model reserves VRAM or RAM:

• llama3.2 (3B): ~2.5GB
• mistral (7B): ~4.5GB
• llama3.1 (8B): ~5.5GB

If the agent runs alongside your database, dev server, and browser, a 7B+ model may cause swapping. Monitor with ollama ps to check loaded models, and unload unused ones with ollama stop <model>.

---

Next Steps

This agent runs on localhost. Shipping it to production requires:

1. A proper model serving stack — replace Ollama with a hosted inference endpoint (Together AI, Fireworks, or a GPU-backed server). Change only the baseURL.
2. Auth — protect the /api/chat route so random visitors can't drain inference budget.
3. Observability — log tool calls, latency per turn, and context window utilization. The AI SDK's onStepFinish callback exposes this data.
4. Rate limiting — local models handle one concurrent request. Queue requests or reject excess with a 429.

The complete code for this tutorial is available in a single page — one Route Handler, one Client Component, one model config. That's the entire surface area of a local AI agent in 2026.