Insanely Easy Local First AI ChatBot With Astro, Ollama and Vercel’s AI Library

Vercel has made a fantastic AI interface library that makes it super easy to create streaming AI interfaces that connect with any LLM. But most of the time we hear about it, it’s using NextJS and OpenAI. Well, to heck with that. Let’s try it with Astro (a lightweight web framework that promotes an Islands Architecture) and Ollama, which allow you to run AI models locally for no cost. Sound good? Let’s give it a go!

“Ride Astro To Planet AI” Thumbnail Image

Getting Ollama Setup

We’ll start with setting up our local AI. To do that first download and install Ollama. On my Mac I used homebrew for that with the command brew install ollama.

Once Ollama is install you can install any LLM you want from their list of models. For this example I used phi3, an LLM from Microsoft. I chose the mini model because it was faster than the larger model. To install it run the command ollama pull phi3. Or if you want to install and then run it just to play around with it interactively use the command ollama run phi3.

Another good model is gemma2 from Google.

Setting Up Astro

Now that we have the AI model installed and running it’s time to build our Astro application. We start by creating the application using pnpm create astro@latest. You can name the application whatever you want. I chose astro-ai. I chose the Empty template because we want to build our own API and UI and any other template would just mean removing a lot of stuff.

With the application set up it’s time to install our AI interface libraries.

pnpm add ai ollama-ai-provider

The ai library is from Vercel and it will be handling the work of creating the stream to send to the client. And the ollama-ai-provider creates the connection between the ai library and the Ollama service.

Creating The API Endpoint

Now that we have our libraries installed we can build out our API endpoint. We start by setting the application into server mode by setting output as server in the astro.config.mjs file.

export default defineConfig({
  output: "server",
});

Normally Astro runs in a static mode. This switch means that now it will run as a server and process every request individually.

Next up we need to create a route for /api/chat. The URL there is important because the useChat hook looks for the endpoint at that location. You can change that, but we’ll just keep it at the default and create a file in src/pages/api/chat.ts.

Into that file we will add our endpoint implementation:

import { createOllama } from "ollama-ai-provider";
import { StreamingTextResponse, streamText } from "ai";
import type { APIRoute } from "astro";

const ollama = createOllama();

export const POST: APIRoute = async ({ request }) => {
  const { messages } = await request.json();

  const result = await streamText({
    model: ollama("phi3"),
    messages,
  });

  return new StreamingTextResponse(result.toAIStream());
};

This code starts off by instantiating an ollama variable using createOllama. We’ll send that as the model to streamText to start our streaming session. When we invoke ollama we also send along the model name, in this case phi3, if you want to use a different model then change that to whatever AI model you want to use.

We also bring in the messages from the JSON content of the POST request and send that to streamText. Those messages contain the previous conversation between the user which would be the human asking the questions, and the assistant which would be the AI.

We finish up by sending back the streaming text response from the AI. That will send back the AI result as a stream of messages that the UI portion of the AI library will process for us. It’s wicked easy. You’ll see.

Building The Chat UI

We’re going to use React and Tailwind for this. To add those to our project we run npx astro add react and npx astro add tailwind . It’s as easy as that.

Since the responses from the AI are in markdown format we need a library to process that. We’ll use pnpm add react-markdown to add react-markdown to the application to handle that.

The UI is a React component that we add into our applicaton as src/pages/Chat.tsx with the contents:

import { useChat } from "ai/react";
import Markdown from "react-markdown";

export default function Chat() {
  const { messages, input, handleInputChange, handleSubmit } = useChat();
  return (
    <div className="flex flex-col">
      {messages.map((m) => (
        <div key={m.id} className="whitespace-pre-wrap text-3xl">
          {m.role === "user" ? "User: " : "AI: "}
          <Markdown>{m.content}</Markdown>
        </div>
      ))}
      <form onSubmit={handleSubmit}>
        <input
          className="w-full p-2 mb-8 border border-gray-300 rounded shadow-xl text-black"
          value={input}
          placeholder="What's your question?"
          onChange={handleInputChange}
        />
      </form>
    </div>
  );
}

At the top we bring in the useChat hook which does the heavy lifting of connecting to the API endpoint. We then invoke that in our application and it gives us back:

• messages — This is the list of all of the messages in this chat.
• input — A text string that tracks the current value of the input field.
• handleInputChange — This function handles onChange events to the input field.
• handleSubmit — This is what we call when we want to send the message to the API.

In the JSX we format all the messages at the top, and then set up a form with an input field down below. When the user presses return or enter on the input the handleSubmit is invoked. That sends the new input to the server, which in turn runs the AI.

Integrating It Into The Page

The last step is to integrate it into our Astro home page. To do that we edit src/pages/index.astro to add an import for our Chat component and the we add it to the page using the client:only directive.

---
import Chat from "./Chat"
---

<html lang="en">
 <head>
  <meta charset="utf-8" />
  <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
  <meta name="viewport" content="width=device-width" />
  <meta name="generator" content={Astro.generator} />
  <title>Astro</title>
 </head>
 <body class="bg-black text-white mx-auto mt-5 max-w-3xl">
  <Chat client:only="react" />
 </body>
</html>

And that’s it. Now all that is left to do is to try it out.

Demo Time

Once we fire it up with pnpm dev we can take a look at it go! Let’s try it out.

Astro AI using Ollama in action!

As you can see, it’s pretty rudimentary, but it’s a good start at making an interactive chat bot. Better yet it’s free, and remarkably easy to build!

What’s Next

You can check out the full source code on github. As you can see, it’s really easy and fun to get started with trying out AI in your application. From here you could try out different AIs. Or you could even try out different view frameworks like Svelte, because the AI library from Vercel is compatible with Svelte as well as Vue. Experiment, have fun and enjoy!