AnythingLLM and Ollama

Singularity Container for AnythingLLM and Ollama

We also provide an image with the AnythingLLM frontend.

Below is the provided Singularity image definition file. You can use it to build the image yourself or use the one provided by Vega.

Note that you require permissions to use --fakeroot to build images Singularity on Vega, which is disabled by default. To enable this for your account contact support@sling.si.

Bootstrap: docker
From: mintplexlabs/anythingllm:latest

%post
    curl -L https://ollama.com/download/ollama-linux-amd64.tgz -o /tmp/ollama.tgz
    tar -xzf /tmp/ollama.tgz -C /usr
    rm /tmp/ollama.tgz

    cp /usr/local/bin/docker-entrypoint.sh /app/run-allm.sh
    cp /app/server/prisma/schema.prisma /app/server/node_modules/.prisma/client/schema.prisma

    mkdir /models /data

    rm -rf /app/server/storage
    rm /app/server/.env
    ln -s /data /app/server/storage
    ln -s /data/.env /app/server/.env

    chmod 777 /models /data
    chmod 777 -R /app

%environment
    # OLLAMA
    export OLLAMA_MODELS="/models"
    export OLLAMA_HOST="127.0.0.1:11433"

    # ANYTHING LLM
    export STORAGE_DIR="/data"

%test
    echo "Checking for Ollama installation..."
    test -f /usr/bin/ollama || exit 1

%runscript
    echo "Starting Anything LLM / Ollama stack"
    echo "Starting Ollama server ..."
    ollama serve > /data/ollama.log 2>&1 &

    echo "Starting AnythingLLM server ..."
    bash /app/run-allm.sh

%help
    This container runs a bundled Ollama and Anything LLM stack.
    Replace the ./models and ./data locations with whatever path you want to use as model and data persistent storage locations
    Usage:
        singularity run --nv --writable-tmpfs -B ./models:/models -B ./data:/data -B ./data/prisma:/app/server/node_modules/.prisma/client allm.sif 

We provide a prebuilt Singularity container containing both the AnythingLLM and Ollama services. To run the container, we must first create folders to store model data for Ollama and create a persistent storage location for AnythingLLM data.

You can use the commands below to create the folders. If you already downloaded some models using Ollama, you can skip creating a new model folder and just use the existing one.

mkdir -p allm/models
mkdir -p allm/data
mkdir -p allm/data/prisma
cd allm

Before you run the service you must first secure a compute node and connect to it.

salloc -n 1 --gres=gpu:1 --partition=gpu -t 01:00:00
ssh {allocated_node}

To run the container in the terminal you can use this command, replacing the bind locations to whatever path you decided to use.

singularity run --nv --writable-tmpfs -B ./models:/models -B ./data:/data -B ./data/prisma:/app/server/node_modules/.prisma/client allm.sif 

You can also modify the port AnythingLLM uses by adding the following flag to the run command.

--env SERVER_PORT="<port>"

The logs of Ollama will also be available in the /data folder you created

To access AnythingLLM on Windows in your local browser you will have to create a tunnel to your node. If using PuTTY on Windows you can set this on the right side under Connection -> SSH -> Tunnels.

You set whatever source port you would like to use on your local machine (for example 8080) and set the destination to \:3001. Set the node to the hostname of whatever node was allocated.

On Mac and Linux you can simply run this command:

ssh -N -f -L <port>:gnXX:3001 <your-username>@login.vega.izum.si