llm

Setup llama.cpp

llama.cpp
├── compose.yaml
└── models
    └── sweep-next-edit-1.5b.q8_0.v2.gguf

The model can be downloaded from: https://huggingface.co/sweepai/sweep-next-edit-1.5B.

• Add nvidia-container-toolkit repo:

  curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
      sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

Install Ollama

1. On the machine which will host the Ollama, create a folder for ollama.

   mkdir ollama
   cd ollama