# Local AI Inference

Canonical HTML: [Read this guide on Rising](https://rising.dev/topics/local-ai-inference/)

Explore an editorial selection of local AI tools on GitHub\. Check model support, memory requirements, deployment options and measurement methods\.

Choose a local inference tool around the model and hardware you actually intend to use\. A runtime, a model\-fit utility and a desktop interface answer different needs\. This guide features selected candidates; their documentation is the source for supported model formats, backends and installation steps\.

5 editorial selections in this guide · 19 tracked in the full category.

Latest observation among these selections: 2026\-10\-27T23\:15\:40\.368Z (UTC).

## What to check before choosing

### Does it support your exact model and hardware?

Check the model architecture and quantization as well as your operating system, processor and accelerator backend\. A model\-family name alone does not confirm that a particular weight file is supported\.

### What resources does the workload require?

Check the documented memory and disk requirements for the intended context length\. Distinguish a model loading successfully from generating useful output at an acceptable speed\.

### What does a performance result measure?

Look for the model version, quantization, hardware, context length and separate prompt\-processing and generation measurements\. Test your own workload before relying on a published speed comparison\.

## Editorial selections

Alphabetical by owner and repository name. This guide includes every editorial selection in this category, with its existing editorial description. The order is not a recommendation or a ranking. Counts below belong to each record's observation date.

The full directory also includes keyword-classified discoveries. [Search, filter and compare all tracked projects in this category](https://rising.dev/projects/#cat=Local%20AI%20Inference).

### [AlexsJones\/llmfit](https://rising.dev/gh/AlexsJones/llmfit/)

TUI that inspects your CPU, RAM, GPU and VRAM and tells you which open models and quantizations will run well\. It can also benchmark real tokens per second\.

Editorial description.

38,424 stars · Rust · Observed 2026\-10\-27T23\:15\:40\.368Z (UTC).

[Read the source on GitHub](https://github.com/AlexsJones/llmfit)

### [antirez\/ds4](https://rising.dev/gh/antirez/ds4/)

"DwarfStar", a small native inference engine from Redis creator antirez for DeepSeek V4 Flash\/PRO, GLM 5\.x and Qwen3\.8 on Metal, CUDA and ROCm\.

Editorial description.

24,116 stars · C · Observed 2026\-10\-27T23\:15\:40\.368Z (UTC).

[Read the source on GitHub](https://github.com/antirez/ds4)

### [jundot\/omlx](https://rising.dev/gh/jundot/omlx/)

Mac menu\-bar LLM server with continuous batching and a hot RAM \/ cold SSD KV cache\.

Editorial description.

23,106 stars · Python · Observed 2026\-10\-27T23\:15\:40\.368Z (UTC).

[Read the source on GitHub](https://github.com/jundot/omlx)

### [JustVugg\/colibri](https://rising.dev/gh/JustVugg/colibri/)

Pure\-C, zero\-dependency engine that runs huge mixture\-of\-experts \(MoE\) models by keeping the active part in RAM and streaming experts from disk\. No GPU needed\.

Editorial description.

44,228 stars · C · Observed 2026\-10\-27T23\:15\:40\.368Z (UTC).

[Read the source on GitHub](https://github.com/JustVugg/colibri)

### [magnitudedev\/magnitude](https://rising.dev/gh/magnitudedev/magnitude/)

Inference engine that auto\-tunes its kernels on your hardware \(Apple Silicon, NVIDIA, AMD, Intel, CPU\)\. The README claims over 2x Ollama speed\.

Editorial description.

7,355 stars · Rust · Observed 2026\-10\-27T23\:15\:40\.368Z (UTC).

[Read the source on GitHub](https://github.com/magnitudedev/magnitude)

## Coverage and limits

Rising does not benchmark these runtimes or verify hardware compatibility\. Local installation also does not by itself establish offline operation; check model downloads, remote endpoints and telemetry settings in the project documentation\.

Category assignments can be wrong. These guides feature editorial selections with reviewed categories and descriptions. The full project directory also includes discovered repositories assigned by keywords; those discoveries are not included in this guide.

Repository observations are dated snapshots, not live measurements. Stars measure attention, not quality. A recent observation does not mean every record in this group was refreshed at the same time.

[Read the measurement and review methodology](https://rising.dev/methodology/), or [report a correction](https://rising.dev/methodology/#corrections).

[Explore all topic guides](https://rising.dev/topics/) · [Browse all tracked projects](https://rising.dev/projects/)