A guide to the category

Local AI Inference

Explore an editorial selection of local AI tools on GitHub. Check model support, memory requirements, deployment options and measurement methods.

Choose a local inference tool around the model and hardware you actually intend to use. A runtime, a model-fit utility and a desktop interface answer different needs. This guide features selected candidates; their documentation is the source for supported model formats, backends and installation steps.

5 editorial selections in this guide · 19 tracked in the full category · Latest observation among these selections:

What to check before choosing

Does it support your exact model and hardware?

Check the model architecture and quantization as well as your operating system, processor and accelerator backend. A model-family name alone does not confirm that a particular weight file is supported.

What resources does the workload require?

Check the documented memory and disk requirements for the intended context length. Distinguish a model loading successfully from generating useful output at an acceptable speed.

What does a performance result measure?

Look for the model version, quantization, hardware, context length and separate prompt-processing and generation measurements. Test your own workload before relying on a published speed comparison.

Editorial selections

Alphabetical by owner and repository name. This guide includes every editorial selection in this category, with its existing editorial description. The order is not a recommendation or a ranking. Counts below belong to each record's observation date.

The full directory also includes keyword-classified discoveries. Search, filter and compare all tracked projects in this category.