An OpenInfer and Arm collaboration — October 2026
OpenInfer is collaborating with Arm to simplify the deployment and scaling of AI inference on Arm-based infrastructure. Through this collaboration, we’re demonstrating how OpenInfer Loom enables developers to run AI models on AWS Graviton processors and integrate Arm-based compute into heterogeneous AI environments. This work supports our shared goal of making agentic AI infrastructure more flexible, efficient, and accessible.
AI agents are only as useful as the models behind them. For many teams, the hardest part of running their own models is serving them with good performance and high throughput: deploying infrastructure, maintaining an inference runtime, handling scale, load balancing and session continuity — all before an agent can call the API endpoint.
OpenInfer Loom handles all of those pieces. The OpenInfer software stack can run entirely on AWS Graviton instances, without GPUs. We provide OpenAI- and Anthropic-compatible APIs, so clients and agents can switch to OpenInfer with no integration work.
From cloud instance to agent endpoint
Connecting an instance to the Loom network is easy. At a high level:
- Connect the nodes. Install the OpenInfer Loom service on the instance with an API key. This installs the inference software stack and registers the node in the Loom network.
- Provision the models. In the Loom dashboard, assign models and their configuration to the available nodes.
- Point your agents. The assigned API endpoint is ready to serve traffic. Loom does the heavy lifting of directing traffic to the right nodes and balancing load.
- Need more capacity? Add more nodes, or define auto-scaling criteria in your cloud provider. Loom supports flexible model mappings that account for dynamic scale, so your agents don’t have to change.
Loom serves the OpenAI (Chat Completions and Responses) and Anthropic (Messages) APIs from a single API host. Not every API feature is supported yet; some, such as image input and generation, are coming soon. Text input, text generation and tool calling are supported, which is enough to serve coding agents.
Any off-the-shelf or proprietary agent or application built for those APIs can migrate by switching the base URL and the API key. The model-discovery endpoint (/v1/models) lists your custom models, ready to reference in API requests or in a UI.
Heterogeneity: Arm and GPUs in one fleet
AWS Graviton nodes join the same fleet as GPU machines on AWS, GCP or your own racks, behind the same endpoint. Teams can put Arm capacity to work alongside their GPUs without running a second serving stack.
How it performs
OpenInfer has new CPU kernels for Arm. On one c8g.4xlarge instance running Qwen3.5-9B with Q8_0 weights, they deliver the first token in 4.8 seconds, ahead of llama.cpp at 5.3 seconds, and process prompts 55% faster than OpenInfer’s previous runtime.
| Setup | Time to first token | Prompt processing | Token generation |
|---|---|---|---|
| OpenInfer, before (Q8_0) | 7.5 s | 87 tokens/s | 18.8 tokens/s |
| OpenInfer, new Arm kernels (Q8_0) | 4.8 s | 135 tokens/s | 22.4 tokens/s |
| llama.cpp (Q8_0) | 5.3 s | 121 tokens/s | 24.4 tokens/s |
| OpenInfer, new Arm kernels (Q4_0) | 5.1 s | 128 tokens/s | 29.1 tokens/s |
| llama.cpp (Q4_0) | 4.5 s | 142 tokens/s | 30.1 tokens/s |
llama.cpp remains ahead on token generation and with Q4_0 weights, and we are continuing to tune OpenInfer for Arm.
About the benchmark: one AWS c8g.4xlarge instance (Graviton4, 16 vCPUs, 32 GiB memory) running Ubuntu 24.04. Qwen3.5-9B from unsloth/Qwen3.5-9B-GGUF on Hugging Face; each engine used the same weights in its own preferred file layout. One request at a time, a prompt of about 650 tokens, 128 output tokens, greedy decoding and 16 threads, served over HTTP and measured by the same streaming client. Each result is the median of 3 runs after a warm-up. llama.cpp commit 23b0202a1, built for Armv9-A with SVE2, I8MM, BF16 and dot-product support, with prompt caching off. “OpenInfer, before” is the OpenInfer runtime without the new Arm kernels.
Get onboarded
Loom is invite-only. Request access at platform.openinfer.io. Once your account is approved, follow the quickstart to set up your deployment.
Questions or want more information? Contact OpenInfer at [email protected].