on-device-llm
Version 1.0.0 · Apple on-device inference for React Native and Expo, with a shared JavaScript toolkit for context management, routing, and cloud endpoints.
The library includes providers, context management, routing, and React hooks. Version 1.0.0 is available on npm. This documentation focuses on Apple inference and the shared toolkit. For Android, use the separate on-device-llm-android package.
What it does
- Providers: access Apple's on-device model or a Chat Completions-compatible endpoint through a common interface.
- Context management: fit conversation history to the model's window using token budgets, sliding windows, or rolling summaries. Apple's model reports a 4K or 8K window depending on the variant; the package handles unknown windows explicitly.
- Routing: choose a provider based on availability, capabilities, context, and task-specific rules, with configurable fallback.
- React hooks: connect availability, chat, and one-shot generation to your interface.
Start with Getting started, then check the requirements and provider support.
Apple and Android are separate packages
This package supplies the Apple native provider. The shared core, openai, and react entry points also serve the Android companion package. Installing this package does not opt your app into Google's ML Kit SDK. The Android package has its own SDK setup and metrics disclosure.
Choose an import
| Import | Purpose |
|---|---|
@taaltreelabs/on-device-llm | All exports for React Native apps; native resolution is lazy |
@taaltreelabs/on-device-llm/core | Provider types, context management, routing, schema normalization, and MockProvider |
@taaltreelabs/on-device-llm/apple | Apple's native model provider |
@taaltreelabs/on-device-llm/openai | OpenAI-compatible HTTP provider |
@taaltreelabs/on-device-llm/react | useAvailability, useChat, and useGenerate |
core and openai have no runtime dependencies and are independent of React Native. They can also be used in Node or a modern browser. The React hooks require React. The Apple provider requires a compatible native build and reports unsupportedPlatform when the capability is unavailable.
Integration guides
| Goal | Guide |
|---|---|
| Keep app state current and conversations within budget | Managing the context window |
| Choose a provider and control fallback | Writing a routing policy |
| Stream responses and handle cancellation | Streaming on React Native |
| Generate schema-shaped objects | Structured output |
| Let the on-device model call app functions | Tool calling |
| Add a backend or build a test double | Writing a custom provider |
Understand the data flow
An on-device request runs through the native provider. A configured cloud provider receives request content when your policy selects it. Cloud summarizers and your tool handlers may also send data off-device. The package does not send prompt or response content to a telemetry service; onRoute reports routing metadata rather than messages.
Choose providers and policies that match your app's requirements, and explain their behavior to users.
Follow development
These docs reflect the package source at 623199f, and the npm 1.0.0 tarball, reviewed September 27, 2026. See Troubleshooting for integration issues and GitHub issues for reproducible problems or feature discussions.