Skip to main content

Getting started

Install the Apple package, configure your native build, and connect a provider to your UI.

Version 1.0.0

Version 1.0.0 is published on npm.

For Android inference, follow the separate Android setup guide. The router and hooks below are shared, but the native provider and SDK setup differ.

1. Check requirements​

The example app uses Expo SDK 57 and React Native 0.86.3. The native module links into apps targeting iOS 16.4 or newer; Apple's on-device model requires iOS 26 or newer at runtime. On-device generation also needs Apple Intelligence-eligible hardware, Apple Intelligence enabled, and downloaded model assets. See Compatibility & limitations.

Use an Expo development client or a native production build. Standard Expo Go does not include this native module.

2. Install from npm​

Run in your application directory:

npm install @taaltreelabs/on-device-llm@1.0.0

This pins the release reviewed here. Commit your lockfile for reproducible installs.

3. Configure the native build​

For an Expo app using generated native projects, merge this plugin entry into your existing app.json configuration:

app.json (excerpt)
{
  "expo": {
    "plugins": [
      "@taaltreelabs/on-device-llm"
    ]
  }
}

No deployment-target override is needed for the default Expo template. The package's bundled plugin configures the scene lifecycle needed by the documented Expo 57 / iOS 27 setup. It patches a recognized Expo AppDelegate template and configures the scene manifest. If prebuild warns about a customized AppDelegate, follow launch-crash troubleshooting.

Apply the configuration and rebuild:

npx expo prebuild --platform ios
npx expo run:ios --device

If you maintain native projects manually, set both the Podfile platform and the app target's IPHONEOS_DEPLOYMENT_TARGET to at least 16.4, then reinstall pods. Check Podfile.lock for OnDeviceLlm: a successful build alone does not prove the module was linked. See native-module troubleshooting.

4. Create a router and connect useChat​

This example tries Apple's model first and uses your configured cloud endpoint when routing and fallback rules allow it. Replace the endpoint and model with your own. Set contextWindow to the actual limit for that model.

Assistant.tsx
import { createAppleProvider } from '@taaltreelabs/on-device-llm/apple';
import { createRouter } from '@taaltreelabs/on-device-llm/core';
import { createOpenAIProvider } from '@taaltreelabs/on-device-llm/openai';
import { useChat } from '@taaltreelabs/on-device-llm/react';
import { fetch as expoFetch } from 'expo/fetch';
import { Button, Text, View } from 'react-native';

const llm = createRouter({
  providers: [
    createAppleProvider(),
    createOpenAIProvider({
      baseUrl: 'https://your-endpoint.example.com/v1',
      model: 'your-model',
      // Configure authentication for your own backend as needed.
      fetch: expoFetch as unknown as typeof fetch,
      contextWindow: 128_000,
    }),
  ],
});

export function Assistant() {
  const { messages, streamingText, status, error, send, stop } = useChat({
    provider: llm,
    systemPrompt: 'You are a concise assistant.',
  });

  return (
    <View>
      {messages.map((message, index) => (
        <Text key={index}>{message.content}</Text>
      ))}
      {streamingText !== undefined && <Text>{streamingText}</Text>}
      {error && <Text>Unable to complete the request. Please try again.</Text>}
      <Button
        title="Ask"
        disabled={status !== 'idle'}
        onPress={() => {
          // useChat exposes failures through error; handle rejected sends too.
          void send('What should I cook tonight?').catch(() => {});
        }}
      />
      <Button title="Stop" disabled={status === 'idle'} onPress={stop} />
    </View>
  );
}

Use your own authenticated backend or a suitable short-lived credential flow for cloud access. Do not put a long-lived provider key in EXPO_PUBLIC_* variables or the mobile bundle.

For an on-device-only integration, pass createAppleProvider() directly as the hook's provider. Handle unavailable devices in your UI; no cloud fallback is configured in that case.

5. Understand what happens on each turn​

useChat fits the conversation to a context budget before streaming, exposes in-flight text as streamingText, and adds completed replies to messages. The router chooses among your configured providers. Availability is a prerequisite, not a guarantee that generation will succeed.

For cloud streaming in React Native, inject expo/fetch as shown above. Without a readable response body, the HTTP provider falls back to a single aggregated text delta. Apple's provider streams through the native bridge and needs no fetch injection.

The router never falls back after its first event has reached the consumer. Cancellation and invalid requests are not retried against another provider. See routing and streaming for the full behavior.

Next steps​