Flutter ExecuTorch On-Device LLM: Llama 3.2 Quantization

When integrating Large Language Models (LLMs) into mobile applications, the biggest hurdles developers face are accumulating API costs per call, service disruption when offline, and privacy concerns. Cloud LLM APIs generate thousands of dollars in fixed monthly costs as your user base grows, while network dead zones like subways or tunnels completely disable app features.

To fundamentally solve this problem, integrating ExecuTorch (PyTorch’s official edge AI engine) and Meta’s Llama 3.2 lightweight on-device models directly into the Flutter ecosystem has emerged as a standard for modern app development in 2026. Built on an ultra-lightweight C++ runtime of just ~50KB, ExecuTorch executes 4-bit group-quantized (INT4) .pte models using Android’s XNNPACK/NPU and iOS’s CoreML hardware accelerators.

This guide walks you through the entire practical workflow: converting Llama 3.2 via 4-bit quantization, establishing Dart FFI bindings using executorch_flutter, implementing smooth 120fps UI streaming on a background Isolate, and benchmarking a 100% cost reduction ($0) compared to cloud APIs.

Key Takeaways

• Official PyTorch Edge Runtime (ExecuTorch): A lightweight C++ runtime under 50KB ported from PyTorch core to mobile environments, seamlessly leveraging mobile NPU and CPU accelerators.
• Llama 3.2 4-bit (INT4) Quantization: Quantizes 1B/3B models down to 1.2GB via 4-bit block quantization, minimizing RAM overhead on mobile devices.
• Dart FFI & Background Isolate: Offloads token generation calculations to an isolated Dart Isolate, preventing UI frame drops (60fps/120fps) during inference.
• $0 API Costs & Offline Functionality: Runs 100% locally without cloud API calls, ensuring zero per-token server costs and complete offline availability.

1. Cloud LLM API vs ExecuTorch On-Device Comparison

Referencing the official PyTorch ExecuTorch documentation and 2026 mobile AI benchmark standards.

[Cloud API Approach] 💸
Mobile App ──(Internet Network)──► OpenAI / Anthropic API ($0.0015/1k tokens + 400ms latency)

[ExecuTorch On-Device Approach] ⭕️
Mobile App ──(Dart FFI)──► ExecuTorch C++ Runtime (.pte Model) ──► NPU/CPU ($0 + Offline)

2. Llama 3.2 Model 4-Bit Quantization and .pte Export (export_llama.py)

Using the export_llm tooling from the PyTorch ExecuTorch repository, convert the Llama 3.2 1B Instruct model into an INT4 4-bit quantized .pte binary file.

# export_llama.py - PyTorch ExecuTorch model quantization script
import torch
from executorch.backends.xnnpack.quantization.subgraph import XNNPACKQuantizer
from executorch.extension.llm.export.builder import LlamaBuilder

def export_quantized_llama():
    print("[1/3] Loading Llama 3.2 1B Instruct model...")
    # Load parameters from Hugging Face or Meta official repo
    builder = LlamaBuilder(
        model_name="meta-llama/Llama-3.2-1B-Instruct",
        checkpoint_path="./checkpoints/llama-3.2-1b/",
        params_path="./checkpoints/llama-3.2-1b/params.json"
    )

    print("[2/3] Applying 4-bit INT4 Group Quantization...")
    # Apply 4-bit group quantization for mobile NPU/CPU execution (group size: 128)
    builder.set_quantization(
        quant_type="int4",
        group_size=128,
        quantizer=XNNPACKQuantizer()
    )

    print("[3/3] Exporting ExecuTorch .pte execution pipeline...")
    exec_program = builder.build()
    
    # Save the .pte binary file to be bundled in the mobile app's assets
    output_path = "./exported/llama3_2_1b_int4.pte"
    with open(output_path, "wb") as f:
        exec_program.write_to_file(f)
        
    print(f"✅ Quantized export complete: {output_path}")

if __name__ == "__main__":
    export_quantized_llama()

# Terminal execution command
python export_llama.py
# → exported/llama3_2_1b_int4.pte (Generates ~1.15GB file)

3. Flutter Integration and Dart FFI Setup (pubspec.yaml, llama_service.dart)

Dependencies and Model Asset Setup (pubspec.yaml)

# pubspec.yaml
name: flutter_ondevice_ai
description: ExecuTorch Llama 3.2 On-Device LLM App

environment:
  sdk: '>=3.5.0 <4.0.0'
  flutter: '>=3.24.0'

dependencies:
  flutter:
    sdk: flutter
  # Official PyTorch ExecuTorch Flutter plugin
  executorch_flutter: ^0.4.0
  flutter_riverpod: ^2.5.1

flutter:
  assets:
    # 4-bit quantized model and tokenizer binaries
    - assets/models/llama3_2_1b_int4.pte
    - assets/models/tokenizer.bin

Isolate-Based Asynchronous On-Device LLM Service (lib/services/llama_service.dart)

Apply the background isolation pattern covered in our Flutter Isolate Heavy Computation Guide to prevent UI frame drops.

// lib/services/llama_service.dart
import 'dart:async';
import 'dart:isolate';
import 'package:executorch_flutter/executorch_flutter.dart';
import 'package:flutter/foundation.dart';

/// DTO for messages passed between UI thread and isolate
class LlamaRequest {
  final String prompt;
  final SendPort sendPort;
  LlamaRequest(this.prompt, this.sendPort);
}

class OnDeviceLlamaService {
  Isolate? _executorchIsolate;
  SendPort? _isolateSendPort;

  /// 1. Spawn background isolate and initialize ExecuTorch runtime
  Future<void> initialize() async {
    final receivePort = ReceivePort();
    _executorchIsolate = await Isolate.spawn(
      _isolateEntryPoint,
      receivePort.sendPort,
    );

    // Receive SendPort created from the isolate
    _isolateSendPort = await receivePort.first as SendPort;
    debugPrint('✅ ExecuTorch background isolate initialized');
  }

  /// 2. Isolate entry point (Runs in an isolated heap memory space)
  static void _isolateEntryPoint(SendPort mainSendPort) async {
    final isolateReceivePort = ReceivePort();
    mainSendPort.send(isolateReceivePort.sendPort);

    // Load ExecuTorch model (C++ FFI binding)
    final runner = await ExecuTorchLlamaRunner.load(
      modelPath: 'assets/models/llama3_2_1b_int4.pte',
      tokenizerPath: 'assets/models/tokenizer.bin',
      temperature: 0.7,
      maxTokens: 512,
    );

    await for (final msg in isolateReceivePort) {
      if (msg is LlamaRequest) {
        // Run inference in background and stream tokens back
        await runner.generateStream(
          prompt: msg.prompt,
          onToken: (token) {
            msg.sendPort.send(token); // Send individual token to UI thread
          },
        );
        msg.sendPort.send(null); // Stream completion signal
      }
    }
  }

  /// 3. Provide token stream to UI components
  Stream<String> generateResponse(String prompt) {
    if (_isolateSendPort == null) {
      throw StateError('ExecuTorch service is not initialized.');
    }

    final responsePort = ReceivePort();
    _isolateSendPort!.send(LlamaRequest(prompt, responsePort.sendPort));

    final controller = StreamController<String>();

    responsePort.listen((message) {
      if (message == null) {
        controller.close();
        responsePort.close();
      } else if (message is String) {
        controller.add(message);
      }
    });

    return controller.stream;
  }
}

UI Component Implementation (lib/views/chat_screen.dart)

// lib/views/chat_screen.dart
import 'package:flutter/material.dart';
import '../services/llama_service.dart';

class OnDeviceChatScreen extends StatefulWidget {
  const OnDeviceChatScreen({super.key});

  @override
  State<OnDeviceChatScreen> createState() => _OnDeviceChatScreenState();
}

class _OnDeviceChatScreenState extends State<OnDeviceChatScreen> {
  final OnDeviceLlamaService _llamaService = OnDeviceLlamaService();
  final TextEditingController _controller = TextEditingController();
  final List<String> _messages = [];
  String _currentStreamResponse = '';
  bool _isGenerating = false;

  @override
  void initState() {
    super.initState();
    _llamaService.initialize();
  }

  void _sendMessage() {
    final text = _controller.text.trim();
    if (text.isEmpty || _isGenerating) return;

    setState(() {
      _messages.add('User: $text');
      _controller.clear();
      _isGenerating = true;
      _currentStreamResponse = '';
    });

    // Receive streaming tokens from background Isolate
    _llamaService.generateResponse(text).listen(
      (token) {
        setState(() {
          _currentStreamResponse += token;
        });
      },
      onDone: () {
        setState(() {
          _messages.add('Llama 3.2: $_currentStreamResponse');
          _currentStreamResponse = '';
          _isGenerating = false;
        });
      },
    );
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('ExecuTorch Llama 3.2 (On-Device $0)'),
        backgroundColor: Colors.indigo,
      ),
      body: Column(
        children: [
          Expanded(
            child: ListView.builder(
              padding: const EdgeInsets.all(16),
              itemCount: _messages.length + (_isGenerating ? 1 : 0),
              itemBuilder: (context, index) {
                if (index < _messages.length) {
                  return Padding(
                    padding: const EdgeInsets.symmetric(vertical: 4),
                    child: Text(_messages[index], style: const TextStyle(fontSize: 16)),
                  );
                } else {
                  return Text('Llama 3.2: $_currentStreamResponse ▌',
                      style: const TextStyle(fontSize: 16, color: Colors.indigo));
                }
              },
            ),
          ),
          Container(
            padding: const EdgeInsets.all(8),
            color: Colors.grey[200],
            child: Row(
              children: [
                Expanded(
                  child: TextField(
                    controller: _controller,
                    decoration: const InputDecoration(hintText: 'Enter your question...'),
                  ),
                ),
                IconButton(
                  icon: const Icon(Icons.send, color: Colors.indigo),
                  onPressed: _sendMessage,
                ),
              ],
            ),
          ),
        ],
      ),
    );
  }
}

To comply with Android 15’s 16KB memory page compatibility standards covered in our Flutter 16KB Page Size Support Guide, the C++ Native shared library (libexecutorch.so) must also be built with 16KB alignment.

4. Production Benchmarks and Resource Comparison

Benchmark results based on 1,000 consecutive inference runs with the Llama 3.2 1B INT4 model on a Snapdragon 8 Gen 3 device (Galaxy S24 Ultra).

5. Enterprise Best Practices & Checklist

Frequently Asked Questions

Does Llama 3.2 3B run smoothly on mobile devices?

On flagship devices with 8GB RAM or more (iPhone 15 Pro, Galaxy S24), the Llama 3.2 3B 4-bit model (~2.1GB) runs smoothly at 15–18 tokens per second. However, to accommodate mid-range and budget devices, we recommend using the 1B model.

Does the C++ ExecuTorch library add too much weight to the app binary size?

The ExecuTorch C++ core runtime itself is extremely compact, ranging from ~50KB to 150KB. However, rather than bundling the .pte model file (1.2GB) directly into the app APK/IPA, we recommend downloading it from a CDN or Cloudflare R2 on first app launch and storing it in local storage.

Is the same .pte file used for both iOS and Android?

A .pte file built with the default XNNPACK CPU backend is 100% cross-platform and shared identically across iOS and Android. However, custom-tuned models utilizing Apple CoreML or Qualcomm Hexagon NPU hardware accelerators must be exported separately for their respective target platform backends.

How is the multilingual performance?

While base Llama 3.2 1B/3B models focus primarily on English, performing LoRA fine-tuning on domain-specific or multilingual datasets before exporting to ExecuTorch 4-bit INT4 yields excellent summarization and conversational performance in on-device environments.