Running On-Device LLMs (Gemma/Phi-3) in Flutter Apps via MediaPipe LLM Inference API

Flutter On-Device LLM MediaPipe Gemma Phi-3 Integration

On-Device AI is revolutionizing mobile app development. While cloud-based LLM APIs (like GPT-4o or Claude) offer high capability, they come with steep API costs, network latency, and privacy compliance risks.

Google’s MediaPipe LLM Inference API overcomes these drawbacks by executing lightweight open models—such as Gemma-2B, Phi-3-Mini, and Falcon-1B—directly on smartphone GPUs and NPUs.

This guide explores integrating MediaPipe LLM Inference into Flutter, utilizing int4 model quantization, streaming token responses, and preventing mobile Out-Of-Memory (OOM) crashes.

Key Takeaways

Benefits of On-Device LLM: Works 100% offline, zero server cost ($0), ultra-low latency, and total privacy assurance.

MediaPipe LLM Inference API: Cross-platform runtime engine for Android & iOS delivering 15–25 tokens/second using mobile GPU acceleration.

Model Quantization: Using 4-bit (int4) .task models reduces memory footprints below 1.5GB RAM for Gemma-2B.

Flutter Native Binding: Streams token responses via MethodChannel/EventChannel straight into Flutter StreamBuilder widgets.

1. Comparing Cloud LLM vs On-Device LLM

Feature	Cloud LLM (OpenAI API)	On-Device LLM (MediaPipe + Gemma)
Server Cost	Scales with tokens ($$$)	$0 (Uses device hardware)
Offline Support	Impossible	100% Offline Capable
Privacy Guarantee	Data leaves device	Zero Data Transfer
First Token Latency	1.5s ~ 4s	< 200ms (Local GPU)

2. Flutter MethodChannel Integration Example

// lib/services/ondevice_llm_service.dart
import 'dart:async';
import 'package:flutter/services.dart';

class OnDeviceLlmService {
  static const MethodChannel _channel = MethodChannel('dev.effidev.mediapipe_llm');
  static const EventChannel _streamChannel = EventChannel('dev.effidev.mediapipe_llm/stream');

  Future<void> initializeModel(String modelPath) async {
    await _channel.invokeMethod('initModel', {'modelPath': modelPath});
  }

  Stream<String> generateResponseStream(String prompt) {
    _channel.invokeMethod('generateResponse', {'prompt': prompt});
    return _streamChannel.receiveBroadcastStream().map((event) => event.toString());
  }
}