Mar 31, 2026 Embedded Development

TensorFlow Lite Deployment in Practice: ESP32-S3 TinyML Edge AI Inference Complete Guide 2026

Why do we need TensorFlow Lite?

Hello everyone, I’m MakerOnsite. Today we’re going to talk about how to run AI models on embedded devices.

You may have already used TensorFlow to train models, but have you ever thought: how do you put a trained model on a Raspberry Pi, ESP32, or microcontroller to run? This is the problem TensorFlow Lite solves.

Traditional TensorFlow models are too large and too slow to run on embedded devices at all. And TFLite is designed specifically for edge devices:

Small size: Models compressed 4-10 times
Fast speed: Optimized for ARM, DSP
Low power consumption: Suitable for battery-powered devices
Offline operation: No internet required, protects privacy

Today we’ll go through the complete TFLite deployment process from scratch.

What do you need to prepare?

Item	Model/Specification	Price
Development Board	Raspberry Pi 4B / Jetson Nano	¥350-800
Or	ESP32-S3 (with AI acceleration)	¥45
Camera	USB Camera / OV2640	¥30-80
Computer	For model training (any)	-
Total		¥425-925

If you just want to try it first, using a computer + CPU can also run through the entire process without needing extra hardware.

Step 1: Train a simple image classification model

We’ll first train a simple model that can recognize “cats” and “dogs”. Here we use TensorFlow 2.x:

import tensorflow as tf
from tensorflow import keras
import numpy as np

# Load preprocessed dataset (using sample data here)
(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()

# Simplification: only take cat and dog classes (actual projects need to prepare your own data)
# Here we use CIFAR-10's cat (5) and dog (3) classes
cat_idx = y_train.flatten() == 5
dog_idx = y_train.flatten() == 3

x_train_cats_dogs = np.concatenate([x_train[cat_idx], x_train[dog_idx]])
y_train_cats_dogs = np.concatenate([y_train[cat_idx], y_train[dog_idx]])

# Normalize
x_train_cats_dogs = x_train_cats_dogs / 255.0

# Build simple CNN model
model = keras.Sequential([
    keras.layers.Conv2D(32, 3, activation='relu', input_shape=(32, 32, 3)),
    keras.layers.MaxPooling2D(),
    keras.layers.Conv2D(64, 3, activation='relu'),
    keras.layers.MaxPooling2D(),
    keras.layers.Flatten(),
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(2, activation='softmax')  # Cat/dog two classes
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train
model.fit(x_train_cats_dogs, y_train_cats_dogs, epochs=10, batch_size=32)

# Save complete model
model.save('cat_dog_model.h5')
print("✅ Model training complete!")

Notes: ⚠️ In actual projects, you need to prepare your own dataset. You can use ImageNet subsets, or take and annotate photos yourself. Training data needs at least 500+ images per class to ensure good results.

Step 2: Convert to TensorFlow Lite format

Trained models cannot be directly used on embedded devices; they need to be converted:

import tensorflow as tf

# Load trained model
model = tf.keras.models.load_model('cat_dog_model.h5')

# Method 1: Basic conversion (no optimization)
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

with open('cat_dog_model.tflite', 'wb') as f:
    f.write(tflite_model)

print(f"✅ Basic conversion complete! Model size: {len(tflite_model)/1024:.2f} KB")

# Method 2: Dynamic range quantization (recommended! Size reduced 4x)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_quant_model = converter.convert()

with open('cat_dog_model_quant.tflite', 'wb') as f:
    f.write(tflite_quant_model)

print(f"✅ Quantization conversion complete! Model size: {len(tflite_quant_model)/1024:.2f} KB")

Conversion results comparison:

Original Keras model: ~50 MB
TFLite basic version: ~12 MB
TFLite quantized version: ~3 MB ⭐

Principle explanation: Quantization converts 32-bit floating-point weights to 8-bit integers, greatly reducing size. Accuracy loss is usually less than 1%, but inference speed improves 2-4 times!

Step 3: Run inference on embedded devices

3.1 Raspberry Pi / Linux devices

import tensorflow.lite as tflite
import numpy as np
from PIL import Image

# Load TFLite model
interpreter = tflite.Interpreter(model_path='cat_dog_model_quant.tflite')
interpreter.allocate_tensors()

# Get input/output information
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Preprocess image
def preprocess_image(image_path):
    img = Image.open(image_path).resize((32, 32))
    img_array = np.array(img, dtype=np.float32) / 255.0
    img_array = np.expand_dims(img_array, axis=0)
    return img_array

# Inference
input_data = preprocess_image('test_image.jpg')
interpreter.set_tensor(input_details[0]['index'], input_data)

interpreter.invoke()  # Execute inference

# Get results
output_data = interpreter.get_tensor(output_details[0]['index'])
prediction = np.argmax(output_data[0])

labels = ['cat', 'dog']
print(f"🎯 Recognition result: {labels[prediction]} (confidence: {output_data[0][prediction]*100:.1f}%)")

3.2 ESP32-S3 (using TensorFlow Lite Micro)

ESP32 has limited resources and needs to use the TFLite Micro version. Here’s the Arduino code:

#include <TensorFlowLite.h>
#include "model.h"  // Model converted to C array
#include "labels.h"

tflite::MicroErrorReporter micro_error_reporter;
tflite::ErrorReporter* error_reporter = &micro_error_reporter;

const tflite::Model* model = ::tflite::GetModel(model_data);
tflite::MicroInterpreter interpreter(model, micro_error_reporter);

// Allocate tensor memory
constexpr int tensor_arena_size = 30 * 1024;
uint8_t tensor_arena[tensor_arena_size];
tflite::MicroTensorAllocator allocator(tensor_arena, tensor_arena_size);
tflite::MicroOpResolver op_resolver;

void setup() {
  Serial.begin(115200);

  // Initialize interpreter
  TfLiteStatus status = interpreter.AllocateTensors(&allocator);
  if (status != kTfLiteOk) {
    Serial.println("❌ Memory allocation failed");
    return;
  }

  Serial.println("✅ TFLite initialization complete");
}

void loop() {
  // Get input tensor
  TfLiteTensor* input = interpreter.input(0);

  // Here you need to read data from the camera and fill input->data
  // Simplified example: fill with random data
  for (int i = 0; i < input->bytes; i++) {
    input->data.uint8[i] = random(0, 255);
  }

  // Execute inference
  TfLiteStatus invoke_status = interpreter.Invoke();
  if (invoke_status != kTfLiteOk) {
    Serial.println("❌ Inference failed");
    return;
  }

  // Get output
  TfLiteTensor* output = interpreter.output(0);
  int predicted_class = 0;
  float max_score = 0;

  for (int i = 0; i < output->dims->data[0]; i++) {
    float score = output->data.f[i];
    if (score > max_score) {
      max_score = score;
      predicted_class = i;
    }
  }

  Serial.print("🎯 Recognition result: ");
  Serial.print(labels[predicted_class]);
  Serial.print(" (confidence: ");
  Serial.print(max_score * 100);
  Serial.println("%)");

  delay(1000);
}

Notes: ⚠️ ESP32-S3’s memory is limited (~512KB SRAM), the model must be controlled within 200KB. It’s recommended to use smaller model architectures (such as MobileNetV1 0.25 width).

Step 4: Performance optimization techniques

4.1 Using delegates for acceleration

Raspberry Pi can use GPU or NPU acceleration:

# Using GPU delegate (requires TensorFlow Lite GPU)
from tensorflow.lite.experimental import load_delegate

interpreter = tflite.Interpreter(
    model_path='cat_dog_model_quant.tflite',
    experimental_delegates=[
        load_delegate('libtensorflowlite_gpu_delegate.so')
    ]
)

# Jetson Nano can use TensorRT delegate
# ESP32 can use ESP-DSP library to accelerate convolution operations

4.2 Model pruning

# Add pruning during training
import tensorflow_model_optimization as tfmot

prune_params = tfmot.sparsity.keras.PruningParams(
    pruning_schedule=tfmot.sparsity.keras.ConstantSparsity(
        0.5,  # Prune 50% of weights
        begin_step=2000,
        frequency=100
    )
)

model_for_pruning = tfmot.sparsity.keras.prune_low_magnitude(
    model, **prune_params
)

4.3 Batch processing optimization

If you need to continuously process multiple images, you can do batch inference:

# Process 4 images at once
batch_input = np.concatenate([img1, img2, img3, img4], axis=0)
interpreter.set_tensor(input_details[0]['index'], batch_input)
interpreter.invoke()
# Throughput increased 2-3 times!

Common problem troubleshooting

Problem 1: Model conversion fails with error “Unsupported ops”

Cause: Model uses operations not supported by TFLite (such as certain custom layers)
Solution:

Use converter._get_unsupported_operations() to view unsupported ops

Replace with operations supported by TFLite
Or use converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS] to enable selective TF ops (will increase model size)

Problem 2: Inference results are all 0 or NaN

Cause: Input data preprocessing is incorrect (normalization method inconsistent with training)
Solution: Ensure preprocessing during inference (normalization, resize, channel order) is exactly the same as during training. If training used 0-1 normalization, inference must also use it!

Problem 3: ESP32 out of memory (OOM)

Cause: tensor_arena allocation too small
Solution:

Increase tensor_arena_size (but ESP32-S3 maximum is only ~500KB)

Use smaller models (MobileNetV1 0.25 or custom tiny CNN)
Enable quantization (int8 quantization halves memory)

Problem 4: Inference speed too slow (>1 second/frame)

Cause: Model too large or not using hardware acceleration
Solution:

Use quantized model (speed 2-4 times faster)

Reduce input resolution (32x32 is 40 times faster than 224x224!)
Use hardware delegates (GPU/NPU/DSP)
Consider switching to lighter model architecture

Summary

Today we went through the complete TensorFlow Lite deployment process:

Use TensorFlow to train an image classification model
Convert model to TFLite format and perform quantization compression
Load model on embedded device and execute inference
Use hardware delegates and batch processing to optimize inference performance

Key points:

Quantization is essential! Size reduced 4x, speed improved 2x, accuracy loss <1%
Input preprocessing must be consistent with training, otherwise results will all be wrong
ESP32 and other microcontrollers need to use TFLite Micro, keep model within 200KB
Raspberry Pi can use GPU delegate for acceleration

Extension suggestions:

Try TensorFlow Lite’s object detection model (MobileNet-SSD)
Deploy on Jetson Nano using TensorRT, speed improved another 5x
Combine with OpenCV for real-time video analysis

Hope this blog post is helpful to you!

Related Resources: