Building On-Device ML for Android Apps | Blogging on Android and ML | Open-Source Developer | Freelancing for ML/Android

medium

github

Shubham Panchal  • Peerlist

SmolChat is an open-source, native-Android application helping users run large-language models on their mobile devices offline and interact with them in a ChatGPT-like interface. - The models and the chats remain completely offline, on-device, ensuring data privacy and no network reliance. - Download any model available in the GGUF format and start using it right away. You can customize the system prompt and inference parameters (temperature, min-p) for controlling the text generation.- Create pre-defined 'Tasks' which help you perform tasks like text summarization and rewriting quickly, without configuring the chats every time.SmolChat is an open-source mobile application, licensed under Apache 2.0. <h3>Technical Details</h3>The app uses its own small JNI binding written over llama.cpp, which is responsible for loading and executing GGUF models. Chat, message and model metadata are stored in a local ObjectBox database. The codebase is written in Kotlin/Compose and follows modern Android development practices.The JNI binding is inspired from the <a target="_blank" rel="noopener noreferrer nofollow" class="text-green-500 rich-link break-all link-break" href="https://github.com/ggerganov/llama.cpp/blob/master/examples/simple-chat/simple-chat.cpp">simple-chat example</a> in llama.cpp.<h3>Demo Video:</h3><ol><li>Interacting with a SmolLM2 360M model for simple question-answering with flight-mode enabled (no connectivity)</li><li>Adding a new model, Qwen2.5 Coder 0.5B and asking it a simple programming question</li><li>Using a prebuilt task to rewrite the given passage in a professional tone, using SmolLM2 1.7B model</li></ol>Project (with an APK built): <a target="_blank" rel="noopener noreferrer nofollow" class="text-green-500 rich-link break-all link-break" href="https://github.com/shubham0204/SmolChat-Android">https://github.com/shubham0204/SmolChat-Android</a>