1 of 9

Akash Paul ML @Airtel Digital

​

github.com/iakashpaul/portal

​

​

​

https://hasgeek.com/fifthelephant/open-source-ai-hackathon/

Portal: Hole in your Palm

2 of 9

Why does this project exist?

  • The list on the right from the Llama.cpp project covers mostly iOS, macOS, PCs & web-UIs.
    • KanTV which is primarily geared towards improving TV watching experience
  • Not many can afford GPUs(hosted/local) or rely on subsidized AI offerings in the short term

​

  • Ability to search better for information must be better than appending “reddit” or “solved” to a search query

​

The acquisition of basic computing skills by any set of children can be achieved through incidental learning provided the learners are given access to a suitable computing facility, with entertaining and motivating content and some minimal (human) guidance - Dr. Sugata Mitra

​

    • https://www.hole-in-the-wall.com/Beginnings.html

​

3 of 9

There has to be a better way for trying LLMs

​

4 of 9

Architecture

Android SDK

Uses Kotlin & Compose for UI elements

Android NDK

Cross compiles Llama.cpp & exposes native functions to the app

LLM Weights

Fetch GGUF Q4 Weights from HuggingFace 🤗

5 of 9

Code overview

Fetches & builds Llama.cpp library & exposes external functions via Java Native Interface(JNI)

​

Uses Kotlin with Compose for UI & DownloadManager library for managing hosted weights

​

6 of 9

Demo

akashpaul.com/assets/portal.apk

7 of 9

Roadmap

Future Roadmap

  • Better cross-device support
  • OpenCL support for Adreno/mobile GPU support
  • Vision Modality - Camera & on-screen content
  • Agentic Task Support using ReACT framework & grammar
  • Publish on Play Store + Indus AppStore

Termux on Android

Managed to get emulated shell to build & run upto 7B models near 3 tok/s & 2B models around 10 tok/s

On emulated shell via CPU compute

Improve Llama.cpp on Android

Updated the app with better configurations for CPU threads, context limit etc.

8 of 9

Indus AppStore

9 of 9

Thanks!

  • Hasgeek, mentors & volunteers for their time & resources!

​

  • Bharat & Akshobhya for the mentor sessions & support on the groups!

​

  • Got to interact with the AI dev community & also contribute to chsasank/device-benchmarks for benchmarking various GPU & CPU compute specs

akashpaul.com/assets/portal.apk