Feature Request: Runtime App Framework for device-ui

[punkle]

Is your feature request related to a problem? Please describe.

As far as I can tell there is no way for developers to add functionality to the device UI without modifying and recompiling the firmware. Even simple additions like a calculator tool requires C++ knowledge, a full build toolchain, and a firmware flash. This puts all feature development burden on the core maintainers and limits what the community can contribute.

Describe the solution you'd like

It would be nice for there to be a sandboxed app framework that allows small applications to be written in a scripting language and loaded from an SD card at runtime. Apps would be browsable from a dedicated screen on the device. The scripting VM should be restrictive enough to protect the main core firmware, and ideally require user confirmation or signature verification before running untrusted code from SD card. A small number of apps could also be compiled into the firmware as defaults.

Describe alternatives you've considered

As far as I can tell there is no alternative solution to this, but Im new to this, so am happy to be corrected.

Additional context

I made a little (AI assisted) POC implementation that uses Berry (https://github.com/berry-lang/berry). It includes an app browser screen, a calculator and a minesweeper clone game as demo apps. It also loads apps from the apps directory on the sd card if present. I have tested this on T-Deck Plus. The POC can be tried by cloning https://github.com/punkle/firmware and running pio run -e t-deck-tft-apps.

The app framework is implemented on this fork https://github.com/punkle/device-ui

I believe this to be a feasible proposition.. but I could be overlooking something in my naive enthusiasm.

[github-actions]

Congratulations for your first issue

[taylorsreid]

This is really cool. I haven't had the chance to test it but you should submit the framework addition as a pull request. I want to develop an app for the T-Deck but I'm not great with C++ and even if I was, I don't particularly want to mess with the firmware.

[punkle]

Ok. I have this app framework working. I need to tidy up the code a but. Here is what I have so far:

App developers would be able to implement bui and/or mui frontend apps as well as mesh event handling and app state stored to flash. Each app must list requested permissions requiring approval (e.g. http-client, mesh-receive:position or mesh-receive:1) and developer signatures. I also managed to implement a single executable CLI that can upload app bundles via serial to flash, or to an sd card.

My local changes are built in the firmware, the device-ui and a new platform-io project I have called mapps. The mapps includes the berry runtime, some shared bindings (e.g. a json parser) and a cli to build/sign the apps.

[taylorsreid]

Berry bindings to send and receive different types of mesh packets (not just text messages) would unlock a lot of potential for apps. Can't wait to see how this turns out!

[punkle]

@taylorsreid what would the apps do when they have received the messages? Can you give me some examples.

[taylorsreid]

Sure, a simple example would be CheckTastic or MeshChess. Receive the other player's move in a packet, and respond with your move in an outgoing packet. Or as a more serious app, the idea I had in mind was a work order / dispatching system, where incidents can be created, assigned, and updated by users over the air.

[punkle]

How would the game/order state be stored/agreed between the users in these cases. E.g. if i make a chess move and send a packet, i think the protocol is fire and forget. So I imagine every node might contain a completely different state.

I guess in chess (with 2 players) its simple, one node acts as the game server. The game server holds the official state. With a more complex system the algorithms would need to be comletely decentalized or we would need a client/server architecture running in the mesh.. which is possible now that is say it out loud.

What are your thoughts?

[punkle]

I have this app framework in a place where I should start pushing up pull requests. It crosses three projects at this point: firmware; device-ui and a new shared platform-io library and cli that I have called mapps. It can handle incoming mesh messages, make http requests, store app state, and can display UIs on MUI and BUI.

If you build t-deck or t-deck-tft from my fork branch it will have the app framework enabled. https://github.com/punkle/firmware/tree/app-framework

Features:

• app developer signing and user approval flows
• app permissions explicitly requested and user approval flows
• http client
• app state
• mesh receive on firmware, agnostic to ui framework (didnt want to implement mesh send just yet)
• UI works on bui and mui
• json parser
• users can install apps to sd card or flash depdending on what is available
• probably more I am missing

To get up an running writing an app you will need the mapps cli that I have created. https://github.com/punkle/mapps

There are some sample apps here to demonstrate the features including a mesh messages dashboard, and a jokes api client. https://github.com/punkle/mapp-apps All four of this work with bui and mui.

[taylorsreid]

There's a few ways to solve the state issue, ultimately it would be up to the developer who is writing the app in Berry how they'd like to maintain state.

With a simple 2 player system I would probably just use message sequence numbers in the payload to ensure that order is correct and correct state is maintained. If one player's device receives a packet with a sequence number that doesn't match the expected sequence number, a packet is sent requesting retransmission of the missing packet. This is how it is done in AX25 packet radio.
Example: Player 1's device is expecting Player 2's next packet to have a sequence number of 4. Instead it receives a packet with a sequence number of 5, meaning that we are missing a packet. An automatic request is sent requesting that the packet with sequence 4 is resent. The game cannot continue until the correct packet is received.

For a more complex system like the work order / dispatching system I would definitely just write a server program with a simple database like sqlite as the single source of truth, and have it interface with the node. I'm toying around with the idea right now using Go.

I'm still waiting on my T-Deck in the mail but I'm looking forward to trying out your firmware and framework.

[punkle]

Yeah interesting. The sequence number helps for the chess game. Regarding the server state.. I didnt realize that sqlite worked on esp32. This could be useful for apps state storage. Its serverless so the memory over head is 0 until query time. However i wonder how it would perform at query time especially with larger queries.

[taylorsreid]

Oh I meant hooking up a node to a PC and running the server and sqlite database on the PC, but from a quick Google search, it does appear that running sqlite on esp32 is possible, albeit possibly slow. That's beyond my current skill set for now though since I'm just starting to get interested in systems programming. That's why the app framework using Berry was so appealing to me, it doesn't require much C++ or hardware knowledge.