Skip to content

extmod/zephyr_ble: Add Zephyr BLE host stack with RP2 port integration.#19

Draft
andrewleech wants to merge 34 commits intoreview/zephyr-ble-corefrom
pr/zephyr-ble-core
Draft

extmod/zephyr_ble: Add Zephyr BLE host stack with RP2 port integration.#19
andrewleech wants to merge 34 commits intoreview/zephyr-ble-corefrom
pr/zephyr-ble-core

Conversation

@andrewleech
Copy link
Owner

Summary

This adds the Zephyr BLE host as a third backend for modbluetooth, integrated as a port-agnostic extmod. The Zephyr host code is compiled against a HAL shim layer (extmod/zephyr_ble/hal/) that replaces Zephyr's kernel primitives (semaphores, work queues, timers, FIFOs, memory slabs) with MicroPython-compatible implementations. This means the Zephyr BLE host runs cooperatively on the main MicroPython task without requiring Zephyr RTOS itself.

This was motivated by limitations in the existing stacks — NimBLE doesn't have an active BLE pre-qualification and is missing some newer BLE features, while BTstack's MicroPython integration lacks pairing/bonding and L2CAP channel support. The Zephyr BLE host stack has active qualification, full feature coverage, and is under active development by multiple silicon vendors.

The RP2 port is the first integration, providing two build variants for Pico W and Pico 2 W:

  • zephyr_ble — cooperative polling from the main task (preferred)
  • zephyr_ble_freertos — HCI processing on a dedicated FreeRTOS task

Also included: a gap_unpair() API addition across all BLE backends, micropython-lib updates for aioble robustness, and bond key persistence via Python secret store callbacks.

flowchart TD
    subgraph "extmod/zephyr_ble (port-agnostic)"
        MOD[modbluetooth_zephyr.c] --> HAL[HAL shim layer]
        HAL --> WORK[work queues]
        HAL --> SEM[semaphores]
        HAL --> TIMER[timers]
        HAL --> FIFO[FIFOs]
        HAL --> H4[HCI H4 transport]
    end

    subgraph "Port integration (e.g. RP2)"
        HCI_DRV[mpzephyrport_rp2.c<br/>CYW43 HCI driver] --> H4
        POLL[main loop polling] --> WORK
    end

    subgraph "Zephyr BLE host (lib/zephyr)"
        HOST[hci_core / gatt / att / smp / l2cap]
    end

    HOST --> MOD
    HCI_DRV --> HOST
Loading

Testing

All 12 BLE multitests passing on Pico W and Pico 2 W (both zephyr_ble variant) with PYBD (NimBLE) as central:
ble_gap_advertise, ble_gap_connect, ble_characteristic, ble_gap_pair, ble_gap_pair_bond, ble_subscribe, ble_irq_calls, ble_gattc_discover_services, ble_l2cap, perf_gatt_notify, perf_l2cap, ble_gap_unpair.

Performance: ~24ms/notification (GATT), ~2184 B/s (L2CAP) on Pico W; ~25ms/notification, ~7956 B/s on Pico 2 W.

Not tested: zephyr_ble_freertos variant on Pico 2 W.

Trade-offs and Alternatives

The HAL shim layer is substantial (~3K lines) because it reimplements Zephyr kernel primitives. The alternative would be running actual Zephyr RTOS, but that would limit this to the Zephyr port only. The shim approach allows any MicroPython port with an HCI transport to use the Zephyr BLE host.

The lib/zephyr submodule adds the full Zephyr BLE host source. Only the host stack files are compiled — no kernel, no drivers, no board support. The submodule is pinned to a specific commit with two small patches (wrapper files for gatt.c and conn.c to expose static internals needed for clean teardown).

MTU is compile-time only (CONFIG_BT_L2CAP_TX_MTU=512); runtime ble.config(mtu=X) is not supported. ble_mtu.py test is skipped.

Generative AI

I used generative AI tools when creating this PR, but a human has checked the code and is responsible for the description above.

robert-hh and others added 15 commits March 8, 2026 23:54
These classes are based on the Quadrature Encoder blocks of the i.MXRT
MCUs.  The i.MXRT 102x has two encoders, the other ones four.  The i.MXRT
101x does not support this function.  It is implemented as two classes,
Encoder and Counter.

The number of pins that can be uses as inputs is limited by the MCU
architecture and the board schematics.  The Encoder class supports:
- Defining the module.
- Defining the input pins.
- Defining a pin for an index signal.
- Defining a pin for a reset signal.
- Defining an output pin showing the compare match signal.
- Setting the number of cycles per revolution (min/max).
- Setting the initial value for the position.
- Setting the counting direction.
- Setting a glitch filter.
- Defining callbacks for getting to a specific position, overrun and
  underrun (starting the next revolution).  These callbacks can be hard
  interrupts to ensure short latency.

The encoder counts all phases of a cycle.  The span for the position is
2**32, for the revolution is 2**16.  The highest input frequency is
CPU-Clock/24.  Note that the "phases" argument is emulated at the API
level (the hardware will always count all phases).

The Counter mode counts single pulses on input A of the Encoder.  The
configuration supports:
- Defining the module.
- Defining the input pin.
- Defining the counting direction, either fixed or controlled by the level
  of an input pin.
- Defining a pin for an index signal.
- Defining an output pin showing the compare match signal.
- Setting the counter value.
- Setting the glitch filter.
- Defining a callback which is called at a certain value.
- Settings for MIMXRT1015. The MIMXRT1015 MCU has only one encoder/counter
  unit.

The counting range is 0 - 2**32-1 and a 16 bit overrun counter.  The
highest input frequency is CPU-Clock/12.

The implementation of the `.irq()` method uses the common code from
`shared/runtime/mpirq.c`, including the `irq().flags()` and
`irq().trigger()` methods.

Signed-off-by: robert-hh <robert@hammelrath.com>
Signed-off-by: robert-hh <robert@hammelrath.com>
This adds MIMXRT-specific arguments and methods, as a superset of the
original Encoder/Counter documentation.

The mimxrt pinout and quickref docs are updated with relevant information.

Signed-off-by: robert-hh <robert@hammelrath.com>
For Teensy 4.x.  The connectivity tests and the falling edge of the counter
test are skipped.

Signed-off-by: robert-hh <robert@hammelrath.com>
Because it requires a different configuration of the pins (in `setUp`).
Eg on mimxrt pins used for an Encoder cannot be read.

Signed-off-by: Damien George <damien@micropython.org>
Because it requires a different configuration of the pins (in `setUp`).
Eg on mimxrt pins used for a `machine.Counter` cannot be read.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: robert-hh <robert@hammelrath.com>
Causing a significant increase of the firmware size.

Signed-off-by: robert-hh <robert@hammelrath.com>
Signed-off-by: Damien George <damien@micropython.org>
This is a linker option, so provided it's added to LDFLAGS then it can be
dropped from CFLAGS without changing the compiler behaviour.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Prior to this fix, f'{{' would raise a SyntaxError.

Signed-off-by: Damien George <damien@micropython.org>
This commit adds support for t-strings by leveraging the existing f-string
parser in the lexer.  It includes:
- t-string parsing in `py/lexer.c`
- new built-in `__template__()` function to construct t-string objects
- new built-in `Template` and `Interpolation` classes which implement all
  the functionality from PEP 750
- new built-in `string` module with `templatelib` sub-module, which
  contains the classes `Template` and `Interpolation`

The way the t-string parser works is that an input t-string like:

    t"hello {name:5}"

is converted character-by-character by the lexer/tokenizer to:

    __template__(("hello ", "",), name, "name", None, "5")

For reference, if it were an f-string it would be converted to:

    "hello {:5}".format(name)

Some properties of this implementation:
- it's enabled by default at the full feature level,
  MICROPY_CONFIG_ROM_LEVEL_AT_LEAST_FULL_FEATURES
- when enabled on a Cortex-M bare-metal port it costs about +3000 bytes
- there are no limits on the size or complexity of t-strings, and it allows
  arbitrary levels of nesting of f-strings and t-strings (up to the memory
  available to the compiler)
- the 'a' (ascii) conversion specifier is not supported (MicroPython does
  not have the built-in `ascii` function)
- space after conversion specifier, eg t"{x!r :10}", is not supported
- arguments to `__template__` and `Interpolation` are not fully validated
  (it's not necessary, it won't crash if the wrong arguments are passed in)

Otherwise the implementation here matches CPython.

Signed-off-by: Damien George <damien@micropython.org>
So `mpy-cross` can compile t-strings.

Signed-off-by: Damien George <damien@micropython.org>
Includes corresponding .exp files because this feature is only available in
Python 3.14+.

Tests for `!a` conversion specifier and space after `!` are not included
because they are not supported by MicroPython.

Signed-off-by: Koudai Aono <koxudaxi@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This documents all of the available functionality in the new
`string.templatelib` module, which is associated with template strings.

Signed-off-by: Koudai Aono <koxudaxi@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
SaintSampo and others added 5 commits March 11, 2026 11:19
The NanoXRP is a small version of the XRP Robot using rp2040.

Signed-off-by: Jacob Williams <jwilliams@experiential.bot>
Signed-off-by: sync-on-luma <spencerc@ayershale.net>
Signed-off-by: EngWill <646689853@qq.com>
Signed-off-by: Damien George <damien@micropython.org>
Includes 4MB and 16MB variants.

Signed-off-by: EngWill <646689853@qq.com>
Signed-off-by: Damien George <damien@micropython.org>
EngineerWill and others added 4 commits March 11, 2026 12:38
Signed-off-by: EngWill <646689853@qq.com>
Signed-off-by: Damien George <damien@micropython.org>
Reclassify failures of tests listed in flaky_tests_to_ignore as "ignored"
instead of retrying them. Ignored tests still run and their output is
reported, but they don't affect the exit code. The ci.sh --exclude lists
for these tests are removed so they run normally.

Signed-off-by: Andrew Leech <andrew.leech@planet-innovation.com>
It's been over 12 years and the project is relatively stable now.

Signed-off-by: Damien George <damien@micropython.org>
This commit adds a section to the top-level README describing MicroPython's
general design philosophy and core values.

Thanks to @projectgus who suggested I add this.

Signed-off-by: Damien George <damien@micropython.org>
@github-actions
Copy link

Code size report:

Reference:  rp2/modules/rp2.py: Don't corrupt globals on asm_pio() exception. [c3ca843]
Comparison: extmod/zephyr_ble: Improve L2CAP CoC throughput with seg_recv and DLE. [merge of eb4a81a]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:   +80 +0.005% ESP32_GENERIC[incl +32(data)]
     mimxrt:    +0 +0.000% TEENSY40
        rp2:  +232 +0.025% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@andrewleech
Copy link
Owner Author

/review

@mpy-reviewer
Copy link

mpy-reviewer bot commented Mar 13, 2026

Review failed. Retry with /review.

europrimus and others added 8 commits March 16, 2026 14:39
'esptool.py' is deprecated, use 'esptool' instead.

Signed-off-by: europrimus <europrimus-dev@c-f.me>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Switch L2CAP CoC from recv+alloc_buf to the seg_recv API, which gives the
application per-PDU callbacks with manual credit control.  The old path
issued one credit per SDU, forcing the peer to wait for the full SDU to
be delivered before sending the next one.  With seg_recv, credits are
issued one per non-last PDU (allowing the peer to pipeline all K-frames
of a single SDU) and one credit per SDU from recvinto() (for the first
PDU of the next SDU), keeping at most one assembled SDU buffered.

Work around a Zephyr bug in l2cap_chan_seg_recv_rx_init() which leaves
rx.mps at zero for seg_recv channels (unlike l2cap_chan_rx_init for the
normal path), causing immediate channel disconnect on the first received
PDU.  Set rx.mps = BT_L2CAP_RX_MTU in l2cap_create_channel() and use
bt_l2cap_chan_give_credits() in accept/connect paths, matching the
pattern from Zephyr's credits_seg_recv test.

Also enable Data Length Extension (DLE) so the controller can negotiate
251-byte PDU payloads, reducing per-PDU overhead.

TX pipeline: allow up to L2CAP_SDU_BUF_COUNT-1 SDUs in flight concurrently
(tracked via tx_in_flight counter) rather than stalling after every send.

On nRF52840 dongle (PCA10059) with PYBD (NimBLE) as central:
  perf_l2cap.py before: ~2,184 B/s
  perf_l2cap.py after:  ~11,518 B/s  (5.3x improvement)
  All 11 BLE multitests pass.

Signed-off-by: Andrew Leech <andrew@alelec.net>
Replace single-SDU L2CAP accumulation buffer with a FIFO that
holds multiple SDUs.  Deep initial credit window (fills rx_buf)
allows the peer to pipeline SDUs without per-SDU credit
round-trips, which is critical for Z2Z throughput where each
credit round-trip costs 2+ connection intervals.

Add deferred L2CAP recv notification (rx_notify_pending) to
avoid re-entrancy between seg_recv_cb and Python IRQ handlers.
Each port's port_run_task must call flush_recv_notify() after
work_process completes.

Disable DLE auto-negotiation (CONFIG_BT_AUTO_DATA_LEN_UPDATE 0)
for CYW43 compatibility — CYW43 disconnects with "Instant
Passed" (0x16) when DLE is negotiated.

Add l2cap_status_cb TX kick via bt_tx_irq_raise() to unblock
queued SDUs when credits arrive.

Run codeformat.py on extmod files.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
@andrewleech andrewleech force-pushed the pr/zephyr-ble-core branch 2 times, most recently from fedbc7f to ffe840c Compare March 16, 2026 05:56
Add ZEPHYR_BLE_POLL_INTERVAL_MS define (default 128ms)
matching NimBLE convention. IRQ-driven ports use poll_now()
for immediate processing; this is a fallback for timer
housekeeping.

Change CONFIG_BT_AUTO_DATA_LEN_UPDATE to #ifndef guard so
ports with capable controllers can override via CFLAGS.

Move random data generation out of the timed window in
perf_l2cap.py and use getrandbits(8) for faster generation.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Cleanup unused functions, macros and debug helpers that were never called
or only conditionally compiled behind disabled feature flags. Removes dead
registry system, PSA crypto stubs, LIFO operations, and various unused
helper functions and inlines from kernel/device/config headers. Also
deletes gatt_pragma.h which is unreferenced.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.