# Research & Root-Cause Log

This document records the investigation that produced the fixes in this
repository. It is kept separate from `README.md` (the rebuild guide) so the
how-to stays clean. Each section maps to a fix number used throughout the
project.

Hardware under test: Apple MacBook Air A1370, Broadcom BCM43224
(PCI `0x14e4:0x4353`, subvendor `0x106b:0x00d1`), N-PHY rev-23 core behind a
BCMA/bhnd bus attached via the `bhndb_pci` PCIe-G1 bridge, on an NVIDIA MCP89
host bridge. FreeBSD 15.1-RELEASE-p2 (amd64).

---

## Fix 5 — Load Broadcom v4 firmware at loader time (firmware "not found" panic)

**Problem.** `if_bwn.ko` has a `pci/bwn_pci` devmatch modalias, so the kernel
AUTO-LOADS it when `bwn_pci0` attaches. The firmware-only `bwn_v4_n.ko` has NO
modalias and loads only via `kld_list`. If the firmware module is missing at
attach time (e.g. `rc.conf` `kld_list` got corrupted/shortened),
`bwn_fw_get()` fails and the driver panics while cleaning up
("firmware bwn_v4_ucode16_mimo not found").

**Fix.** Add `bwn_v4_n_load="YES"` to `/boot/loader.conf` so the firmware
module is loaded at the loader stage, before any device attach. This makes
firmware availability independent of `rc.conf`/`kld_list` ordering.

**Verification.** With `bwn_v4_n` loaded, creating `wlan1` on `bwn0` loads
`ucode16_mimo` with no "not found" error.

---

## Fix 6 — Kernel panic on kldunload after a "device timeout" (vmcore.4)

**Symptom.** `kldunload if_bwn` (or any detach while a wlan existed on bwn0)
panicked with a page fault in `node_cleanup` / `ieee80211_ageq_remove`
(fault addr `0x488`), always after a `bwn0: device timeout`.

**Root cause.** TX never completes because MSI interrupts are not delivered,
so the TX slot keeps its `mbuf` + `ieee80211_node` (`mt_ni`) reference —
`bwn_dma_handle_txeof` (the only place that clears `mt_ni`) never runs.
`bwn_dma_stop()` only halts the DMA engine and never frees queued frames, so
the node reference lingers. When net80211 later frees that node (scan cancel /
VAP teardown) the slot points at freed memory; then `bwn_detach` ->
`bwn_dma_free` -> `bwn_dma_ringfree` -> `bwn_dma_free_descbufs` ->
`ieee80211_free_node(stale mt_ni)` walks freed memory and panics.

**Fix.** Reclaim pending frames (and their node references) in
`bwn_dma_stop()`, which runs from both the detach path
(`bwn_stop` -> `bwn_core_exit`) and every HW reset. This drops node references
while the node is still alive (before `ieee80211_ifdetach` frees nodes).
`bwn_dma_free_descbufs()` is idempotent (NULLs `mt_m`/`mt_ni`), so the later
detach free finds empty slots.

---

## Fix 7 — TX node reference ownership + unconditional reclaim + MSI/INTx (vmcore.5)

**Symptom.** After Fix 6 a fresh reboot still panicked on `kldunload`
(vmcore.5): page fault in `ieee80211_ratectl_node_deinit` (fault addr `0x30`,
`vap->iv_rate` deref), `ni->ni_refcnt = 1`, node's `ni_vap` already torn down.
Also, with MSI never delivered, the chip showed "device timeout" and passed no
traffic.

**Root cause.** The driver relied on the mbuf's `rcvif` node reference (owned
by the net80211 stack) for the lifetime of a queued TX frame. The stack can
release that reference at any time (scan cancel, VAP teardown, node reclaim)
while the frame is still in the ring, because TX completion never runs when
interrupts are not delivered. Also `bwn_dma_free_descbufs()` early-returned on
`if (!dr->dr_usedslot)`; `dr_usedslot` is only a free-slot accounting hint and
is not a reliable indicator of pending frames, so the reclaim could be
skipped.

Separately, `bhndb_pci` enables MSI for the BCM43224, but on this host (NVIDIA
MCP89 Apple MacBook Air) MSI is never delivered.

**Fix.**
1. `bwn_dma_tx_start()` / `bwn_pio_tx_start()` now take the driver's OWN
   reference via `ieee80211_ref_node(ni)` before stashing it in `mt->mt_ni` /
   `tp->tp_ni`. `ieee80211_tx_complete()` in the txeof path releases exactly
   one reference, so accounting stays balanced and detach cannot double-free.
2. Removed the `if (!dr->dr_usedslot) return;` early-return from
   `bwn_dma_free_descbufs()` so reclaim unconditionally walks every slot.
3. Added `BHNDB_PCI_QUIRK_NO_MSI` for BCM43224 in `bhndb_pci.c` /
   `bhndb_pcivar.h` so the bridge falls back to legacy INTx — TX interrupts
   are then delivered and the "device timeout" root cause is gone.

---

## Fix 8 — Remove RX-ring reclaim from bwn_dma_stop (vmcore.6 heap corruption)

**Symptom.** After Fix 6 the bwn *detach* panic was gone (clean `bwn0:
detached` cycles), but ~47 min into a session the box panicked with a
DIFFERENT crash: a TCP stack NULL-deref inside `tcp_m_copym` while processing
an inbound packet on wlan0/rtwn0 (the SSH link). The SSH socket's send buffer
was corrupted (`so_snd.sb_mb == NULL` while `sb_ccc == 126404`; `tp` seq
numbers garbage).

**Root cause.** Fix 6's reclaim in `bwn_dma_stop()` walked **all six rings
including the RX ring**. `bwn_dma_stop()` runs on **every** `bwn_hwreset`
(every "device timeout" — and on this chip MSI TX interrupts are not
delivered, so timeouts fire constantly). `bwn_dma_free_descbufs()` frees the
RX mbuf WITHOUT clearing the DMA descriptor's physical address. The RX ring is
persistent (refilled in place by `bwn_rxeof()` -> `bwn_dma_newbuf()`, never
re-allocated) and `bwn_dma_cleanup()` only zeroes the ring *control* register.
So after a reset the descriptor still pointed at the freed page; when RX was
re-enabled the NIC DMAd received frames into freed kernel memory -> heap
corruption that eventually clobbered the SSH socket. The console log showed
~60 lines of `bwn0: bwn_dma_free_descbufs: not TX?` (one per RX slot whose
mbuf had just been freed) immediately before the panic — the smoking gun.

**Fix.** Restrict the reclaim in `bwn_dma_stop()` (factored into a new helper
`bwn_dma_reclaim_tx()`) to the **TX rings only** (wme BK/BE/VI/VO + mcast).
The RX ring is never reclaimed there. `bwn_dma_reclaim_tx()` is also called
from `bwn_detach()` BEFORE `ieee80211_ifdetach()` (defence-in-depth on top of
the ref_node fix in Fix 7).

**Verification.** Disassembly: `bwn_dma_stop` has 6 ringstops (RX + 5 TX) +
exactly 5 `bwn_dma_free_descbufs` calls (TX only; RX ring no longer
reclaimed).

---

## Why the MSI quirk matters (Fix 7, point 3)

On this host the `bhndb_pci` bridge enables MSI for the BCM43224, but the
interrupts are never delivered (no `bwn`/`MSI` line in `vmstat -i`). Because
the only place that clears a TX slot's node reference is the TX-completion
path (`bwn_dma_handle_txeof`), TX frames never complete -> "device timeout"
and the cascade of use-after-free / heap-corruption panics. Forcing legacy
INTx (the `NO_MSI` quirk) makes TX interrupts arrive, so completions run
normally. The reclaim/ref_node fixes (6-8) are defence-in-depth that keep the
driver robust even if completions are delayed.