Troubleshooting
The TV’s NPU is 400× slower than expected
Section titled “The TV’s NPU is 400× slower than expected”Symptom: a model that should take ~22 ms per image on the TV’s NPU takes ~9 s, timings.invoke is far above what
benchmark_model reports, and nothing errors.
Cause: given a file path, TFLite passes the model file’s descriptor to the vendor NN HAL. SELinux may not let the
HAL read app files on shared storage (FUSE): on the tested TCL TV, mtk_hal_neuralnetworks was denied fuse reads.
NNAPI then silently runs the model on its in-process CPU reference path.
Fix: load delegated models from a memory-mapped buffer, not by path (the gateway does this). To confirm, look for
avc: denied lines naming the model file:
adb logcat -b all | grep 'avc: denied' | grep -i fuseThe registry records this as a works with tweaks result for efficientnet-lite4-int8 on the TV.
The TV node will not start listening: port already in use
Section titled “The TV node will not start listening: port already in use”Symptom: a TV node configured with "port": 8443 fails to bind, or answers with something that is not the gateway.
Cause: on Google TV, port 8443 is already used by the system (Google Cast). The phone’s default port clashes on the TV.
Fix: use the WebSocket node transport with "listen": false (the default): the node opens no port at all. If the
TV must listen (HTTPS self-registration), pick another port such as 9443.
The TV stops answering a few minutes after power-off
Section titled “The TV stops answering a few minutes after power-off”Symptom: the TV node goes offline about 3 minutes after the screen is switched off, even though the gateway holds a wake lock.
Cause: TVs enter a low-power standby that suspends apps and networking regardless of wake locks. Depending on the firmware, settings such as Low Power Standby or network standby control this.
Fix: treat a TV as an “only while switched on” node, and keep an always-on node (a phone on a charger, a mini PC) for models that must always answer. The hub drops an offline node from routing after 90 s and routes to the next one.
fp32 models fail on the TV’s NPU
Section titled “fp32 models fail on the TV’s NPU”Symptom: mobilenet-v2-fp32 reports fails on the MediaTek MDLA.
Cause: the MDLA does not run fp32 graphs, and NNAPI rejects the model for the NPU without fp16 relaxation.
Fix: the nnapi build enables setAllowFp16(true) (11.2 ms afterwards), or ship an int8 model. Set
nnapiAccelerator in the manifest (e.g. "mtk-mdla_shim") to disable NNAPI’s own CPU fallback, so a model that
cannot run on the NPU fails loudly instead of running slowly.
Big models crash the TV
Section titled “Big models crash the TV”Symptom: loading a 330 MB fp32 CLIP model reboots the TV.
Cause: only about 700 MB of its 2.4 GB RAM was free. Set a conservative memoryBudgetMB (e.g. 300) on TV nodes and
keep large models on the phone or a server node.
The phone node disconnects on battery
Section titled “The phone node disconnects on battery”Doze cuts the WebSocket. Set "keepAwake": true or exempt the app from battery optimisation. On external power the
wake lock is held automatically.
Node shows “rejected token” or keeps backing off
Section titled “Node shows “rejected token” or keeps backing off”- 401/403 on the upgrade: the key is wrong, revoked, or not a node key. The node retries at the slowest backoff (every 30–60 s).
- Close code 4409: the same node connected again elsewhere; the older session was superseded.
- Close code 4429: the hub’s node limit is reached.
Clients see 502 peer_unavailable or 504 timeout
Section titled “Clients see 502 peer_unavailable or 504 timeout”The node went away mid-request or did not answer within requestSec (60 s). Check the node’s status screen and the
hub’s Nodes page. Only transport failures are retried on another node; a node’s own error is passed through.
Behind Cloudflare, every client has the same IP
Section titled “Behind Cloudflare, every client has the same IP”AIGW_TRUSTED_PROXIES must be the address your reverse proxy connects from. Check with
docker service logs ai-gateway_hub | grep '"path":"/healthz"': once it is right, public requests show the visitor’s
address.
The website’s registry page shows “snapshot” instead of live data
Section titled “The website’s registry page shows “snapshot” instead of live data”The website’s origin is not in the hub’s AIGW_PUBLIC_ORIGINS, or the hub is unreachable. The page falls back to the
build-time snapshot on purpose.