Compare commits
1 Commits
main
...
jonny-docs
| Author | SHA1 | Date | |
|---|---|---|---|
| 81b0f28507 |
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -99,5 +99,4 @@ cython_debug/
|
||||||
*.sigmf-meta
|
*.sigmf-meta
|
||||||
*.blue
|
*.blue
|
||||||
*.wav
|
*.wav
|
||||||
/images/
|
images/
|
||||||
!docs/source/images/**
|
|
||||||
|
|
|
||||||
223
Agent TX Streaming Handoff.md
Normal file
223
Agent TX Streaming Handoff.md
Normal file
|
|
@ -0,0 +1,223 @@
|
||||||
|
|
||||||
|
# Agent TX Streaming — `ria-toolkit-oss` Handoff
|
||||||
|
|
||||||
|
**Paired repo:** `ria-hub` (this doc lives here, but it's written for the Claude working in `ria-toolkit-oss`)
|
||||||
|
**Source of truth for the overall design:** [Agent TX Streaming - Cross-Repo Plan.md](./Agent%20TX%20Streaming%20-%20Cross-Repo%20Plan.md) — read that first.
|
||||||
|
**Status (ria-hub side):** landed 2026-04-16. Ready to talk to a TX-capable agent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Your job
|
||||||
|
|
||||||
|
Implement Part A of the plan in `ria-toolkit-oss` (§A1–A8). The hub is already speaking the protocol below and waiting for an agent that can:
|
||||||
|
|
||||||
|
1. Accept hub → agent binary TX buffers over an existing WebSocket.
|
||||||
|
2. Enforce the operator-configured TX interlocks (`tx_enabled`, `tx_max_gain_db`, `tx_max_duration_s`, `tx_allowed_freq_ranges`).
|
||||||
|
3. Drive `sdr.init_tx` / `_stream_tx` so a real Pluto transmits what the hub sends.
|
||||||
|
4. Report status back as `tx_status` JSON frames.
|
||||||
|
|
||||||
|
Full-duplex with the existing RX session on the same `app_id` must keep working.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What ria-hub does (so you know what's on the other end of the wire)
|
||||||
|
|
||||||
|
You don't need to know any of this to do your work — but if you hit a wall, here's the mental model:
|
||||||
|
|
||||||
|
| Hub-side concept | What it does |
|
||||||
|
|---|---|
|
||||||
|
| `AgentTxSink` (Python, Celery worker) | Mirrors `AgentDataSource`. Publishes `tx_start`/`tx_stop`/`tx_configure` control JSON and raw binary IQ buffers intended for the agent. |
|
||||||
|
| `/screens/agent/ws` FastAPI endpoint | The WebSocket you already connect to. Now also pumps hub → agent binary TX frames and republishes your `tx_status` JSON upstream to the Celery task. |
|
||||||
|
| Redis channels (`agent:tx_iq:*`, `agent:events:*`) | Internal to the hub. You will never see them. Everything reaches you as WS frames. |
|
||||||
|
| Capability gate | Hub refuses to launch a TX app unless it's seen a recent heartbeat from you with `"tx" ∈ capabilities` and `tx_enabled: true`. |
|
||||||
|
| Audit log (`AgentTxAudit`) | Hub persists who started what transmission at what frequency and gain. Your error messages in `tx_status` end up in that record. |
|
||||||
|
|
||||||
|
**Bottom line: from your process, this is still the same WebSocket you've been using. You're just getting new message types and a new class of binary frames going the other direction.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Protocol contract (the only thing you actually need)
|
||||||
|
|
||||||
|
All additions. Existing RX messages (`start`/`stop`/`configure` + agent → hub binary) keep their current semantics — do not touch them.
|
||||||
|
|
||||||
|
### Hub → agent
|
||||||
|
|
||||||
|
**JSON control frames** (text WS frames):
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// Arm TX. Call sdr.init_tx with this radio_config and start _stream_tx.
|
||||||
|
// After this you'll start receiving binary frames (see below) that go into
|
||||||
|
// the stream callback.
|
||||||
|
{
|
||||||
|
"type": "tx_start",
|
||||||
|
"app_id": "app-abc",
|
||||||
|
"radio_config": {
|
||||||
|
"device": "pluto",
|
||||||
|
"identifier": "ip:192.168.3.1",
|
||||||
|
"tx_sample_rate": 1000000,
|
||||||
|
"tx_center_frequency": 2450000000,
|
||||||
|
"tx_gain": -20, // dB. Pluto: negative = attenuation.
|
||||||
|
"tx_bandwidth": 1000000, // optional
|
||||||
|
"buffer_size": 1024, // optional; complex samples per buffer
|
||||||
|
"underrun_policy": "pause" // "pause" | "zero" | "repeat"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop TX, drain queue, pause_tx. RX session on the same app_id (if any)
|
||||||
|
// stays alive.
|
||||||
|
{ "type": "tx_stop", "app_id": "app-abc" }
|
||||||
|
|
||||||
|
// Apply parameter changes at the next buffer boundary. Any subset of
|
||||||
|
// radio_config fields.
|
||||||
|
{ "type": "tx_configure", "app_id": "app-abc", "radio_config": { "tx_gain": -25 } }
|
||||||
|
|
||||||
|
// Advisory — safe to ignore. Hub publishes this whenever it RPUSHes a new
|
||||||
|
// binary buffer; it was wired so the WS bridge wakes up promptly. You do
|
||||||
|
// NOT need to act on it. Consider it a keepalive.
|
||||||
|
{ "type": "tx_data_available", "app_id": "app-abc" }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Binary frames** (binary WS frames):
|
||||||
|
|
||||||
|
* Raw interleaved `float32` IQ samples in `[-1, 1]`.
|
||||||
|
* One frame = one buffer.
|
||||||
|
* Byte length is always `num_complex_samples × 8` (8 bytes per complex sample: two float32s).
|
||||||
|
* **Only valid between `tx_start` and `tx_stop`.** If you receive a binary frame outside that window, drop it and log WARN — don't crash, don't panic.
|
||||||
|
|
||||||
|
Format validator is already in `ria_toolkit_oss.sdr.sdr._verify_sample_format` — reuse it.
|
||||||
|
|
||||||
|
### Agent → hub
|
||||||
|
|
||||||
|
**JSON status frames** (text WS frames). Use the existing `send_json` path:
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// Lifecycle — emit on every state transition.
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "armed" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "transmitting" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "underrun" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "done" }
|
||||||
|
|
||||||
|
// Errors — include a human-readable message. Hub surfaces it to the UI
|
||||||
|
// and writes it into the audit record.
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "error",
|
||||||
|
"message": "gain -5 exceeds tx_max_gain_db=-15" }
|
||||||
|
```
|
||||||
|
|
||||||
|
**States** (hub assumes this vocabulary):
|
||||||
|
* `armed` — `init_tx` done, callback started, queue empty, nothing transmitting yet.
|
||||||
|
* `transmitting` — at least one buffer has flowed through the callback.
|
||||||
|
* `underrun` — queue drained; what you do next depends on `underrun_policy`:
|
||||||
|
* `"pause"` → call `pause_tx()`, emit `underrun`, stay paused until the hub sends a fresh `tx_start`.
|
||||||
|
* `"zero"` → continue with `np.zeros(...)` fills, still emit `underrun` once so the hub can show the indicator.
|
||||||
|
* `"repeat"` → loop the last good buffer, emit `underrun` once.
|
||||||
|
* `done` — clean stop after `tx_stop`.
|
||||||
|
* `error` — capability rejection or hardware failure. Include `message`.
|
||||||
|
|
||||||
|
**Extended heartbeat** — you are already sending heartbeats. Grow the payload:
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{
|
||||||
|
"type": "heartbeat",
|
||||||
|
"hardware": ["mock", "pluto"],
|
||||||
|
"status": "streaming", // unchanged semantics
|
||||||
|
"capabilities": ["rx", "tx"], // NEW — derived from tx_enabled + SDR class having init_tx
|
||||||
|
"tx_enabled": true, // NEW — mirror of cfg.tx_enabled
|
||||||
|
"tx_max_gain_db": -10, // NEW — optional, from agent config
|
||||||
|
"tx_max_duration_s": 60, // NEW — optional
|
||||||
|
"tx_allowed_freq_ranges": [[2.4e9, 2.5e9]], // NEW — optional
|
||||||
|
"sessions": { // NEW — optional per-session snapshot
|
||||||
|
"rx": { "app_id": "app-abc", "state": "streaming" },
|
||||||
|
"tx": { "app_id": "app-abc", "state": "transmitting" }
|
||||||
|
},
|
||||||
|
"app_id": "app-abc" // keep for back-compat
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The hub reads these fields, stores them on `ScreensAgent`, and gates TX launches on them. **If you don't advertise `tx` in `capabilities` and `tx_enabled: true`, the hub will refuse to start any TX app with HTTP 400 — no WS traffic will be generated.**
|
||||||
|
|
||||||
|
### Backpressure model (what happens when you can't keep up)
|
||||||
|
|
||||||
|
* The hub caps its outbound TX queue at 200 buffers. If it fills, the hub either blocks on `write()` or drops the oldest buffer — both are benign for you.
|
||||||
|
* On the agent side, enforce your own cap (plan §A2 suggests 8 buffers). When full, `await ws.send` on the hub will slow via TCP/WS backpressure. You don't need an application-level flow-control message.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation roadmap (mapped to the Cross-Repo Plan)
|
||||||
|
|
||||||
|
Work in the order below. Each row is a single PR-sized unit.
|
||||||
|
|
||||||
|
| # | Plan ref | Deliverable | Acceptance |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | §A3 | `AgentConfig` gains `tx_enabled`, `tx_max_gain_db`, `tx_max_duration_s`, `tx_allowed_freq_ranges`. `save()` keeps 0600. | Unit test: round-trip through `~/.ria/agent.json`. |
|
||||||
|
| 2 | §A4 | `ria-agent register --allow-tx --tx-max-gain-db -10 --tx-max-duration 60` persists into config. `ria-agent stream --allow-tx` is a runtime override. | Integration test: `ria-agent register --allow-tx` then `cat ~/.ria/agent.json` shows fields. |
|
||||||
|
| 3 | §A1 | `ws_client.run()` grows a `on_binary: Callable[[bytes], Awaitable[None]]` parameter. Reconnect + heartbeat + malformed-frame behavior unchanged. | Existing `test_ws_client.py` still passes; new `test_ws_client_binary.py` asserts bytes reach the handler. |
|
||||||
|
| 4 | §A2 | Replace flat `self._sdr` / `self._app_id` state in `streamer.py` with `RxSession` + `TxSession` dataclasses. SDR instances cached by `(device, identifier)` so RX+TX share one handle on the same device. | Unit test: creating a TxSession on the same device as an active RxSession reuses the same SDR object. |
|
||||||
|
| 5 | §A2 | `_handle_tx_start`, `_handle_tx_stop`, `_handle_tx_configure` + `on_binary(data)` → `self._tx.queue.put(data)`. TX loop runs `_stream_tx` in an executor thread with a thread-safe `queue.Queue` adapter. | Integration test against MockSDR: tx_start → 10 binary frames → tx_stop produces exactly those samples through the callback. |
|
||||||
|
| 6 | §A2 | Underrun handling: `"pause"` / `"zero"` / `"repeat"` fills. Emits `tx_status: underrun` exactly once per drain event. | Unit test per policy against a slow producer. |
|
||||||
|
| 7 | §A2 | Cap enforcement **before** opening the SDR: reject with `tx_status: error` if `tx_enabled=False`, gain exceeds cap, freq outside allowed ranges, or duration cap exceeded (watchdog in TX loop calls `tx_stop` after `tx_max_duration_s`). | Unit test per rejection path; SDR is never opened when rejection fires. |
|
||||||
|
| 8 | §A5 | Heartbeat grows `capabilities`, `tx_enabled`, optional caps, `sessions`. | Integration test: start agent with `--allow-tx`, connect, verify heartbeat payload. |
|
||||||
|
| 9 | §A6 | Audit the Pluto driver's `_tx_lock` + `_param_lock` interaction to ensure concurrent RX + TX on the same `adi.Pluto` doesn't race on attribute writes. `MockSDR.init_tx` already exists — no change needed. | Stress test: 30 seconds of concurrent RX + TX on MockSDR with `_param_lock` instrumented for contention. |
|
||||||
|
| 10 | §A7 | Test matrix per plan: `test_streamer_tx`, `test_tx_safety`, `test_tx_underrun`, `test_full_duplex`, `test_ws_client_binary`, `test_integration_tx`. | All green in CI. |
|
||||||
|
| 11 | §A8 | Docs: new `docs/agent_tx_protocol.md` OR extended section in existing agent protocol doc. Regulatory disclaimer included. | Lints + renders. |
|
||||||
|
|
||||||
|
**Ship order advice:** 1 → 2 → 3 → 4 → (5 || 6) → 7 → 8 → 9 → 10 → 11. Steps 1–3 are strict prerequisites for everything else. Steps 5 and 6 can parallelize. Step 7 can't land without 5.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verification loop (how to prove the two sides talk)
|
||||||
|
|
||||||
|
Once you've implemented §A1–A7, use this to close the loop with the live hub:
|
||||||
|
|
||||||
|
1. On the agent host, run `ria-agent register --allow-tx --tx-max-gain-db -10 --tx-max-duration 60` then `ria-agent stream`.
|
||||||
|
2. Confirm the hub has seen the heartbeat: `curl $HUB/screens/agents/json | jq '.agents[] | select(.agent_id==...) | {tx_enabled, capabilities}'` should show `tx_enabled: true` and `["rx","tx"]`.
|
||||||
|
3. Create a Screens app whose manifest contains a `dataSink.type == "agent"` pointing at your `agent_id`, or use the composer UI with a `PlutoTXOp` in the graph + the new TX-sink agent picker.
|
||||||
|
4. `POST /screens/apps/{id}/start` on the hub. You should observe, in order:
|
||||||
|
1. `tx_start` JSON on your WS.
|
||||||
|
2. Binary frames arriving (if the hub-side operator is actually generating buffers — may no-op for now since the operator refactor is planned but not done).
|
||||||
|
3. Your `tx_status: armed` JSON emitted back.
|
||||||
|
5. Stop the app. You should receive `tx_stop` and emit `tx_status: done`.
|
||||||
|
6. Provoke a rejection: set `tx_max_gain_db: -15` in your config, then start a TX app with `tx_gain: -5`. The hub should return `HTTP 400` from `/start` without any WS traffic — capability gate fires first. If you make it past the gate and it's still wrong, emit `tx_status: error` and the hub will surface the message to the UI.
|
||||||
|
|
||||||
|
**Useful hub-side greps if something is wrong:**
|
||||||
|
* `grep -r "tx_status" controller/app/modules/screens/` — see how the hub parses your frames.
|
||||||
|
* `grep -r "tx_enabled" controller/app/modules/screens/` — see what heartbeat fields the hub reads.
|
||||||
|
* `controller/app/modules/screens/agent_ws.py:200-290` — the WS handler's JSON dispatch.
|
||||||
|
* `controller/app/modules/screens/data_sinks.py` — what the hub publishes on each control frame.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open questions (from the original plan that still apply)
|
||||||
|
|
||||||
|
Answered since the plan was written:
|
||||||
|
* ✅ **Operator name:** `PlutoTXOp` (PascalCase, stored in the hub's MongoDB `ops` collection via the application packager).
|
||||||
|
* ✅ **Redis channel naming:** kept `agent:*` prefix on the hub side — you never see this.
|
||||||
|
* ✅ **Status plumbing:** `tx_status` frames get republished on a hub-internal pub/sub and surface to the UI through the existing SSE stream. You just send the frames; the hub does the rest.
|
||||||
|
|
||||||
|
Still open (flag when you have a preference):
|
||||||
|
* **Bulk + loop fast-path.** If the hub's TX operator turns out to be a fixed recording played on loop, we could add a `{ "type": "tx_start", ..., "loop": true }` variant where the hub sends the buffer once and the agent uses the existing `tx_recording` path. Protocol-compatible with the streaming version. Defer until a real use case demands it.
|
||||||
|
* **Multi-app-per-agent.** Out of scope for v1 (§Non-goals). If/when needed: prefix binary frames with a 4-byte session header and bump a `protocol_version` in the heartbeat.
|
||||||
|
* **TX clock drift.** Relying on generous queue depth + stable local networks for v1. Longer term may need agent-side resampling.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What lives in `ria-hub` now (reference)
|
||||||
|
|
||||||
|
You don't need to read any of this, but if you're curious or need to debug the integration, these are the load-bearing bits on the hub side:
|
||||||
|
|
||||||
|
| Path | What |
|
||||||
|
|---|---|
|
||||||
|
| `controller/app/modules/screens/data_sinks.py` | `AgentTxSink`, `LocalPlutoTxSink`, `build_data_sink` |
|
||||||
|
| `controller/app/modules/screens/agent_ws.py` | `_forward_tx_binary`, heartbeat parsing, `tx_status` republish |
|
||||||
|
| `controller/app/modules/screens/graph_derivation.py` | `_pluto_tx_spec_mapping`, `_SDR_SINK_MAP`, `_derive_data_sink` |
|
||||||
|
| `controller/app/modules/screens/routes.py` | `_check_agent_tx_capability`, `AgentTxAudit` write, `POST /apps/{id}/sink-agent` |
|
||||||
|
| `controller/app/modules/screens/models.py` | `ScreensAgent` TX fields, `AgentTxAudit` document |
|
||||||
|
| `schemas/screens/app_manifest.schema.json` | `dataSink` schema block |
|
||||||
|
| `web_src/js/components/screens/components/TxConsentModal.vue` | Pre-transmit consent dialog |
|
||||||
|
| `web_src/js/components/screens/components/SinkPanel.vue` | TX-capable agent picker + live `tx_status` indicator |
|
||||||
|
| `web_src/js/components/screens/ScreensApp.vue` | Consent gate + `tx_status` forwarding to children |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Regulatory note (keep this in your docs too)
|
||||||
|
|
||||||
|
Transmission is regulated in every jurisdiction. The agent-side interlocks (`tx_enabled`, caps, freq ranges) exist so the operator can configure safe defaults for an agent's physical location. They are not a substitute for licensing or for respecting local regulations. The hub shows a consent modal and writes an audit log so actions are attributable. None of this is a legal compliance layer — it's defense-in-depth.
|
||||||
19
CHANGELOG.md
19
CHANGELOG.md
|
|
@ -1,24 +1,5 @@
|
||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
## [0.1.7] - 2026-05-26
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- **Human-readable agent names** — `ria-agent register` now generates a default `adjective-colour-animal` name (e.g. `swift-teal-falcon`) via the new `namegen` module when `--name` is omitted, instead of registering with an empty string.
|
|
||||||
- **Structured registration error messages** — `ria-agent register` translates hub responses into actionable English for the known failure reasons (`invalid_key`, `expired`, `revoked`, `already_consumed`) and rate-limit (`HTTP 429`) responses, instead of surfacing raw `HTTP 4xx` text.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- **`ria-agent register` `--api-key` help** — now describes the personal `ria_reg_*` registration key flow (minted from **Settings → RIA Agents** on the hub, shown once at mint time). The legacy shared `[wac] API_KEY` is still accepted by the hub for back-compat, but the CLI documents the per-user flow as preferred.
|
|
||||||
- **`ria-agent register` success output** — now prints both the hub-assigned agent ID and the chosen name: `Registered agent: <id> (<name>)`.
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- **`ria-agent register` blocked by Cloudflare on hubs behind it** — set an explicit `User-Agent` (`ria-agent/<package-version> (+https://riahub.ai/qoherent/ria-toolkit-oss)`) so the request isn't rejected as `Python-urllib/<ver>` (Cloudflare Browser Integrity Check returns HTTP 403, edge error code 1010). Version is read from package metadata so it tracks releases automatically.
|
|
||||||
- **`ria-agent register` could hang indefinitely** — added a 15-second timeout to the hub request; previously `urllib`'s default of no timeout meant a stuck hub would block the CLI forever.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## [0.1.0] - 2026-02-20
|
## [0.1.0] - 2026-02-20
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
|
||||||
|
|
@ -159,7 +159,7 @@ Finally, RIA Toolkit OSS can be installed directly from the source code. This ap
|
||||||
Once the project is installed, you can import modules, functions, and classes from the Toolkit for use in your Python code. For example, you can use the following import statement to access the `Recording` object:
|
Once the project is installed, you can import modules, functions, and classes from the Toolkit for use in your Python code. For example, you can use the following import statement to access the `Recording` object:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
```
|
```
|
||||||
|
|
||||||
Additional usage information is provided in the project documentation: [RIA Toolkit OSS Documentation](https://ria-toolkit-oss.readthedocs.io/).
|
Additional usage information is provided in the project documentation: [RIA Toolkit OSS Documentation](https://ria-toolkit-oss.readthedocs.io/).
|
||||||
|
|
@ -185,7 +185,7 @@ RIA Toolkit OSS is developed and maintained by [Qoherent](https://qoherent.ai/),
|
||||||
If you are doing research with RIA Toolkit OSS, please cite the project:
|
If you are doing research with RIA Toolkit OSS, please cite the project:
|
||||||
|
|
||||||
```
|
```
|
||||||
[1] Qoherent Inc., "Radio Intelligence Apps Toolkit OSS," 2026. [Online]. Available: https://riahub.ai/qoherent/ria-toolkit-oss
|
[1] Qoherent Inc., "Radio Intelligence Apps Toolkit OSS," 2025. [Online]. Available: https://riahub.ai/qoherent/ria-toolkit-oss
|
||||||
```
|
```
|
||||||
|
|
||||||
If you like what we're doing, don't forget to give the project a star! ⭐
|
If you like what we're doing, don't forget to give the project a star! ⭐
|
||||||
|
|
|
||||||
|
|
@ -6,10 +6,10 @@ This is a practical reference for the ``ria`` CLI from ``ria-toolkit-oss``.
|
||||||
|
|
||||||
**Scope of this guide:**
|
**Scope of this guide:**
|
||||||
|
|
||||||
* **Installation and SDR driver prerequisites** — how to install RIA Toolkit OSS and configure the system drivers your hardware requires
|
* Installation and setup
|
||||||
* **End-to-end CLI workflow** — a step-by-step walkthrough from hardware discovery through capture, annotation, and processing
|
* End-to-end CLI workflows
|
||||||
* **Full command reference** — options, flags, and examples for every ``ria`` command
|
* Full command reference for CLI features
|
||||||
* **Python scripting preview** — using the toolkit API directly without the CLI
|
* Brief scripting section
|
||||||
|
|
||||||
**Official resources:**
|
**Official resources:**
|
||||||
|
|
||||||
|
|
@ -18,15 +18,76 @@ This is a practical reference for the ``ria`` CLI from ``ria-toolkit-oss``.
|
||||||
* `PyPI package <https://pypi.org/project/ria-toolkit-oss/>`_
|
* `PyPI package <https://pypi.org/project/ria-toolkit-oss/>`_
|
||||||
* `RIA Hub Conda package <https://riahub.ai/qoherent/-/packages/conda/ria-toolkit-oss>`_
|
* `RIA Hub Conda package <https://riahub.ai/qoherent/-/packages/conda/ria-toolkit-oss>`_
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
:depth: 2
|
||||||
|
:backlinks: none
|
||||||
|
|
||||||
|
|
||||||
1) Installation and Setup
|
1) Installation and Setup
|
||||||
==========================
|
==========================
|
||||||
|
|
||||||
Before using the ``ria`` CLI, follow the :doc:`Installation <installation>` guide to
|
1.1 Installation with Conda
|
||||||
install RIA Toolkit OSS and any SDR drivers required for your hardware.
|
----------------------------
|
||||||
|
|
||||||
|
RIA Toolkit OSS is available as a Conda package on RIA Hub. This is typically the easiest
|
||||||
|
path when using SDR tooling that depends on native/system libraries.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
conda update --force conda
|
||||||
|
conda config --add channels https://riahub.ai/api/packages/qoherent/conda
|
||||||
|
conda activate base
|
||||||
|
conda install ria-toolkit-oss
|
||||||
|
|
||||||
|
Verify:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
conda list | grep ria-toolkit-oss
|
||||||
|
|
||||||
|
|
||||||
1.1 SDR driver prerequisites
|
1.2 Installation with pip
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
Use pip unless you specifically need to edit toolkit source.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
python3 -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install --upgrade pip
|
||||||
|
pip install ria-toolkit-oss
|
||||||
|
|
||||||
|
Verify CLI entrypoint:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
ria --help
|
||||||
|
|
||||||
|
``pyproject.toml`` defines two script entry points:
|
||||||
|
|
||||||
|
* ``ria``
|
||||||
|
* ``ria-tools``
|
||||||
|
|
||||||
|
Both point to the same CLI module (``ria_toolkit_oss_cli.cli:cli``).
|
||||||
|
|
||||||
|
|
||||||
|
1.3 Optional install from source
|
||||||
|
----------------------------------
|
||||||
|
|
||||||
|
Use this for local development or testing unreleased changes.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
git clone https://riahub.ai/qoherent/ria-toolkit-oss.git
|
||||||
|
cd ria-toolkit-oss
|
||||||
|
python3 -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install -e .
|
||||||
|
|
||||||
|
|
||||||
|
1.4 SDR driver prerequisites
|
||||||
-----------------------------
|
-----------------------------
|
||||||
|
|
||||||
Toolkit package install does not install all system SDR drivers. Install vendor/runtime
|
Toolkit package install does not install all system SDR drivers. Install vendor/runtime
|
||||||
|
|
@ -34,22 +95,11 @@ dependencies for the hardware you use.
|
||||||
|
|
||||||
Examples (depends on device and OS):
|
Examples (depends on device and OS):
|
||||||
|
|
||||||
.. list-table::
|
* USRP: UHD drivers
|
||||||
:widths: 25 75
|
* Pluto: libiio / IIO utilities
|
||||||
:header-rows: 1
|
* BladeRF: libbladeRF
|
||||||
|
* HackRF: libhackrf
|
||||||
* - Device
|
* RTL-SDR: librtlsdr
|
||||||
- Driver Package
|
|
||||||
* - USRP
|
|
||||||
- UHD drivers
|
|
||||||
* - Pluto
|
|
||||||
- libiio / IIO utilities
|
|
||||||
* - BladeRF
|
|
||||||
- libbladeRF
|
|
||||||
* - HackRF
|
|
||||||
- libhackrf
|
|
||||||
* - RTL-SDR
|
|
||||||
- librtlsdr
|
|
||||||
|
|
||||||
See repo docs under ``docs/source/sdr_guides/*`` and your OS package instructions.
|
See repo docs under ``docs/source/sdr_guides/*`` and your OS package instructions.
|
||||||
|
|
||||||
|
|
@ -69,34 +119,18 @@ Top-level CLI follows this model:
|
||||||
|
|
||||||
**Top-level commands:**
|
**Top-level commands:**
|
||||||
|
|
||||||
.. list-table::
|
* ``discover``
|
||||||
:widths: 25 75
|
* ``init``
|
||||||
:header-rows: 1
|
* ``capture``
|
||||||
|
* ``view``
|
||||||
* - Command
|
* ``annotate`` (group)
|
||||||
- Purpose
|
* ``convert``
|
||||||
* - :ref:`discover <cmd-discover>`
|
* ``split``
|
||||||
- Probe SDR drivers and enumerate attached hardware
|
* ``combine``
|
||||||
* - :ref:`init <cmd-init>`
|
* ``generate`` (group)
|
||||||
- Create and manage user metadata defaults
|
* ``transform`` (group)
|
||||||
* - :ref:`capture <cmd-capture>`
|
* ``transmit``
|
||||||
- Record IQ samples from a connected SDR
|
* ``synth`` (alias of ``generate`` in command bindings)
|
||||||
* - :ref:`view <cmd-view>`
|
|
||||||
- Generate visualizations from IQ files
|
|
||||||
* - :ref:`annotate <cmd-annotate>`
|
|
||||||
- Label signal regions manually or with auto-detection (group)
|
|
||||||
* - :ref:`convert <cmd-convert>`
|
|
||||||
- Convert between IQ file formats
|
|
||||||
* - :ref:`split <cmd-split>`
|
|
||||||
- Split, trim, or extract recordings
|
|
||||||
* - :ref:`combine <cmd-combine>`
|
|
||||||
- Merge multiple recordings by concatenation or addition
|
|
||||||
* - :ref:`generate / synth <cmd-generate>`
|
|
||||||
- Generate synthetic IQ signals (group; ``synth`` is an alias)
|
|
||||||
* - :ref:`transform <cmd-transform>`
|
|
||||||
- Apply augmentations or impairments to recordings (group)
|
|
||||||
* - :ref:`transmit <cmd-transmit>`
|
|
||||||
- Transmit IQ through a TX-capable SDR
|
|
||||||
|
|
||||||
|
|
||||||
3) Quick End-to-End Workflow
|
3) Quick End-to-End Workflow
|
||||||
|
|
@ -124,8 +158,10 @@ provenance fields.
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
ria init
|
ria init
|
||||||
ria init --author "Jane Doe" --project "rf-campaign-1" --location "Lab-A" # non-interactive
|
# or non-interactive
|
||||||
ria init --show # show config
|
ria init --author "Jane Doe" --project "rf-campaign-1" --location "Lab-A"
|
||||||
|
# show config
|
||||||
|
ria init --show
|
||||||
|
|
||||||
|
|
||||||
3.3 Capture IQ
|
3.3 Capture IQ
|
||||||
|
|
@ -191,14 +227,13 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
ria transmit -d hackrf -f 2.44G -s 2e6 --input capture.sigmf-data
|
ria transmit -d hackrf -f 2.44G -s 2e6 --input capture.sigmf-data
|
||||||
ria transmit -d hackrf --generate lfm --continuous # generated waveform
|
# or generated waveform
|
||||||
|
ria transmit -d hackrf --generate lfm --continuous
|
||||||
|
|
||||||
|
|
||||||
4) Command Reference
|
4) Command Reference
|
||||||
=====================
|
=====================
|
||||||
|
|
||||||
.. _cmd-discover:
|
|
||||||
|
|
||||||
4.1 ``discover``
|
4.1 ``discover``
|
||||||
-----------------
|
-----------------
|
||||||
|
|
||||||
|
|
@ -228,8 +263,6 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
||||||
hidden in default output.
|
hidden in default output.
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-init:
|
|
||||||
|
|
||||||
4.2 ``init``
|
4.2 ``init``
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
|
|
@ -276,8 +309,6 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
||||||
generate metadata, and YAML config loading paths).
|
generate metadata, and YAML config loading paths).
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-capture:
|
|
||||||
|
|
||||||
4.3 ``capture``
|
4.3 ``capture``
|
||||||
----------------
|
----------------
|
||||||
|
|
||||||
|
|
@ -351,8 +382,6 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
||||||
ria capture -c capture_config.yaml
|
ria capture -c capture_config.yaml
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-view:
|
|
||||||
|
|
||||||
4.4 ``view``
|
4.4 ``view``
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
|
|
@ -413,21 +442,7 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
||||||
ria view capture.npy --type full --title "Test Capture" --format pdf
|
ria view capture.npy --type full --title "Test Capture" --format pdf
|
||||||
ria view capture.npy --show --no-save
|
ria view capture.npy --show --no-save
|
||||||
ria view old.npy --legacy --type simple
|
ria view old.npy --legacy --type simple
|
||||||
ria view recordings\qam64_35.npy --type simple
|
|
||||||
ria view recordings\qam64_35.npy --type full
|
|
||||||
|
|
||||||
.. figure:: ../images/recordings/qam64_35.png
|
|
||||||
:alt: Example output of ria view recordings\qam64_35.npy --type simple
|
|
||||||
|
|
||||||
Output of ``ria view recordings\qam64_35.npy --type simple``
|
|
||||||
|
|
||||||
.. figure:: ../images/recordings/qam64_35-full.png
|
|
||||||
:alt: Example output of ria view recordings\qam64_35.npy --type full
|
|
||||||
|
|
||||||
Output of ``ria view recordings\qam64_35.npy --type full``
|
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-annotate:
|
|
||||||
|
|
||||||
4.5 ``annotate`` group
|
4.5 ``annotate`` group
|
||||||
-----------------------
|
-----------------------
|
||||||
|
|
@ -444,30 +459,8 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
||||||
|
|
||||||
ria annotate <subcommand> ...
|
ria annotate <subcommand> ...
|
||||||
|
|
||||||
**Subcommands:**
|
**Subcommands:** ``list``, ``add``, ``remove``, ``clear``, ``energy``, ``cusum``,
|
||||||
|
``threshold``, ``separate``
|
||||||
.. list-table::
|
|
||||||
:widths: 25 75
|
|
||||||
:header-rows: 1
|
|
||||||
|
|
||||||
* - Subcommand
|
|
||||||
- Purpose
|
|
||||||
* - ``list``
|
|
||||||
- Inspect all annotations on a recording
|
|
||||||
* - ``add``
|
|
||||||
- Add one annotation with explicit sample-domain bounds
|
|
||||||
* - ``remove``
|
|
||||||
- Remove one annotation by index
|
|
||||||
* - ``clear``
|
|
||||||
- Remove all annotations from a recording
|
|
||||||
* - ``energy``
|
|
||||||
- Auto-detect regions above the estimated noise floor
|
|
||||||
* - ``cusum``
|
|
||||||
- Auto-detect regime changes using change-point detection
|
|
||||||
* - ``threshold``
|
|
||||||
- Auto-detect regions using normalized magnitude thresholding
|
|
||||||
* - ``separate``
|
|
||||||
- Decompose annotations into narrower spectral components
|
|
||||||
|
|
||||||
**General behavior:**
|
**General behavior:**
|
||||||
|
|
||||||
|
|
@ -594,16 +587,8 @@ annotations.
|
||||||
ria annotate add capture.sigmf-data --start 10000 --count 5000 --label burst
|
ria annotate add capture.sigmf-data --start 10000 --count 5000 --label burst
|
||||||
ria annotate energy capture.sigmf-data --label signal --threshold 1.3
|
ria annotate energy capture.sigmf-data --label signal --threshold 1.3
|
||||||
ria annotate cusum capture.sigmf-data --min-duration 5
|
ria annotate cusum capture.sigmf-data --min-duration 5
|
||||||
ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%
|
|
||||||
ria annotate separate capture.sigmf-data --indices 0,1 --verbose
|
ria annotate separate capture.sigmf-data --indices 0,1 --verbose
|
||||||
|
|
||||||
.. figure:: ../images/recordings/sample_recording3_annotated.png
|
|
||||||
:alt: Example output of ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%
|
|
||||||
|
|
||||||
Output of ``ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%``
|
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-convert:
|
|
||||||
|
|
||||||
4.6 ``convert``
|
4.6 ``convert``
|
||||||
----------------
|
----------------
|
||||||
|
|
@ -644,8 +629,6 @@ inferred from the output file extension.
|
||||||
ria convert old.npy --format sigmf --legacy --overwrite
|
ria convert old.npy --format sigmf --legacy --overwrite
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-split:
|
|
||||||
|
|
||||||
4.7 ``split``
|
4.7 ``split``
|
||||||
--------------
|
--------------
|
||||||
|
|
||||||
|
|
@ -687,8 +670,6 @@ Choose exactly one operation per invocation:
|
||||||
ria split annotated.sigmf-data --extract-annotations --annotation-label payload
|
ria split annotated.sigmf-data --extract-annotations --annotation-label payload
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-combine:
|
|
||||||
|
|
||||||
4.8 ``combine``
|
4.8 ``combine``
|
||||||
----------------
|
----------------
|
||||||
|
|
||||||
|
|
@ -736,8 +717,6 @@ Choose exactly one operation per invocation:
|
||||||
ria combine signal.npy pattern.npy out.npy --mode add --align-mode repeat-spaced --repeat-spacing 10000
|
ria combine signal.npy pattern.npy out.npy --mode add --align-mode repeat-spaced --repeat-spacing 10000
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-generate:
|
|
||||||
|
|
||||||
4.9 ``generate`` group (and ``synth`` alias)
|
4.9 ``generate`` group (and ``synth`` alias)
|
||||||
---------------------------------------------
|
---------------------------------------------
|
||||||
|
|
||||||
|
|
@ -749,34 +728,15 @@ Choose exactly one operation per invocation:
|
||||||
|
|
||||||
``ria synth ...`` is an alias for ``ria generate ...``.
|
``ria synth ...`` is an alias for ``ria generate ...``.
|
||||||
|
|
||||||
**Usage:**
|
**Shape:**
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
ria generate <subcommand> [subcommand options] [common options]
|
ria generate <subcommand> [subcommand options] [common options]
|
||||||
|
|
||||||
**Available subcommands:**
|
**Available subcommands:**
|
||||||
|
``tone``, ``noise``, ``chirp``, ``square``, ``sawtooth``, ``qam``, ``apsk``, ``pam``,
|
||||||
.. list-table::
|
``fsk``, ``ook``, ``oqpsk``, ``gmsk``, ``psk``
|
||||||
:widths: 30 70
|
|
||||||
:header-rows: 1
|
|
||||||
|
|
||||||
* - Subcommand(s)
|
|
||||||
- Description
|
|
||||||
* - ``tone``
|
|
||||||
- Clean sinusoidal calibration/reference source
|
|
||||||
* - ``noise``
|
|
||||||
- Baseline noise floor data or controlled additive-noise synthesis
|
|
||||||
* - ``chirp``
|
|
||||||
- Sweep-based radar/sonar-style signals and bandwidth occupancy tests
|
|
||||||
* - ``square``, ``sawtooth``
|
|
||||||
- Periodic waveform primitives
|
|
||||||
* - ``qam``, ``apsk``, ``pam``, ``psk``
|
|
||||||
- Digital modulation families with pulse-shaping filter support
|
|
||||||
* - ``fsk``
|
|
||||||
- Frequency-shift keying with configurable tone spacing
|
|
||||||
* - ``ook``, ``oqpsk``, ``gmsk``
|
|
||||||
- On-off keying and continuous-phase modulation schemes
|
|
||||||
|
|
||||||
**Common options shared across all generators:**
|
**Common options shared across all generators:**
|
||||||
|
|
||||||
|
|
@ -800,16 +760,22 @@ Multipath and IQ imbalance flags apply impairment-style post-processing during g
|
||||||
|
|
||||||
Options: ``--frequency``, ``--amplitude``, ``--phase``
|
Options: ``--frequency``, ``--amplitude``, ``--phase``
|
||||||
|
|
||||||
|
Clean sinusoidal calibration/reference source.
|
||||||
|
|
||||||
``noise``
|
``noise``
|
||||||
~~~~~~~~~~
|
~~~~~~~~~~
|
||||||
|
|
||||||
Options: ``--noise-type {gaussian,uniform}``, ``--power``
|
Options: ``--noise-type {gaussian,uniform}``, ``--power``
|
||||||
|
|
||||||
|
Baseline noise floor data or controlled additive-noise synthesis.
|
||||||
|
|
||||||
``chirp``
|
``chirp``
|
||||||
~~~~~~~~~~
|
~~~~~~~~~~
|
||||||
|
|
||||||
Options: ``--bandwidth`` (required), ``--period`` (required), ``--type {up,down,up_down}``
|
Options: ``--bandwidth`` (required), ``--period`` (required), ``--type {up,down,up_down}``
|
||||||
|
|
||||||
|
Sweep-based radar/sonar-style signals and bandwidth occupancy tests.
|
||||||
|
|
||||||
``square``
|
``square``
|
||||||
~~~~~~~~~~~
|
~~~~~~~~~~~
|
||||||
|
|
||||||
|
|
@ -860,8 +826,6 @@ symbol transition sharpness).
|
||||||
ria synth psk -s 2e6 -r 100e3 -M 8 -N 8000 -o psk8.npy
|
ria synth psk -s 2e6 -r 100e3 -M 8 -N 8000 -o psk8.npy
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-transform:
|
|
||||||
|
|
||||||
4.10 ``transform`` group
|
4.10 ``transform`` group
|
||||||
-------------------------
|
-------------------------
|
||||||
|
|
||||||
|
|
@ -870,7 +834,7 @@ symbol transition sharpness).
|
||||||
* Apply algorithmic transforms to existing recordings.
|
* Apply algorithmic transforms to existing recordings.
|
||||||
* Run reusable augmentations/impairments for dataset diversity and robustness testing.
|
* Run reusable augmentations/impairments for dataset diversity and robustness testing.
|
||||||
|
|
||||||
**Usage:**
|
**Shape:**
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
|
|
@ -931,8 +895,6 @@ inspect parameter hints. ``--view`` writes a PNG preview alongside transform out
|
||||||
ria transform custom my_filter in.npy out.npy --transform-dir ./my_transforms --params cutoff=0.2
|
ria transform custom my_filter in.npy out.npy --transform-dir ./my_transforms --params cutoff=0.2
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-transmit:
|
|
||||||
|
|
||||||
4.11 ``transmit``
|
4.11 ``transmit``
|
||||||
------------------
|
------------------
|
||||||
|
|
||||||
|
|
@ -1031,7 +993,17 @@ experiment-specific fields on the CLI.
|
||||||
ria generate noise --config generate.yaml
|
ria generate noise --config generate.yaml
|
||||||
|
|
||||||
|
|
||||||
6) Version Notes
|
6) Practical Tips and Safety
|
||||||
|
=============================
|
||||||
|
|
||||||
|
* Use ``ria discover`` before capture/transmit sessions.
|
||||||
|
* Keep TX gain conservative first; validate with attenuators/dummy loads when needed.
|
||||||
|
* Prefer SigMF for interoperable metadata and annotations.
|
||||||
|
* For long workflows, keep outputs organized by campaign directories and consistent prefixes.
|
||||||
|
* Use ``--verbose`` when debugging device init or driver issues.
|
||||||
|
|
||||||
|
|
||||||
|
7) Version Notes
|
||||||
=================
|
=================
|
||||||
|
|
||||||
These notes are based on the current implementation and should be re-validated against future
|
These notes are based on the current implementation and should be re-validated against future
|
||||||
|
|
@ -1044,19 +1016,18 @@ releases.
|
||||||
3. Multiple non-CLI modules still import ``utils.*``, which can create runtime dependency
|
3. Multiple non-CLI modules still import ``utils.*``, which can create runtime dependency
|
||||||
coupling when using only ``ria-toolkit-oss`` in isolation.
|
coupling when using only ``ria-toolkit-oss`` in isolation.
|
||||||
|
|
||||||
.. tip::
|
If you observe unexpected import errors after install, check the package version and
|
||||||
If you observe unexpected import errors after install, check the package version and
|
changelog, then test ``ria --help`` in a clean virtual environment.
|
||||||
changelog, then test ``ria --help`` in a clean virtual environment.
|
|
||||||
|
|
||||||
|
|
||||||
7) Brief Scripting (Python) Preview
|
8) Brief Scripting (Python) Preview
|
||||||
=====================================
|
=====================================
|
||||||
|
|
||||||
For quick non-CLI use:
|
For quick non-CLI use:
|
||||||
|
|
||||||
.. code-block:: python
|
.. code-block:: python
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
from ria_toolkit_oss.io import load_recording, to_sigmf
|
from ria_toolkit_oss.io import load_recording, to_sigmf
|
||||||
from ria_toolkit_oss.transforms import iq_augmentations, iq_impairments
|
from ria_toolkit_oss.transforms import iq_augmentations, iq_impairments
|
||||||
|
|
||||||
|
|
@ -1066,3 +1037,47 @@ For quick non-CLI use:
|
||||||
to_sigmf(imp, filename="capture_awgn", path=".")
|
to_sigmf(imp, filename="capture_awgn", path=".")
|
||||||
|
|
||||||
You can also call annotation algorithms and block-generator primitives from Python directly.
|
You can also call annotation algorithms and block-generator primitives from Python directly.
|
||||||
|
|
||||||
|
|
||||||
|
9) Cheat Sheet
|
||||||
|
===============
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
# Install
|
||||||
|
pip install ria-toolkit-oss
|
||||||
|
|
||||||
|
# Discover
|
||||||
|
ria discover -v
|
||||||
|
|
||||||
|
# Init defaults
|
||||||
|
ria init --author "Jane" --project "rf1" --location "Lab-A"
|
||||||
|
|
||||||
|
# Capture
|
||||||
|
ria capture -d pluto -f 2.44G -s 2e6 -n 1000000 -o cap.sigmf-data
|
||||||
|
|
||||||
|
# View
|
||||||
|
ria view cap.sigmf-data --type simple
|
||||||
|
|
||||||
|
# Annotate
|
||||||
|
ria annotate energy cap.sigmf-data --threshold 1.2
|
||||||
|
ria annotate list cap.sigmf-data --verbose
|
||||||
|
|
||||||
|
# Convert
|
||||||
|
ria convert cap.sigmf-data cap.npy
|
||||||
|
|
||||||
|
# Split
|
||||||
|
ria split cap.sigmf-data --split-every 100000 --output-dir chunks
|
||||||
|
|
||||||
|
# Combine
|
||||||
|
ria combine chunks/a.npy chunks/b.npy merged.npy
|
||||||
|
|
||||||
|
# Generate
|
||||||
|
ria generate qam -s 2e6 -r 100e3 -M 16 -N 5000 -o qam16.npy
|
||||||
|
|
||||||
|
# Transform
|
||||||
|
ria transform augment channel_swap cap.npy
|
||||||
|
ria transform impair add_awgn_to_signal cap.npy --params snr=10
|
||||||
|
|
||||||
|
# Transmit
|
||||||
|
ria transmit -d hackrf --input cap.sigmf-data -f 2.44G -s 2e6
|
||||||
|
|
|
||||||
568
docs/agent_tx_implementation_plan.md
Normal file
568
docs/agent_tx_implementation_plan.md
Normal file
|
|
@ -0,0 +1,568 @@
|
||||||
|
# Agent TX Streaming — Implementation Plan (`ria-toolkit-oss`)
|
||||||
|
|
||||||
|
**Scope:** Part A of [agent_tx_plan.md](./agent_tx_plan.md). This repo only.
|
||||||
|
**Goal:** Make the agent accept hub-originated TX control + binary IQ, stream it to the SDR in full duplex with RX, and enforce agent-local safety caps.
|
||||||
|
**Acceptance:** `pytest tests/agent/` green; `ria-agent stream --allow-tx` accepts a `tx_start` against MockSDR and round-trips binary frames to `_stream_tx`.
|
||||||
|
|
||||||
|
Each phase below lands independently. After every phase the existing agent tests must still pass (no regressions), and the phase's own new tests must be green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Preconditions
|
||||||
|
|
||||||
|
- `--allow-tx` is opt-in at CLI level. Default config has `tx_enabled=False`; the agent will reject all TX control frames from the hub.
|
||||||
|
- Pluto FDD: one `adi.Pluto` instance serves both RX and TX. We share the SDR between sessions keyed by `(device, identifier)`.
|
||||||
|
- Known pre-existing bug: [`sdr/pluto.py:151`](../src/ria_toolkit_oss/sdr/pluto.py#L151) sets `_rx_initialized = False` inside `init_tx`. Our streamer's RX path (`sdr.rx(n)`) does not read this flag, so FDD still works. Leave the bug for a separate follow-up; do not refactor Pluto in this plan.
|
||||||
|
|
||||||
|
## Glossary
|
||||||
|
|
||||||
|
- **Session** = a `(app_id, direction)` pair held by the agent: one `RxSession` or one `TxSession`.
|
||||||
|
- **Direction** = `"rx"` (agent → hub binary) or `"tx"` (hub → agent binary).
|
||||||
|
- **Shared SDR** = when the same `(device, identifier)` is referenced by an RX and TX session concurrently; both sessions hold the same driver instance.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — WS binary ingress
|
||||||
|
|
||||||
|
**Why first:** protocol-plumbing only. No behavior change for existing RX, but unblocks every later phase.
|
||||||
|
|
||||||
|
### Touches
|
||||||
|
|
||||||
|
- `src/ria_toolkit_oss/agent/ws_client.py` — add optional `on_binary` callback to `WsClient.run`.
|
||||||
|
- `tests/agent/test_ws_client.py` — add a "server sends binary, handler receives" case.
|
||||||
|
|
||||||
|
### Shape
|
||||||
|
|
||||||
|
```python
|
||||||
|
# ws_client.py
|
||||||
|
BinaryHandler = Callable[[bytes], Awaitable[None]]
|
||||||
|
|
||||||
|
async def run(
|
||||||
|
self,
|
||||||
|
on_message: MessageHandler,
|
||||||
|
heartbeat: HeartbeatBuilder,
|
||||||
|
on_binary: BinaryHandler | None = None, # NEW, default preserves old behavior
|
||||||
|
) -> None:
|
||||||
|
...
|
||||||
|
async for raw in self._ws:
|
||||||
|
if isinstance(raw, bytes):
|
||||||
|
if on_binary is not None:
|
||||||
|
try:
|
||||||
|
await on_binary(raw)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("on_binary handler raised; dropping frame")
|
||||||
|
else:
|
||||||
|
logger.debug("Discarding unexpected %d-byte binary frame", len(raw))
|
||||||
|
continue
|
||||||
|
# ... existing JSON dispatch unchanged
|
||||||
|
```
|
||||||
|
|
||||||
|
### Acceptance
|
||||||
|
|
||||||
|
- New test: local `websockets` server pushes a binary frame after JSON handshake → handler sees exact bytes.
|
||||||
|
- Existing `test_ws_client.py` cases still pass with `on_binary=None`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2 — Config + CLI TX opt-in
|
||||||
|
|
||||||
|
**Why second:** small, isolated, and gives the rest of the phases a real `AgentConfig.tx_enabled` / caps to read.
|
||||||
|
|
||||||
|
### Touches
|
||||||
|
|
||||||
|
- `src/ria_toolkit_oss/agent/config.py`
|
||||||
|
- `src/ria_toolkit_oss/agent/cli.py`
|
||||||
|
- `tests/agent/test_config.py`
|
||||||
|
- new `tests/agent/test_cli_tx.py`
|
||||||
|
|
||||||
|
### config.py
|
||||||
|
|
||||||
|
Extend the dataclass and preserve backward-compat for old JSON files (the existing `extra` trick already handles unknown keys, but we want these fields promoted to first-class):
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class AgentConfig:
|
||||||
|
hub_url: str = ""
|
||||||
|
agent_id: str = ""
|
||||||
|
token: str = ""
|
||||||
|
name: str = ""
|
||||||
|
insecure: bool = False
|
||||||
|
api_key: str = ""
|
||||||
|
# NEW — TX interlocks
|
||||||
|
tx_enabled: bool = False
|
||||||
|
tx_max_gain_db: float | None = None
|
||||||
|
tx_max_duration_s: float | None = None
|
||||||
|
tx_allowed_freq_ranges: list[list[float]] | None = None # JSON-friendly list-of-lists
|
||||||
|
extra: dict = field(default_factory=dict)
|
||||||
|
```
|
||||||
|
|
||||||
|
Update `load()` to pull the new fields and `save()` to emit them. Preserve the `0o600` chmod behavior.
|
||||||
|
|
||||||
|
### cli.py
|
||||||
|
|
||||||
|
Two entry points need flags:
|
||||||
|
|
||||||
|
```
|
||||||
|
ria-agent register --hub ... --api-key ...
|
||||||
|
[--allow-tx]
|
||||||
|
[--tx-max-gain-db VALUE]
|
||||||
|
[--tx-max-duration-s VALUE]
|
||||||
|
[--tx-freq-range LO HI] # repeatable: --tx-freq-range 2.4e9 2.5e9 --tx-freq-range 5.7e9 5.8e9
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
ria-agent stream
|
||||||
|
[--allow-tx] # runtime override: sets cfg.tx_enabled for this process only
|
||||||
|
```
|
||||||
|
|
||||||
|
In `_cmd_register`: after successful server registration, populate `cfg.tx_enabled=bool(args.allow_tx)` and caps from argparse before `_config.save(cfg)`.
|
||||||
|
|
||||||
|
In `_cmd_stream`: `if args.allow_tx: cfg.tx_enabled = True` (before passing `cfg` to the streamer — which requires plumbing `cfg` in, see Phase 3).
|
||||||
|
|
||||||
|
### Acceptance
|
||||||
|
|
||||||
|
- `test_config.py` round-trip: new fields serialize → deserialize cleanly; missing fields in old JSON default correctly.
|
||||||
|
- `test_cli_tx.py`: `register --allow-tx --tx-max-gain-db -10` writes expected JSON; `stream --allow-tx` sets runtime flag without touching disk.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3 — Streamer refactor: session model (RX behavior preserved)
|
||||||
|
|
||||||
|
**Why third:** the TX work needs a session-based state machine. Doing the refactor before wiring TX keeps the diff reviewable and keeps the RX regression surface contained.
|
||||||
|
|
||||||
|
**Goal:** replace the flat state (`self._sdr`, `self._app_id`, `self._capture_task`, `self._pending_config`, `self._status`) with explicit session objects and an SDR registry, without changing any observable RX behavior.
|
||||||
|
|
||||||
|
### Touches
|
||||||
|
|
||||||
|
- `src/ria_toolkit_oss/agent/streamer.py` (bulk of work)
|
||||||
|
- `src/ria_toolkit_oss/agent/hardware.py` (heartbeat grows `capabilities` + optional `sessions` snapshot)
|
||||||
|
- `src/ria_toolkit_oss/agent/cli.py` (plumb `cfg` into the streamer)
|
||||||
|
- `tests/agent/test_streamer.py` + `test_hardware.py` — update for new heartbeat shape; keep all RX assertions.
|
||||||
|
|
||||||
|
### Data model
|
||||||
|
|
||||||
|
```python
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class RxSession:
|
||||||
|
app_id: str
|
||||||
|
sdr: Any
|
||||||
|
device_key: tuple[str, str | None] # (device, identifier)
|
||||||
|
buffer_size: int
|
||||||
|
task: asyncio.Task
|
||||||
|
pending_config: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TxSession:
|
||||||
|
app_id: str
|
||||||
|
sdr: Any
|
||||||
|
device_key: tuple[str, str | None]
|
||||||
|
buffer_size: int
|
||||||
|
queue: queue.Queue # thread-safe; bytes -> np.complex64 buffers
|
||||||
|
stop_event: threading.Event
|
||||||
|
task: asyncio.Task # wraps run_in_executor(sdr._stream_tx, ...)
|
||||||
|
underrun_policy: str = "pause"
|
||||||
|
pending_config: dict = field(default_factory=dict)
|
||||||
|
last_buffer: np.ndarray | None = None # for "repeat" policy
|
||||||
|
started_at: float = 0.0
|
||||||
|
max_duration_s: float | None = None
|
||||||
|
```
|
||||||
|
|
||||||
|
### SDR registry (ref-counted)
|
||||||
|
|
||||||
|
```python
|
||||||
|
class _SdrRegistry:
|
||||||
|
def __init__(self, factory):
|
||||||
|
self._factory = factory # (device, identifier) -> SDR
|
||||||
|
self._instances: dict[tuple[str, str|None], tuple[Any, int]] = {}
|
||||||
|
self._lock = threading.Lock()
|
||||||
|
|
||||||
|
def acquire(self, device: str, identifier: str | None):
|
||||||
|
key = (device, identifier)
|
||||||
|
with self._lock:
|
||||||
|
if key in self._instances:
|
||||||
|
sdr, rc = self._instances[key]
|
||||||
|
self._instances[key] = (sdr, rc + 1)
|
||||||
|
return sdr, key
|
||||||
|
sdr = self._factory(device, identifier)
|
||||||
|
self._instances[key] = (sdr, 1)
|
||||||
|
return sdr, key
|
||||||
|
|
||||||
|
def release(self, key: tuple[str, str|None]) -> bool:
|
||||||
|
with self._lock:
|
||||||
|
sdr, rc = self._instances[key]
|
||||||
|
if rc <= 1:
|
||||||
|
del self._instances[key]
|
||||||
|
return True # caller should close()
|
||||||
|
self._instances[key] = (sdr, rc - 1)
|
||||||
|
return False
|
||||||
|
```
|
||||||
|
|
||||||
|
### Streamer state
|
||||||
|
|
||||||
|
```python
|
||||||
|
class Streamer:
|
||||||
|
def __init__(self, ws, cfg: AgentConfig, sdr_factory=None):
|
||||||
|
self.ws = ws
|
||||||
|
self._cfg = cfg
|
||||||
|
self._registry = _SdrRegistry(sdr_factory or _default_sdr_factory)
|
||||||
|
self._rx: RxSession | None = None
|
||||||
|
self._tx: TxSession | None = None
|
||||||
|
```
|
||||||
|
|
||||||
|
### Message dispatch
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def on_message(self, msg: dict) -> None:
|
||||||
|
t = msg.get("type")
|
||||||
|
handlers = {
|
||||||
|
"start": self._handle_rx_start,
|
||||||
|
"stop": self._handle_rx_stop,
|
||||||
|
"configure": self._handle_rx_configure,
|
||||||
|
# TX handlers stubbed here in Phase 3, implemented in Phase 4
|
||||||
|
"tx_start": self._handle_tx_start,
|
||||||
|
"tx_stop": self._handle_tx_stop,
|
||||||
|
"tx_configure": self._handle_tx_configure,
|
||||||
|
}
|
||||||
|
handler = handlers.get(t)
|
||||||
|
if handler is None:
|
||||||
|
logger.warning("Unknown server message type: %r", t)
|
||||||
|
return
|
||||||
|
await handler(msg)
|
||||||
|
```
|
||||||
|
|
||||||
|
Rename internals: `_handle_start → _handle_rx_start`, `_handle_stop → _handle_rx_stop`, etc. Behavior unchanged — just reading/writing `self._rx` in place of the old flat attributes, and going through the registry for acquire/release.
|
||||||
|
|
||||||
|
### Heartbeat
|
||||||
|
|
||||||
|
```python
|
||||||
|
# streamer.py
|
||||||
|
def build_heartbeat(self) -> dict:
|
||||||
|
status = "streaming" if (self._rx or self._tx) else "idle"
|
||||||
|
sessions: dict = {}
|
||||||
|
if self._rx: sessions["rx"] = {"app_id": self._rx.app_id, "state": "streaming"}
|
||||||
|
if self._tx: sessions["tx"] = {"app_id": self._tx.app_id, "state": self._tx_state()}
|
||||||
|
return heartbeat_payload(
|
||||||
|
status=status,
|
||||||
|
app_id=(self._rx or self._tx).app_id if (self._rx or self._tx) else None,
|
||||||
|
cfg=self._cfg,
|
||||||
|
sessions=sessions or None,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Update `hardware.heartbeat_payload` to take `cfg` (for `capabilities`/`tx_enabled`) and optional `sessions`. Keep unknown-arg compatibility — existing tests can pass `cfg=AgentConfig()` to get the old shape minus the new fields.
|
||||||
|
|
||||||
|
### Phase 3 acceptance
|
||||||
|
|
||||||
|
- All existing `test_streamer.py` / `test_integration.py` / `test_hardware.py` cases pass, with the heartbeat additions asserted in `test_hardware.py` (capabilities = `["rx"]` when `tx_enabled=False`).
|
||||||
|
- New test: two `start` messages in sequence with same `(device, identifier)` both succeed without recreating the SDR (registry hit). (This is a Phase 3 bonus — confirms the registry works before TX consumes it.)
|
||||||
|
- New test: `tx_start` with `tx_enabled=False` returns `tx_status: error` (handler stubs can do just this much in Phase 3, full implementation lands in Phase 4).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4 — TX implementation
|
||||||
|
|
||||||
|
**Why fourth:** now that binary arrives, config exists, and sessions exist, wire up real TX.
|
||||||
|
|
||||||
|
### Touches
|
||||||
|
|
||||||
|
- `src/ria_toolkit_oss/agent/streamer.py`
|
||||||
|
- Potentially a small helper module `src/ria_toolkit_oss/agent/_tx_loop.py` if streamer.py gets unwieldy.
|
||||||
|
- `tests/agent/test_streamer_tx.py`, `test_tx_safety.py`, `test_tx_underrun.py`, `test_full_duplex.py`
|
||||||
|
|
||||||
|
### Binary ingress
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def on_binary(self, data: bytes) -> None:
|
||||||
|
if self._tx is None:
|
||||||
|
logger.debug("Dropping %d-byte binary frame: no TX session", len(data))
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
self._tx.queue.put(data, timeout=2.0) # backpressure: block if full
|
||||||
|
except queue.Full:
|
||||||
|
logger.warning("TX queue stalled; dropping frame (agent side)")
|
||||||
|
```
|
||||||
|
|
||||||
|
Wire this in via `ws.run(..., on_binary=self.on_binary)` — change `run_streamer()`'s `ws.run` call accordingly.
|
||||||
|
|
||||||
|
### `_handle_tx_start`
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def _handle_tx_start(self, msg: dict) -> None:
|
||||||
|
app_id = msg.get("app_id") or ""
|
||||||
|
cfg_radio = dict(msg.get("radio_config") or {})
|
||||||
|
|
||||||
|
# 1) interlocks
|
||||||
|
if not self._cfg.tx_enabled:
|
||||||
|
return await self._send_tx_error(app_id, "tx disabled on this agent")
|
||||||
|
gain = cfg_radio.get("tx_gain")
|
||||||
|
if self._cfg.tx_max_gain_db is not None and gain is not None and float(gain) > self._cfg.tx_max_gain_db:
|
||||||
|
return await self._send_tx_error(app_id, f"tx_gain {gain} exceeds cap {self._cfg.tx_max_gain_db}")
|
||||||
|
freq = cfg_radio.get("tx_center_frequency")
|
||||||
|
if self._cfg.tx_allowed_freq_ranges and freq is not None:
|
||||||
|
if not any(lo <= float(freq) <= hi for lo, hi in self._cfg.tx_allowed_freq_ranges):
|
||||||
|
return await self._send_tx_error(app_id, f"tx_center_frequency {freq} outside allowed ranges")
|
||||||
|
|
||||||
|
if self._tx is not None:
|
||||||
|
return await self._send_tx_error(app_id, "tx already active on this agent")
|
||||||
|
|
||||||
|
# 2) device
|
||||||
|
device = cfg_radio.pop("device", None)
|
||||||
|
identifier = cfg_radio.pop("identifier", None)
|
||||||
|
buffer_size = int(cfg_radio.pop("buffer_size", 1024))
|
||||||
|
underrun_policy = cfg_radio.pop("underrun_policy", "pause")
|
||||||
|
if not device:
|
||||||
|
return await self._send_tx_error(app_id, "tx_start missing radio_config.device")
|
||||||
|
|
||||||
|
try:
|
||||||
|
sdr, device_key = self._registry.acquire(device, identifier)
|
||||||
|
_apply_sdr_config(sdr, cfg_radio) # sets tx_* attributes via alias map
|
||||||
|
# explicit init_tx if the driver supports it
|
||||||
|
if hasattr(sdr, "init_tx"):
|
||||||
|
sdr.init_tx(
|
||||||
|
sample_rate=cfg_radio.get("tx_sample_rate"),
|
||||||
|
center_frequency=cfg_radio.get("tx_center_frequency"),
|
||||||
|
gain=cfg_radio.get("tx_gain"),
|
||||||
|
channel=cfg_radio.get("tx_channel", 0),
|
||||||
|
gain_mode=cfg_radio.get("tx_gain_mode", "manual"),
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
self._registry.release(device_key)
|
||||||
|
logger.exception("Failed to init TX on %r", device)
|
||||||
|
return await self._send_tx_error(app_id, f"tx init failed: {exc}")
|
||||||
|
|
||||||
|
# 3) build session + launch loop
|
||||||
|
self._tx = TxSession(
|
||||||
|
app_id=app_id,
|
||||||
|
sdr=sdr,
|
||||||
|
device_key=device_key,
|
||||||
|
buffer_size=buffer_size,
|
||||||
|
queue=queue.Queue(maxsize=8),
|
||||||
|
stop_event=threading.Event(),
|
||||||
|
task=None, # filled below
|
||||||
|
underrun_policy=underrun_policy,
|
||||||
|
max_duration_s=self._cfg.tx_max_duration_s,
|
||||||
|
started_at=time.monotonic(),
|
||||||
|
)
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
self._tx.task = loop.run_in_executor(None, self._tx_executor_body)
|
||||||
|
|
||||||
|
await self._send_tx_status(app_id, "armed")
|
||||||
|
# streamer transitions to "transmitting" on the first buffer consumed in the thread;
|
||||||
|
# schedule a tiny watchdog that emits that status when queue count rises.
|
||||||
|
```
|
||||||
|
|
||||||
|
### TX executor body
|
||||||
|
|
||||||
|
Runs in a worker thread. Blocks in the SDR's `_stream_tx` driven by our callback that pulls from the queue.
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _tx_executor_body(self) -> None:
|
||||||
|
sdr = self._tx.sdr
|
||||||
|
try:
|
||||||
|
sdr._stream_tx(self._tx_callback)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("TX stream crashed")
|
||||||
|
# surface via asyncio side
|
||||||
|
asyncio.run_coroutine_threadsafe(
|
||||||
|
self._send_tx_status(self._tx.app_id, "error", "stream crashed"),
|
||||||
|
asyncio.get_event_loop(),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _tx_callback(self, num_samples):
|
||||||
|
tx = self._tx
|
||||||
|
if tx is None or tx.stop_event.is_set():
|
||||||
|
sdr = tx.sdr if tx else None
|
||||||
|
if sdr is not None:
|
||||||
|
sdr.pause_tx()
|
||||||
|
return _silence(num_samples)
|
||||||
|
|
||||||
|
# duration watchdog
|
||||||
|
if tx.max_duration_s is not None and (time.monotonic() - tx.started_at) > tx.max_duration_s:
|
||||||
|
tx.stop_event.set()
|
||||||
|
tx.sdr.pause_tx()
|
||||||
|
_schedule(self._send_tx_status(tx.app_id, "done", "max duration reached"))
|
||||||
|
return _silence(num_samples)
|
||||||
|
|
||||||
|
try:
|
||||||
|
raw = tx.queue.get(timeout=0.1)
|
||||||
|
except queue.Empty:
|
||||||
|
return self._underrun_fill(tx, num_samples)
|
||||||
|
|
||||||
|
samples = np.frombuffer(raw, dtype=np.float32)
|
||||||
|
# interleaved float32 -> complex64
|
||||||
|
if samples.size % 2 != 0 or samples.size // 2 != num_samples:
|
||||||
|
# malformed / wrong-sized frame; underrun this cycle
|
||||||
|
logger.warning("TX frame size mismatch: got %d floats, expected %d", samples.size, num_samples * 2)
|
||||||
|
return self._underrun_fill(tx, num_samples)
|
||||||
|
complex_samples = samples.reshape(-1, 2).view(np.complex64).reshape(-1)
|
||||||
|
tx.last_buffer = complex_samples
|
||||||
|
return complex_samples
|
||||||
|
```
|
||||||
|
|
||||||
|
Helper `_underrun_fill`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _underrun_fill(self, tx: TxSession, num_samples: int):
|
||||||
|
if tx.underrun_policy == "zero":
|
||||||
|
return np.zeros(num_samples, dtype=np.complex64)
|
||||||
|
if tx.underrun_policy == "repeat" and tx.last_buffer is not None:
|
||||||
|
return tx.last_buffer[:num_samples] if tx.last_buffer.size >= num_samples \
|
||||||
|
else np.concatenate([tx.last_buffer,
|
||||||
|
np.zeros(num_samples - tx.last_buffer.size, dtype=np.complex64)])
|
||||||
|
# "pause" (default)
|
||||||
|
tx.stop_event.set()
|
||||||
|
tx.sdr.pause_tx()
|
||||||
|
_schedule(self._send_tx_status(tx.app_id, "underrun"))
|
||||||
|
return np.zeros(num_samples, dtype=np.complex64)
|
||||||
|
```
|
||||||
|
|
||||||
|
`_schedule()` is a tiny wrapper around `asyncio.run_coroutine_threadsafe` that resolves the loop once at streamer construction.
|
||||||
|
|
||||||
|
### `_handle_tx_stop`
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def _handle_tx_stop(self, msg: dict) -> None:
|
||||||
|
tx = self._tx
|
||||||
|
if tx is None:
|
||||||
|
return
|
||||||
|
tx.stop_event.set()
|
||||||
|
tx.sdr.pause_tx()
|
||||||
|
# drain the queue so the executor thread wakes
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
tx.queue.get_nowait()
|
||||||
|
except queue.Empty:
|
||||||
|
pass
|
||||||
|
# wait up to ~1s for the executor thread to finish
|
||||||
|
if tx.task is not None:
|
||||||
|
try:
|
||||||
|
await asyncio.wait_for(asyncio.wrap_future(tx.task), timeout=1.0)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
logger.warning("TX executor did not exit within 1s after stop")
|
||||||
|
|
||||||
|
# release SDR reference
|
||||||
|
should_close = self._registry.release(tx.device_key)
|
||||||
|
if should_close:
|
||||||
|
try:
|
||||||
|
tx.sdr.close()
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Error closing SDR on tx_stop")
|
||||||
|
|
||||||
|
self._tx = None
|
||||||
|
await self._send_tx_status(msg.get("app_id") or "", "done")
|
||||||
|
```
|
||||||
|
|
||||||
|
### `_handle_tx_configure`
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def _handle_tx_configure(self, msg: dict) -> None:
|
||||||
|
if self._tx is None:
|
||||||
|
return
|
||||||
|
self._tx.pending_config.update(msg.get("radio_config") or {})
|
||||||
|
```
|
||||||
|
|
||||||
|
Consume `pending_config` at the top of `_tx_callback` before pulling from the queue (same pattern as RX's `_capture_loop`), using `_apply_sdr_config` with tx aliases.
|
||||||
|
|
||||||
|
### `_apply_sdr_config` — extend alias map
|
||||||
|
|
||||||
|
```python
|
||||||
|
_CONFIG_ATTR_MAP = {
|
||||||
|
# existing RX aliases...
|
||||||
|
"sample_rate": ("sample_rate", "rx_sample_rate"),
|
||||||
|
"center_frequency": ("center_freq", "rx_center_frequency"),
|
||||||
|
"gain": ("gain", "rx_gain"),
|
||||||
|
"bandwidth": ("bandwidth", "rx_bandwidth"),
|
||||||
|
# NEW TX aliases
|
||||||
|
"tx_sample_rate": ("tx_sample_rate",),
|
||||||
|
"tx_center_frequency": ("tx_center_frequency", "tx_lo"),
|
||||||
|
"tx_gain": ("tx_gain",),
|
||||||
|
"tx_bandwidth": ("tx_bandwidth",),
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Pluto has `set_tx_sample_rate`, `set_tx_center_frequency`, `set_tx_gain` — those are called by `init_tx` using the attribute values, so setting attributes via `_apply_sdr_config` + calling `init_tx` is sufficient.
|
||||||
|
|
||||||
|
### Phase 4 acceptance
|
||||||
|
|
||||||
|
- `test_streamer_tx.py`: full happy path — `tx_start` against MockSDR → 3 binary frames → verify `_stream_tx` callback received them in order → `tx_stop` → session cleared, SDR closed.
|
||||||
|
- `test_tx_safety.py`: one test per cap — `tx_enabled=False`, gain cap, freq range, duplicate session. Each produces a `tx_status: error` JSON; registry shows zero outstanding acquires.
|
||||||
|
- `test_tx_underrun.py`: three tests — `pause` (session ends, `underrun` emitted), `zero` (callback returns zeros, no status change), `repeat` (callback returns last buffer).
|
||||||
|
- `test_full_duplex.py`: against MockSDR, send `start` + `tx_start` with same `(device=mock, identifier=None)` → registry ref-count = 2 → both sessions stream independently → stop one, other still runs → stop second, SDR closed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 5 — Integration + docs
|
||||||
|
|
||||||
|
**Touches:**
|
||||||
|
|
||||||
|
- `tests/agent/test_integration_tx.py` — end-to-end with a real local `websockets` server + MockSDR. Mirror `test_integration.py`'s shape: register → heartbeat with `tx_enabled=True` → tx_start → 3 binary frames → tx_stop.
|
||||||
|
- `docs/agent_tx_protocol.md` — short, user-facing: message types, binary format, heartbeat additions, interlock config, CLI examples. Link from [screens_agent_handoff.md](./screens_agent_handoff.md).
|
||||||
|
- `README.md` (if it mentions agent subcommands) — add `--allow-tx` usage.
|
||||||
|
|
||||||
|
**Real-Pluto smoke test** (manual, not in CI):
|
||||||
|
|
||||||
|
1. `ria-agent register --hub http://hub:3005 --api-key KEY --allow-tx --tx-max-gain-db -10 --tx-freq-range 2.4e9 2.5e9`
|
||||||
|
2. `ria-agent stream`
|
||||||
|
3. From a Python REPL with `websockets`, open the hub WS on the agent's behalf (bypass hub during dev), send a `tx_start` + binary frames of a 1kHz tone → confirm carrier on a spectrum analyzer at the configured frequency.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File-by-file summary
|
||||||
|
|
||||||
|
| File | Phase | Change |
|
||||||
|
|---|---|---|
|
||||||
|
| `src/ria_toolkit_oss/agent/ws_client.py` | 1 | Add `on_binary` callback. |
|
||||||
|
| `src/ria_toolkit_oss/agent/config.py` | 2 | Add `tx_enabled`, `tx_max_gain_db`, `tx_max_duration_s`, `tx_allowed_freq_ranges`. |
|
||||||
|
| `src/ria_toolkit_oss/agent/cli.py` | 2, 4 | `--allow-tx` + cap flags on `register`; `--allow-tx` on `stream`; plumb `cfg` into `Streamer`. |
|
||||||
|
| `src/ria_toolkit_oss/agent/hardware.py` | 3 | `heartbeat_payload(cfg, sessions)` with `capabilities`, `tx_enabled`. |
|
||||||
|
| `src/ria_toolkit_oss/agent/streamer.py` | 3, 4 | Session refactor, SDR registry, TX dispatch, TX loop, underrun fills, `_apply_sdr_config` TX aliases. |
|
||||||
|
| `src/ria_toolkit_oss/agent/_tx_loop.py` | 4 (opt) | Extracted TX callback helpers if streamer.py > ~400 lines. |
|
||||||
|
| `tests/agent/test_ws_client.py` | 1 | Binary-frame case. |
|
||||||
|
| `tests/agent/test_config.py` | 2 | Round-trip new fields. |
|
||||||
|
| `tests/agent/test_cli_tx.py` | 2 | New — `--allow-tx` flag handling. |
|
||||||
|
| `tests/agent/test_hardware.py` | 3 | Heartbeat `capabilities` + `sessions`. |
|
||||||
|
| `tests/agent/test_streamer.py` | 3 | Refactor for session model; RX assertions unchanged. |
|
||||||
|
| `tests/agent/test_streamer_tx.py` | 4 | New — TX happy path. |
|
||||||
|
| `tests/agent/test_tx_safety.py` | 4 | New — cap enforcement. |
|
||||||
|
| `tests/agent/test_tx_underrun.py` | 4 | New — pause/zero/repeat policies. |
|
||||||
|
| `tests/agent/test_full_duplex.py` | 4 | New — shared SDR ref count. |
|
||||||
|
| `tests/agent/test_integration_tx.py` | 5 | New — real `websockets` server E2E. |
|
||||||
|
| `docs/agent_tx_protocol.md` | 5 | New — operator-facing protocol doc. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation gotchas (do not skip)
|
||||||
|
|
||||||
|
1. **Asyncio ↔ thread bridge.** The SDR's `_stream_tx` is synchronous and runs in an executor thread. Its callback must not `await`. Use `queue.Queue` (thread-safe) for inbound buffers and `asyncio.run_coroutine_threadsafe(coro, loop)` to emit `tx_status` from inside the thread. Resolve `loop` once at streamer construction; don't call `get_event_loop()` from the thread.
|
||||||
|
|
||||||
|
2. **`sdr.pause_tx()` from inside the callback.** Pluto's `_stream_tx` loop condition is `while self._enable_tx is True`. Calling `pause_tx()` inside the callback sets `_enable_tx = False` so the NEXT iteration exits. That's fine — it may emit one trailing zero-filled buffer. Document this; don't try to exit mid-callback.
|
||||||
|
|
||||||
|
3. **Queue drain on stop.** When `_handle_tx_stop` sets `stop_event` and pauses TX, the executor thread may still be blocked in `queue.get(timeout=0.1)`. Draining the queue does not unblock a timed get. Rely on the 100ms timeout; the thread exits on the next iteration. Don't try to clever-inject a poison pill.
|
||||||
|
|
||||||
|
4. **Interleaved float32 → complex64 conversion.** `np.frombuffer(buf, dtype=np.float32).view(np.complex64)` is zero-copy and correct when `buf.size` is a multiple of 8 bytes. Validate size first; mismatched size = underrun for that cycle, don't crash the thread.
|
||||||
|
|
||||||
|
5. **MockSDR's `_stream_tx`** ([sdr/mock.py:96-100](../src/ria_toolkit_oss/sdr/mock.py#L96-L100)) calls `callback(self.rx_buffer_size)` — it passes a *size*, not samples. The TX callback contract is "I am given `num_samples`, I return that many complex64 samples." `test_streamer_tx` must respect this: the test's `sdr.tx_buffer_size` (if used) doesn't affect what the callback receives from mock. Simplest path: set `MockSDR.rx_buffer_size = buffer_size` in the test harness before `_stream_tx` is invoked, so the TX callback receives the right size.
|
||||||
|
|
||||||
|
6. **`init_tx` on MockSDR vs Pluto.** MockSDR's `init_tx` [sets attributes and flips `_tx_initialized = True`](../src/ria_toolkit_oss/sdr/mock.py#L70-L81). Pluto's does the same plus `_rx_initialized = False` (the FDD bug). For full-duplex tests we currently target MockSDR only — Pluto FDD will work because our RX path ignores `_rx_initialized`, but the real-Pluto smoke test is the only validation. Call that out in the PR description.
|
||||||
|
|
||||||
|
7. **Don't block the event loop.** `asyncio.wait_for(asyncio.wrap_future(tx.task), timeout=1.0)` in `_handle_tx_stop` is non-blocking from the loop's perspective — the 1s cap prevents a misbehaving driver from stalling heartbeat/RX.
|
||||||
|
|
||||||
|
8. **Heartbeat during TX.** The existing heartbeat loop runs on a 30s timer. Sessions snapshot is cheap; no locking needed if we read `self._rx`/`self._tx` references atomically (Python ref swap is GIL-safe for single field reads).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollout
|
||||||
|
|
||||||
|
1. Open a single PR per phase (1 → 2 → 3 → 4 → 5), each green on its own.
|
||||||
|
2. Phase 3 is the riskiest diff (RX refactor). Get a second reviewer if possible; the regression surface is all of current RX behavior.
|
||||||
|
3. After Phase 4 merges, `ria-agent stream --allow-tx` is a usable toy — you can hand-drive it from a Python REPL with `websockets` to validate against real hardware before the hub side is ready.
|
||||||
|
4. Phase 5 closes the loop and ships the user-facing docs.
|
||||||
|
|
||||||
|
## Out of scope (explicit)
|
||||||
|
|
||||||
|
- **Multi-app-per-agent** — one RX + one TX per agent in v1. Adding session IDs to binary frames is a v2 protocol bump.
|
||||||
|
- **Other TX drivers** (HackRF, USRP, bladeRF) — wiring `_CONFIG_ATTR_MAP` entries and verifying `_stream_tx` behavior per-driver. Tackle when the hub has an operator that targets them.
|
||||||
|
- **Resampling / clock drift** — agent treats the hub-supplied samples as authoritative. Drift manifests as underruns; the underrun policy is the only mitigation.
|
||||||
|
- **Fixing Pluto's `init_tx` `_rx_initialized = False` reset** — pre-existing, not triggered by our RX path, left for a separate cleanup.
|
||||||
420
docs/agent_tx_plan.md
Normal file
420
docs/agent_tx_plan.md
Normal file
|
|
@ -0,0 +1,420 @@
|
||||||
|
# Agent TX Streaming — Cross-Repo Plan
|
||||||
|
|
||||||
|
**Repos:** `ria-toolkit-oss`, `ria-hub`, Screens frontend
|
||||||
|
**Status:** Proposal / pre-implementation
|
||||||
|
**Prerequisites:** The RX-streaming work from [screens_agent_handoff.md](./screens_agent_handoff.md) and [screens_agent_streamer_plan.md](./screens_agent_streamer_plan.md) is landed (agent WS protocol, `AgentDataSource`, `/screens/agents/register`, `/screens/agent/ws`).
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Let a Screens app running on the hub drive a **remote agent's Pluto** (or other TX-capable SDR) to transmit — streaming IQ buffers end-to-end from an operator like `plutoTXoperator` into the agent's `sdr.tx()` path. Mirror image of what `AgentDataSource` already does for RX.
|
||||||
|
|
||||||
|
## Non-goals (v1)
|
||||||
|
|
||||||
|
- Multi-tenant radio sharing (one app owns the radio at a time per agent).
|
||||||
|
- Bulk/upload-once TX — superseded by streaming per request.
|
||||||
|
- Arbitrary waveform generation in the agent. The agent is dumb pipe + hardware control; signal generation stays on the hub.
|
||||||
|
|
||||||
|
## Key design decisions
|
||||||
|
|
||||||
|
| # | Decision | Value |
|
||||||
|
|---|---|---|
|
||||||
|
| D1 | **Delivery mode** | **Streaming**. Hub pushes binary IQ buffers continuously over the existing WS; agent's `_stream_tx` callback pulls them from an in-agent queue. |
|
||||||
|
| D2 | **Full-duplex** | **Yes.** A single `app_id` may own both an RX session and a TX session on the same agent concurrently. Same physical SDR handle serves both (Pluto is FDD-capable; `init_rx` and `init_tx` are independent on one `adi.Pluto` instance). |
|
||||||
|
| D3 | **Safety caps** | **Agent-enforced.** `~/.ria/agent.json` holds `tx_enabled`, `tx_max_gain_db`, `tx_max_duration_s`, optional `tx_allowed_freq_ranges: [[low,high], …]`. Agent rejects `tx_start` frames that violate any of these, independent of what the hub sends. |
|
||||||
|
| D4 | **Buffer format** | Interleaved float32 IQ, range `[-1, 1]` — same as RX. Format-validated by `ria_toolkit_oss.sdr.sdr._verify_sample_format`. |
|
||||||
|
| D5 | **Protocol evolution** | Keep existing RX messages (`start`/`stop`/`configure`) unchanged for back-compat. Add parallel `tx_start`/`tx_stop`/`tx_configure`. Heartbeat grows to advertise capabilities. |
|
||||||
|
| D6 | **Underrun policy** | Default `pause`: if the TX queue empties, agent calls `pause_tx()` and emits `tx_status: underrun`. Hub must recover by sending a fresh `tx_start` + buffers. Configurable per session via `radio_config.underrun_policy ∈ {"pause", "zero", "repeat"}`. |
|
||||||
|
| D7 | **Backpressure** | Rely on TCP/WS backpressure. Agent caps inbound TX queue at 8 buffers; `await ws.send` on the hub side slows when the agent doesn't drain. No application-level flow control in v1. |
|
||||||
|
| D8 | **Session identity** | `app_id` identifies a Screens app. Each app has at most one RX session and one TX session per agent. Binary direction disambiguates: agent → hub binary = RX IQ; hub → agent binary = TX IQ. |
|
||||||
|
|
||||||
|
## Protocol specification
|
||||||
|
|
||||||
|
Additions only. Existing RX messages from [screens_agent_handoff.md §Phase 4](./screens_agent_handoff.md) are unchanged.
|
||||||
|
|
||||||
|
### Hub → agent (JSON)
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// Arm the TX side. Agent calls init_tx, starts the stream_tx thread with an empty queue.
|
||||||
|
// After this, hub sends binary TX buffers on the same WS.
|
||||||
|
{
|
||||||
|
"type": "tx_start",
|
||||||
|
"app_id": "app-abc",
|
||||||
|
"radio_config": {
|
||||||
|
"device": "pluto",
|
||||||
|
"identifier": "ip:192.168.3.1",
|
||||||
|
"tx_sample_rate": 1000000,
|
||||||
|
"tx_center_frequency": 2450000000,
|
||||||
|
"tx_gain": -20, // dB, negative = attenuation on Pluto
|
||||||
|
"tx_bandwidth": 1000000, // optional
|
||||||
|
"buffer_size": 1024,
|
||||||
|
"underrun_policy": "pause" // "pause" | "zero" | "repeat"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Apply parameter changes at the next buffer boundary.
|
||||||
|
{ "type": "tx_configure", "app_id": "app-abc", "radio_config": { "tx_gain": -25 } }
|
||||||
|
|
||||||
|
// Stop TX, drain queue, pause_tx, release TX side (RX may continue if a separate RX session is live).
|
||||||
|
{ "type": "tx_stop", "app_id": "app-abc" }
|
||||||
|
```
|
||||||
|
|
||||||
|
### Hub → agent (binary)
|
||||||
|
|
||||||
|
Raw interleaved float32 IQ in `[-1, 1]`. One WS frame = one buffer = `buffer_size` complex samples = `buffer_size * 2 * 4` bytes. Delivered only between `tx_start` and `tx_stop`. Binary frames arriving outside that window are discarded and logged at WARN.
|
||||||
|
|
||||||
|
### Agent → hub (JSON)
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// Lifecycle events.
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "armed" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "transmitting" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "underrun" } // queue empty; TX paused
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "done" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-abc", "state": "error", "message": "gain -5 exceeds tx_max_gain_db=-15" }
|
||||||
|
|
||||||
|
// Reject reasons from agent-enforced caps/interlocks are surfaced via tx_status:error.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Heartbeat extension
|
||||||
|
|
||||||
|
Existing `{type: heartbeat, hardware[], status}` grows:
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{
|
||||||
|
"type": "heartbeat",
|
||||||
|
"hardware": ["mock", "pluto"],
|
||||||
|
"status": "streaming", // unchanged semantics
|
||||||
|
"capabilities": ["rx", "tx"], // NEW — derived from tx_enabled + SDR class having init_tx
|
||||||
|
"tx_enabled": true, // NEW — mirrors config flag
|
||||||
|
"sessions": { // NEW — optional per-session snapshot
|
||||||
|
"rx": { "app_id": "app-abc", "state": "streaming" },
|
||||||
|
"tx": { "app_id": "app-abc", "state": "transmitting" }
|
||||||
|
},
|
||||||
|
"app_id": "app-abc" // kept for back-compat
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part A — `ria-toolkit-oss` (this repo)
|
||||||
|
|
||||||
|
### A1. `agent/ws_client.py`
|
||||||
|
|
||||||
|
Currently the WS client drops server → agent binary (`ws_client.py:77-79`). Add a binary handler alongside the JSON one.
|
||||||
|
|
||||||
|
```python
|
||||||
|
BinaryHandler = Callable[[bytes], Awaitable[None]]
|
||||||
|
|
||||||
|
async def run(
|
||||||
|
self,
|
||||||
|
on_message: MessageHandler,
|
||||||
|
heartbeat: HeartbeatBuilder,
|
||||||
|
on_binary: BinaryHandler | None = None,
|
||||||
|
) -> None:
|
||||||
|
...
|
||||||
|
async for raw in self._ws:
|
||||||
|
if isinstance(raw, bytes):
|
||||||
|
if on_binary is not None:
|
||||||
|
await on_binary(raw)
|
||||||
|
continue
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep the reconnect, heartbeat, and malformed-frame behavior unchanged.
|
||||||
|
|
||||||
|
### A2. `agent/streamer.py` — add TX sessions
|
||||||
|
|
||||||
|
Replace the flat `self._sdr` / `self._app_id` / `self._capture_task` state with a session model:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class RxSession:
|
||||||
|
app_id: str
|
||||||
|
sdr: Any
|
||||||
|
buffer_size: int
|
||||||
|
task: asyncio.Task
|
||||||
|
pending_config: dict
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TxSession:
|
||||||
|
app_id: str
|
||||||
|
sdr: Any
|
||||||
|
queue: asyncio.Queue[bytes] # bounded, maxsize=8
|
||||||
|
task: asyncio.Task # runs _stream_tx in executor
|
||||||
|
underrun_policy: str
|
||||||
|
pending_config: dict
|
||||||
|
bytes_transmitted: int = 0
|
||||||
|
started_at: float = 0.0 # for tx_max_duration_s enforcement
|
||||||
|
```
|
||||||
|
|
||||||
|
The streamer holds `self._rx: RxSession | None` and `self._tx: TxSession | None`. SDR instances are cached by `(device, identifier)` — when RX and TX name the same device, both sessions share one handle (matters for Pluto FDD).
|
||||||
|
|
||||||
|
**New handlers**:
|
||||||
|
|
||||||
|
- `_handle_tx_start(msg)` — check `cfg.tx_enabled`, validate gain/duration/freq against caps, open/resolve SDR, `sdr.init_tx(...)`, start `_tx_loop`, emit `tx_status: armed`.
|
||||||
|
- `_handle_tx_stop(msg)` — cancel TX task, `sdr.pause_tx()`, drain queue, release SDR if no RX session on it, emit `tx_status: done`.
|
||||||
|
- `_handle_tx_configure(msg)` — stash into `self._tx.pending_config`, applied at next buffer boundary (same pattern as RX).
|
||||||
|
- `on_binary(data)` — if `self._tx`: `await self._tx.queue.put(data)` (awaiting here is the backpressure mechanism). Else: log and drop.
|
||||||
|
|
||||||
|
**TX loop** (runs in an executor thread via `loop.run_in_executor`, like the RX capture loop):
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _tx_callback(num_samples: int) -> np.ndarray:
|
||||||
|
# Called by sdr._stream_tx on every buffer boundary.
|
||||||
|
try:
|
||||||
|
raw = self._tx_queue_sync.get(timeout=0.1)
|
||||||
|
except queue.Empty:
|
||||||
|
return self._underrun_fill(num_samples) # policy-driven
|
||||||
|
samples = np.frombuffer(raw, dtype=np.float32).view(np.complex64)
|
||||||
|
if len(samples) < num_samples:
|
||||||
|
return _pad_zero(samples, num_samples)
|
||||||
|
return samples[:num_samples]
|
||||||
|
```
|
||||||
|
|
||||||
|
Use a thread-safe `queue.Queue` for the TX side (the `asyncio.Queue` lives on the event loop; the executor thread reads from a sibling `queue.Queue` fed by a tiny asyncio→threading adapter).
|
||||||
|
|
||||||
|
**Underrun fills**:
|
||||||
|
- `"pause"`: signal the main loop to call `sdr.pause_tx()`, emit `tx_status: underrun`, exit the callback.
|
||||||
|
- `"zero"`: return `np.zeros(num_samples, dtype=np.complex64)`.
|
||||||
|
- `"repeat"`: return the last good buffer (cached). If no buffer yet: zeros.
|
||||||
|
|
||||||
|
**Cap enforcement** in `_handle_tx_start` (before opening the SDR):
|
||||||
|
|
||||||
|
```python
|
||||||
|
if not self._cfg.tx_enabled:
|
||||||
|
return await self._send_error_tx(app_id, "tx disabled on this agent")
|
||||||
|
if (cap := self._cfg.tx_max_gain_db) is not None and tx_gain > cap:
|
||||||
|
return await self._send_error_tx(app_id, f"gain {tx_gain} exceeds cap {cap}")
|
||||||
|
if (cap := self._cfg.tx_max_duration_s) is not None:
|
||||||
|
# enforced by a watchdog in _tx_loop that calls tx_stop after cap seconds
|
||||||
|
...
|
||||||
|
for (lo, hi) in self._cfg.tx_allowed_freq_ranges or []:
|
||||||
|
if lo <= tx_center_frequency <= hi:
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
if self._cfg.tx_allowed_freq_ranges:
|
||||||
|
return await self._send_error_tx(app_id, f"freq {tx_center_frequency} outside allowed ranges")
|
||||||
|
```
|
||||||
|
|
||||||
|
### A3. `agent/config.py`
|
||||||
|
|
||||||
|
Extend `AgentConfig`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class AgentConfig:
|
||||||
|
# existing fields…
|
||||||
|
tx_enabled: bool = False
|
||||||
|
tx_max_gain_db: float | None = None
|
||||||
|
tx_max_duration_s: float | None = None
|
||||||
|
tx_allowed_freq_ranges: list[tuple[float, float]] | None = None
|
||||||
|
```
|
||||||
|
|
||||||
|
`save()` preserves existing 0600 perms.
|
||||||
|
|
||||||
|
### A4. `agent/cli.py`
|
||||||
|
|
||||||
|
- `ria-agent register --allow-tx --tx-max-gain-db -10 --tx-max-duration 60` — persist the interlock into config.
|
||||||
|
- `ria-agent stream --allow-tx` — runtime override (sets `cfg.tx_enabled=True` for the life of the process without writing config).
|
||||||
|
- `ria-agent detect` unchanged.
|
||||||
|
|
||||||
|
### A5. `agent/hardware.py`
|
||||||
|
|
||||||
|
```python
|
||||||
|
def heartbeat_payload(status, app_id=None, *, cfg: AgentConfig, sessions: dict | None = None) -> dict:
|
||||||
|
caps = ["rx"]
|
||||||
|
if cfg.tx_enabled:
|
||||||
|
caps.append("tx")
|
||||||
|
payload = {
|
||||||
|
"type": "heartbeat",
|
||||||
|
"hardware": available_devices(),
|
||||||
|
"status": status,
|
||||||
|
"capabilities": caps,
|
||||||
|
"tx_enabled": cfg.tx_enabled,
|
||||||
|
}
|
||||||
|
if app_id:
|
||||||
|
payload["app_id"] = app_id
|
||||||
|
if sessions:
|
||||||
|
payload["sessions"] = sessions
|
||||||
|
return payload
|
||||||
|
```
|
||||||
|
|
||||||
|
### A6. SDR layer
|
||||||
|
|
||||||
|
- **Audit**: [`sdr/pluto.py`](../src/ria_toolkit_oss/sdr/pluto.py) `tx_recording` + `_stream_tx` paths already use `_tx_lock` (line 31, 323, 360). Double-check concurrent-with-RX behavior: the `adi.Pluto` Python object is not thread-safe for arbitrary attribute writes, so all `set_tx_*` / `set_rx_*` must go through the shared `_param_lock` (already present at [`sdr/sdr.py:44`](../src/ria_toolkit_oss/sdr/sdr.py#L44)). Verify `rx()` in a loop + `_stream_tx` in another thread don't step on each other.
|
||||||
|
- **MockSDR** already has `init_tx` + `_stream_tx` (`sdr/mock.py:70-100`). No changes needed for mock-based tests.
|
||||||
|
- **Other TX-capable drivers** (blade, usrp, hackrf): out of scope for v1; leave their `init_tx` as-is.
|
||||||
|
|
||||||
|
### A7. Tests (`tests/agent/`)
|
||||||
|
|
||||||
|
- `test_streamer_tx.py` — `tx_start` → binary frames → `_stream_tx` callback pulls correct samples → `tx_stop` cleans up.
|
||||||
|
- `test_tx_safety.py` — cap violations (gain, duration, freq, `tx_enabled=False`) each produce `tx_status: error` and never open the SDR.
|
||||||
|
- `test_tx_underrun.py` — each policy (`pause`, `zero`, `repeat`) exercised against a fake slow producer.
|
||||||
|
- `test_full_duplex.py` — one `app_id` sends `start` + `tx_start`; both sessions share one MockSDR; both produce their expected frames; stopping one does not stop the other.
|
||||||
|
- `test_ws_client_binary.py` — binary frames now reach the binary handler.
|
||||||
|
- `test_integration_tx.py` — end-to-end against local `websockets` server + MockSDR.
|
||||||
|
|
||||||
|
### A8. Docs
|
||||||
|
|
||||||
|
- Add a TX section to any existing agent protocol doc (or create `docs/agent_tx_protocol.md`).
|
||||||
|
- Include a regulatory disclaimer: the operator is responsible for transmissions. The agent is an enabler, not a policy layer beyond the interlocks.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part B — `ria-hub`
|
||||||
|
|
||||||
|
> Paths below are conceptual — confirm against the actual module layout in `ria-hub` before editing. Anchor points reference the RX handoff at [screens_agent_handoff.md §Part B](./screens_agent_handoff.md).
|
||||||
|
|
||||||
|
### B1. `AgentTxSink` (new)
|
||||||
|
|
||||||
|
Mirror of `AgentDataSource`. Location: `controller/app/modules/screens/data_sinks.py` (or wherever output sinks live in `ria-hub`).
|
||||||
|
|
||||||
|
Responsibilities:
|
||||||
|
- `prepare(radio_config)` — send `tx_start` via Redis pub/sub on `screens:agent:{agent_id}:tx` → WS proxy → agent.
|
||||||
|
- `write(buffer: np.ndarray | bytes)` — convert to interleaved float32 bytes, send as binary over the WS. Awaits on WS backpressure.
|
||||||
|
- `configure(partial_radio_config)` — send `tx_configure`.
|
||||||
|
- `close()` — send `tx_stop`.
|
||||||
|
- Subscribes to the agent's `tx_status` frames (via the same Redis pub/sub channel used for RX status today) and surfaces state back to the orchestrator. An `error` state aborts the Celery task.
|
||||||
|
|
||||||
|
### B2. Refactor `plutoTXoperator`
|
||||||
|
|
||||||
|
The existing operator presumably calls `radio.tx(...)` against a directly-attached Pluto. Abstract the "output" into an injectable sink:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class PlutoTxOperator:
|
||||||
|
def __init__(self, sink: TxSink, ...):
|
||||||
|
self.sink = sink # AgentTxSink when dataSink.type == "agent", else LocalPlutoTxSink
|
||||||
|
|
||||||
|
def run(self, ...):
|
||||||
|
self.sink.prepare(self.radio_config)
|
||||||
|
while not stop:
|
||||||
|
buf = self._generate_next_buffer()
|
||||||
|
self.sink.write(buf)
|
||||||
|
self.sink.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
The local path (existing direct-hardware behavior) becomes `LocalPlutoTxSink`, a thin wrapper around the current `radio.tx` calls. No behavior change for existing deployments.
|
||||||
|
|
||||||
|
`build_data_sink()` (to match `build_data_source()` from B1/B6) routes on `dataSink.type`.
|
||||||
|
|
||||||
|
### B3. Manifest schema
|
||||||
|
|
||||||
|
Add `dataSink` alongside `dataSource` in the manifest. New `type: "agent"`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"dataSource": { "type": "agent", "device": "pluto", "agent_id": "agent-abc", "params": { "sample_rate": 1000000, "center_frequency": 2450000000, "gain": 40 } },
|
||||||
|
"dataSink": { "type": "agent", "device": "pluto", "agent_id": "agent-abc", "params": { "tx_sample_rate": 1000000, "tx_center_frequency": 2450000000, "tx_gain": -20, "underrun_policy": "pause" } }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Update Pydantic models + JSON schema validators in `controller/app/modules/screens/graph_derivation.py` (or equivalent). When `dataSource.agent_id == dataSink.agent_id` and both target `pluto` with the same `identifier`, the agent will naturally share one SDR handle — no special-casing needed on the hub side.
|
||||||
|
|
||||||
|
### B4. WS endpoint extensions
|
||||||
|
|
||||||
|
`/screens/agent/ws` already exists. Add:
|
||||||
|
|
||||||
|
- Support for hub → agent **binary frames** (currently binary is agent → hub only). FastAPI's `WebSocket.send_bytes` works directly; just route binary from the Redis pub/sub channel through to the WS.
|
||||||
|
- New Redis pub/sub channel `screens:agent:{agent_id}:tx` for outbound TX control JSON + a separate `screens:agent:{agent_id}:tx_bin` for outbound binary. (Two channels because many Redis brokers don't love mixing binary into text-keyed channels; if your deployment uses Redis 6+ with `SUBSCRIBE` that handles bytes, one channel is fine.)
|
||||||
|
|
||||||
|
### B5. Celery wiring
|
||||||
|
|
||||||
|
When `dataSink.type == "agent"`, the Celery task that runs the TX-containing graph uses `AgentTxSink` instead of a local sink. The operator code (`plutoTXoperator`) is unchanged because the sink abstraction hides the difference.
|
||||||
|
|
||||||
|
Full-duplex: a single task with both `dataSource.type == "agent"` and `dataSink.type == "agent"` pointing at the same agent spawns both the RX consumer loop (existing `AgentDataSource.next_chunk` via BLPOP) and the TX producer loop (`AgentTxSink.write`). Both sides are wired up before any capture frames are sent.
|
||||||
|
|
||||||
|
### B6. Capability gating
|
||||||
|
|
||||||
|
Before any control path sends `tx_start`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
agent = get_agent(agent_id)
|
||||||
|
if agent.last_heartbeat.age > 60: # stale
|
||||||
|
raise HTTPException(503, "agent not responding")
|
||||||
|
if "tx" not in agent.last_heartbeat.capabilities:
|
||||||
|
raise HTTPException(400, "agent has not opted in to transmission (tx_enabled=false)")
|
||||||
|
```
|
||||||
|
|
||||||
|
Surface clear errors to the Screens UI so the user knows it's an agent config issue, not an app config issue.
|
||||||
|
|
||||||
|
### B7. Audit log
|
||||||
|
|
||||||
|
New MongoDB collection `agent_tx_audit`:
|
||||||
|
|
||||||
|
```
|
||||||
|
{
|
||||||
|
agent_id, app_id, user_id,
|
||||||
|
center_frequency_hz, tx_gain_db, duration_s, num_samples,
|
||||||
|
started_at, ended_at, terminal_status, // "done" | "error" | "underrun" | "cancelled"
|
||||||
|
error_message?
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Write on every `tx_start`. Update on terminal `tx_status`. Index on `{agent_id, started_at}` for admin-view queries.
|
||||||
|
|
||||||
|
### B8. Registration — no change needed
|
||||||
|
|
||||||
|
`POST /screens/agents/register` and `~/.ria/agent.json` already cover credential storage. The TX interlock (`tx_enabled`, caps) is written by the *agent operator* via `ria-agent register --allow-tx`; the hub only reads the heartbeat to learn whether an agent will accept TX.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part C — Screens (Vue 3 frontend)
|
||||||
|
|
||||||
|
### C1. App composer
|
||||||
|
|
||||||
|
- **Agent picker** (exists from RX work) grows a "TX capable" filter toggle; hides agents whose heartbeat `capabilities` lacks `"tx"`.
|
||||||
|
- When the graph contains `plutoTXoperator` (or any future TX operator):
|
||||||
|
- Render a **dataSink** section mirroring dataSource.
|
||||||
|
- Fields: device, agent_id, identifier, tx_sample_rate, tx_center_frequency, tx_gain, underrun_policy.
|
||||||
|
- Validation: tx_center_frequency within radio band; tx_gain within agent-advertised max (read from heartbeat when available).
|
||||||
|
|
||||||
|
### C2. Run-time UI
|
||||||
|
|
||||||
|
- **Consent modal** on "Start" for any app whose manifest contains a `dataSink.type: "agent"`:
|
||||||
|
> "This app will transmit on **2.450 GHz** at **-20 dB** through agent **lab-pluto-01**. I confirm this transmission is permitted under my local radio regulations."
|
||||||
|
Required checkbox, cannot be remembered across apps.
|
||||||
|
- **TX status indicator** in the running-app view: shows `armed` / `transmitting` / `underrun` state from `tx_status` frames. Red banner on `underrun` or `error`.
|
||||||
|
- **Stop TX button** always visible during transmission; fires `tx_stop` immediately. Separate from "Stop app" (which also stops RX).
|
||||||
|
|
||||||
|
### C3. Admin view
|
||||||
|
|
||||||
|
Extend the agents list from B8 of the RX handoff:
|
||||||
|
|
||||||
|
- Column: **TX**: `enabled` / `disabled` / `in-use by app X`.
|
||||||
|
- Agent detail page: show `tx_max_gain_db`, `tx_max_duration_s`, `tx_allowed_freq_ranges`, and the last 10 rows from `agent_tx_audit` filtered to this agent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollout order
|
||||||
|
|
||||||
|
1. **Part A §A1-A3, A7** — agent-side TX session + binary ingress + safety, all behind `--allow-tx`. Mock-based tests. Shippable standalone; no consumer yet.
|
||||||
|
2. **Part B §B1-B5** — hub sink + manifest + WS extension + Celery wiring. End-to-end test: Screens app with `plutoTXoperator` + agent sink → real Pluto in the lab → verify carrier on a spectrum analyzer.
|
||||||
|
3. **Part B §B6-B7** — capability gating + audit log. Blocks general release, not lab use.
|
||||||
|
4. **Part C §C1** — composer UI for TX apps.
|
||||||
|
5. **Part C §C2-C3** — consent modal + admin view. Gate for first non-internal user.
|
||||||
|
|
||||||
|
Parts A + B can land on parallel branches and meet at step 2's integration test. Part C can start in parallel with B once the manifest shape in B3 is stable.
|
||||||
|
|
||||||
|
## Test matrix (integration)
|
||||||
|
|
||||||
|
| Scenario | Expected |
|
||||||
|
|---|---|
|
||||||
|
| App with RX only, agent connected | RX as today (regression guard) |
|
||||||
|
| App with TX only, agent `tx_enabled=True` | TX starts, underrun → pause, stop cleans up |
|
||||||
|
| App with RX + TX same agent, same device | One Pluto handle serves both; independent gains/frequencies |
|
||||||
|
| App with TX, agent `tx_enabled=False` | Hub rejects at gate with 400; no WS traffic generated |
|
||||||
|
| App with TX, gain exceeds agent cap | `tx_status: error`; SDR never opened |
|
||||||
|
| Hub stops sending TX buffers mid-stream | `underrun` emitted after queue drains; agent paused cleanly |
|
||||||
|
| WS drops during TX | Agent cancels TX task, pauses hardware, reconnects; hub must re-issue `tx_start` |
|
||||||
|
| Agent process killed during TX | Hardware stops (existing `close()` already handles this; verify `_tx_lock` released) |
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- **Waveform source**: is `plutoTXoperator` a real-time generator emitting on a clock, or does it synthesize a fixed recording and loop? If the latter, worth exposing a "bulk + loop" fast-path — hub sends the buffer once, agent loops it via existing `tx_recording`. Same protocol (`tx_start` + one buffer + `loop: true`), much less WS traffic.
|
||||||
|
- **Multi-app-per-agent**: out of scope for v1 (§Non-goals). When needed: add a session id to binary frames (4-byte prefix: magic + stream_id + reserved), bump a `protocol_version` in the heartbeat.
|
||||||
|
- **Streaming TX clock drift**: if hub and agent sample clocks drift, repeating zeros on underrun is audible/visible in the spectrum. Longer term: agent-side resampling or PLL, both expensive. v1: rely on generous queue depth + stable local networks.
|
||||||
|
- **Other TX-capable SDRs**: HackRF, USRP, bladeRF. The `_CONFIG_ATTR_MAP` in [`agent/streamer.py:169-175`](../src/ria_toolkit_oss/agent/streamer.py#L169-L175) will need per-driver entries when those come online.
|
||||||
|
|
||||||
|
## Regulatory note
|
||||||
|
|
||||||
|
Transmission is regulated in every jurisdiction. The agent-side interlocks (`tx_enabled`, caps, freq ranges) exist so the operator can configure safe defaults for an agent's physical location. They are not a substitute for licensing or for respecting local regulations. The hub's consent modal and audit log exist so actions are attributable. None of this is a legal compliance layer — it's a defense-in-depth mechanism.
|
||||||
185
docs/agent_tx_protocol.md
Normal file
185
docs/agent_tx_protocol.md
Normal file
|
|
@ -0,0 +1,185 @@
|
||||||
|
# Agent TX Protocol
|
||||||
|
|
||||||
|
Operator-facing reference for the TX streaming extensions to the agent
|
||||||
|
WebSocket protocol. Implementation plan: [agent_tx_implementation_plan.md](./agent_tx_implementation_plan.md).
|
||||||
|
Cross-repo design: [agent_tx_plan.md](./agent_tx_plan.md).
|
||||||
|
|
||||||
|
> **Regulatory note.** Transmission is regulated in every jurisdiction. The
|
||||||
|
> agent-side interlocks documented below let you configure safe defaults
|
||||||
|
> for your deployment. They do not replace licensing or responsibility
|
||||||
|
> for your own emissions. The RIA Hub's consent modal and audit log make
|
||||||
|
> actions attributable — they are not a legal-compliance layer.
|
||||||
|
|
||||||
|
## Opt-in
|
||||||
|
|
||||||
|
TX is **disabled by default**. The hub cannot make the agent transmit unless
|
||||||
|
the operator has explicitly opted in on the agent host.
|
||||||
|
|
||||||
|
Two equivalent opt-in paths:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Persist to ~/.ria/agent.json so the agent always allows TX.
|
||||||
|
ria-agent register --hub http://HUB:3005 --api-key KEY \
|
||||||
|
--allow-tx \
|
||||||
|
--tx-max-gain-db -10 \
|
||||||
|
--tx-max-duration-s 60 \
|
||||||
|
--tx-freq-range 2.4e9 2.5e9 \
|
||||||
|
--tx-freq-range 5.7e9 5.8e9
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Runtime-only override (does not touch disk).
|
||||||
|
ria-agent stream --allow-tx
|
||||||
|
```
|
||||||
|
|
||||||
|
Caps:
|
||||||
|
|
||||||
|
| Flag | Config key | Effect |
|
||||||
|
|---|---|---|
|
||||||
|
| `--tx-max-gain-db VALUE` | `tx_max_gain_db` | Reject any `tx_start` whose `tx_gain > VALUE` |
|
||||||
|
| `--tx-max-duration-s VALUE` | `tx_max_duration_s` | Auto-stop any TX session after `VALUE` seconds (watchdog in the TX loop) |
|
||||||
|
| `--tx-freq-range LO HI` (repeatable) | `tx_allowed_freq_ranges` | Reject any `tx_start` whose `tx_center_frequency` falls outside all configured ranges |
|
||||||
|
|
||||||
|
The agent enforces each cap **before** opening the SDR. A violating
|
||||||
|
`tx_start` produces a `tx_status: error` frame and never touches hardware.
|
||||||
|
|
||||||
|
## Heartbeat advertisement
|
||||||
|
|
||||||
|
Every heartbeat now includes:
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{
|
||||||
|
"type": "heartbeat",
|
||||||
|
"hardware": ["mock", "pluto"],
|
||||||
|
"status": "streaming",
|
||||||
|
"capabilities": ["rx", "tx"], // "tx" present only when tx_enabled=True
|
||||||
|
"tx_enabled": true,
|
||||||
|
"sessions": { // omitted when no session is live
|
||||||
|
"rx": { "app_id": "app-1", "state": "streaming" },
|
||||||
|
"tx": { "app_id": "app-1", "state": "transmitting" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Hubs should read `capabilities` to decide whether to surface TX operators
|
||||||
|
against this agent in the Screens app composer.
|
||||||
|
|
||||||
|
## Control messages
|
||||||
|
|
||||||
|
### Hub → agent (JSON)
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// Arm the TX side. Agent validates interlocks, opens/resolves the SDR,
|
||||||
|
// and transitions into "armed". The next binary frames are consumed as
|
||||||
|
// TX IQ buffers.
|
||||||
|
{
|
||||||
|
"type": "tx_start",
|
||||||
|
"app_id": "app-1",
|
||||||
|
"radio_config": {
|
||||||
|
"device": "pluto",
|
||||||
|
"identifier": "ip:192.168.3.1",
|
||||||
|
"tx_sample_rate": 1000000,
|
||||||
|
"tx_center_frequency": 2450000000,
|
||||||
|
"tx_gain": -20, // dB; Pluto uses negative attenuation
|
||||||
|
"tx_bandwidth": 1000000, // optional
|
||||||
|
"buffer_size": 1024,
|
||||||
|
"underrun_policy": "pause" // "pause" (default) | "zero" | "repeat"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Update parameters at the next buffer boundary. No re-arm needed.
|
||||||
|
{ "type": "tx_configure", "app_id": "app-1",
|
||||||
|
"radio_config": { "tx_gain": -25 } }
|
||||||
|
|
||||||
|
// Stop TX, drain the inbound queue, pause_tx, release the SDR (if no RX
|
||||||
|
// session is still using it). A new tx_start can follow immediately.
|
||||||
|
{ "type": "tx_stop", "app_id": "app-1" }
|
||||||
|
```
|
||||||
|
|
||||||
|
### Hub → agent (binary)
|
||||||
|
|
||||||
|
- Raw interleaved float32 IQ, normalised to `[-1, 1]`.
|
||||||
|
- One WebSocket frame = one buffer = `buffer_size` complex samples =
|
||||||
|
`buffer_size * 2 * 4` bytes.
|
||||||
|
- Accepted only while a TX session is live. Frames outside that window
|
||||||
|
are logged and dropped.
|
||||||
|
- Malformed frames (odd float count, wrong size) trigger one underrun
|
||||||
|
cycle but do not crash the stream.
|
||||||
|
|
||||||
|
### Agent → hub (JSON)
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
{ "type": "tx_status", "app_id": "app-1", "state": "armed" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-1", "state": "transmitting" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-1", "state": "underrun" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-1", "state": "done" }
|
||||||
|
{ "type": "tx_status", "app_id": "app-1", "state": "error",
|
||||||
|
"message": "tx_gain -5 exceeds cap -15.0" }
|
||||||
|
```
|
||||||
|
|
||||||
|
Transitions:
|
||||||
|
|
||||||
|
```
|
||||||
|
tx_start tx_stop
|
||||||
|
—————————————————▶ armed ▶ transmitting ——————————▶ done
|
||||||
|
│ │
|
||||||
|
│ │ queue empties + policy="pause"
|
||||||
|
│ ▼
|
||||||
|
│ underrun ▶ done (auto-teardown)
|
||||||
|
│
|
||||||
|
└─ interlock / init failure ▶ error (no session)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Underrun policies
|
||||||
|
|
||||||
|
When the inbound TX queue is empty at a buffer boundary:
|
||||||
|
|
||||||
|
| Policy | Behavior |
|
||||||
|
|---|---|
|
||||||
|
| `pause` *(default)* | Callback returns silence, calls `pause_tx()`, flips the session into `underrun`. Watchdog emits `tx_status: underrun` + `tx_status: done` and tears down. Hub must re-issue `tx_start` to resume. |
|
||||||
|
| `zero` | Callback returns a zero-filled buffer. Session stays alive; no status change. Carrier continues with dead air. |
|
||||||
|
| `repeat` | Callback returns the most recently transmitted buffer. If no buffer has arrived yet, falls back to zero for that cycle. |
|
||||||
|
|
||||||
|
Choose `pause` for correctness-sensitive workloads (any data modulation
|
||||||
|
where zero-fill or repeat corrupts the stream). Choose `zero` or `repeat`
|
||||||
|
for continuous-carrier use cases where brief stalls are acceptable.
|
||||||
|
|
||||||
|
## Concurrent RX + TX
|
||||||
|
|
||||||
|
A single `app_id` may hold both an RX session (`start`/`stop`) and a TX
|
||||||
|
session (`tx_start`/`tx_stop`) on the same agent at the same time. When
|
||||||
|
both reference the same `(device, identifier)`, the agent shares a single
|
||||||
|
driver instance between the two sessions (ref-counted release on stop).
|
||||||
|
|
||||||
|
Multi-app sharing of one SDR is not supported in v1. A second `tx_start`
|
||||||
|
with a different `app_id` while another TX session is live produces
|
||||||
|
`tx_status: error "tx already active on this agent"`.
|
||||||
|
|
||||||
|
## Buffer format recap
|
||||||
|
|
||||||
|
- **Direction** is the only framing: hub → agent binary means TX,
|
||||||
|
agent → hub binary means RX.
|
||||||
|
- **Layout**: `[I0, Q0, I1, Q1, …]` as little-endian float32.
|
||||||
|
- **Size**: `buffer_size * 2 * 4` bytes. Mismatched sizes are treated as
|
||||||
|
a single-cycle underrun (malformed frame).
|
||||||
|
- **Range**: samples must lie in `[-1, 1]`. Out-of-range values are
|
||||||
|
transmitted as-is; the SDR driver may clip.
|
||||||
|
|
||||||
|
## Configuration reference
|
||||||
|
|
||||||
|
`~/.ria/agent.json` is written by `ria-agent register` and read by
|
||||||
|
`ria-agent stream`. Minimum schema with TX:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"hub_url": "https://hub.example.com",
|
||||||
|
"agent_id": "agent-abc123",
|
||||||
|
"token": "rha_...",
|
||||||
|
"tx_enabled": true,
|
||||||
|
"tx_max_gain_db": -10.0,
|
||||||
|
"tx_max_duration_s": 60,
|
||||||
|
"tx_allowed_freq_ranges": [[2.4e9, 2.5e9], [5.7e9, 5.8e9]]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
File permissions are enforced to `0600` by `save()`.
|
||||||
232
docs/per_user_registration_keys_plan.md
Normal file
232
docs/per_user_registration_keys_plan.md
Normal file
|
|
@ -0,0 +1,232 @@
|
||||||
|
---
|
||||||
|
name: Per-user agent registration keys
|
||||||
|
description: Replace the shared [wac] API_KEY with per-user registration keys issued from the RIA Agents page on RIA Hub.
|
||||||
|
type: plan
|
||||||
|
---
|
||||||
|
|
||||||
|
# Per-user agent registration keys — plan
|
||||||
|
|
||||||
|
**Status:** design only; nothing implemented.
|
||||||
|
**Owner (toolkit side):** `ria-toolkit-oss`
|
||||||
|
**Owner (hub side):** `ria-hub` / `controller`
|
||||||
|
**Related:** [screens_agent_handoff.md](./screens_agent_handoff.md), [agent_tx_protocol.md](./agent_tx_protocol.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context (current state)
|
||||||
|
|
||||||
|
Today, `ria-agent register` calls `POST {hub_url}/screens/agents/register` with
|
||||||
|
an `X-API-Key` header ([cli.py:41-64](../src/ria_toolkit_oss/agent/cli.py#L41-L64)).
|
||||||
|
The hub validates that header against a single shared secret — `[wac] API_KEY`
|
||||||
|
in the hub's `app.ini` ([legacy_executor.py:821-822](../src/ria_toolkit_oss/agent/legacy_executor.py#L821-L822)).
|
||||||
|
The hub responds with `{agent_id, token}`; the agent persists both to
|
||||||
|
`~/.ria/agent.json` and uses `token` as the bearer on the WS handshake
|
||||||
|
afterwards.
|
||||||
|
|
||||||
|
Consequences of the shared secret:
|
||||||
|
|
||||||
|
- Every agent operator holds the same key → no per-user attribution in logs.
|
||||||
|
- Revoking one operator forces a rotation across every deployed agent.
|
||||||
|
- Key-in-CLI-history leaks escalate to the whole fleet.
|
||||||
|
- Nothing ties a registered agent to a human in the hub's user table.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
A user signs into `riahub.ai`, opens an **RIA Agents** page, mints a key, and
|
||||||
|
uses it once with `ria-agent register`. The resulting agent is owned by that
|
||||||
|
user; the key can be revoked without affecting anyone else's agents.
|
||||||
|
|
||||||
|
The agent-side `token` returned by `/screens/agents/register` keeps its current
|
||||||
|
role (bearer for the WS handshake). Only the *registration* credential
|
||||||
|
changes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## User flow
|
||||||
|
|
||||||
|
1. User signs into `https://riahub.ai`.
|
||||||
|
2. User navigates to **Settings → RIA Agents** (or a top-level `/agents`
|
||||||
|
page — see open question O1).
|
||||||
|
3. User clicks **Generate registration key**. A modal shows the key **once**,
|
||||||
|
with copy-to-clipboard. Only a prefix + hash is stored server-side.
|
||||||
|
4. User runs, on the agent host:
|
||||||
|
```
|
||||||
|
ria-agent register --hub https://riahub.ai --api-key ria_reg_<...>
|
||||||
|
```
|
||||||
|
5. Hub validates the key, creates an agent row owned by the user, marks the
|
||||||
|
key as `consumed` (one-shot) or bumps `last_used_at` (multi-use — see O2),
|
||||||
|
and returns `{agent_id, token}` exactly as today.
|
||||||
|
6. The agent list on the same page shows the new agent's `name`, `hardware[]`,
|
||||||
|
`last_heartbeat`, and **Revoke** / **Rename** actions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scope split
|
||||||
|
|
||||||
|
### Toolkit (`ria-toolkit-oss`)
|
||||||
|
|
||||||
|
The CLI already sends `X-API-Key`, so no protocol change is required. Two
|
||||||
|
small quality-of-life changes:
|
||||||
|
|
||||||
|
| # | Change | File |
|
||||||
|
|---|--------|------|
|
||||||
|
| T1 | Update `--api-key` help text and [cli.py:8 docstring](../src/ria_toolkit_oss/agent/cli.py#L8) to say "personal registration key from the RIA Agents page" rather than "Hub API key". | [agent/cli.py](../src/ria_toolkit_oss/agent/cli.py) |
|
||||||
|
| T2 | On registration failure, if the response body is JSON with a `reason` field (`invalid_key` / `expired` / `already_consumed` / `revoked`), surface it verbatim instead of the raw `HTTPError`. Makes user-facing errors actionable. | [agent/cli.py:56-61](../src/ria_toolkit_oss/agent/cli.py#L56-L61) |
|
||||||
|
|
||||||
|
No change to `config.py`, `ws_client.py`, or the streamer — the `token`
|
||||||
|
returned by register is still what authenticates the WS connection.
|
||||||
|
|
||||||
|
### Hub (`ria-hub` / `controller`)
|
||||||
|
|
||||||
|
Paths below are inferred from [screens_agent_handoff.md](./screens_agent_handoff.md)
|
||||||
|
(`controller/app/modules/...`). Hub team should sanity-check before starting.
|
||||||
|
|
||||||
|
#### Prior art — check RIA Conductor first
|
||||||
|
|
||||||
|
The RIA Conductor feature is believed to already implement similar key
|
||||||
|
generation (likely for authenticating conductors to the hub). **Before
|
||||||
|
building anything in this section, read the Conductor key code** and decide
|
||||||
|
whether to:
|
||||||
|
|
||||||
|
- **Reuse** it as-is (shared `registration_keys` table, `kind` column
|
||||||
|
discriminating `conductor` vs. `agent`) — preferred if the shapes line up.
|
||||||
|
- **Extract** the hashing / minting / revoke primitives into a shared
|
||||||
|
`registration_keys` module that both features depend on.
|
||||||
|
- **Fork** a parallel `agent_registration_keys` table — only if the
|
||||||
|
Conductor model is materially different (e.g. per-org scoping, different
|
||||||
|
lifetime rules) and forcing a merge would distort one or both features.
|
||||||
|
|
||||||
|
Whichever path is chosen should be decided up front and noted on the PR, so
|
||||||
|
we don't end up with two near-identical key subsystems by accident. The
|
||||||
|
security notes below (argon2id, one-time reveal, rate limits, audit logging)
|
||||||
|
apply regardless of which path is taken — confirm Conductor already does
|
||||||
|
these; if not, the fix belongs in the shared code, not this feature.
|
||||||
|
|
||||||
|
#### Data model
|
||||||
|
|
||||||
|
New collection (Mongo) or table (if Postgres is used for users):
|
||||||
|
|
||||||
|
```
|
||||||
|
registration_keys
|
||||||
|
_id
|
||||||
|
user_id # FK to hub users
|
||||||
|
name # user-supplied label, e.g. "lab laptop"
|
||||||
|
key_prefix # first 8 chars of the plaintext, for UI display
|
||||||
|
key_hash # argon2id or bcrypt of the full plaintext
|
||||||
|
created_at
|
||||||
|
expires_at # optional; null = no expiry
|
||||||
|
consumed_at # null until first successful registration (if one-shot)
|
||||||
|
revoked_at # null unless explicitly revoked
|
||||||
|
last_used_at # updated on every successful use (if multi-use)
|
||||||
|
```
|
||||||
|
|
||||||
|
Augment the existing agents collection with `owner_user_id` (FK) and
|
||||||
|
`registered_via_key_id` (FK to `registration_keys._id`).
|
||||||
|
|
||||||
|
Decide O2 before building: one-shot vs. reusable. Recommendation: one-shot by
|
||||||
|
default with an optional "reusable for N days" toggle, since one-shot is the
|
||||||
|
lower-blast-radius default and matches how GitHub/Gitea deploy keys behave.
|
||||||
|
|
||||||
|
#### Endpoints
|
||||||
|
|
||||||
|
| # | Endpoint | Notes |
|
||||||
|
|---|----------|-------|
|
||||||
|
| H1 | `POST /api/v1/user/registration-keys` | Auth: session cookie. Body: `{name, expires_in_days?, reusable?}`. Returns plaintext key **once**. |
|
||||||
|
| H2 | `GET /api/v1/user/registration-keys` | Auth: session cookie. Lists the caller's keys (prefix + metadata, never plaintext). |
|
||||||
|
| H3 | `DELETE /api/v1/user/registration-keys/{id}` | Auth: session cookie. Revokes. |
|
||||||
|
| H4 | `POST /screens/agents/register` (existing) | Change auth: look up `X-API-Key` by hash instead of string-compare against `[wac] API_KEY`. Reject if revoked / expired / consumed. Set `owner_user_id` on the new agent row. |
|
||||||
|
| H5 | `GET /api/v1/user/agents` | Auth: session cookie. Lists the caller's agents for the UI. |
|
||||||
|
| H6 | `DELETE /api/v1/user/agents/{id}` | Auth: session cookie. De-registers and closes any live WS. |
|
||||||
|
|
||||||
|
H4 is the only backwards-incompatible change. See the migration section for
|
||||||
|
how to ship it without breaking existing deployments.
|
||||||
|
|
||||||
|
#### Frontend
|
||||||
|
|
||||||
|
New page — **Settings → RIA Agents** — two panels:
|
||||||
|
|
||||||
|
- **Registration keys:** table (name, prefix, created, expires, last used,
|
||||||
|
revoke button) + "Generate" button that opens the one-time-reveal modal.
|
||||||
|
- **Agents:** table (name, hardware, status, last heartbeat, rename, revoke).
|
||||||
|
|
||||||
|
Matches the existing Gitea-style Settings sidebar if RIA Hub is Gitea-based
|
||||||
|
(O3).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Migration from the shared `[wac] API_KEY`
|
||||||
|
|
||||||
|
The shared key is likely in use on every existing deployment. To avoid a
|
||||||
|
flag day:
|
||||||
|
|
||||||
|
1. **Dual-accept window.** H4 accepts *either* a per-user key (lookup by
|
||||||
|
hash) *or* the legacy `[wac] API_KEY` string. When the legacy key is used,
|
||||||
|
the resulting agent has `owner_user_id = null` and a warning is logged.
|
||||||
|
2. **Admin UI surfaces "unowned" agents** so an admin can re-assign them or
|
||||||
|
ask owners to re-register.
|
||||||
|
3. **Deprecation window of one release**, then H4 rejects the legacy key and
|
||||||
|
the `[wac] API_KEY` config is removed from `app.ini`.
|
||||||
|
|
||||||
|
No toolkit-side migration needed — existing `~/.ria/agent.json` files already
|
||||||
|
store the post-registration `token`, which keeps working regardless of how
|
||||||
|
registration itself was authenticated.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Security notes
|
||||||
|
|
||||||
|
- Store `key_hash` with a password hash (argon2id), not a fast hash. The key
|
||||||
|
is a secret-equivalent: treat it like a password.
|
||||||
|
- Plaintext key format: `ria_reg_<base64url of 32 random bytes>`. Prefix makes
|
||||||
|
the purpose obvious in leaked logs and lets scanners (trufflehog etc.)
|
||||||
|
recognize it.
|
||||||
|
- One-time reveal in the UI — never persist or re-display the plaintext.
|
||||||
|
- Rate-limit H4 per source IP and per `key_prefix` to blunt brute-force on
|
||||||
|
leaked prefixes. Lock a key out after N failed attempts in M minutes.
|
||||||
|
- Log every H4 call (success + failure, with key prefix and source IP)
|
||||||
|
to the audit trail.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- **O1.** Where does the page live? A top-level `/agents` route is
|
||||||
|
discoverable; `/user/settings/agents` matches Gitea's existing IA. Pick
|
||||||
|
before F7 (frontend task).
|
||||||
|
- **O2.** One-shot vs. reusable keys (default and whether both are offered).
|
||||||
|
Recommendation above; needs product sign-off.
|
||||||
|
- **O3.** Is RIA Hub's web UI really a Gitea fork? URL patterns
|
||||||
|
(`/qoherent/-/packages/...`, `.git` clones) suggest yes, but the "Settings"
|
||||||
|
integration plan depends on confirming this. If it isn't, F7 is a standalone
|
||||||
|
page instead.
|
||||||
|
- **O4.** Does the agent bearer `token` need per-user scoping too, or is
|
||||||
|
ownership-at-registration enough? Today the token is opaque and not tied
|
||||||
|
to a user in the WS handler. Probably fine to defer until after per-user
|
||||||
|
keys ship.
|
||||||
|
- **O5.** Should admins be able to mint keys on behalf of other users (for
|
||||||
|
onboarding)? If yes, H1 needs an admin-scoped variant.
|
||||||
|
- **O6.** Conductor reuse decision — reuse / extract / fork. Must be answered
|
||||||
|
before any hub-side code lands. See "Prior art" above.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- SSO / OIDC for agent-to-hub auth (current `token` bearer is kept as-is).
|
||||||
|
- Per-agent capability scoping beyond what `--allow-tx` already does at
|
||||||
|
registration time.
|
||||||
|
- Fleet provisioning (N agents from one key); covered instead by "reusable"
|
||||||
|
flag in O2 if that's the chosen default.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## MVP cut
|
||||||
|
|
||||||
|
If the hub team wants the smallest shippable slice:
|
||||||
|
|
||||||
|
- H1, H2, H3, H4 (with dual-accept), H5.
|
||||||
|
- Frontend: registration-keys panel only; reuse the existing agents admin
|
||||||
|
view if one already exists.
|
||||||
|
- T1 toolkit copy-change.
|
||||||
|
|
||||||
|
Defer H6, rename flows, T2, and audit logging to a follow-up.
|
||||||
104
docs/ria_app_hub_handoff.md
Normal file
104
docs/ria_app_hub_handoff.md
Normal file
|
|
@ -0,0 +1,104 @@
|
||||||
|
# `ria-app` Hub-Side Handoff
|
||||||
|
|
||||||
|
**Repo:** `ria-hub`
|
||||||
|
**Goal:** Make containerized apps built by Application Composer self-describing so the new `ria-app` CLI in `ria-toolkit-oss` can auto-configure GPU/USB/network flags at `docker run` time. No user copy-paste of flags.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context — what exists today
|
||||||
|
|
||||||
|
In `ria-toolkit-oss` (branch `screens-connection`) there is now a `ria-app` CLI:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ria-app configure --registry registry.riahub.ai --namespace qoherent
|
||||||
|
ria-app pull <app>[:tag]
|
||||||
|
ria-app run <app>[:tag] [--config config.yaml]
|
||||||
|
ria-app list
|
||||||
|
ria-app logs <app> [-f]
|
||||||
|
ria-app stop <app>
|
||||||
|
```
|
||||||
|
|
||||||
|
`ria-app run` inspects OCI image labels and auto-adds runtime flags:
|
||||||
|
|
||||||
|
| Label | Value (example) | Effect |
|
||||||
|
|---|---|---|
|
||||||
|
| `ria.profile` | `native-x86`, `nvidia-x86`, `holoscan` | `nvidia`/`holoscan`/`cuda` → adds `--gpus all` |
|
||||||
|
| `ria.hardware` | comma list: `pluto,usrp,rtlsdr,hackrf,bladerf,thinkrf` | USB-attached SDRs → `--device /dev/bus/usb`; networked SDRs → `--net host` |
|
||||||
|
| `ria.app` | `<app-name>` | Used by `ria-app list` to filter images |
|
||||||
|
| `ria.version` | `<git sha or semver>` | Informational |
|
||||||
|
|
||||||
|
If the labels are missing, `ria-app run` still works but can't auto-configure — the user has to pass `--docker-args ...` themselves. So the value here is entirely in getting CI to stamp the labels.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What to change in `ria-hub`
|
||||||
|
|
||||||
|
### 1. Stamp OCI labels on every built image
|
||||||
|
|
||||||
|
In the Application Composer build flow (follow the path from `application_composer.go:172` `ComposerBuildTrigger` → generated `.riahub/workflows/*.yml` → `sample-build-tools` `full_generator.py` → `Dockerfile` emission), add `LABEL` instructions to the generated Dockerfile. The values should be computed from the app JSON the user submitted, not hard-coded:
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
LABEL ria.app="${APP_NAME}"
|
||||||
|
LABEL ria.profile="${PROFILE}" # native-x86 | nvidia-x86 | holoscan | ...
|
||||||
|
LABEL ria.hardware="${HARDWARE_CSV}" # e.g. "pluto,usrp" (empty string if none)
|
||||||
|
LABEL ria.version="${GIT_SHA}"
|
||||||
|
LABEL ria.operators="${OPERATORS_CSV}" # optional, nice for debugging
|
||||||
|
```
|
||||||
|
|
||||||
|
`HARDWARE_CSV` derivation: walk the operator graph in the submitted app JSON and collect the set of hardware backends that any operator requires. The mapping from operator → hardware tag should live next to the existing `operator_generator.py` apt-dep resolution (that code already knows, per operator, whether it needs `libuhd-dev`, `libad9361-dev`, `libhackrf-dev`, `librtlsdr-dev`, etc.). Reuse that table — just emit the short tag (`usrp`, `pluto`, `hackrf`, `rtlsdr`) alongside the apt package name.
|
||||||
|
|
||||||
|
Allowed hardware tags (must match what `ria-app` recognizes):
|
||||||
|
|
||||||
|
- `pluto`, `rtlsdr`, `hackrf`, `bladerf` → USB
|
||||||
|
- `usrp`, `thinkrf` → network
|
||||||
|
- (extend here when new SDR backends are added)
|
||||||
|
|
||||||
|
If an operator needs both (e.g. Pluto over USB *and* its iio network endpoint), list it once — `ria-app` already applies both USB and host-net when `pluto` appears.
|
||||||
|
|
||||||
|
### 2. Prefer `LABEL` over `ARG`-only
|
||||||
|
|
||||||
|
The CI job likely already passes things like `APP_NAME` and `GIT_SHA` as build args. Those args disappear after build unless promoted to `LABEL`. Make sure each of the five labels above ends up in the final image layer (verify with `docker image inspect --format '{{json .Config.Labels}}' <ref>`).
|
||||||
|
|
||||||
|
### 3. Push with both `:<sha>` and `:latest` tags
|
||||||
|
|
||||||
|
`ria-app` defaults to `:latest` when the user omits a tag. If CI only pushes immutable SHA tags today, also push `:latest` on main-branch builds so `ria-app run my-classifier` Just Works.
|
||||||
|
|
||||||
|
### 4. (Optional but recommended) App index endpoint
|
||||||
|
|
||||||
|
Add `GET /apps` to the hub API returning something like:
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"name": "my-classifier",
|
||||||
|
"image": "registry.riahub.ai/qoherent/my-classifier:latest",
|
||||||
|
"profile": "nvidia-x86",
|
||||||
|
"hardware": ["pluto"],
|
||||||
|
"updated_at": "2026-04-14T10:00:00Z"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
This lets `ria-app list --remote` show available apps without the user knowing image names. Not required for MVP — skip if it adds scope.
|
||||||
|
|
||||||
|
### 5. (Optional) Ship a default `config.yaml` inside the image at a known path
|
||||||
|
|
||||||
|
`ria-app run --config <path>` mounts the user's config to `/config/config.yaml` and sets `RIA_CONFIG=/config/config.yaml`. The runtime already falls back to an embedded config per your handoff notes, so this just needs to keep working — no change unless you want to standardize the embedded path.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Acceptance checklist
|
||||||
|
|
||||||
|
- [ ] A Composer-built image for a native-x86 app with a Pluto operator has labels: `ria.profile=native-x86`, `ria.hardware=pluto`, `ria.app=<name>`, `ria.version=<sha>`.
|
||||||
|
- [ ] A Composer-built image for an nvidia-x86 app has `ria.profile=nvidia-x86`.
|
||||||
|
- [ ] `docker image inspect --format '{{json .Config.Labels}}' <ref>` shows all five labels.
|
||||||
|
- [ ] `:latest` tag is pushed for main-branch builds.
|
||||||
|
- [ ] Running `ria-app run <app>` on a user's machine starts the container with the right `--gpus` / `--device` / `--net` flags without the user passing anything beyond the app name.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Anything on the `ria-toolkit-oss` side — the CLI is already implemented on branch `screens-connection`.
|
||||||
|
- Changes to the generated C++ code, CMakeLists, or runtime config lookup.
|
||||||
|
- Artifact downloads — we're distributing via the container registry only.
|
||||||
257
docs/screens_agent_handoff.md
Normal file
257
docs/screens_agent_handoff.md
Normal file
|
|
@ -0,0 +1,257 @@
|
||||||
|
# Screens Agent Streamer — Hand-off
|
||||||
|
|
||||||
|
**Branch:** `screens-connection` in `ria-toolkit-oss`
|
||||||
|
**Status:** Part A complete (tests passing, 25/25). Part B is pending in the `ria-hub` repo.
|
||||||
|
**Related docs:** [screens_agent_streamer_plan.md](./screens_agent_streamer_plan.md), [../screens_connection_updates.md](../screens_connection_updates.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What's done (Part A — this repo)
|
||||||
|
|
||||||
|
### Phase 1 — SDR foundation
|
||||||
|
|
||||||
|
- Added `ria_toolkit_oss.sdr.detect_available() -> dict[str, type]` that probes
|
||||||
|
every driver module and returns the map of importable driver classes.
|
||||||
|
Importability is used as a proxy for "user has installed this driver's
|
||||||
|
optional dep"; it does **not** probe for physical hardware.
|
||||||
|
- Added `SdrDisconnectedError` (subclass of `SDRError`) plus a
|
||||||
|
`translate_disconnect(exc)` helper in [sdr/sdr.py](../src/ria_toolkit_oss/sdr/sdr.py).
|
||||||
|
Pattern-matches USB/device-drop exceptions (`ENODEV`/`EIO`, "no such device",
|
||||||
|
"broken pipe", etc.) and converts them so the streamer can distinguish a
|
||||||
|
real hardware failure from a transient error.
|
||||||
|
- Audited every driver under `sdr/` for GUI imports — none found. All drivers
|
||||||
|
are headless-clean at import time.
|
||||||
|
- `Pluto.rx(num_samples)` now wraps `self.radio.rx()` with
|
||||||
|
`translate_disconnect`. The same one-liner pattern can be applied to other
|
||||||
|
drivers (hackrf/rtlsdr/usrp/blade/thinkrf) when they get wired to the
|
||||||
|
streamer — deferred until each is needed.
|
||||||
|
|
||||||
|
**Not done (Phase 1 Task 4):** SigMF recording validation across radio types —
|
||||||
|
needs real captures from each device, out of scope without hardware.
|
||||||
|
|
||||||
|
### Phase 2 — Agent package restructure
|
||||||
|
|
||||||
|
Moved the former `src/ria_toolkit_oss/agent.py` into a package:
|
||||||
|
|
||||||
|
```
|
||||||
|
src/ria_toolkit_oss/agent/
|
||||||
|
├── __init__.py # re-exports NodeAgent + main for back-compat
|
||||||
|
├── legacy_executor.py # former agent.py, unchanged behavior
|
||||||
|
├── streamer.py # new WebSocket IQ streamer
|
||||||
|
├── ws_client.py # persistent WS client + heartbeat + reconnect
|
||||||
|
├── hardware.py # wraps sdr.detect_available() for heartbeat payloads
|
||||||
|
├── config.py # ~/.ria/agent.json load/save (0600 perms)
|
||||||
|
└── cli.py # unified CLI (run / stream / detect / register)
|
||||||
|
```
|
||||||
|
|
||||||
|
Back-compat preserved: `from ria_toolkit_oss.agent import NodeAgent` still
|
||||||
|
works, and bare `ria-agent ...` with no subcommand (or with legacy flags
|
||||||
|
like `--hub`) falls through to the original long-poll executor.
|
||||||
|
|
||||||
|
### Phase 3 — Streamer implementation
|
||||||
|
|
||||||
|
- `ws_client.WsClient` — `async run()` loop, JSON heartbeats on a timer,
|
||||||
|
auto-reconnect on drop, bearer-token auth header on connect.
|
||||||
|
- `streamer.Streamer` — handles `start` / `stop` / `configure` messages,
|
||||||
|
opens the SDR via an injectable `sdr_factory`, runs capture in an executor
|
||||||
|
thread, and ships raw interleaved float32 IQ bytes per `rx()` call.
|
||||||
|
- `hardware.heartbeat_payload(status, app_id)` — `{type, hardware[], status}`.
|
||||||
|
- `config.AgentConfig` — dataclass round-tripped through JSON, unknown keys
|
||||||
|
preserved in an `extra` dict.
|
||||||
|
- CLI subcommands: `run` (legacy), `stream`, `detect`, `register`.
|
||||||
|
|
||||||
|
### Phase 4 — Protocol
|
||||||
|
|
||||||
|
Matches `screens_connection_updates.md` §"WebSocket Protocol" exactly:
|
||||||
|
|
||||||
|
| Direction | Message |
|
||||||
|
|-----------|---------|
|
||||||
|
| A → S (JSON) | `{type: heartbeat, hardware[], status}` |
|
||||||
|
| A → S (JSON) | `{type: status, status, app_id}` |
|
||||||
|
| A → S (JSON) | `{type: error, app_id, message}` |
|
||||||
|
| A → S (binary) | raw interleaved float32 IQ, one frame per `rx()` |
|
||||||
|
| S → A (JSON) | `{type: start, app_id, radio_config}` |
|
||||||
|
| S → A (JSON) | `{type: stop, app_id}` |
|
||||||
|
| S → A (JSON) | `{type: configure, app_id, radio_config}` — applied at next boundary |
|
||||||
|
|
||||||
|
**Auth decision:** bearer token in `Authorization` header on the initial
|
||||||
|
handshake. (Open questions in the plan around first-frame auth / mid-buffer
|
||||||
|
`configure` / backpressure remain open.)
|
||||||
|
|
||||||
|
### Phase 5 — Tests
|
||||||
|
|
||||||
|
25 tests, all green under `poetry run pytest tests/agent/`:
|
||||||
|
|
||||||
|
- `test_hardware.py` — `detect_available`, heartbeat payload shape
|
||||||
|
- `test_config.py` — round-trip, missing-file fallback, extra-key preservation
|
||||||
|
- `test_streamer.py` — start/stream/stop against `MockSDR` + fake WS, error
|
||||||
|
frames, configure queueing
|
||||||
|
- `test_disconnect.py` — `translate_disconnect` patterns + streamer reports
|
||||||
|
`SDR disconnected:` and closes the SDR
|
||||||
|
- `test_ws_client.py` — real local `websockets` server: heartbeat on connect,
|
||||||
|
auto-reconnect after server drop, malformed-frame resilience
|
||||||
|
- `test_integration.py` — end-to-end heartbeat → start → 3 binary IQ frames → stop
|
||||||
|
- `test_legacy.py` — regression: `NodeAgent` still importable
|
||||||
|
|
||||||
|
### Dependency / build changes
|
||||||
|
|
||||||
|
- Added `websockets (>=12.0,<14.0)` to `[tool.poetry.group.agent.dependencies]`.
|
||||||
|
- Repointed the `ria-agent` console script from `ria_toolkit_oss.agent:main`
|
||||||
|
to `ria_toolkit_oss.agent.cli:main` (the unified CLI, which still calls
|
||||||
|
through to the legacy `main` for back-compat).
|
||||||
|
|
||||||
|
### Files changed
|
||||||
|
|
||||||
|
```
|
||||||
|
M pyproject.toml
|
||||||
|
M poetry.lock
|
||||||
|
M src/ria_toolkit_oss/sdr/__init__.py
|
||||||
|
M src/ria_toolkit_oss/sdr/sdr.py
|
||||||
|
M src/ria_toolkit_oss/sdr/pluto.py
|
||||||
|
R src/ria_toolkit_oss/agent.py -> src/ria_toolkit_oss/agent/legacy_executor.py
|
||||||
|
A src/ria_toolkit_oss/agent/{__init__,cli,config,hardware,streamer,ws_client}.py
|
||||||
|
A tests/agent/{__init__,test_config,test_disconnect,test_hardware,
|
||||||
|
test_integration,test_legacy,test_streamer,test_ws_client}.py
|
||||||
|
A docs/screens_agent_streamer_plan.md
|
||||||
|
A docs/screens_agent_handoff.md (this file)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What's left (Part B — `ria-hub` repo)
|
||||||
|
|
||||||
|
In priority order. B1 + B2 + B3 + B6 are the MVP; everything else hardens.
|
||||||
|
|
||||||
|
### Server-side (MVP)
|
||||||
|
|
||||||
|
| # | Task | File / area |
|
||||||
|
|---|------|-------------|
|
||||||
|
| B1 | Implement `AgentDataSource` (DataSource ABC, reads IQ from WS connection) and register in `build_data_source()` | `controller/app/modules/screens/data_sources.py` |
|
||||||
|
| B2 | Add `"agent"` to `dataSource.type` enum in manifest schema; update Pydantic / JSON schema validators | `controller/app/modules/screens/graph_derivation.py` + schema files |
|
||||||
|
| B3 | Agent WebSocket endpoint `POST /api/agent/ws` (or `GET` with Upgrade) — accepts agent connections, auths on bearer token, bridges the connection to the Celery task's `AgentDataSource` | new `controller/app/modules/agent/routes.py` |
|
||||||
|
| B6 | Celery wiring: when `dataSource.type == "agent"`, look up the connected agent by `agent_id`, forward `radio_config` as a `start` message, and feed received IQ bytes into the inference loop via `AgentDataSource.next_chunk()` | `controller/app/modules/screens/tasks.py` |
|
||||||
|
|
||||||
|
### Server-side (hardening)
|
||||||
|
|
||||||
|
| # | Task |
|
||||||
|
|---|------|
|
||||||
|
| B4 | Agent registry — MongoDB collection: `agent_id`, `hardware[]`, `last_heartbeat`, `online`, registration tokens |
|
||||||
|
| B5 | `POST /api/agent/register` returning `{agent_id, token}` for `~/.ria/agent.json` |
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
|
||||||
|
| # | Task |
|
||||||
|
|---|------|
|
||||||
|
| B7 | Device / agent picker in the Screens app config (Vue 3) |
|
||||||
|
| B8 | Agents list / status admin view |
|
||||||
|
|
||||||
|
### Protocol contract (keep in lockstep with Part A)
|
||||||
|
|
||||||
|
- Binary frames are interleaved float32 IQ, one frame per `radio.rx()` call.
|
||||||
|
- `radio_config` is forwarded verbatim from manifest `dataSource.params`.
|
||||||
|
Minimum keys the agent handles: `device`, `identifier`, `sample_rate`,
|
||||||
|
`center_frequency`, `gain`, `buffer_size`.
|
||||||
|
- `configure` messages from the server apply at the next capture boundary
|
||||||
|
(current agent implementation).
|
||||||
|
- Agent authenticates with bearer token in `Authorization` header on the
|
||||||
|
handshake.
|
||||||
|
|
||||||
|
### Still-open protocol questions (pin down before B3 lands)
|
||||||
|
|
||||||
|
- Auth frame as fallback when proxies strip `Authorization`?
|
||||||
|
- Mid-buffer `configure` application for tighter retune latency?
|
||||||
|
- Backpressure policy when the server's inference loop is slower than the
|
||||||
|
agent's `rx()` cadence — drop, queue, or pause the agent via a `pause`
|
||||||
|
control frame?
|
||||||
|
|
||||||
|
### Manifest example (new `agent` mode)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"dataSource": {
|
||||||
|
"type": "agent",
|
||||||
|
"device": "pluto",
|
||||||
|
"agent_id": "agent-abc123",
|
||||||
|
"params": {
|
||||||
|
"identifier": "ip:192.168.3.1",
|
||||||
|
"sample_rate": 1000000,
|
||||||
|
"center_frequency": 2450000000,
|
||||||
|
"gain": 40,
|
||||||
|
"buffer_size": 1024
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"preprocess": "magnitude_phase_window_stats",
|
||||||
|
"config": {"inference": {"knownDevices": [], "interval": 1}}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline invariant
|
||||||
|
|
||||||
|
**No changes to `inference_core.py`, `preprocessors.py`, or
|
||||||
|
`run_onnx_chain_loop()`.** `AgentDataSource` is a drop-in replacement for
|
||||||
|
`SdrDataSource`; everything downstream is unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Local test setup
|
||||||
|
|
||||||
|
### 1. Branch layout
|
||||||
|
|
||||||
|
- `ria-toolkit-oss` — branch **`screens-connection`** (this work).
|
||||||
|
- `ria-hub` — whatever branch you'll be doing Part B on (create a matching
|
||||||
|
`screens-connection` branch there to keep the hand-off clean).
|
||||||
|
|
||||||
|
### 2. Install the toolkit branch into your local RIA Hub
|
||||||
|
|
||||||
|
The hub declares `ria-toolkit-oss` as a git dep in its `pyproject.toml`.
|
||||||
|
Point it at this branch:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# ria-hub/pyproject.toml
|
||||||
|
ria-toolkit-oss = { git = "https://riahub.ai/qoherent/ria-toolkit-oss.git", branch = "screens-connection" }
|
||||||
|
```
|
||||||
|
|
||||||
|
Or for a fully local dev loop (edits visible without reinstall):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# from ria-hub repo
|
||||||
|
poetry run pip install -e /home/qrf/ria-toolkit-oss
|
||||||
|
# plus the agent extra so websockets is available server-side too
|
||||||
|
poetry run pip install websockets
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Commit this branch and push (optional, for CI / shared dev)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/qrf/ria-toolkit-oss
|
||||||
|
git add -A
|
||||||
|
git commit -m "Add WebSocket IQ streamer agent (Part A)"
|
||||||
|
git push -u origin screens-connection
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Try the agent locally against a mock server
|
||||||
|
|
||||||
|
You can exercise the whole Part A stack without RIA Hub by running the
|
||||||
|
integration test:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/qrf/ria-toolkit-oss
|
||||||
|
poetry run pytest tests/agent/test_integration.py -v
|
||||||
|
```
|
||||||
|
|
||||||
|
Or manually, once Part B lands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# on the user machine (mock hardware is fine)
|
||||||
|
ria-agent register --url https://localhost:8000 --token <tok>
|
||||||
|
ria-agent detect # lists: mock, pluto, ... whatever's importable
|
||||||
|
ria-agent stream --url ws://localhost:8000/api/agent/ws
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. End-to-end integration test (after Part B MVP)
|
||||||
|
|
||||||
|
1. Start RIA Hub (FastAPI + Celery + Mongo + Redis) with the Part B branch.
|
||||||
|
2. Run `ria-agent stream` on your laptop.
|
||||||
|
3. Create a Screens app with `dataSource.type: "agent"` and `device: "mock"`.
|
||||||
|
4. Start the app; confirm the agent logs "streaming" and the SSE stream
|
||||||
|
shows inference metrics flowing.
|
||||||
156
docs/screens_agent_streamer_plan.md
Normal file
156
docs/screens_agent_streamer_plan.md
Normal file
|
|
@ -0,0 +1,156 @@
|
||||||
|
# Screens Agent Streamer — Implementation Plan
|
||||||
|
|
||||||
|
**Source doc:** [screens_connection_updates.md](../screens_connection_updates.md)
|
||||||
|
**Created:** 2026-04-13
|
||||||
|
**Goal:** Add a thin WebSocket-based IQ streaming agent to `ria-toolkit-oss`, alongside the existing long-poll `NodeAgent`, and wire up the RIA Hub server to consume it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architectural Decision
|
||||||
|
|
||||||
|
The existing [src/ria_toolkit_oss/agent.py](../src/ria_toolkit_oss/agent.py) (`NodeAgent`) uses HTTP long-polling and runs ONNX inference locally. It stays as-is.
|
||||||
|
|
||||||
|
The new **streamer agent** described in `screens_connection_updates.md` is a *different* execution mode — thin, WebSocket-based, server-driven inference. It will be added as a new submodule and exposed as a new CLI subcommand. Both modes coexist; users pick one based on deployment needs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part A — ria-toolkit-oss (this repo)
|
||||||
|
|
||||||
|
### Phase 1 — SDR foundation
|
||||||
|
|
||||||
|
| # | Task | File(s) | Priority |
|
||||||
|
|---|------|---------|----------|
|
||||||
|
| 1 | Add `detect_available() -> dict[str, type]` that probes every driver without importing GUI deps | [src/ria_toolkit_oss/sdr/__init__.py](../src/ria_toolkit_oss/sdr/__init__.py) | P0 |
|
||||||
|
| 2 | Audit each SDR driver for headless cleanliness (no matplotlib/Qt at import time) | `src/ria_toolkit_oss/sdr/{pluto,hackrf,rtlsdr,usrp,blade,thinkrf}.py` | P1 |
|
||||||
|
| 3 | Raise typed `SdrDisconnectedError` from `radio.rx()` on USB drop instead of crashing | `src/ria_toolkit_oss/sdr/sdr.py` + drivers | P1 |
|
||||||
|
| 4 | Validate `load_recording` against SigMF captures from each radio type | `tests/io/` | P2 |
|
||||||
|
|
||||||
|
### Phase 2 — Agent package restructure (non-breaking)
|
||||||
|
|
||||||
|
Promote the existing module to a package and add the streamer next to it:
|
||||||
|
|
||||||
|
```
|
||||||
|
src/ria_toolkit_oss/
|
||||||
|
├── agent.py # DELETE after move
|
||||||
|
└── agent/
|
||||||
|
├── __init__.py # re-export NodeAgent for back-compat
|
||||||
|
├── legacy_executor.py # former agent.py (NodeAgent, unchanged behavior)
|
||||||
|
├── streamer.py # NEW — thin IQ streamer loop
|
||||||
|
├── ws_client.py # NEW — persistent WS client + heartbeat + reconnect
|
||||||
|
├── hardware.py # NEW — wraps sdr.detect_available()
|
||||||
|
├── config.py # shared: ~/.ria/agent.json load/save
|
||||||
|
└── cli.py # unified CLI with subcommands
|
||||||
|
```
|
||||||
|
|
||||||
|
Back-compat requirement: `from ria_toolkit_oss.agent import NodeAgent` must keep working. Keep the `ria-agent` console script entry point; expose both modes via subcommands.
|
||||||
|
|
||||||
|
### Phase 3 — Streamer implementation
|
||||||
|
|
||||||
|
| # | Task | File | Priority |
|
||||||
|
|---|------|------|----------|
|
||||||
|
| 5 | `ws_client.py` — persistent WebSocket, auto-reconnect, heartbeat loop | `src/ria_toolkit_oss/agent/ws_client.py` | P0 |
|
||||||
|
| 6 | `streamer.py` — main loop: receive `start` → open SDR → `radio.rx()` → send binary IQ → handle `stop`/`configure` | `src/ria_toolkit_oss/agent/streamer.py` | P0 |
|
||||||
|
| 7 | `hardware.py` — heartbeat payload builder, uses `sdr.detect_available()` | `src/ria_toolkit_oss/agent/hardware.py` | P0 |
|
||||||
|
| 8 | `config.py` — `~/.ria/agent.json` read/write, registration token storage | `src/ria_toolkit_oss/agent/config.py` | P1 |
|
||||||
|
| 9 | CLI subcommands: `ria-agent register`, `detect`, `stream` (new), `run` (legacy long-poll) | `src/ria_toolkit_oss/agent/cli.py` | P0 |
|
||||||
|
| 10 | Add `websockets` to dependencies | [pyproject.toml](../pyproject.toml) | P0 |
|
||||||
|
|
||||||
|
### Phase 4 — WebSocket protocol
|
||||||
|
|
||||||
|
Implement exactly the messages from `screens_connection_updates.md` §"WebSocket Protocol":
|
||||||
|
|
||||||
|
**Agent → Server (JSON control):**
|
||||||
|
```json
|
||||||
|
{"type": "heartbeat", "hardware": ["pluto", "hackrf"], "status": "idle"}
|
||||||
|
{"type": "status", "status": "streaming", "app_id": "abc"}
|
||||||
|
{"type": "error", "app_id": "abc", "message": "USB device disconnected"}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Agent → Server (binary data):** raw interleaved float32 IQ bytes per `radio.rx()` call.
|
||||||
|
|
||||||
|
**Server → Agent (JSON control):**
|
||||||
|
```json
|
||||||
|
{"type": "start", "app_id": "...", "radio_config": {"device": "pluto", ...}}
|
||||||
|
{"type": "stop", "app_id": "..."}
|
||||||
|
{"type": "configure", "app_id": "...", "radio_config": {"center_frequency": 915000000}}
|
||||||
|
```
|
||||||
|
|
||||||
|
The agent does **not** interpret manifests, models, or preprocessing — it just applies `radio_config` to `ria_toolkit_oss.sdr.<device>`.
|
||||||
|
|
||||||
|
### Phase 5 — Tests
|
||||||
|
|
||||||
|
| # | Task | Location |
|
||||||
|
|---|------|----------|
|
||||||
|
| 11 | Unit: streamer loop against `sdr.mock` + fake WS | `tests/agent/test_streamer.py` |
|
||||||
|
| 12 | Unit: `ws_client` reconnect/heartbeat timing | `tests/agent/test_ws_client.py` |
|
||||||
|
| 13 | Unit: `hardware.detect_available()` and heartbeat payload | `tests/agent/test_hardware.py` |
|
||||||
|
| 14 | Integration: local `websockets` server → mock SDR → full start/stream/stop cycle | `tests/agent/test_integration.py` |
|
||||||
|
| 15 | Regression: `NodeAgent` still importable and functional | `tests/agent/test_legacy.py` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Part B — RIA Hub (hand-off to separate session)
|
||||||
|
|
||||||
|
> **Give this section to Claude in the ria-hub repo session.** It depends on Part A shipping first (or at minimum, stubbed protocol).
|
||||||
|
|
||||||
|
### Server-side tasks
|
||||||
|
|
||||||
|
| # | Task | File / Area | Priority |
|
||||||
|
|---|------|-------------|----------|
|
||||||
|
| B1 | Implement `AgentDataSource` (DataSource ABC, reads IQ from WebSocket connection) and register it in `build_data_source()` | `controller/app/modules/screens/data_sources.py` | P0 |
|
||||||
|
| B2 | Add `"agent"` to `dataSource.type` enum in manifest schema; update Pydantic/JSON schema validators | `controller/app/modules/screens/graph_derivation.py` + manifest schema files | P0 |
|
||||||
|
| B3 | Add agent WebSocket endpoint `POST /api/agent/ws` — accepts agent connections, auth via registration token, bridges the connection to the Celery task's `AgentDataSource` | `controller/app/modules/agent/routes.py` (new) | P0 |
|
||||||
|
| B4 | Agent registry: MongoDB collection tracking `agent_id`, hardware list, last heartbeat, online status, registration tokens | `controller/app/modules/agent/models.py` (new) | P1 |
|
||||||
|
| B5 | Registration endpoint `POST /api/agent/register` returning agent credentials for `~/.ria/agent.json` | `controller/app/modules/agent/routes.py` | P1 |
|
||||||
|
| B6 | Celery task wiring: when manifest `dataSource.type == "agent"`, look up the connected agent by `agent_id`, forward `radio_config` to it, and feed received IQ chunks into the inference loop via `AgentDataSource.next_chunk()` | `controller/app/modules/screens/tasks.py` | P0 |
|
||||||
|
| B7 | Device/agent picker UI in Screens app config (Vue 3) | frontend screens panel | P2 |
|
||||||
|
| B8 | Agents list/status admin view | frontend admin area | P2 |
|
||||||
|
|
||||||
|
### Protocol contract (must match Part A exactly)
|
||||||
|
|
||||||
|
See `screens_connection_updates.md` §"WebSocket Protocol". Key invariants:
|
||||||
|
- Binary frames are interleaved float32 IQ, one frame per `radio.rx()` call.
|
||||||
|
- `radio_config` is derived from the manifest's `dataSource.params` and forwarded verbatim.
|
||||||
|
- The server sends `configure` for retune-without-stop flow; the agent is responsible for applying it at the next capture boundary.
|
||||||
|
|
||||||
|
### Manifest example (new mode)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"dataSource": {
|
||||||
|
"type": "agent",
|
||||||
|
"device": "pluto",
|
||||||
|
"agent_id": "agent-abc123",
|
||||||
|
"params": {
|
||||||
|
"identifier": "ip:192.168.3.1",
|
||||||
|
"sample_rate": 1000000,
|
||||||
|
"center_frequency": 2450000000,
|
||||||
|
"gain": 40,
|
||||||
|
"buffer_size": 1024
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"preprocess": "magnitude_phase_window_stats",
|
||||||
|
"config": {"inference": {"knownDevices": [], "interval": 1}}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline invariant
|
||||||
|
|
||||||
|
**No changes to `inference_core.py`, `preprocessors.py`, or `run_onnx_chain_loop()`.** `AgentDataSource` is a drop-in for `SdrDataSource`; everything downstream is unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollout Order
|
||||||
|
|
||||||
|
1. Part A Phase 1 (SDR foundation) — safe to land independently.
|
||||||
|
2. Part A Phases 2–3 (agent package + streamer) — lands the `ria-agent stream` CLI; no server needed to build/test against a mock WS.
|
||||||
|
3. Part B B1, B2, B3, B6 — minimum viable server path; lets a real agent connect end-to-end.
|
||||||
|
4. Part A Phase 5 integration test against a dev RIA Hub instance.
|
||||||
|
5. Part B B4, B5 (registry + registration) — hardens multi-agent deployments.
|
||||||
|
6. Part B B7, B8 (UI) — operator-facing polish.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- Auth model for the WebSocket handshake: bearer token in header, query param, or first-message auth frame?
|
||||||
|
- Should `configure` apply mid-buffer or only at capture boundaries? (Doc implies boundaries; confirm for retune latency budget.)
|
||||||
|
- Backpressure policy when the server's inference loop is slower than the agent's `rx()` cadence — drop frames, queue, or pause?
|
||||||
|
|
@ -1,7 +1,6 @@
|
||||||
/* Change the hex values below to customize heading colours */
|
/* Change the hex values below to customize heading colours */
|
||||||
|
|
||||||
.rst-content { color: #e0e0e0; }
|
.rst-content h1 { color: #2c3e50; }
|
||||||
.rst-content h1 { color: #ffffff; }
|
|
||||||
.rst-content h2,
|
.rst-content h2,
|
||||||
.rst-content h2 a { color: #ffffff !important; font-size: 22px !important; }
|
.rst-content h2 a { color: #ffffff !important; font-size: 22px !important; }
|
||||||
|
|
||||||
|
|
@ -23,20 +22,8 @@
|
||||||
.rst-content .admonition.warning p {
|
.rst-content .admonition.warning p {
|
||||||
color: #ffffff !important;
|
color: #ffffff !important;
|
||||||
}
|
}
|
||||||
.rst-content h4 { color: #cccccc; }
|
.rst-content h4 { color: #404040; }
|
||||||
|
|
||||||
.highlight * { color: #ffffff !important; }
|
.highlight * { color: #ffffff !important; }
|
||||||
|
|
||||||
.ria-cmd { color: #2980b9 !important; }
|
.ria-cmd { color: #2980b9 !important; }
|
||||||
|
|
||||||
|
|
||||||
/* Table header text */
|
|
||||||
.rst-content table.docutils th {
|
|
||||||
color: #ffffff !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Remove alternating row background colors from tables */
|
|
||||||
.rst-content table.docutils td,
|
|
||||||
.rst-content table.docutils tr:nth-child(2n-1) td {
|
|
||||||
background-color: transparent !important;
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -12,9 +12,9 @@ sys.path.insert(0, os.path.abspath(os.path.join('..', '..')))
|
||||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
|
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
|
||||||
|
|
||||||
project = 'ria-toolkit-oss'
|
project = 'ria-toolkit-oss'
|
||||||
copyright = '2026, Qoherent Inc'
|
copyright = '2025, Qoherent Inc'
|
||||||
author = 'Qoherent Inc.'
|
author = 'Qoherent Inc.'
|
||||||
release = '0.1.7'
|
release = '0.1.5'
|
||||||
|
|
||||||
# -- General configuration ---------------------------------------------------
|
# -- General configuration ---------------------------------------------------
|
||||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
|
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
.. _sdr_examples:
|
.. _examples:
|
||||||
|
|
||||||
############
|
############
|
||||||
SDR Examples
|
SDR Examples
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@ In this example, we initialize the `Blade` SDR, configure it to record a signal
|
||||||
|
|
||||||
import time
|
import time
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.sdr.blade import Blade
|
from ria_toolkit_oss.sdr.blade import Blade
|
||||||
|
|
||||||
my_radio = Blade()
|
my_radio = Blade()
|
||||||
|
|
|
||||||
|
|
@ -21,7 +21,7 @@ Code
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.sdr.blade import Blade
|
from ria_toolkit_oss.sdr.blade import Blade
|
||||||
|
|
||||||
# Parameters
|
# Parameters
|
||||||
|
|
|
||||||
BIN
docs/source/images/recordings/qam64_35-full.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/qam64_35-full.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/qam64_35.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/qam64_35.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3-full.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3-full.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_annotated.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_annotated.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_awgn.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_awgn.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_cusum.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_cusum.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_threshold.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_threshold.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording5_after_separate.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording5_after_separate.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording5_before_separate.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording5_before_separate.png
(Stored with Git LFS)
Binary file not shown.
File diff suppressed because it is too large
Load Diff
|
|
@ -4,26 +4,7 @@ Installation
|
||||||
RIA Hub Toolkit OSS can be installed either as a Conda package or as a standard Python package.
|
RIA Hub Toolkit OSS can be installed either as a Conda package or as a standard Python package.
|
||||||
|
|
||||||
Please note that SDR drivers must be installed separately. Refer to the relevant guide in the
|
Please note that SDR drivers must be installed separately. Refer to the relevant guide in the
|
||||||
:ref:`SDR Guides <sdr_guides>` section of the documentation for additional setup instructions.
|
:ref:`SDR Guides <sdr_guides>` section of the documentation for addition setup instructions.
|
||||||
|
|
||||||
Common driver packages by device (exact package names depend on your OS):
|
|
||||||
|
|
||||||
.. list-table::
|
|
||||||
:widths: 25 75
|
|
||||||
:header-rows: 1
|
|
||||||
|
|
||||||
* - Device
|
|
||||||
- Driver Package
|
|
||||||
* - USRP
|
|
||||||
- UHD drivers
|
|
||||||
* - Pluto
|
|
||||||
- libiio / IIO utilities
|
|
||||||
* - BladeRF
|
|
||||||
- libbladeRF
|
|
||||||
* - HackRF
|
|
||||||
- libhackrf
|
|
||||||
* - RTL-SDR
|
|
||||||
- librtlsdr
|
|
||||||
|
|
||||||
We want your experience with RIA Toolkit OSS to be as smooth and frictionless as possible. If you run into any
|
We want your experience with RIA Toolkit OSS to be as smooth and frictionless as possible. If you run into any
|
||||||
issues during installation, please reach out to our support team: ``support@qoherent.ai``.
|
issues during installation, please reach out to our support team: ``support@qoherent.ai``.
|
||||||
|
|
@ -103,22 +84,12 @@ Please follow the steps below to install RIA Toolkit OSS using pip:
|
||||||
python -m venv venv
|
python -m venv venv
|
||||||
venv\Scripts\activate
|
venv\Scripts\activate
|
||||||
|
|
||||||
2. Upgrade pip and install RIA Toolkit OSS:
|
2. Install RIA Toolkit OSS from PyPI with pip:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
pip install --upgrade pip
|
|
||||||
pip install ria-toolkit-oss
|
pip install ria-toolkit-oss
|
||||||
|
|
||||||
3. Verify the CLI is available:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
ria --help
|
|
||||||
|
|
||||||
A successful install prints the top-level help text. ``pyproject.toml`` registers two
|
|
||||||
entrypoints — ``ria`` and ``ria-tools`` — that both point to the same CLI module.
|
|
||||||
|
|
||||||
RIA Toolkit OSS can also be installed from RIA Hub. However, RIA Hub does not yet support a proxy or cache for public packages.
|
RIA Toolkit OSS can also be installed from RIA Hub. However, RIA Hub does not yet support a proxy or cache for public packages.
|
||||||
We intend to add this missing functionality soon. In the meantime, please use the ``--no-deps`` option with pip to skip automatic
|
We intend to add this missing functionality soon. In the meantime, please use the ``--no-deps`` option with pip to skip automatic
|
||||||
dependency installation, and then manually install each dependency afterward.
|
dependency installation, and then manually install each dependency afterward.
|
||||||
|
|
@ -148,6 +119,3 @@ Follow the steps below to install RIA Toolkit OSS from source:
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
pip install .
|
pip install .
|
||||||
|
|
||||||
For local development, use ``pip install -e .`` instead to install in editable mode
|
|
||||||
so local changes take effect immediately without reinstalling.
|
|
||||||
|
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
Data Package (ria_toolkit_oss.data)
|
|
||||||
=======================================
|
|
||||||
|
|
||||||
.. |br| raw:: html
|
|
||||||
|
|
||||||
<br />
|
|
||||||
|
|
||||||
.. automodule:: ria_toolkit_oss.data
|
|
||||||
:members:
|
|
||||||
:undoc-members:
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
Radio Dataset SubPackage
|
|
||||||
------------------------
|
|
||||||
|
|
||||||
.. automodule:: ria_toolkit_oss.data.datasets
|
|
||||||
:members:
|
|
||||||
:undoc-members:
|
|
||||||
:show-inheritance:
|
|
||||||
|
|
||||||
.. toctree::
|
|
||||||
:maxdepth: 2
|
|
||||||
|
|
||||||
Dataset License SubModule <ria_toolkit_oss.data.datasets.license>
|
|
||||||
Radio Datasets <radio_datasets>
|
|
||||||
|
|
@ -11,15 +11,15 @@ The Radio Dataset Framework provides a software interface to access and manipula
|
||||||
the need for users to interface with the source files directly. Instead, users initialize and interact with a Python
|
the need for users to interface with the source files directly. Instead, users initialize and interact with a Python
|
||||||
object, while the complexities of efficient data retrieval and source file manipulation are managed behind the scenes.
|
object, while the complexities of efficient data retrieval and source file manipulation are managed behind the scenes.
|
||||||
|
|
||||||
Ria Toolkit OSS includes an abstract class called :py:obj:`ria_toolkit_oss.data.datasets.RadioDataset`, which defines common properties and
|
Ria Toolkit OSS includes an abstract class called :py:obj:`ria_toolkit_oss.datatypes.datasets.RadioDataset`, which defines common properties and
|
||||||
behaviors for all radio datasets. :py:obj:`ria_toolkit_oss.data.datasets.RadioDataset` can be considered a blueprint for all
|
behaviors for all radio datasets. :py:obj:`ria_toolkit_oss.datatypes.datasets.RadioDataset` can be considered a blueprint for all
|
||||||
other radio dataset classes. This class is then subclassed to define more specific blueprints for different types
|
other radio dataset classes. This class is then subclassed to define more specific blueprints for different types
|
||||||
of radio datasets. For example, :py:obj:`ria_toolkit_oss.data.datasets.IQDataset`, which is tailored for machine learning tasks
|
of radio datasets. For example, :py:obj:`ria_toolkit_oss.datatypes.datasets.IQDataset`, which is tailored for machine learning tasks
|
||||||
involving the processing of signals represented as IQ (In-phase and Quadrature) samples.
|
involving the processing of signals represented as IQ (In-phase and Quadrature) samples.
|
||||||
|
|
||||||
Then, in the various project backends, there are concrete dataset classes, which inherit from both Ria Toolkit OSS and the base
|
Then, in the various project backends, there are concrete dataset classes, which inherit from both Ria Toolkit OSS and the base
|
||||||
dataset class from the respective backend. For example, the :py:obj:`TorchIQDataset` class extends both
|
dataset class from the respective backend. For example, the :py:obj:`TorchIQDataset` class extends both
|
||||||
:py:obj:`ria_toolkit_oss.data.datasets.IQDataset` from Ria Toolkit OSS and :py:obj:`torch.ria_toolkit_oss.data.IterableDataset` from
|
:py:obj:`ria_toolkit_oss.datatypes.datasets.IQDataset` from Ria Toolkit OSS and :py:obj:`torch.ria_toolkit_oss.datatypes.IterableDataset` from
|
||||||
PyTorch, providing a concrete dataset class tailored for IQ datasets and optimized for the PyTorch backend.
|
PyTorch, providing a concrete dataset class tailored for IQ datasets and optimized for the PyTorch backend.
|
||||||
|
|
||||||
Dataset initialization
|
Dataset initialization
|
||||||
|
|
@ -130,7 +130,7 @@ Dataset processing and manipulation
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
All radio datasets support methods tailored specifically for radio processing. These methods are backend-independent,
|
All radio datasets support methods tailored specifically for radio processing. These methods are backend-independent,
|
||||||
inherited from the blueprints in Ria Toolkit OSS like :py:obj:`ria_toolkit_oss.data.datasets.RadioDataset`.
|
inherited from the blueprints in Ria Toolkit OSS like :py:obj:`ria_toolkit_oss.datatypes.datasets.RadioDataset`.
|
||||||
|
|
||||||
For example, we can trim down the length of the examples from 1,024 to 512 samples, and then augment the dataset:
|
For example, we can trim down the length of the examples from 1,024 to 512 samples, and then augment the dataset:
|
||||||
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
Dataset License SubModule
|
Dataset License SubModule
|
||||||
=========================
|
=========================
|
||||||
|
|
||||||
.. automodule:: ria_toolkit_oss.data.datasets.license
|
.. automodule:: ria_toolkit_oss.datatypes.datasets.license
|
||||||
:members:
|
:members:
|
||||||
:undoc-members:
|
:undoc-members:
|
||||||
:show-inheritance:
|
:show-inheritance:
|
||||||
|
|
@ -0,0 +1,25 @@
|
||||||
|
Datatypes Package (ria_toolkit_oss.datatypes)
|
||||||
|
=============================================
|
||||||
|
|
||||||
|
.. |br| raw:: html
|
||||||
|
|
||||||
|
<br />
|
||||||
|
|
||||||
|
.. automodule:: ria_toolkit_oss.datatypes
|
||||||
|
:members:
|
||||||
|
:undoc-members:
|
||||||
|
:show-inheritance:
|
||||||
|
|
||||||
|
Radio Dataset SubPackage
|
||||||
|
------------------------
|
||||||
|
|
||||||
|
.. automodule:: ria_toolkit_oss.datatypes.datasets
|
||||||
|
:members:
|
||||||
|
:undoc-members:
|
||||||
|
:show-inheritance:
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:maxdepth: 2
|
||||||
|
|
||||||
|
Dataset License SubModule <ria_toolkit_oss.datatypes.datasets.license>
|
||||||
|
Radio Datasets <radio_datasets>
|
||||||
|
|
@ -11,7 +11,7 @@ class and function signatures, and doctest examples where available.
|
||||||
:maxdepth: 2
|
:maxdepth: 2
|
||||||
:caption: Contents:
|
:caption: Contents:
|
||||||
|
|
||||||
Data Package <data/ria_toolkit_oss.data>
|
Datatypes Package <datatypes/ria_toolkit_oss.datatypes>
|
||||||
SDR Package <ria_toolkit_oss.sdr>
|
SDR Package <ria_toolkit_oss.sdr>
|
||||||
IO Package <ria_toolkit_oss.io>
|
IO Package <ria_toolkit_oss.io>
|
||||||
Transforms Package <ria_toolkit_oss.transforms>
|
Transforms Package <ria_toolkit_oss.transforms>
|
||||||
|
|
|
||||||
|
|
@ -1,87 +1,77 @@
|
||||||
.. _blade:
|
.. _blade:
|
||||||
|
|
||||||
BladeRF
|
BladeRF
|
||||||
=======
|
=======
|
||||||
|
|
||||||
The BladeRF is a versatile software-defined radio (SDR) platform developed by Nuand. It is designed for a wide
|
The BladeRF is a versatile software-defined radio (SDR) platform developed by Nuand. It is designed for a wide
|
||||||
range of applications, from wireless communication research to field deployments. BladeRF devices are known
|
range of applications, from wireless communication research to field deployments. BladeRF devices are known
|
||||||
for their high performance, flexibility, and extensive open-source support, making them suitable for both
|
for their high performance, flexibility, and extensive open-source support, making them suitable for both
|
||||||
hobbyists and professionals. The BladeRF is based on the Analog Devices AD9361 RF transceiver, which provides
|
hobbyists and professionals. The BladeRF is based on the Analog Devices AD9361 RF transceiver, which provides
|
||||||
wide frequency coverage and high bandwidth.
|
wide frequency coverage and high bandwidth.
|
||||||
|
|
||||||
Supported Models
|
Supported Models
|
||||||
----------------
|
----------------
|
||||||
|
|
||||||
- **BladeRF 2.0 Micro xA4:** A compact model with a 49 kLE FPGA, ideal for portable applications.
|
- **BladeRF 2.0 Micro xA4:** A compact model with a 49 kLE FPGA, ideal for portable applications.
|
||||||
- **BladeRF 2.0 Micro xA9:** A higher-end version of the Micro with a 115 kLE FPGA, offering more processing power in a small form factor.
|
- **BladeRF 2.0 Micro xA9:** A higher-end version of the Micro with a 115 kLE FPGA, offering more processing power in a small form factor.
|
||||||
|
|
||||||
Key Features
|
Key Features
|
||||||
------------
|
------------
|
||||||
|
|
||||||
- **Frequency Range:** Typically from 47 MHz to 6 GHz, covering a wide range of wireless communication bands.
|
- **Frequency Range:** Typically from 47 MHz to 6 GHz, covering a wide range of wireless communication bands.
|
||||||
- **Bandwidth:** Up to 56 MHz, allowing for wideband signal processing.
|
- **Bandwidth:** Up to 56 MHz, allowing for wideband signal processing.
|
||||||
- **FPGA:** Integrated FPGA (varies by model) for real-time processing and custom logic development.
|
- **FPGA:** Integrated FPGA (varies by model) for real-time processing and custom logic development.
|
||||||
- **Connectivity:** USB 3.0 interface for high-speed data transfer, with options for GPIO, SPI, and other I/O.
|
- **Connectivity:** USB 3.0 interface for high-speed data transfer, with options for GPIO, SPI, and other I/O.
|
||||||
|
|
||||||
Hackability
|
Hackability
|
||||||
-----------
|
-----------
|
||||||
|
|
||||||
- **Expansion:** The BladeRF features GPIO, expansion headers, and add-on boards, allowing users to extend the
|
- **Expansion:** The BladeRF features GPIO, expansion headers, and add-on boards, allowing users to extend the
|
||||||
functionality of the device for specific applications, such as additional RF front ends.
|
functionality of the device for specific applications, such as additional RF front ends.
|
||||||
- **Frequency and Bandwidth Modification:** Advanced users can modify the BladeRF's settings and firmware to
|
- **Frequency and Bandwidth Modification:** Advanced users can modify the BladeRF's settings and firmware to
|
||||||
explore different frequency bands and optimize the bandwidth for their specific use cases.
|
explore different frequency bands and optimize the bandwidth for their specific use cases.
|
||||||
|
|
||||||
Limitations
|
Limitations
|
||||||
-----------
|
-----------
|
||||||
|
|
||||||
- The complexity of FPGA development may present a steep learning curve for users unfamiliar with hardware
|
- The complexity of FPGA development may present a steep learning curve for users unfamiliar with hardware
|
||||||
description languages (HDL).
|
description languages (HDL).
|
||||||
- Bandwidth is capped at 56 MHz, which might not be sufficient for ultra-wideband applications.
|
- Bandwidth is capped at 56 MHz, which might not be sufficient for ultra-wideband applications.
|
||||||
- USB 3.0 connectivity is required for optimal performance; using USB 2.0 will significantly limit data
|
- USB 3.0 connectivity is required for optimal performance; using USB 2.0 will significantly limit data
|
||||||
transfer rates.
|
transfer rates.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
No additional Python packages are required for BladeRF beyond the base RIA Toolkit OSS installation.
|
1. Activate your Radioconda environment.
|
||||||
|
|
||||||
1. Install the system library:
|
.. code-block:: bash
|
||||||
|
|
||||||
.. code-block:: bash
|
conda activate <your-env-name>
|
||||||
|
|
||||||
sudo apt install libbladerf-dev
|
2. Install the base dependencies and drivers (*Easy method*):
|
||||||
|
|
||||||
For a more complete installation including CLI tools and FPGA images, use the Nuand PPA:
|
.. code-block:: bash
|
||||||
|
|
||||||
.. code-block:: bash
|
sudo add-apt-repository ppa:nuandllc/bladerf
|
||||||
|
sudo apt-get update
|
||||||
sudo add-apt-repository ppa:nuandllc/bladerf
|
sudo apt-get install bladerf
|
||||||
sudo apt-get update
|
sudo apt-get install libbladerf-dev
|
||||||
sudo apt-get install bladerf libbladerf-dev
|
sudo apt-get install bladerf-fpga-hostedxa4 # Necessary for installation of bladeRF 2.0 Micro A4.
|
||||||
sudo apt-get install bladerf-fpga-hostedxa4 # Necessary for BladeRF 2.0 Micro xA4
|
|
||||||
|
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||||
2. Install udev rules:
|
|
||||||
|
.. code-block:: bash
|
||||||
For most users:
|
|
||||||
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bladerf1.rules /etc/udev/rules.d/88-radioconda-nuand-bladerf1.rules
|
||||||
.. code-block:: bash
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bladerf2.rules /etc/udev/rules.d/88-radioconda-nuand-bladerf2.rules
|
||||||
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bootloader.rules /etc/udev/rules.d/88-radioconda-nuand-bootloader.rules
|
||||||
sudo udevadm control --reload
|
sudo udevadm control --reload
|
||||||
sudo udevadm trigger
|
sudo udevadm trigger
|
||||||
|
|
||||||
For **Radioconda** users, create symlinks from your conda environment instead:
|
Further Information
|
||||||
|
-------------------
|
||||||
.. code-block:: bash
|
|
||||||
|
- `Official BladeRF Website <https://www.nuand.com/>`_
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bladerf1.rules /etc/udev/rules.d/88-radioconda-nuand-bladerf1.rules
|
- `BladeRF GitHub Repository <https://github.com/Nuand/bladeRF>`_
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bladerf2.rules /etc/udev/rules.d/88-radioconda-nuand-bladerf2.rules
|
- `BladeRF Setup with Radioconda <https://github.com/radioconda/radioconda-installer?tab=readme-ov-file#bladerf>`_
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bootloader.rules /etc/udev/rules.d/88-radioconda-nuand-bootloader.rules
|
|
||||||
sudo udevadm control --reload
|
|
||||||
sudo udevadm trigger
|
|
||||||
|
|
||||||
Further Information
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
- `Official BladeRF Website <https://www.nuand.com/>`_
|
|
||||||
- `BladeRF GitHub Repository <https://github.com/Nuand/bladeRF>`_
|
|
||||||
- `BladeRF Setup with Radioconda <https://github.com/radioconda/radioconda-installer?tab=readme-ov-file#bladerf>`_
|
|
||||||
|
|
|
||||||
|
|
@ -1,88 +1,83 @@
|
||||||
.. _hackrf:
|
.. _hackrf:
|
||||||
|
|
||||||
HackRF
|
HackRF
|
||||||
======
|
======
|
||||||
|
|
||||||
The HackRF One is a portable and affordable software-defined radio developed by Great Scott Gadgets. It is an
|
The HackRF One is a portable and affordable software-defined radio developed by Great Scott Gadgets. It is an
|
||||||
open source hardware platform that is designed to enable test and development of modern and next generation
|
open source hardware platform that is designed to enable test and development of modern and next generation
|
||||||
radio technologies.
|
radio technologies.
|
||||||
|
|
||||||
The HackRF is based on the Analog Devices MAX2839 transceiver chip, which supports both transmission and
|
The HackRF is based on the Analog Devices MAX2839 transceiver chip, which supports both transmission and
|
||||||
reception of signals across a wide frequency range, combined with a MAX5864 RF front-end chip and a
|
reception of signals across a wide frequency range, combined with a MAX5864 RF front-end chip and a
|
||||||
RFFC5072 wideband synthesizer/VCO.
|
RFFC5072 wideband synthesizer/VCO.
|
||||||
|
|
||||||
Supported models
|
Supported models
|
||||||
----------------
|
----------------
|
||||||
|
|
||||||
- **HackRF One:** The standard model with a frequency range of 1 MHz to 6 GHz and a bandwidth of up to 20 MHz.
|
- **HackRF One:** The standard model with a frequency range of 1 MHz to 6 GHz and a bandwidth of up to 20 MHz.
|
||||||
- **Opera Cake for HackRF:** An antenna switching add-on board for HackRF One that is configured with command-line software.
|
- **Opera Cake for HackRF:** An antenna switching add-on board for HackRF One that is configured with command-line software.
|
||||||
|
|
||||||
Key features
|
Key features
|
||||||
------------
|
------------
|
||||||
|
|
||||||
- **Frequency Range:** 1 MHz to 6 GHz.
|
- **Frequency Range:** 1 MHz to 6 GHz.
|
||||||
- **Bandwidth:** 2 MHz to 20 MHz.
|
- **Bandwidth:** 2 MHz to 20 MHz.
|
||||||
- **Connectivity:** USB 2.0 interface with support for power, data, and firmware updates.
|
- **Connectivity:** USB 2.0 interface with support for power, data, and firmware updates.
|
||||||
- **Software Support:** Compatible with GNU Radio, SDR#, and other SDR frameworks.
|
- **Software Support:** Compatible with GNU Radio, SDR#, and other SDR frameworks.
|
||||||
- **Onboard Processing:** ARM-based LPC4320 processor for digital signal processing and interfacing over USB.
|
- **Onboard Processing:** ARM-based LPC4320 processor for digital signal processing and interfacing over USB.
|
||||||
|
|
||||||
Hackability
|
Hackability
|
||||||
-----------
|
-----------
|
||||||
|
|
||||||
.. todo::
|
.. todo::
|
||||||
|
|
||||||
Add information regarding HackRF hackability
|
Add information regarding HackRF hackability
|
||||||
|
|
||||||
Limitations
|
Limitations
|
||||||
-----------
|
-----------
|
||||||
|
|
||||||
- Bandwidth is limited to 20 MHz.
|
- Bandwidth is limited to 20 MHz.
|
||||||
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
HackRF is supported out of the box after installing RIA Toolkit OSS.
|
1. Activate your Radioconda environment:
|
||||||
|
|
||||||
1. Ensure ``libhackrf`` is installed at the system level. On most Ubuntu installations this is already
|
.. code-block:: bash
|
||||||
present. If not:
|
|
||||||
|
conda activate <your-env-name>
|
||||||
.. code-block:: bash
|
|
||||||
|
2. Install the System Package (Ubuntu / Debian):
|
||||||
sudo apt install libhackrf-dev
|
|
||||||
|
.. code-block:: bash
|
||||||
2. Install udev rules to allow non-root device access:
|
|
||||||
|
sudo apt-get update
|
||||||
For most users:
|
sudo apt-get install hackrf
|
||||||
|
|
||||||
.. code-block:: bash
|
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||||
|
|
||||||
sudo udevadm control --reload
|
.. code-block:: bash
|
||||||
sudo udevadm trigger
|
|
||||||
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/53-hackrf.rules /etc/udev/rules.d/53-radioconda-hackrf.rules
|
||||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
sudo udevadm control --reload
|
||||||
|
sudo udevadm trigger
|
||||||
.. code-block:: bash
|
|
||||||
|
Make sure your user account belongs to the plugdev group in order to access your device:
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/53-hackrf.rules /etc/udev/rules.d/53-radioconda-hackrf.rules
|
|
||||||
sudo udevadm control --reload
|
.. code-block:: bash
|
||||||
sudo udevadm trigger
|
|
||||||
|
sudo usermod -a -G plugdev <user>
|
||||||
Make sure your user account belongs to the ``plugdev`` group in order to access your device:
|
|
||||||
|
.. note::
|
||||||
.. code-block:: bash
|
|
||||||
|
You may have to restart your system for changes to take effect.
|
||||||
sudo usermod -a -G plugdev <user>
|
|
||||||
|
Further information
|
||||||
.. note::
|
-------------------
|
||||||
|
|
||||||
You may have to restart your system for group membership changes to take effect.
|
- `Official HackRF Website <https://greatscottgadgets.com/hackrf/>`_
|
||||||
|
- `HackRF Project Documentation <https://hackrf.readthedocs.io/en/latest/>`_
|
||||||
Further information
|
- `HackRF Software Installation Guide <https://hackrf.readthedocs.io/en/latest/installing_hackrf_software.html>`_
|
||||||
-------------------
|
- `HackRF GitHub Repository <https://github.com/greatscottgadgets/hackrf>`_
|
||||||
|
- `HackRF Setup with Radioconda <https://github.com/radioconda/radioconda-installer?tab=readme-ov-file#hackrf>`_
|
||||||
- `Official HackRF Website <https://greatscottgadgets.com/hackrf/>`_
|
|
||||||
- `HackRF Project Documentation <https://hackrf.readthedocs.io/en/latest/>`_
|
|
||||||
- `HackRF Software Installation Guide <https://hackrf.readthedocs.io/en/latest/installing_hackrf_software.html>`_
|
|
||||||
- `HackRF GitHub Repository <https://github.com/greatscottgadgets/hackrf>`_
|
|
||||||
- `HackRF Setup with Radioconda <https://github.com/radioconda/radioconda-installer?tab=readme-ov-file#hackrf>`_
|
|
||||||
|
|
|
||||||
|
|
@ -1,123 +1,116 @@
|
||||||
.. _pluto:
|
.. _pluto:
|
||||||
|
|
||||||
PlutoSDR
|
PlutoSDR
|
||||||
========
|
========
|
||||||
|
|
||||||
The ADALM-PLUTO (PlutoSDR) is a portable and affordable software-defined radio developed by Analog Devices.
|
The ADALM-PLUTO (PlutoSDR) is a portable and affordable software-defined radio developed by Analog Devices.
|
||||||
It is designed for learning, experimenting, and prototyping in the field of wireless communication. The PlutoSDR
|
It is designed for learning, experimenting, and prototyping in the field of wireless communication. The PlutoSDR
|
||||||
is popular among students, educators, and hobbyists due to its versatility and ease of use.
|
is popular among students, educators, and hobbyists due to its versatility and ease of use.
|
||||||
|
|
||||||
The PlutoSDR is based on the AD9363 transceiver chip, which supports both transmission and reception of signals
|
The PlutoSDR is based on the AD9363 transceiver chip, which supports both transmission and reception of signals
|
||||||
across a wide frequency range. The device is supported by a robust open-source ecosystem, making it ideal for
|
across a wide frequency range. The device is supported by a robust open-source ecosystem, making it ideal for
|
||||||
hands-on learning and rapid prototyping.
|
hands-on learning and rapid prototyping.
|
||||||
|
|
||||||
Supported models
|
Supported models
|
||||||
----------------
|
----------------
|
||||||
|
|
||||||
- **ADALM-PLUTO:** The standard model with a frequency range of 325 MHz to 3.8 GHz and a bandwidth of up to 20 MHz.
|
- **ADALM-PLUTO:** The standard model with a frequency range of 325 MHz to 3.8 GHz and a bandwidth of up to 20 MHz.
|
||||||
- **Modified ADALM-PLUTO:** Some users modify their PlutoSDR to extend the frequency range to approximately 70 MHz
|
- **Modified ADALM-PLUTO:** Some users modify their PlutoSDR to extend the frequency range to approximately 70 MHz
|
||||||
to 6 GHz by applying firmware patches with unqualified RF performance.
|
to 6 GHz by applying firmware patches with unqualified RF performance.
|
||||||
|
|
||||||
Key features
|
Key features
|
||||||
------------
|
------------
|
||||||
|
|
||||||
- **Frequency Range:** 325 MHz to 3.8 GHz (standard), expandable with modifications.
|
- **Frequency Range:** 325 MHz to 3.8 GHz (standard), expandable with modifications.
|
||||||
- **Bandwidth:** Up to 20 MHz, can be increased to 56 MHz with firmware modifications.
|
- **Bandwidth:** Up to 20 MHz, can be increased to 56 MHz with firmware modifications.
|
||||||
- **Connectivity:** USB 2.0 interface with support for power, data, and firmware updates.
|
- **Connectivity:** USB 2.0 interface with support for power, data, and firmware updates.
|
||||||
- **Software Support:** Compatible with GNU Radio, MATLAB, Simulink, and other SDR frameworks.
|
- **Software Support:** Compatible with GNU Radio, MATLAB, Simulink, and other SDR frameworks.
|
||||||
- **Onboard Processing:** Integrated ARM Cortex-A9 processor for custom applications and signal processing.
|
- **Onboard Processing:** Integrated ARM Cortex-A9 processor for custom applications and signal processing.
|
||||||
|
|
||||||
Hackability
|
Hackability
|
||||||
------------
|
------------
|
||||||
|
|
||||||
- **Frequency Range and Bandwidth:** The default frequency range of 325 MHz to 3.8 GHz can be expanded to
|
- **Frequency Range and Bandwidth:** The default frequency range of 325 MHz to 3.8 GHz can be expanded to
|
||||||
approximately 70 MHz to 6 GHz, and the bandwidth can be increased from 20 MHz to 56 MHz by modifying
|
approximately 70 MHz to 6 GHz, and the bandwidth can be increased from 20 MHz to 56 MHz by modifying
|
||||||
the device's firmware.
|
the device's firmware.
|
||||||
- **2x2 MIMO:** On Rev C models, users can unlock 2x2 MIMO (Multiple Input Multiple Output) functionality by
|
- **2x2 MIMO:** On Rev C models, users can unlock 2x2 MIMO (Multiple Input Multiple Output) functionality by
|
||||||
wiring UFL to SMA connectors to the device's PCB, effectively turning the device into a dual-channel SDR.
|
wiring UFL to SMA connectors to the device's PCB, effectively turning the device into a dual-channel SDR.
|
||||||
|
|
||||||
Limitations
|
Limitations
|
||||||
-----------
|
-----------
|
||||||
|
|
||||||
- Bandwidth is limited to 20 MHz by default, but can be increased to 56 MHz with modifications, which may
|
- Bandwidth is limited to 20 MHz by default, but can be increased to 56 MHz with modifications, which may
|
||||||
affect stability.
|
affect stability.
|
||||||
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
The PlutoSDR is supported out of the box after installing RIA Toolkit OSS. The required Python package
|
1. Activate your Radioconda environment:
|
||||||
(``pyadi-iio``) is included in the toolkit's dependencies.
|
|
||||||
|
.. code-block:: bash
|
||||||
1. Ensure ``libiio`` is installed at the system level. On most Ubuntu installations this is already present.
|
|
||||||
If not:
|
conda activate <your-env-name>
|
||||||
|
|
||||||
.. code-block:: bash
|
2. Install system dependencies:
|
||||||
|
|
||||||
sudo apt install libiio-dev libiio-utils libiio0
|
.. code-block:: bash
|
||||||
|
|
||||||
.. note::
|
sudo apt-get update
|
||||||
|
sudo apt-get install -y \
|
||||||
PlutoSDR devices are discoverable over both USB and network (mDNS). Network discovery uses Avahi — if
|
build-essential \
|
||||||
``avahi-daemon`` is not running, network discovery will be skipped but USB discovery still works.
|
git \
|
||||||
|
libxml2-dev \
|
||||||
2. Install a ``udev`` rule to allow non-root device access:
|
bison \
|
||||||
|
flex \
|
||||||
For most users:
|
libcdk5-dev \
|
||||||
|
cmake \
|
||||||
.. code-block:: bash
|
libusb-1.0-0-dev \
|
||||||
|
libavahi-client-dev \
|
||||||
sudo udevadm control --reload
|
libavahi-common-dev \
|
||||||
sudo udevadm trigger
|
libaio-dev
|
||||||
|
|
||||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/90-libiio.rules /etc/udev/rules.d/90-radioconda-libiio.rules
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/90-libiio.rules /etc/udev/rules.d/90-radioconda-libiio.rules
|
||||||
sudo udevadm control --reload
|
sudo udevadm control --reload
|
||||||
sudo udevadm trigger
|
sudo udevadm trigger
|
||||||
|
|
||||||
Once you can communicate with the hardware, you may want to perform the post-install steps detailed on
|
Once you can talk to the hardware, you may want to perform the post-install steps detailed on the `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_.
|
||||||
the `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_.
|
|
||||||
|
4. (Optional) Building ``libiio`` or ``libad9361-iio`` from source:
|
||||||
3. (Optional) Building ``libiio`` or ``libad9361-iio`` from source:
|
|
||||||
|
This step is only required if you want the latest version of these libraries not provided in Radioconda.
|
||||||
This step is only required if you need a version not available via ``apt``. First install build
|
|
||||||
dependencies:
|
.. code-block:: bash
|
||||||
|
|
||||||
.. code-block:: bash
|
# Build libiio from source
|
||||||
|
cd ~
|
||||||
sudo apt-get install -y build-essential git libxml2-dev bison flex libcdk5-dev cmake \
|
git clone --branch v0.23 https://github.com/analogdevicesinc/libiio.git
|
||||||
libusb-1.0-0-dev libavahi-client-dev libavahi-common-dev libaio-dev
|
cd libiio
|
||||||
|
mkdir -p build
|
||||||
.. code-block:: bash
|
cd build
|
||||||
|
cmake -DPYTHON_BINDINGS=ON ..
|
||||||
# Build libiio from source
|
make -j"$(nproc)"
|
||||||
cd ~
|
sudo make install
|
||||||
git clone --branch v0.23 https://github.com/analogdevicesinc/libiio.git
|
sudo ldconfig
|
||||||
cd libiio
|
|
||||||
mkdir -p build
|
.. code-block:: bash
|
||||||
cd build
|
|
||||||
cmake -DPYTHON_BINDINGS=ON ..
|
# Build libad9361-iio from source
|
||||||
make -j"$(nproc)"
|
cd ~
|
||||||
sudo make install
|
git clone https://github.com/analogdevicesinc/libad9361-iio.git
|
||||||
sudo ldconfig
|
cd libad9361-iio
|
||||||
|
mkdir -p build
|
||||||
.. code-block:: bash
|
cd build
|
||||||
|
cmake ..
|
||||||
# Build libad9361-iio from source
|
make -j"$(nproc)"
|
||||||
cd ~
|
sudo make install
|
||||||
git clone https://github.com/analogdevicesinc/libad9361-iio.git
|
|
||||||
cd libad9361-iio
|
Further information
|
||||||
mkdir -p build
|
-------------------
|
||||||
cd build
|
|
||||||
cmake ..
|
- `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_
|
||||||
make -j"$(nproc)"
|
- `PlutoSDR Setup with Radioconda <https://github.com/radioconda/radioconda-installer?tab=readme-ov-file#iio-pluto-sdr>`_
|
||||||
sudo make install
|
|
||||||
|
|
||||||
Further information
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
- `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_
|
|
||||||
- `PlutoSDR Setup with Radioconda <https://github.com/radioconda/radioconda-installer?tab=readme-ov-file#iio-pluto-sdr>`_
|
|
||||||
|
|
@ -30,111 +30,71 @@ Limitations
|
||||||
- Sensitivity and performance can vary depending on the specific model and components.
|
- Sensitivity and performance can vary depending on the specific model and components.
|
||||||
- Requires external software for signal processing and analysis.
|
- Requires external software for signal processing and analysis.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
1. If you previously had RTL-SDR drivers installed, purge them first:
|
1. Activate your Radioconda environment:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
conda activate <your-env-name>
|
||||||
|
|
||||||
|
2. Purge drivers:
|
||||||
|
|
||||||
|
If you already have other drivers installed, purge them from your system.
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo apt purge ^librtlsdr
|
sudo apt purge ^librtlsdr
|
||||||
sudo rm -rvf /usr/lib/librtlsdr*
|
sudo rm -rvf /usr/lib/librtlsdr*
|
||||||
sudo rm -rvf /usr/include/rtl-sdr*
|
sudo rm -rvf /usr/include/rtl-sdr*
|
||||||
sudo rm -rvf /usr/local/lib/librtlsdr*
|
sudo rm -rvf /usr/local/lib/librtlsdr*
|
||||||
sudo rm -rvf /usr/local/include/rtl-sdr*
|
sudo rm -rvf /usr/local/include/rtl-sdr*
|
||||||
sudo rm -rvf /usr/local/include/rtl_*
|
sudo rm -rvf /usr/local/include/rtl_*
|
||||||
sudo rm -rvf /usr/local/bin/rtl_*
|
sudo rm -rvf /usr/local/bin/rtl_*
|
||||||
|
|
||||||
2. Install build dependencies:
|
3. Install RTL-SDR Blog drivers:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo apt install libusb-1.0-0-dev git cmake pkg-config build-essential
|
sudo apt-get install libusb-1.0-0-dev git cmake pkg-config build-essential
|
||||||
|
git clone https://github.com/osmocom/rtl-sdr
|
||||||
3. Build ``librtlsdr`` from source:
|
cd rtl-sdr
|
||||||
|
mkdir build
|
||||||
The standard ``librtlsdr`` package available via ``apt`` is missing symbols required by the Python
|
cd build
|
||||||
bindings. Build from the **rtl-sdr-blog fork**:
|
cmake ../ -DINSTALL_UDEV_RULES=ON
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
git clone https://github.com/rtlsdrblog/rtl-sdr-blog.git
|
|
||||||
cd rtl-sdr-blog
|
|
||||||
mkdir build && cd build
|
|
||||||
cmake .. -DINSTALL_UDEV_RULES=ON
|
|
||||||
make
|
make
|
||||||
sudo make install
|
sudo make install
|
||||||
sudo cp ../rtl-sdr.rules /etc/udev/rules.d/
|
sudo cp ../rtl-sdr.rules /etc/udev/rules.d/
|
||||||
sudo ldconfig
|
sudo ldconfig
|
||||||
|
|
||||||
.. important::
|
4. Blacklist the DVB-T modules that would otherwise claim the device:
|
||||||
|
|
||||||
Do not use the osmocom ``rtl-sdr`` repository or the Ubuntu ``librtlsdr-dev`` apt package. Neither
|
|
||||||
provides the ``rtlsdr_set_dithering`` symbol that the Python bindings require.
|
|
||||||
|
|
||||||
4. Blacklist the kernel DVB driver:
|
|
||||||
|
|
||||||
The kernel DVB-T driver (``dvb_usb_rtl28xxu``) claims the RTL-SDR device and prevents ``librtlsdr``
|
|
||||||
from accessing it.
|
|
||||||
|
|
||||||
For most users:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
echo 'blacklist dvb_usb_rtl28xxu' | sudo tee /etc/modprobe.d/blacklist-rtlsdr.conf
|
|
||||||
sudo modprobe -r dvb_usb_rtl28xxu
|
|
||||||
|
|
||||||
For **Radioconda** users, a blacklist configuration is already provided in your conda environment:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf /etc/modprobe.d/radioconda-rtl-sdr-blacklist.conf
|
sudo ln -s $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf /etc/modprobe.d/radioconda-rtl-sdr-blacklist.conf
|
||||||
sudo modprobe -r $(cat $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf | sed -n -e 's/^blacklist //p')
|
sudo modprobe -r $(cat $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf | sed -n -e 's/^blacklist //p')
|
||||||
|
|
||||||
If ``modprobe -r`` fails with "Module is in use", unplug the RTL-SDR dongle, run the command again,
|
.. note::
|
||||||
then plug it back in. Alternatively, reboot — the blacklist takes effect on next boot.
|
|
||||||
|
|
||||||
.. note::
|
In addition to the Radioconda blacklist file, some systems also require
|
||||||
|
manually blacklisting the following DVB-T modules to prevent them from
|
||||||
|
claiming the device:
|
||||||
|
|
||||||
Some systems also require blacklisting additional DVB-T modules. Add these entries to your
|
- ``dvb_usb_rtl28xxu``
|
||||||
blacklist configuration if needed:
|
- ``rtl2832``
|
||||||
|
- ``rtl2830``
|
||||||
|
|
||||||
- ``rtl2832``
|
Add these entries to ``rtlsdr.conf`` (or create the file at
|
||||||
- ``rtl2830``
|
``/etc/modprobe.d/rtlsdr.conf``) if they are not already present.
|
||||||
|
|
||||||
5. Reload udev rules:
|
5. Install a udev rule by creating a link into your radioconda installation:
|
||||||
|
|
||||||
For most users (rules are installed by the build step above):
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo udevadm control --reload
|
|
||||||
sudo udevadm trigger
|
|
||||||
|
|
||||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/rtl-sdr.rules /etc/udev/rules.d/radioconda-rtl-sdr.rules
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/rtl-sdr.rules /etc/udev/rules.d/radioconda-rtl-sdr.rules
|
||||||
sudo udevadm control --reload
|
sudo udevadm control --reload
|
||||||
sudo udevadm trigger
|
sudo udevadm trigger
|
||||||
|
|
||||||
6. Install Python packages:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
pip install pyrtlsdr==0.3.0
|
|
||||||
pip install setuptools==69.5.1
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
``pyrtlsdr`` 0.4.0 references a ``rtlsdr_set_dithering`` symbol not present in standard
|
|
||||||
``librtlsdr`` builds. Version 0.3.0 works correctly.
|
|
||||||
|
|
||||||
``pyrtlsdr`` 0.3.0 depends on ``pkg_resources``, which was removed in ``setuptools`` >= 82.
|
|
||||||
Pinning to 69.5.1 ensures ``pkg_resources`` is available.
|
|
||||||
|
|
||||||
Further Information
|
Further Information
|
||||||
-------------------
|
-------------------
|
||||||
- `RTL-SDR Official Website <https://www.rtl-sdr.com/>`_
|
- `RTL-SDR Official Website <https://www.rtl-sdr.com/>`_
|
||||||
- `RTL-SDR Documentation <https://www.rtl-sdr.com/rtl-sdr-quick-start-guide/>`_
|
- `RTL-SDR Documentation <https://www.rtl-sdr.com/rtl-sdr-quick-start-guide/>`_
|
||||||
|
|
@ -39,48 +39,18 @@ Limitations
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux)
|
||||||
---------------------------------
|
---------------------------------
|
||||||
|
|
||||||
ThinkRF devices require the ``pyrf`` package, which is written in Python 2 syntax and must be patched
|
Install PyRF
|
||||||
after installation to work with Python 3.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
``lib2to3`` was fully removed in Python 3.13. ThinkRF support is currently limited to
|
|
||||||
**Python 3.12 and below**.
|
|
||||||
|
|
||||||
1. Install ``lib2to3``:
|
|
||||||
|
|
||||||
On some distributions (including Ubuntu 24.04+), ``lib2to3`` is not included by default:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo apt install python3-lib2to3
|
pip install 'pyrf>=2.8.0'
|
||||||
|
|
||||||
2. Install ``pyrf``:
|
Convert PyRF scripts to Python 3
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
pip install pyrf
|
cd ../scripts
|
||||||
|
./convert_pyrf_to_python3.sh
|
||||||
3. Patch ``pyrf`` for Python 3:
|
|
||||||
|
|
||||||
The ``pyrf`` package contains Python 2 syntax throughout (e.g., ``dict.iteritems()``, ``print``
|
|
||||||
statements). Run the following to automatically convert the entire package to Python 3:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
python -c "
|
|
||||||
from lib2to3.refactor import RefactoringTool, get_fixers_from_package
|
|
||||||
import pyrf, os
|
|
||||||
pyrf_path = os.path.dirname(pyrf.__file__)
|
|
||||||
fixers = get_fixers_from_package('lib2to3.fixes')
|
|
||||||
tool = RefactoringTool(fixers)
|
|
||||||
tool.refactor_dir(pyrf_path, write=True)
|
|
||||||
print('Done')
|
|
||||||
"
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
This patches the entire ``pyrf`` package in place, which is required for the driver to fully load.
|
|
||||||
|
|
||||||
Further Information
|
Further Information
|
||||||
-------------------
|
-------------------
|
||||||
|
|
|
||||||
|
|
@ -1,155 +1,92 @@
|
||||||
.. _usrp:
|
.. _usrp:
|
||||||
|
|
||||||
USRP
|
USRP
|
||||||
====
|
====
|
||||||
|
|
||||||
The USRP (Universal Software Radio Peripheral) product line is a series of software-defined radios (SDRs)
|
The USRP (Universal Software Radio Peripheral) product line is a series of software-defined radios (SDRs)
|
||||||
developed by Ettus Research. These devices are widely used in academia, industry, and research for various
|
developed by Ettus Research. These devices are widely used in academia, industry, and research for various
|
||||||
wireless communication applications, ranging from simple experimentation to complex signal processing tasks.
|
wireless communication applications, ranging from simple experimentation to complex signal processing tasks.
|
||||||
|
|
||||||
USRP devices offer a flexible platform that can be used with various software frameworks, including GNU Radio
|
USRP devices offer a flexible platform that can be used with various software frameworks, including GNU Radio
|
||||||
and the USRP Hardware Driver (UHD). The product line includes both entry-level models for hobbyists and
|
and the USRP Hardware Driver (UHD). The product line includes both entry-level models for hobbyists and
|
||||||
advanced models for professional and research use.
|
advanced models for professional and research use.
|
||||||
|
|
||||||
Supported models
|
Supported models
|
||||||
----------------
|
----------------
|
||||||
|
|
||||||
- **USRP B200/B210:** Compact, single-board, full-duplex, with a wide frequency range.
|
- **USRP B200/B210:** Compact, single-board, full-duplex, with a wide frequency range.
|
||||||
- **USRP N200/N210:** High-performance models with increased bandwidth and connectivity options.
|
- **USRP N200/N210:** High-performance models with increased bandwidth and connectivity options.
|
||||||
- **USRP X300/X310:** High-end models featuring large bandwidth, multiple MIMO channels, and support for GPSDO.
|
- **USRP X300/X310:** High-end models featuring large bandwidth, multiple MIMO channels, and support for GPSDO.
|
||||||
- **USRP E310/E320:** Embedded devices with onboard processing capabilities.
|
- **USRP E310/E320:** Embedded devices with onboard processing capabilities.
|
||||||
- **USRP B200mini:** Ultra-compact model for portable and embedded applications.
|
- **USRP B200mini:** Ultra-compact model for portable and embedded applications.
|
||||||
|
|
||||||
Key features
|
Key features
|
||||||
------------
|
------------
|
||||||
|
|
||||||
- **Frequency Range:** Typically covers from DC to 6 GHz, depending on the model and daughter boards used.
|
- **Frequency Range:** Typically covers from DC to 6 GHz, depending on the model and daughter boards used.
|
||||||
- **Bandwidth:** Varies by model, up to 160 MHz in some high-end versions.
|
- **Bandwidth:** Varies by model, up to 160 MHz in some high-end versions.
|
||||||
- **Connectivity:** Includes USB 3.0, Ethernet, and PCIe interfaces depending on the model.
|
- **Connectivity:** Includes USB 3.0, Ethernet, and PCIe interfaces depending on the model.
|
||||||
- **Software Support:** Compatible with UHD, GNU Radio, and other SDR frameworks.
|
- **Software Support:** Compatible with UHD, GNU Radio, and other SDR frameworks.
|
||||||
|
|
||||||
Hackability
|
Hackability
|
||||||
-----------
|
-----------
|
||||||
|
|
||||||
- The UHD library is fully open source and can be modified to meet user untention.
|
- The UHD library is fully open source and can be modified to meet user untention.
|
||||||
- Certain USRP models have "RFNoC" which streamlines the inclusion of custom FPGA processing in a USRP.
|
- Certain USRP models have "RFNoC" which streamlines the inclusion of custom FPGA processing in a USRP.
|
||||||
|
|
||||||
Limitations
|
Limitations
|
||||||
-----------
|
-----------
|
||||||
|
|
||||||
- Some models may have limited bandwidth or processing capabilities.
|
- Some models may have limited bandwidth or processing capabilities.
|
||||||
- Compatibility with certain software tools may vary depending on the version of the UHD.
|
- Compatibility with certain software tools may vary depending on the version of the UHD.
|
||||||
- Price range can be a consideration, especially for high-end models.
|
- Price range can be a consideration, especially for high-end models.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
USRP devices require the UHD (USRP Hardware Driver) library with Python bindings. There is no pip-installable
|
1. Activate your Radioconda environment:
|
||||||
UHD package — it must either be installed via conda or built from source.
|
|
||||||
|
.. code-block:: bash
|
||||||
**Option A: Install via conda (recommended for conda environments)**
|
|
||||||
|
conda activate <your-env-name>
|
||||||
.. code-block:: bash
|
|
||||||
|
2. Install UHD and Python bindings:
|
||||||
conda install conda-forge::uhd
|
|
||||||
|
.. code-block:: bash
|
||||||
**Option B: Build from source (required for pip/venv environments)**
|
|
||||||
|
conda install conda-forge::uhd
|
||||||
The Python bindings must target the same Python version used in your virtual environment.
|
|
||||||
|
3. Download UHD images:
|
||||||
1. Install build dependencies:
|
|
||||||
|
.. code-block:: bash
|
||||||
.. code-block:: bash
|
|
||||||
|
uhd_images_downloader
|
||||||
sudo apt install cmake build-essential libboost-all-dev libusb-1.0-0-dev \
|
|
||||||
python3-dev python3-numpy libncurses-dev
|
4. Verify access to your device:
|
||||||
|
|
||||||
2. Install the Mako template library into your virtual environment (used by UHD's build system):
|
.. code-block:: bash
|
||||||
|
|
||||||
.. code-block:: bash
|
uhd_find_devices
|
||||||
|
|
||||||
pip install mako
|
For USB devices only (e.g. B series), install a ``udev`` rule by creating a link into your Radioconda installation.
|
||||||
|
|
||||||
3. Clone and build UHD with your virtual environment activated:
|
.. code-block:: bash
|
||||||
|
|
||||||
.. code-block:: bash
|
sudo ln -s $CONDA_PREFIX/lib/uhd/utils/uhd-usrp.rules /etc/udev/rules.d/radioconda-uhd-usrp.rules
|
||||||
|
sudo udevadm control --reload
|
||||||
git clone https://github.com/EttusResearch/uhd.git
|
sudo udevadm trigger
|
||||||
cd uhd
|
|
||||||
git checkout v4.7.0.0
|
5. (Optional) Update firmware/FPGA images:
|
||||||
cd host
|
|
||||||
mkdir build && cd build
|
.. code-block:: bash
|
||||||
cmake -DENABLE_PYTHON_API=ON -DPYTHON_EXECUTABLE=$(which python3) ..
|
|
||||||
make -j$(nproc)
|
uhd_usrp_probe
|
||||||
sudo make install
|
|
||||||
sudo ldconfig
|
This will ensure your device is running the latest firmware and FPGA versions.
|
||||||
|
|
||||||
.. important::
|
Further information
|
||||||
|
-------------------
|
||||||
Run the ``cmake`` command with your virtual environment activated so ``$(which python3)`` points
|
|
||||||
to the correct interpreter. Before running ``make``, verify the cmake output includes::
|
- `Official USRP Website <https://www.ettus.com/>`_
|
||||||
|
- `USRP Documentation <https://kb.ettus.com/USRP_Hardware_Driver_and_Interfaces>`_
|
||||||
-- * LibUHD - Python API → must say "Enabling"
|
- `USRP Setup with Radioconda <https://github.com/radioconda/radioconda-installer?tab=readme-ov-file#uhd-ettus-usrp>`_
|
||||||
-- Python interpreter: .../your-venv/bin/python3
|
|
||||||
|
|
||||||
If "LibUHD - Python API" is not listed under enabled components, the Python bindings will not be
|
|
||||||
built. The build typically takes 10–30 minutes.
|
|
||||||
|
|
||||||
4. Copy the Python bindings into your virtual environment if ``import uhd`` fails after installation:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
cp -r ~/uhd/host/build/python/uhd ~/.venv/lib/python3.XX/site-packages/
|
|
||||||
|
|
||||||
Replace ``python3.XX`` with your Python version (e.g., ``python3.12``).
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
If you have a pre-existing UHD installation built against a different Python version, you will see
|
|
||||||
a circular import error. The bindings must match the Python version in your virtual environment exactly.
|
|
||||||
|
|
||||||
**After either installation method:**
|
|
||||||
|
|
||||||
1. Download UHD FPGA/firmware images:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
uhd_images_downloader
|
|
||||||
|
|
||||||
2. Verify device access:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
uhd_find_devices
|
|
||||||
|
|
||||||
For USB devices (e.g. B-series), install a ``udev`` rule.
|
|
||||||
|
|
||||||
For most users:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
sudo udevadm control --reload
|
|
||||||
sudo udevadm trigger
|
|
||||||
|
|
||||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/uhd/utils/uhd-usrp.rules /etc/udev/rules.d/radioconda-uhd-usrp.rules
|
|
||||||
sudo udevadm control --reload
|
|
||||||
sudo udevadm trigger
|
|
||||||
|
|
||||||
3. (Optional) Update firmware/FPGA images:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
uhd_usrp_probe
|
|
||||||
|
|
||||||
This will ensure your device is running the latest firmware and FPGA versions.
|
|
||||||
|
|
||||||
Further information
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
- `Official USRP Website <https://www.ettus.com/>`_
|
|
||||||
- `USRP Documentation <https://kb.ettus.com/USRP_Hardware_Driver_and_Interfaces>`_
|
|
||||||
- `USRP Setup with Radioconda <https://github.com/radioconda/radioconda-installer?tab=readme-ov-file#uhd-ettus-usrp>`_
|
|
||||||
|
|
|
||||||
123
docs/spectrogram_dashboard_op_bug.md
Normal file
123
docs/spectrogram_dashboard_op_bug.md
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
# Bug: `SpectrogramDashboardOp` destructor calls `std::terminate`
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
`SpectrogramDashboardOp` spawns an HTTP server thread during setup but its destructor
|
||||||
|
does not `join()` or `detach()` it. Per the C++ standard, destroying a joinable
|
||||||
|
`std::thread` calls `std::terminate()` — so **any** shutdown path kills the app:
|
||||||
|
init failure, Ctrl-C, or normal exit at end of `main`.
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
Built app (`new_dashboard`) crashes on shutdown with this backtrace:
|
||||||
|
|
||||||
|
```
|
||||||
|
#3 __GI_raise
|
||||||
|
#4 __GI_abort
|
||||||
|
#5 libstdc++ (std::terminate handler)
|
||||||
|
#6 libstdc++
|
||||||
|
#7 std::terminate()
|
||||||
|
#8 std::thread::~thread()
|
||||||
|
#9 ria::ops::SpectrogramDashboardOp::~SpectrogramDashboardOp()
|
||||||
|
#10 __gnu_cxx::new_allocator<SpectrogramDashboardOp>::destroy(...)
|
||||||
|
...
|
||||||
|
#23 ria::Pipeline::~Pipeline()
|
||||||
|
#24 main
|
||||||
|
```
|
||||||
|
|
||||||
|
The stack shows the failure is entirely inside the op's own destructor — not
|
||||||
|
downstream of any flow / port-wiring issue. The op's startup message
|
||||||
|
`HTTP server started on port 8080` prints just before the crash, confirming the
|
||||||
|
server thread is running and joinable when destruction begins.
|
||||||
|
|
||||||
|
## Reproduction
|
||||||
|
|
||||||
|
1. Build any RIA app that includes `SpectrogramDashboardOp`.
|
||||||
|
2. Run the container; it crashes with `terminate called without an active exception`
|
||||||
|
regardless of whether other operators succeed or fail.
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
|
||||||
|
Standard C++ invariant:
|
||||||
|
|
||||||
|
> If a `std::thread` object is destroyed while still `joinable()`, the destructor
|
||||||
|
> calls `std::terminate()`.
|
||||||
|
> — [cppreference.com/w/cpp/thread/thread/~thread](https://en.cppreference.com/w/cpp/thread/thread/~thread)
|
||||||
|
|
||||||
|
The destructor needs to (a) signal the server to stop, (b) wait for the thread
|
||||||
|
to exit, and (c) join it before the `std::thread` member is destroyed.
|
||||||
|
|
||||||
|
## Fix
|
||||||
|
|
||||||
|
In `SpectrogramDashboardOp`:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
SpectrogramDashboardOp::~SpectrogramDashboardOp() {
|
||||||
|
// 1. Tell the HTTP server / websocket server to stop accepting
|
||||||
|
// and to return from its serve loop. Exact call depends on the
|
||||||
|
// HTTP library in use:
|
||||||
|
// - cpp-httplib: server_.stop();
|
||||||
|
// - Boost.Beast: acceptor_.close(); io_context_.stop();
|
||||||
|
// - custom: shutdown_flag_.store(true); close(listen_fd_);
|
||||||
|
if (server_) {
|
||||||
|
server_->stop();
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Join the thread if it was ever started.
|
||||||
|
if (http_thread_.joinable()) {
|
||||||
|
http_thread_.join();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
If multiple threads are owned (e.g. separate WebSocket broadcaster, update-rate
|
||||||
|
timer), join **each** of them.
|
||||||
|
|
||||||
|
## Related checks
|
||||||
|
|
||||||
|
While fixing this op, audit any other operator in the same repo that owns a
|
||||||
|
thread:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep -rn "std::thread " src/
|
||||||
|
```
|
||||||
|
|
||||||
|
For each match, confirm the owning class's destructor does:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
if (thread_.joinable()) thread_.join();
|
||||||
|
```
|
||||||
|
|
||||||
|
plus whatever shutdown signal is needed to make the thread actually return.
|
||||||
|
|
||||||
|
## Acceptance
|
||||||
|
|
||||||
|
- `SpectrogramDashboardOp` destructor joins all spawned threads.
|
||||||
|
- A RIA app containing this op exits cleanly on `Ctrl-C` with no
|
||||||
|
`terminate called without an active exception` message.
|
||||||
|
- Forcing an init failure (e.g. a bad `websocket_port`) produces a readable
|
||||||
|
exception message instead of `SIGABRT`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prompt to paste into Claude Code (in the op's repo)
|
||||||
|
|
||||||
|
> `SpectrogramDashboardOp` has a latent bug: its destructor lets a joinable
|
||||||
|
> `std::thread` (the HTTP server thread that prints "HTTP server started on
|
||||||
|
> port 8080") go out of scope, which per the C++ standard calls
|
||||||
|
> `std::terminate()`. This makes any built RIA app containing this op crash on
|
||||||
|
> every shutdown path — init failure, normal exit, and Ctrl-C — with the
|
||||||
|
> unhelpful message `terminate called without an active exception`. Stack trace
|
||||||
|
> at the point of abort goes through `std::thread::~thread()` →
|
||||||
|
> `SpectrogramDashboardOp::~SpectrogramDashboardOp()`.
|
||||||
|
>
|
||||||
|
> Fix the destructor: (a) signal the HTTP server to stop (e.g. `server_->stop()`
|
||||||
|
> for cpp-httplib, or close the listening socket + set a shutdown flag), then
|
||||||
|
> (b) `if (http_thread_.joinable()) http_thread_.join();`. Apply the same pattern
|
||||||
|
> to any other `std::thread` members the op owns (WebSocket broadcaster, rate
|
||||||
|
> timer, etc.). Then grep for other `std::thread` members in this repo and audit
|
||||||
|
> their owners' destructors for the same bug.
|
||||||
|
>
|
||||||
|
> Acceptance: the op's destructor joins every thread it starts; a test that
|
||||||
|
> constructs and immediately destroys the op exits cleanly; Ctrl-C on a running
|
||||||
|
> app produces no `terminate` message.
|
||||||
1411
poetry.lock
generated
1411
poetry.lock
generated
File diff suppressed because it is too large
Load Diff
|
|
@ -1,6 +1,6 @@
|
||||||
[project]
|
[project]
|
||||||
name = "ria-toolkit-oss"
|
name = "ria-toolkit-oss"
|
||||||
version = "0.1.7"
|
version = "0.1.5"
|
||||||
description = "An open-source version of the RIA Toolkit, including the fundamental tools to get started developing, testing, and deploying radio intelligence applications"
|
description = "An open-source version of the RIA Toolkit, including the fundamental tools to get started developing, testing, and deploying radio intelligence applications"
|
||||||
license = { text = "AGPL-3.0-only" }
|
license = { text = "AGPL-3.0-only" }
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
|
|
@ -50,7 +50,7 @@ dependencies = [
|
||||||
"pyyaml (>=6.0.3,<7.0.0)",
|
"pyyaml (>=6.0.3,<7.0.0)",
|
||||||
"click (>=8.1.0,<9.0.0)",
|
"click (>=8.1.0,<9.0.0)",
|
||||||
"matplotlib (>=3.8.0,<4.0.0)",
|
"matplotlib (>=3.8.0,<4.0.0)",
|
||||||
"paramiko (>=3.5.1)"
|
"paramiko (>=4.0.0)"
|
||||||
]
|
]
|
||||||
|
|
||||||
# [project.optional-dependencies] Commented out to prevent Tox tests from failing
|
# [project.optional-dependencies] Commented out to prevent Tox tests from failing
|
||||||
|
|
@ -149,11 +149,6 @@ exclude = '''
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
[tool.pytest.ini_options]
|
||||||
pythonpath = ["src"]
|
pythonpath = ["src"]
|
||||||
filterwarnings = [
|
|
||||||
# FastAPI emits this internally when handling 422 responses; the constant
|
|
||||||
# is not yet renamed in the installed starlette version, so we can't migrate.
|
|
||||||
"ignore:'HTTP_422_UNPROCESSABLE_ENTITY' is deprecated:DeprecationWarning",
|
|
||||||
]
|
|
||||||
|
|
||||||
[tool.isort]
|
[tool.isort]
|
||||||
profile = "black"
|
profile = "black"
|
||||||
|
|
|
||||||
170
screens_connection_updates.md
Normal file
170
screens_connection_updates.md
Normal file
|
|
@ -0,0 +1,170 @@
|
||||||
|
# Agent CLI Simplification — Handoff to ria-toolkit-oss
|
||||||
|
|
||||||
|
**Repo:** `ria-toolkit-oss`, branch `screens-connection`
|
||||||
|
**Goal:** Reduce agent setup from 3+ commands to 2 simple ones.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current UX (painful)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Step 1: Register via curl against FastAPI directly
|
||||||
|
curl -X POST http://hub:8005/screens/agents/register \
|
||||||
|
-H 'X-API-Key: supersecretapikey' \
|
||||||
|
-d '{"name": "my-agent"}'
|
||||||
|
# → {"agent_id": "agent-55cf3c5b8137f6f3", "token": "45Hbt..."}
|
||||||
|
|
||||||
|
# Step 2: Manually save credentials
|
||||||
|
ria-agent register \
|
||||||
|
--url http://hub:8005 \
|
||||||
|
--token 45HbtlpVDX7_XTF47biDcLcyiVmM51icEZVJ7J_UrEE \
|
||||||
|
--agent-id agent-55cf3c5b8137f6f3
|
||||||
|
|
||||||
|
# Step 3: Stream (with manual URL construction)
|
||||||
|
ria-agent stream \
|
||||||
|
--url "ws://hub:8005/screens/agent/ws?agent_id=agent-55cf3c5b8137f6f3" \
|
||||||
|
--token 45HbtlpVDX7_XTF47biDcLcyiVmM51icEZVJ7J_UrEE
|
||||||
|
```
|
||||||
|
|
||||||
|
Problems:
|
||||||
|
- User must know the FastAPI port (8005), not just the hub URL (3005)
|
||||||
|
- `register` subcommand only saves locally — doesn't call the server
|
||||||
|
- `_derive_ws_url` builds `/api/agent/ws/{agent_id}` but server endpoint is `/screens/agent/ws?agent_id=...`
|
||||||
|
- User must copy-paste agent_id and token between commands
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Target UX
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# One-time setup: register with the hub (hits server, saves config)
|
||||||
|
ria-agent register --hub http://whitehorse:3005 --api-key supersecretapikey --name lab-pluto
|
||||||
|
|
||||||
|
# Stream (reads config, connects automatically)
|
||||||
|
ria-agent stream
|
||||||
|
```
|
||||||
|
|
||||||
|
That's it. Two commands, no copy-pasting.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Changes needed in ria-toolkit-oss
|
||||||
|
|
||||||
|
### 1. `cli.py` — Make `register` call the server
|
||||||
|
|
||||||
|
Current `_cmd_register` just saves to `~/.ria/agent.json`. It should:
|
||||||
|
|
||||||
|
1. POST to `{hub_url}/screens/agents/register` with `X-API-Key` header
|
||||||
|
2. Receive `{agent_id, token}` from the server
|
||||||
|
3. Save everything to `~/.ria/agent.json`
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _cmd_register(args: argparse.Namespace) -> int:
|
||||||
|
import urllib.request
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
hub_url = args.hub.rstrip("/")
|
||||||
|
api_key = args.api_key
|
||||||
|
|
||||||
|
# Call the server to register
|
||||||
|
url = f"{hub_url}/screens/agents/register"
|
||||||
|
body = _json.dumps({"name": args.name or ""}).encode()
|
||||||
|
req = urllib.request.Request(
|
||||||
|
url,
|
||||||
|
data=body,
|
||||||
|
headers={
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"X-API-Key": api_key,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req) as resp:
|
||||||
|
data = _json.loads(resp.read())
|
||||||
|
except Exception as e:
|
||||||
|
print(f"error: registration failed: {e}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
agent_id = data["agent_id"]
|
||||||
|
token = data["token"]
|
||||||
|
|
||||||
|
# Save to config
|
||||||
|
cfg = _config.load()
|
||||||
|
cfg.hub_url = hub_url
|
||||||
|
cfg.agent_id = agent_id
|
||||||
|
cfg.token = token
|
||||||
|
if args.name:
|
||||||
|
cfg.name = args.name
|
||||||
|
cfg.insecure = bool(args.insecure)
|
||||||
|
path = _config.save(cfg)
|
||||||
|
|
||||||
|
print(f"Registered agent: {agent_id}")
|
||||||
|
print(f"Credentials saved to {path}")
|
||||||
|
return 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Update the argparse for `register`:
|
||||||
|
```python
|
||||||
|
p_reg = sub.add_parser("register", help="Register agent with RIA Hub and save credentials")
|
||||||
|
p_reg.add_argument("--hub", required=True, help="RIA Hub URL (e.g. http://whitehorse:3005)")
|
||||||
|
p_reg.add_argument("--api-key", required=True, help="Hub API key for authentication")
|
||||||
|
p_reg.add_argument("--name", default=None, help="Human-friendly agent name")
|
||||||
|
p_reg.add_argument("--insecure", action="store_true", help="Skip TLS verification")
|
||||||
|
```
|
||||||
|
|
||||||
|
Remove `--url`, `--token`, `--agent-id` from register — those are now server-generated.
|
||||||
|
|
||||||
|
### 2. `cli.py` — Fix `_derive_ws_url`
|
||||||
|
|
||||||
|
Current (wrong):
|
||||||
|
```python
|
||||||
|
suffix = f"/api/agent/ws/{agent_id}" if agent_id else "/api/agent/ws"
|
||||||
|
```
|
||||||
|
|
||||||
|
Should be:
|
||||||
|
```python
|
||||||
|
suffix = f"/screens/agent/ws?agent_id={agent_id}" if agent_id else "/screens/agent/ws"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. `cli.py` — Make `stream` zero-arg by default
|
||||||
|
|
||||||
|
Current `_cmd_stream` already loads config and derives the URL — it just needs the URL fix above. After that, bare `ria-agent stream` works if `register` was run first.
|
||||||
|
|
||||||
|
### 4. `config.py` — Add `api_key` field (optional)
|
||||||
|
|
||||||
|
Add `api_key: str = ""` to `AgentConfig` so the hub API key can be persisted for re-registration or other API calls. Not strictly required but useful.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Changes already done in ria-hub (Part B)
|
||||||
|
|
||||||
|
The server side is ready:
|
||||||
|
|
||||||
|
- `POST /screens/agents/register` — accepts `{"name": "..."}` with `X-API-Key` header, returns `{"agent_id": "...", "token": "..."}`
|
||||||
|
- `GET /screens/agent/ws?agent_id=...` — WebSocket endpoint, authenticates via `Authorization: Bearer {token}` header
|
||||||
|
- Agent token is hashed (SHA-256) and stored in MongoDB; lookup happens on WS connect
|
||||||
|
|
||||||
|
The Go proxy for `/screens/agents/register` through port 3005 still needs to be added (currently agents must hit FastAPI port 8005 directly). That's a ria-hub task, not ria-toolkit-oss.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary of file changes
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|------|--------|
|
||||||
|
| `src/ria_toolkit_oss/agent/cli.py` | `register` calls server API, new flags `--hub`/`--api-key`; fix `_derive_ws_url` path |
|
||||||
|
| `src/ria_toolkit_oss/agent/config.py` | Optional: add `api_key` field to `AgentConfig` |
|
||||||
|
| `tests/agent/test_cli.py` | Update register tests for new server-calling behavior |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Validated E2E flow (what works today)
|
||||||
|
|
||||||
|
We tested the full pipeline on whitehorse with a real Pluto SDR:
|
||||||
|
|
||||||
|
1. Agent connects via WebSocket with bearer token auth ✅
|
||||||
|
2. Server sends `start` with `radio_config` via Redis pub/sub → agent ✅
|
||||||
|
3. Agent opens Pluto, streams interleaved float32 IQ via binary WS frames ✅
|
||||||
|
4. FastAPI pushes frames to Redis list, Celery worker's `AgentDataSource.next_chunk()` BLPOP reads them ✅
|
||||||
|
5. Inference loop runs on live agent data identically to direct SDR mode ✅
|
||||||
|
|
||||||
|
The only manual friction is the multi-step registration and URL construction — which these CLI changes eliminate.
|
||||||
|
|
@ -5,11 +5,8 @@ Subcommands:
|
||||||
- ``ria-agent run [legacy args]`` — legacy long-poll NodeAgent (unchanged).
|
- ``ria-agent run [legacy args]`` — legacy long-poll NodeAgent (unchanged).
|
||||||
- ``ria-agent stream`` — new WebSocket-based IQ streamer.
|
- ``ria-agent stream`` — new WebSocket-based IQ streamer.
|
||||||
- ``ria-agent detect`` — print SDR drivers whose modules import cleanly.
|
- ``ria-agent detect`` — print SDR drivers whose modules import cleanly.
|
||||||
- ``ria-agent register --hub URL --api-key KEY`` — register with the hub
|
- ``ria-agent register --hub URL --api-key KEY`` — register with the hub and
|
||||||
using a personal registration key (minted from **Settings → RIA Agents**
|
save credentials (and optional TX interlocks) to ``~/.ria/agent.json``.
|
||||||
on the hub, shown once at mint time) and save credentials (and optional
|
|
||||||
TX interlocks) to ``~/.ria/agent.json``. The hub also accepts the legacy
|
|
||||||
shared ``[wac] API_KEY`` for back-compat, but that path is deprecated.
|
|
||||||
|
|
||||||
Invoking ``ria-agent`` with no subcommand falls through to the legacy
|
Invoking ``ria-agent`` with no subcommand falls through to the legacy
|
||||||
long-poll behavior for back-compatibility with existing deployments.
|
long-poll behavior for back-compatibility with existing deployments.
|
||||||
|
|
@ -28,75 +25,9 @@ from .hardware import available_devices
|
||||||
from .legacy_executor import main as _legacy_main
|
from .legacy_executor import main as _legacy_main
|
||||||
from .namegen import generate_agent_name
|
from .namegen import generate_agent_name
|
||||||
|
|
||||||
DEFAULT_HUB_URL = "https://riahub.ai"
|
|
||||||
|
|
||||||
_LEGACY_ALIASES = {"--hub", "--key", "--name", "--device", "--insecure", "--log-level", "--config"}
|
_LEGACY_ALIASES = {"--hub", "--key", "--name", "--device", "--insecure", "--log-level", "--config"}
|
||||||
|
|
||||||
|
|
||||||
def _user_agent() -> str:
|
|
||||||
"""Build the User-Agent header for hub requests.
|
|
||||||
|
|
||||||
Set explicitly so we don't fall back to Python's default `Python-urllib/<ver>`,
|
|
||||||
which is blocked by Cloudflare's Browser Integrity Check on `riahub.ai`
|
|
||||||
(HTTP 403 edge code 1010). Version is read from package metadata so it
|
|
||||||
tracks releases instead of going stale.
|
|
||||||
"""
|
|
||||||
from importlib.metadata import PackageNotFoundError, version
|
|
||||||
|
|
||||||
try:
|
|
||||||
pkg_version = version("ria-toolkit-oss")
|
|
||||||
except PackageNotFoundError:
|
|
||||||
pkg_version = "unknown"
|
|
||||||
return f"ria-agent/{pkg_version} (+https://riahub.ai/qoherent/ria-toolkit-oss)"
|
|
||||||
|
|
||||||
|
|
||||||
# How long to wait on the hub before giving up. The register endpoint is a
|
|
||||||
# small DB lookup + insert; anything past this is a stuck hub, not a slow one.
|
|
||||||
_REGISTER_TIMEOUT_S = 15
|
|
||||||
|
|
||||||
|
|
||||||
REGISTRATION_REASON_MESSAGES = {
|
|
||||||
"invalid_key": ("Registration key not recognized. Generate a fresh key from " "Settings → RIA Agents on the hub."),
|
|
||||||
"expired": ("This registration key has expired. Generate a new one from " "Settings → RIA Agents on the hub."),
|
|
||||||
"revoked": ("This registration key was revoked. Generate a new one from " "Settings → RIA Agents on the hub."),
|
|
||||||
"already_consumed": (
|
|
||||||
"This single-use registration key has already been used. "
|
|
||||||
"Generate a new one, or mint a reusable key instead."
|
|
||||||
),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _explain_registration_failure(status: int, body: bytes) -> str:
|
|
||||||
"""Return a human-readable explanation for a failed register call."""
|
|
||||||
try:
|
|
||||||
parsed = json.loads(body) if body else None
|
|
||||||
except ValueError:
|
|
||||||
parsed = None
|
|
||||||
|
|
||||||
if status == 429:
|
|
||||||
# 429 carries a plain string detail, never a reason code.
|
|
||||||
if isinstance(parsed, dict) and parsed.get("detail"):
|
|
||||||
detail = parsed["detail"]
|
|
||||||
else:
|
|
||||||
detail = body.decode("utf-8", "replace") or "rate limited"
|
|
||||||
return f"Registration rate-limited by the hub: {detail}"
|
|
||||||
|
|
||||||
if not isinstance(parsed, dict):
|
|
||||||
text = body.decode("utf-8", "replace")
|
|
||||||
return f"HTTP {status}: {text or 'no body'}"
|
|
||||||
|
|
||||||
detail = parsed.get("detail")
|
|
||||||
if isinstance(detail, dict):
|
|
||||||
reason = detail.get("reason")
|
|
||||||
if reason in REGISTRATION_REASON_MESSAGES:
|
|
||||||
return REGISTRATION_REASON_MESSAGES[reason]
|
|
||||||
if reason:
|
|
||||||
return f"Registration rejected ({reason})"
|
|
||||||
if isinstance(detail, str) and detail:
|
|
||||||
return f"Registration rejected: {detail}"
|
|
||||||
return f"HTTP {status}: {parsed}"
|
|
||||||
|
|
||||||
|
|
||||||
def _cmd_detect(_args: argparse.Namespace) -> int:
|
def _cmd_detect(_args: argparse.Namespace) -> int:
|
||||||
devices = available_devices()
|
devices = available_devices()
|
||||||
if not devices:
|
if not devices:
|
||||||
|
|
@ -108,7 +39,6 @@ def _cmd_detect(_args: argparse.Namespace) -> int:
|
||||||
|
|
||||||
|
|
||||||
def _cmd_register(args: argparse.Namespace) -> int:
|
def _cmd_register(args: argparse.Namespace) -> int:
|
||||||
import urllib.error
|
|
||||||
import urllib.request
|
import urllib.request
|
||||||
|
|
||||||
hub_url = args.hub.rstrip("/")
|
hub_url = args.hub.rstrip("/")
|
||||||
|
|
@ -121,20 +51,11 @@ def _cmd_register(args: argparse.Namespace) -> int:
|
||||||
headers={
|
headers={
|
||||||
"Content-Type": "application/json",
|
"Content-Type": "application/json",
|
||||||
"X-API-Key": args.api_key,
|
"X-API-Key": args.api_key,
|
||||||
"User-Agent": _user_agent(),
|
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
with urllib.request.urlopen(req, timeout=_REGISTER_TIMEOUT_S) as resp:
|
with urllib.request.urlopen(req) as resp:
|
||||||
data = json.loads(resp.read())
|
data = json.loads(resp.read())
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
try:
|
|
||||||
err_body = e.read()
|
|
||||||
except Exception:
|
|
||||||
err_body = b""
|
|
||||||
msg = _explain_registration_failure(e.code, err_body)
|
|
||||||
print(f"error: registration failed: {msg}", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"error: registration failed: {e}", file=sys.stderr)
|
print(f"error: registration failed: {e}", file=sys.stderr)
|
||||||
return 1
|
return 1
|
||||||
|
|
@ -159,7 +80,7 @@ def _cmd_register(args: argparse.Namespace) -> int:
|
||||||
cfg.tx_allowed_freq_ranges = [[float(lo), float(hi)] for lo, hi in freq_ranges]
|
cfg.tx_allowed_freq_ranges = [[float(lo), float(hi)] for lo, hi in freq_ranges]
|
||||||
path = _config.save(cfg)
|
path = _config.save(cfg)
|
||||||
|
|
||||||
print(f"Registered agent: {agent_id} ({name})")
|
print(f"Registered agent: {agent_id}")
|
||||||
if cfg.tx_enabled:
|
if cfg.tx_enabled:
|
||||||
caps: list[str] = []
|
caps: list[str] = []
|
||||||
if cfg.tx_max_gain_db is not None:
|
if cfg.tx_max_gain_db is not None:
|
||||||
|
|
@ -219,17 +140,8 @@ def main() -> None:
|
||||||
sub.add_parser("detect", help="List available SDR drivers")
|
sub.add_parser("detect", help="List available SDR drivers")
|
||||||
|
|
||||||
p_reg = sub.add_parser("register", help="Register agent with RIA Hub and save credentials")
|
p_reg = sub.add_parser("register", help="Register agent with RIA Hub and save credentials")
|
||||||
p_reg.add_argument("--hub", default=DEFAULT_HUB_URL, help=f"RIA Hub URL (default: {DEFAULT_HUB_URL})")
|
p_reg.add_argument("--hub", required=True, help="RIA Hub URL (e.g. http://whitehorse:3005)")
|
||||||
p_reg.add_argument(
|
p_reg.add_argument("--api-key", dest="api_key", required=True, help="Hub API key")
|
||||||
"--api-key",
|
|
||||||
dest="api_key",
|
|
||||||
required=True,
|
|
||||||
help=(
|
|
||||||
"Personal registration key from the RIA Agents page on the hub "
|
|
||||||
"(format: ria_reg_...). Shown once when generated; save it then. "
|
|
||||||
"The legacy shared API key is also accepted but deprecated."
|
|
||||||
),
|
|
||||||
)
|
|
||||||
p_reg.add_argument("--name", default=None, help="Human-friendly agent name")
|
p_reg.add_argument("--name", default=None, help="Human-friendly agent name")
|
||||||
p_reg.add_argument("--insecure", action="store_true", help="Skip TLS verification")
|
p_reg.add_argument("--insecure", action="store_true", help="Skip TLS verification")
|
||||||
p_reg.add_argument(
|
p_reg.add_argument(
|
||||||
|
|
|
||||||
|
|
@ -68,7 +68,7 @@ _HEARTBEAT_INTERVAL = 30 # seconds between heartbeats
|
||||||
_POLL_TIMEOUT = 30 # server-side long-poll duration
|
_POLL_TIMEOUT = 30 # server-side long-poll duration
|
||||||
_POLL_CLIENT_TIMEOUT = 40 # client read timeout — slightly longer than server
|
_POLL_CLIENT_TIMEOUT = 40 # client read timeout — slightly longer than server
|
||||||
_RECONNECT_PAUSE = 5 # seconds to wait after a poll error before retrying
|
_RECONNECT_PAUSE = 5 # seconds to wait after a poll error before retrying
|
||||||
_CHUNK_SIZE = 10 * 1024 * 1024 # 10 MB per chunk — fast enough for git-LFS to process within timeout
|
_CHUNK_SIZE = 50 * 1024 * 1024 # 50 MB — well below Cloudflare's 100 MB limit
|
||||||
_DIRECT_THRESHOLD = 90 * 1024 * 1024 # files above this use chunked upload
|
_DIRECT_THRESHOLD = 90 * 1024 * 1024 # files above this use chunked upload
|
||||||
_CAPTURE_SAMPLES = 4096 # IQ samples per inference window
|
_CAPTURE_SAMPLES = 4096 # IQ samples per inference window
|
||||||
_IDLE_LABELS = frozenset({"noise", "idle", "no_signal", "unknown_protocol", "background"})
|
_IDLE_LABELS = frozenset({"noise", "idle", "no_signal", "unknown_protocol", "background"})
|
||||||
|
|
@ -93,24 +93,16 @@ class NodeAgent:
|
||||||
name: str,
|
name: str,
|
||||||
sdr_device: str = "unknown",
|
sdr_device: str = "unknown",
|
||||||
insecure: bool = False,
|
insecure: bool = False,
|
||||||
role: str = "general",
|
|
||||||
session_code: str | None = None,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
self.hub_url = hub_url.rstrip("/")
|
self.hub_url = hub_url.rstrip("/")
|
||||||
self.api_key = api_key
|
self.api_key = api_key
|
||||||
self.name = name
|
self.name = name
|
||||||
self.sdr_device = sdr_device
|
self.sdr_device = sdr_device
|
||||||
self.insecure = insecure
|
self.insecure = insecure
|
||||||
self.role = role
|
|
||||||
self.session_code = session_code
|
|
||||||
|
|
||||||
self.node_id: str | None = None
|
self.node_id: str | None = None
|
||||||
self._stop = threading.Event()
|
self._stop = threading.Event()
|
||||||
|
|
||||||
# ── TX state ────────────────────────────────────────────────────────
|
|
||||||
self._tx_stop = threading.Event()
|
|
||||||
self._tx_thread: threading.Thread | None = None
|
|
||||||
|
|
||||||
# ── Inference state ─────────────────────────────────────────────────
|
# ── Inference state ─────────────────────────────────────────────────
|
||||||
# Protected by _inf_lock for cross-thread model swaps.
|
# Protected by _inf_lock for cross-thread model swaps.
|
||||||
self._inf_lock = threading.Lock()
|
self._inf_lock = threading.Lock()
|
||||||
|
|
@ -180,27 +172,19 @@ class NodeAgent:
|
||||||
capabilities = ["campaign"]
|
capabilities = ["campaign"]
|
||||||
if self._ort_available:
|
if self._ort_available:
|
||||||
capabilities.append("inference")
|
capabilities.append("inference")
|
||||||
if self.role == "tx":
|
resp = self._post(
|
||||||
capabilities.append("transmit")
|
"/composer/nodes/register",
|
||||||
payload: dict = {
|
json={
|
||||||
"name": self.name,
|
"name": self.name,
|
||||||
"sdr_device": self.sdr_device,
|
"sdr_device": self.sdr_device,
|
||||||
"ria_toolkit_version": self._ria_version,
|
"ria_toolkit_version": self._ria_version,
|
||||||
"capabilities": capabilities,
|
"capabilities": capabilities,
|
||||||
"role": self.role,
|
},
|
||||||
}
|
timeout=15,
|
||||||
if self.session_code:
|
)
|
||||||
payload["session_code"] = self.session_code
|
|
||||||
resp = self._post("/composer/nodes/register", json=payload, timeout=15)
|
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
self.node_id = resp.json()["node_id"]
|
self.node_id = resp.json()["node_id"]
|
||||||
logger.info(
|
logger.info("Registered as %r (node_id=%s)", self.name, self.node_id)
|
||||||
"Registered as %r (node_id=%s, role=%s%s)",
|
|
||||||
self.name,
|
|
||||||
self.node_id,
|
|
||||||
self.role,
|
|
||||||
f", session_code={self.session_code!r}" if self.session_code else "",
|
|
||||||
)
|
|
||||||
|
|
||||||
def _deregister(self) -> None:
|
def _deregister(self) -> None:
|
||||||
if not self.node_id:
|
if not self.node_id:
|
||||||
|
|
@ -261,10 +245,9 @@ class NodeAgent:
|
||||||
if command == "run_campaign":
|
if command == "run_campaign":
|
||||||
campaign_id: str = cmd.get("campaign_id") or str(uuid.uuid4())
|
campaign_id: str = cmd.get("campaign_id") or str(uuid.uuid4())
|
||||||
config_dict: dict = cmd.get("payload") or {}
|
config_dict: dict = cmd.get("payload") or {}
|
||||||
skip_local_tx: bool = bool(cmd.get("skip_local_tx", False))
|
|
||||||
threading.Thread(
|
threading.Thread(
|
||||||
target=self._run_campaign,
|
target=self._run_campaign,
|
||||||
args=(campaign_id, config_dict, skip_local_tx),
|
args=(campaign_id, config_dict),
|
||||||
daemon=True,
|
daemon=True,
|
||||||
name=f"campaign-{campaign_id[:8]}",
|
name=f"campaign-{campaign_id[:8]}",
|
||||||
).start()
|
).start()
|
||||||
|
|
@ -286,17 +269,6 @@ class NodeAgent:
|
||||||
self._stop_inference()
|
self._stop_inference()
|
||||||
elif command == "configure_inference":
|
elif command == "configure_inference":
|
||||||
self._queue_sdr_config(cmd)
|
self._queue_sdr_config(cmd)
|
||||||
elif command == "start_transmit":
|
|
||||||
threading.Thread(
|
|
||||||
target=self._start_transmit,
|
|
||||||
args=(cmd,),
|
|
||||||
daemon=True,
|
|
||||||
name="ria-start-tx",
|
|
||||||
).start()
|
|
||||||
elif command == "stop_transmit":
|
|
||||||
self._stop_transmit()
|
|
||||||
elif command == "configure_transmit":
|
|
||||||
logger.info("configure_transmit received — will apply on next step boundary")
|
|
||||||
else:
|
else:
|
||||||
logger.warning("Unknown command %r — ignored", command)
|
logger.warning("Unknown command %r — ignored", command)
|
||||||
|
|
||||||
|
|
@ -304,7 +276,7 @@ class NodeAgent:
|
||||||
# Campaign execution
|
# Campaign execution
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
def _run_campaign(self, campaign_id: str, config_dict: dict, skip_local_tx: bool = False) -> None:
|
def _run_campaign(self, campaign_id: str, config_dict: dict) -> None:
|
||||||
try:
|
try:
|
||||||
from ria_toolkit_oss.orchestration.campaign import CampaignConfig
|
from ria_toolkit_oss.orchestration.campaign import CampaignConfig
|
||||||
from ria_toolkit_oss.orchestration.executor import CampaignExecutor
|
from ria_toolkit_oss.orchestration.executor import CampaignExecutor
|
||||||
|
|
@ -316,10 +288,10 @@ class NodeAgent:
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
logger.info("Campaign %s starting (skip_local_tx=%s)", campaign_id[:8], skip_local_tx)
|
logger.info("Campaign %s starting", campaign_id[:8])
|
||||||
try:
|
try:
|
||||||
config = CampaignConfig.from_dict(config_dict)
|
config = CampaignConfig.from_dict(config_dict)
|
||||||
executor = CampaignExecutor(config, skip_local_tx=skip_local_tx)
|
executor = CampaignExecutor(config)
|
||||||
result = executor.run()
|
result = executor.run()
|
||||||
logger.info("Campaign %s completed — uploading recordings", campaign_id[:8])
|
logger.info("Campaign %s completed — uploading recordings", campaign_id[:8])
|
||||||
self._upload_recordings(campaign_id, config, result)
|
self._upload_recordings(campaign_id, config, result)
|
||||||
|
|
@ -329,58 +301,6 @@ class NodeAgent:
|
||||||
logger.error("Campaign %s failed: %s", campaign_id[:8], exc)
|
logger.error("Campaign %s failed: %s", campaign_id[:8], exc)
|
||||||
self._report_campaign_status(campaign_id, "failed", error=str(exc))
|
self._report_campaign_status(campaign_id, "failed", error=str(exc))
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# TX execution
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _start_transmit(self, cmd: dict) -> None:
|
|
||||||
"""Execute a synthetic transmit campaign using TxExecutor.
|
|
||||||
|
|
||||||
The command payload mirrors a TransmitterConfig dict with an optional
|
|
||||||
``schedule`` of steps. Each step synthesises a signal and transmits it
|
|
||||||
via the local SDR in TX mode.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
from ria_toolkit_oss.orchestration.tx_executor import TxExecutor
|
|
||||||
except ImportError as exc:
|
|
||||||
logger.error("start_transmit: TxExecutor not available: %s", exc)
|
|
||||||
return
|
|
||||||
|
|
||||||
if self._tx_thread and self._tx_thread.is_alive():
|
|
||||||
logger.warning("start_transmit: TX already running — ignoring duplicate command")
|
|
||||||
return
|
|
||||||
|
|
||||||
self._tx_stop.clear()
|
|
||||||
campaign_id: str = cmd.get("campaign_id") or str(uuid.uuid4())
|
|
||||||
executor = TxExecutor(
|
|
||||||
config=cmd,
|
|
||||||
sdr_device=self.sdr_device,
|
|
||||||
stop_event=self._tx_stop,
|
|
||||||
)
|
|
||||||
self._tx_thread = threading.Thread(
|
|
||||||
target=self._run_tx_campaign,
|
|
||||||
args=(executor, campaign_id),
|
|
||||||
daemon=True,
|
|
||||||
name=f"tx-campaign-{campaign_id[:8]}",
|
|
||||||
)
|
|
||||||
self._tx_thread.start()
|
|
||||||
|
|
||||||
def _run_tx_campaign(self, executor: Any, campaign_id: str) -> None:
|
|
||||||
try:
|
|
||||||
executor.run()
|
|
||||||
logger.info("TX campaign %s completed", campaign_id[:8])
|
|
||||||
self._report_campaign_status(campaign_id, "completed")
|
|
||||||
except Exception as exc:
|
|
||||||
logger.error("TX campaign %s failed: %s", campaign_id[:8], exc)
|
|
||||||
self._report_campaign_status(campaign_id, "failed", error=str(exc))
|
|
||||||
|
|
||||||
def _stop_transmit(self) -> None:
|
|
||||||
"""Signal the TX loop to stop gracefully."""
|
|
||||||
self._tx_stop.set()
|
|
||||||
if self._tx_thread and self._tx_thread.is_alive():
|
|
||||||
self._tx_thread.join(timeout=5.0)
|
|
||||||
logger.info("TX stopped")
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Inference — model loading
|
# Inference — model loading
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
@ -659,18 +579,13 @@ class NodeAgent:
|
||||||
base_url = f"{self.hub_url}/datasets/upload"
|
base_url = f"{self.hub_url}/datasets/upload"
|
||||||
steps = (result.get("steps") if isinstance(result, dict) else getattr(result, "steps", None)) or []
|
steps = (result.get("steps") if isinstance(result, dict) else getattr(result, "steps", None)) or []
|
||||||
|
|
||||||
output_obj = getattr(config, "output", None)
|
|
||||||
folder = getattr(output_obj, "folder", None)
|
|
||||||
campaign_name: str = folder if folder is not None else (getattr(config, "name", None) or "")
|
|
||||||
for step in steps:
|
for step in steps:
|
||||||
output_path: str | None = getattr(step, "output_path", None)
|
output_path: str | None = getattr(step, "output_path", None)
|
||||||
if not output_path:
|
if not output_path:
|
||||||
continue
|
continue
|
||||||
device_id: str = getattr(step, "transmitter_id", "") or ""
|
device_id: str = getattr(step, "transmitter_id", "") or ""
|
||||||
for fpath in _sigmf_files(output_path):
|
for fpath in _sigmf_files(output_path):
|
||||||
basename = os.path.basename(fpath)
|
filename = os.path.basename(fpath)
|
||||||
path_parts = [p for p in (campaign_name, device_id) if p]
|
|
||||||
filename = "/".join(path_parts + [basename])
|
|
||||||
metadata = {
|
metadata = {
|
||||||
"filename": filename,
|
"filename": filename,
|
||||||
"repo_owner": repo_owner,
|
"repo_owner": repo_owner,
|
||||||
|
|
@ -756,7 +671,7 @@ class NodeAgent:
|
||||||
headers=headers,
|
headers=headers,
|
||||||
files={"file": (filename, chunk, "application/octet-stream")},
|
files={"file": (filename, chunk, "application/octet-stream")},
|
||||||
data={**metadata, "upload_id": upload_id, "chunk_index": i, "total_chunks": total_chunks},
|
data={**metadata, "upload_id": upload_id, "chunk_index": i, "total_chunks": total_chunks},
|
||||||
timeout=(30, None), # 30s connect, no read timeout — server may take minutes on final chunk
|
timeout=120,
|
||||||
verify=verify,
|
verify=verify,
|
||||||
)
|
)
|
||||||
if not resp.ok:
|
if not resp.ok:
|
||||||
|
|
@ -933,21 +848,6 @@ def main() -> None:
|
||||||
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
|
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
|
||||||
help="Logging verbosity (default: INFO)",
|
help="Logging verbosity (default: INFO)",
|
||||||
)
|
)
|
||||||
parser.add_argument(
|
|
||||||
"--role",
|
|
||||||
default=None,
|
|
||||||
choices=["general", "rx", "tx"],
|
|
||||||
help=("Node role reported to the hub. " "'tx' enables synthetic transmission commands. " "Default: general"),
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--session-code",
|
|
||||||
default=None,
|
|
||||||
metavar="CODE",
|
|
||||||
help=(
|
|
||||||
"3-word session code to pair this TX agent with a waiting campaign, "
|
|
||||||
"e.g. 'amber-peak-transmit'. Supplied by the campaign UI."
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
|
@ -961,8 +861,6 @@ def main() -> None:
|
||||||
device = args.device or cfg.get("device", "unknown")
|
device = args.device or cfg.get("device", "unknown")
|
||||||
insecure = args.insecure if args.insecure is not None else cfg.get("insecure", False)
|
insecure = args.insecure if args.insecure is not None else cfg.get("insecure", False)
|
||||||
log_level = args.log_level or cfg.get("log_level", "INFO")
|
log_level = args.log_level or cfg.get("log_level", "INFO")
|
||||||
role = args.role or cfg.get("role", "general")
|
|
||||||
session_code = args.session_code or cfg.get("session_code")
|
|
||||||
|
|
||||||
if not hub:
|
if not hub:
|
||||||
parser.error("--hub is required (or set 'hub' in the config file)")
|
parser.error("--hub is required (or set 'hub' in the config file)")
|
||||||
|
|
@ -990,8 +888,6 @@ def main() -> None:
|
||||||
name=name,
|
name=name,
|
||||||
sdr_device=device,
|
sdr_device=device,
|
||||||
insecure=insecure,
|
insecure=insecure,
|
||||||
role=role,
|
|
||||||
session_code=session_code,
|
|
||||||
)
|
)
|
||||||
agent.run()
|
agent.run()
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
from ria_toolkit_oss.data.annotation import Annotation
|
from ria_toolkit_oss.datatypes.annotation import Annotation
|
||||||
|
|
||||||
# TODO figure out how to transfer labels in the merge case
|
# TODO figure out how to transfer labels in the merge case
|
||||||
|
|
||||||
|
|
@ -45,14 +45,7 @@ def is_annotation_contained(inner: Annotation, outer: Annotation) -> bool:
|
||||||
outer_sample_stop = outer.sample_start + outer.sample_count
|
outer_sample_stop = outer.sample_start + outer.sample_count
|
||||||
|
|
||||||
if inner.sample_start > outer.sample_start and inner_sample_stop < outer_sample_stop:
|
if inner.sample_start > outer.sample_start and inner_sample_stop < outer_sample_stop:
|
||||||
if (
|
if inner.freq_lower_edge > outer.freq_lower_edge and inner.freq_upper_edge < outer.freq_upper_edge:
|
||||||
inner.freq_lower_edge is not None
|
|
||||||
and inner.freq_upper_edge is not None
|
|
||||||
and outer.freq_lower_edge is not None
|
|
||||||
and outer.freq_upper_edge is not None
|
|
||||||
and inner.freq_lower_edge > outer.freq_lower_edge
|
|
||||||
and inner.freq_upper_edge < outer.freq_upper_edge
|
|
||||||
):
|
|
||||||
return True
|
return True
|
||||||
|
|
||||||
return False
|
return False
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@ from typing import Optional
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Annotation, Recording
|
from ria_toolkit_oss.datatypes import Annotation, Recording
|
||||||
|
|
||||||
|
|
||||||
def annotate_with_cusum(
|
def annotate_with_cusum(
|
||||||
|
|
@ -24,7 +24,7 @@ def annotate_with_cusum(
|
||||||
changes between a low and high amplitude.
|
changes between a low and high amplitude.
|
||||||
|
|
||||||
:param recording: A ``Recording`` object to annotate.
|
:param recording: A ``Recording`` object to annotate.
|
||||||
:type recording: ``ria_toolkit_oss.data.Recording``
|
:type recording: ``ria_toolkit_oss.datatypes.Recording``
|
||||||
:param label: Label for the detected segments.
|
:param label: Label for the detected segments.
|
||||||
:type label: str
|
:type label: str
|
||||||
:param window_size: The length (in samples) of the moving average window.
|
:param window_size: The length (in samples) of the moving average window.
|
||||||
|
|
@ -38,13 +38,7 @@ def annotate_with_cusum(
|
||||||
:type annotation_type: str
|
:type annotation_type: str
|
||||||
"""
|
"""
|
||||||
|
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Supply it with --sample-rate when using the CLI, or set "
|
|
||||||
"recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||||
|
|
||||||
# Create an object of the time segmenter
|
# Create an object of the time segmenter
|
||||||
|
|
|
||||||
|
|
@ -6,13 +6,12 @@ and occupied bandwidth calculation following ITU-R SM.328 standard.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import warnings
|
|
||||||
from typing import Tuple
|
from typing import Tuple
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from scipy.signal import filtfilt
|
from scipy.signal import filtfilt
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Annotation, Recording
|
from ria_toolkit_oss.datatypes import Annotation, Recording
|
||||||
|
|
||||||
|
|
||||||
def detect_signals_energy(
|
def detect_signals_energy(
|
||||||
|
|
@ -120,17 +119,6 @@ def detect_signals_energy(
|
||||||
if active:
|
if active:
|
||||||
boundaries.append((start, len(smoothed_power) - start))
|
boundaries.append((start, len(smoothed_power) - start))
|
||||||
|
|
||||||
if not boundaries and noise_floor > 0:
|
|
||||||
peak = float(np.max(smoothed_power))
|
|
||||||
dynamic_range = peak / noise_floor
|
|
||||||
if dynamic_range < threshold_factor:
|
|
||||||
warnings.warn(
|
|
||||||
f"detect_signals_energy: no signal boundaries found — dynamic range {dynamic_range:.2f}x is below "
|
|
||||||
f"the threshold factor {threshold_factor}x. The signal may be constant-envelope (e.g. CW or chirp). "
|
|
||||||
"If the entire recording is signal, use 'ria annotate cusum' to segment it as a single region.",
|
|
||||||
stacklevel=2,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Merge boundaries that are closer than min_distance
|
# Merge boundaries that are closer than min_distance
|
||||||
merged_boundaries = []
|
merged_boundaries = []
|
||||||
if boundaries:
|
if boundaries:
|
||||||
|
|
@ -147,13 +135,7 @@ def detect_signals_energy(
|
||||||
merged_boundaries.append((start, length))
|
merged_boundaries.append((start, length))
|
||||||
|
|
||||||
# Create annotations from detected boundaries
|
# Create annotations from detected boundaries
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Supply it with --sample-rate when using the CLI, or set "
|
|
||||||
"recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||||
|
|
||||||
# Validate frequency method
|
# Validate frequency method
|
||||||
|
|
@ -369,12 +351,7 @@ def annotate_with_obw(
|
||||||
>>> annotated = annotate_with_obw(recording, label="signal_obw")
|
>>> annotated = annotate_with_obw(recording, label="signal_obw")
|
||||||
"""
|
"""
|
||||||
signal = recording.data[0]
|
signal = recording.data[0]
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Set recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_freq = recording.metadata.get("center_frequency", 0)
|
center_freq = recording.metadata.get("center_frequency", 0)
|
||||||
|
|
||||||
# Calculate OBW
|
# Calculate OBW
|
||||||
|
|
|
||||||
|
|
@ -49,14 +49,13 @@ allowing splitting of overlapping signals into separate training samples.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import warnings
|
|
||||||
from typing import List, Optional, Tuple
|
from typing import List, Optional, Tuple
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from scipy import ndimage
|
from scipy import ndimage
|
||||||
from scipy import signal as scipy_signal
|
from scipy import signal as scipy_signal
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Annotation, Recording
|
from ria_toolkit_oss.datatypes import Annotation, Recording
|
||||||
|
|
||||||
|
|
||||||
def find_spectral_components(
|
def find_spectral_components(
|
||||||
|
|
@ -402,13 +401,7 @@ def split_recording_annotations(
|
||||||
return recording
|
return recording
|
||||||
|
|
||||||
signal = recording.data[0]
|
signal = recording.data[0]
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Supply it with --sample-rate when using the CLI, or set "
|
|
||||||
"recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_frequency = recording.metadata.get("center_frequency", 0.0)
|
center_frequency = recording.metadata.get("center_frequency", 0.0)
|
||||||
|
|
||||||
# Build new annotation list
|
# Build new annotation list
|
||||||
|
|
@ -432,11 +425,8 @@ def split_recording_annotations(
|
||||||
else:
|
else:
|
||||||
# No components found, keep original
|
# No components found, keep original
|
||||||
new_annotations.append(anno)
|
new_annotations.append(anno)
|
||||||
except Exception as e:
|
except Exception:
|
||||||
warnings.warn(
|
# Split failed for any reason, keep original
|
||||||
f"split_recording_annotations: failed to split annotation at index {i} ({e}); keeping original.",
|
|
||||||
stacklevel=2,
|
|
||||||
)
|
|
||||||
new_annotations.append(anno)
|
new_annotations.append(anno)
|
||||||
else:
|
else:
|
||||||
# Not in split list, keep as-is
|
# Not in split list, keep as-is
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
|
||||||
def qualify_slice_from_annotations(recording: Recording, slice_length: int):
|
def qualify_slice_from_annotations(recording: Recording, slice_length: int):
|
||||||
|
|
@ -24,7 +24,7 @@ def qualify_slice_from_annotations(recording: Recording, slice_length: int):
|
||||||
|
|
||||||
output_recordings = []
|
output_recordings = []
|
||||||
|
|
||||||
for i in range(len(recording.data[0]) // slice_length):
|
for i in range((len(recording.data[0]) // slice_length) - 1):
|
||||||
start_index = slice_length * i
|
start_index = slice_length * i
|
||||||
end_index = slice_length * (i + 1)
|
end_index = slice_length * (i + 1)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from scipy.signal import butter, lfilter
|
from scipy.signal import butter, lfilter
|
||||||
|
|
||||||
from ria_toolkit_oss.data.annotation import Annotation
|
from ria_toolkit_oss.datatypes.annotation import Annotation
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
|
|
||||||
|
|
||||||
def isolate_signal(recording: Recording, annotation: Annotation) -> Recording:
|
def isolate_signal(recording: Recording, annotation: Annotation) -> Recording:
|
||||||
|
|
@ -35,24 +35,17 @@ def isolate_signal(recording: Recording, annotation: Annotation) -> Recording:
|
||||||
|
|
||||||
isolation_bw = anno_bw
|
isolation_bw = anno_bw
|
||||||
|
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Set recording.sample_rate before calling isolate_signal."
|
|
||||||
)
|
|
||||||
|
|
||||||
# frequency shift the center of the box about zero
|
# frequency shift the center of the box about zero
|
||||||
shifted_signal_slice = frequency_shift_iq_samples(
|
shifted_signal_slice = frequency_shift_iq_samples(
|
||||||
iq_samples=signal_slice,
|
iq_samples=signal_slice,
|
||||||
sample_rate=sample_rate,
|
sample_rate=recording.metadata["sample_rate"],
|
||||||
shift_frequency=-1 * anno_base_center_freq,
|
shift_frequency=-1 * anno_base_center_freq,
|
||||||
)
|
)
|
||||||
|
|
||||||
# filter
|
# filter
|
||||||
if isolation_bw < sample_rate - 1:
|
if isolation_bw < recording.metadata["sample_rate"] - 1:
|
||||||
filtered_signal = apply_complex_lowpass_filter(
|
filtered_signal = apply_complex_lowpass_filter(
|
||||||
signal=shifted_signal_slice, cutoff_frequency=isolation_bw, sample_rate=sample_rate
|
signal=shifted_signal_slice, cutoff_frequency=isolation_bw, sample_rate=recording.metadata["sample_rate"]
|
||||||
)
|
)
|
||||||
|
|
||||||
else:
|
else:
|
||||||
|
|
|
||||||
|
|
@ -42,12 +42,11 @@ classification or demodulation stages.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import warnings
|
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Annotation, Recording
|
from ria_toolkit_oss.datatypes import Annotation, Recording
|
||||||
|
|
||||||
|
|
||||||
def _find_ranges(indices, max_gap):
|
def _find_ranges(indices, max_gap):
|
||||||
|
|
@ -217,22 +216,11 @@ def threshold_qualifier(
|
||||||
"""
|
"""
|
||||||
# Extract signal and metadata
|
# Extract signal and metadata
|
||||||
sample_data = recording.data[channel]
|
sample_data = recording.data[channel]
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Supply it with --sample-rate when using the CLI, or set "
|
|
||||||
"recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||||
|
|
||||||
n_samples = len(sample_data)
|
|
||||||
|
|
||||||
if window_size is None:
|
if window_size is None:
|
||||||
window_size = max(64, int(sample_rate * 0.001))
|
window_size = max(64, int(sample_rate * 0.001))
|
||||||
# Cap at 1% of signal length so short recordings aren't over-smoothed into
|
|
||||||
# a flat envelope that collapses the dynamic range below the early-exit guard.
|
|
||||||
window_size = min(window_size, max(64, n_samples // 100))
|
|
||||||
|
|
||||||
# --- 1. SIGNAL CONDITIONING ---
|
# --- 1. SIGNAL CONDITIONING ---
|
||||||
# Convert to power (Magnitude squared)
|
# Convert to power (Magnitude squared)
|
||||||
|
|
@ -249,12 +237,6 @@ def threshold_qualifier(
|
||||||
# Soft early exit: keep a guard for low-contrast noise, but compute it from
|
# Soft early exit: keep a guard for low-contrast noise, but compute it from
|
||||||
# the quieter tail of the envelope so burst-heavy captures are not rejected.
|
# the quieter tail of the envelope so burst-heavy captures are not rejected.
|
||||||
if dynamic_range_ratio < 1.5:
|
if dynamic_range_ratio < 1.5:
|
||||||
warnings.warn(
|
|
||||||
f"threshold_qualifier: dynamic range ratio {dynamic_range_ratio:.2f} is below 1.5 — "
|
|
||||||
"the signal appears to be constant-envelope or pure noise, so no burst boundaries can be found. "
|
|
||||||
"If the entire recording is signal, use 'ria annotate cusum' to segment it as a single region.",
|
|
||||||
stacklevel=2,
|
|
||||||
)
|
|
||||||
return Recording(data=recording.data, metadata=recording.metadata, annotations=recording.annotations)
|
return Recording(data=recording.data, metadata=recording.metadata, annotations=recording.annotations)
|
||||||
|
|
||||||
trigger_val = noise_floor + threshold * (max_power - noise_floor)
|
trigger_val = noise_floor + threshold * (max_power - noise_floor)
|
||||||
|
|
@ -314,7 +296,7 @@ def threshold_qualifier(
|
||||||
# burst energy does not bleed through the long window into adjacent regions,
|
# burst energy does not bleed through the long window into adjacent regions,
|
||||||
# which would inflate macro_residual_max and push the trigger above the
|
# which would inflate macro_residual_max and push the trigger above the
|
||||||
# faint burst's average power.
|
# faint burst's average power.
|
||||||
macro_window_size = min(max(window_size * 16, int(sample_rate * 0.02)), max(window_size * 2, n_samples // 4))
|
macro_window_size = max(window_size * 16, int(sample_rate * 0.02))
|
||||||
macro_kernel = np.ones(macro_window_size, dtype=np.float64) / macro_window_size
|
macro_kernel = np.ones(macro_window_size, dtype=np.float64) / macro_window_size
|
||||||
# Expand each annotated range by half the macro window on both sides so that
|
# Expand each annotated range by half the macro window on both sides so that
|
||||||
# the long convolution cannot "see" the leading/trailing edges of already-
|
# the long convolution cannot "see" the leading/trailing edges of already-
|
||||||
|
|
|
||||||
|
|
@ -1,144 +1,128 @@
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
from typing import Any, Optional
|
from typing import Any, Optional
|
||||||
|
|
||||||
from sigmf import SigMFFile
|
from sigmf import SigMFFile
|
||||||
|
|
||||||
|
|
||||||
class Annotation:
|
class Annotation:
|
||||||
"""Signal annotations are labels or additional information associated with specific data points or segments within
|
"""Signal annotations are labels or additional information associated with specific data points or segments within
|
||||||
a signal. These annotations could be used for tasks like supervised learning, where the goal is to train a model
|
a signal. These annotations could be used for tasks like supervised learning, where the goal is to train a model
|
||||||
to recognize patterns or characteristics in the signal associated with these annotations.
|
to recognize patterns or characteristics in the signal associated with these annotations.
|
||||||
|
|
||||||
Annotations can be used to label interesting points in your recording.
|
Annotations can be used to label interesting points in your recording.
|
||||||
|
|
||||||
:param sample_start: The index of the starting sample of the annotation.
|
:param sample_start: The index of the starting sample of the annotation.
|
||||||
:type sample_start: int
|
:type sample_start: int
|
||||||
:param sample_count: The index of the ending sample of the annotation, inclusive.
|
:param sample_count: The index of the ending sample of the annotation, inclusive.
|
||||||
:type sample_count: int
|
:type sample_count: int
|
||||||
:param freq_lower_edge: The lower frequency of the annotation. Optional; None if not specified in source.
|
:param freq_lower_edge: The lower frequency of the annotation.
|
||||||
:type freq_lower_edge: float, optional
|
:type freq_lower_edge: float
|
||||||
:param freq_upper_edge: The upper frequency of the annotation. Optional; None if not specified in source.
|
:param freq_upper_edge: The upper frequency of the annotation.
|
||||||
:type freq_upper_edge: float, optional
|
:type freq_upper_edge: float
|
||||||
:param label: The label that will be displayed with the bounding box in compatible viewers including IQEngine.
|
:param label: The label that will be displayed with the bounding box in compatible viewers including IQEngine.
|
||||||
Defaults to an emtpy string.
|
Defaults to an emtpy string.
|
||||||
:type label: str, optional
|
:type label: str, optional
|
||||||
:param comment: A human-readable comment. Defaults to an empty string.
|
:param comment: A human-readable comment. Defaults to an empty string.
|
||||||
:type comment: str, optional
|
:type comment: str, optional
|
||||||
:param detail: A dictionary of user defined annotation-specific metadata. Defaults to None.
|
:param detail: A dictionary of user defined annotation-specific metadata. Defaults to None.
|
||||||
:type detail: dict, optional
|
:type detail: dict, optional
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
sample_start: int,
|
sample_start: int,
|
||||||
sample_count: int,
|
sample_count: int,
|
||||||
freq_lower_edge: Optional[float] = None,
|
freq_lower_edge: float,
|
||||||
freq_upper_edge: Optional[float] = None,
|
freq_upper_edge: float,
|
||||||
label: Optional[str] = "",
|
label: Optional[str] = "",
|
||||||
comment: Optional[str] = "",
|
comment: Optional[str] = "",
|
||||||
detail: Optional[dict] = None,
|
detail: Optional[dict] = None,
|
||||||
):
|
):
|
||||||
"""Initialize a new Annotation instance."""
|
"""Initialize a new Annotation instance."""
|
||||||
self.sample_start = int(sample_start)
|
self.sample_start = int(sample_start)
|
||||||
self.sample_count = int(sample_count)
|
self.sample_count = int(sample_count)
|
||||||
self.freq_lower_edge = float(freq_lower_edge) if freq_lower_edge is not None else None
|
self.freq_lower_edge = float(freq_lower_edge)
|
||||||
self.freq_upper_edge = float(freq_upper_edge) if freq_upper_edge is not None else None
|
self.freq_upper_edge = float(freq_upper_edge)
|
||||||
self.label = str(label)
|
self.label = str(label)
|
||||||
self.comment = str(comment)
|
self.comment = str(comment)
|
||||||
|
|
||||||
if detail is None:
|
if detail is None:
|
||||||
self.detail = {}
|
self.detail = {}
|
||||||
elif not _is_jsonable(detail):
|
elif not _is_jsonable(detail):
|
||||||
raise ValueError(f"Detail object is not json serializable: {detail}")
|
raise ValueError(f"Detail object is not json serializable: {detail}")
|
||||||
else:
|
else:
|
||||||
self.detail = detail
|
self.detail = detail
|
||||||
|
|
||||||
def is_valid(self) -> bool:
|
def is_valid(self) -> bool:
|
||||||
"""
|
"""
|
||||||
Verify ``sample_count > 0`` and the ``freq_lower_edge < freq_upper_edge``.
|
Check that the annotation sample count is > 0 and the freq_lower_edge<freq_upper_edge.
|
||||||
|
|
||||||
:returns: True if valid, False if not.
|
:returns: True if valid, False if not.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.freq_lower_edge is None or self.freq_upper_edge is None:
|
return self.sample_count > 0 and self.freq_lower_edge < self.freq_upper_edge
|
||||||
return self.sample_count > 0
|
|
||||||
return self.sample_count > 0 and self.freq_lower_edge < self.freq_upper_edge
|
def overlap(self, other):
|
||||||
|
"""
|
||||||
def overlap(self, other):
|
Quantify how much the bounding box in this annotation overlaps with another annotation.
|
||||||
"""
|
|
||||||
Quantify how much the bounding box in this annotation overlaps with another annotation.
|
:param other: The other annotation.
|
||||||
|
:type other: Annotation
|
||||||
:param other: The other annotation.
|
|
||||||
:type other: Annotation
|
:returns: The area of the overlap in samples*frequency, or 0 if they do not overlap."""
|
||||||
|
|
||||||
:returns: The area of the overlap in samples*frequency, or 0 if they do not overlap."""
|
sample_overlap_start = max(self.sample_start, other.sample_start)
|
||||||
|
sample_overlap_end = min(self.sample_start + self.sample_count, other.sample_start + other.sample_count)
|
||||||
if (
|
|
||||||
self.freq_lower_edge is None
|
freq_overlap_start = max(self.freq_lower_edge, other.freq_lower_edge)
|
||||||
or self.freq_upper_edge is None
|
freq_overlap_end = min(self.freq_upper_edge, other.freq_upper_edge)
|
||||||
or other.freq_lower_edge is None
|
|
||||||
or other.freq_upper_edge is None
|
if freq_overlap_start >= freq_overlap_end or sample_overlap_start >= sample_overlap_end:
|
||||||
):
|
return 0
|
||||||
return 0
|
else:
|
||||||
|
return (sample_overlap_end - sample_overlap_start) * (freq_overlap_end - freq_overlap_start)
|
||||||
sample_overlap_start = max(self.sample_start, other.sample_start)
|
|
||||||
sample_overlap_end = min(self.sample_start + self.sample_count, other.sample_start + other.sample_count)
|
def area(self):
|
||||||
|
"""
|
||||||
freq_overlap_start = max(self.freq_lower_edge, other.freq_lower_edge)
|
The 'area' of the bounding box, samples*frequency.
|
||||||
freq_overlap_end = min(self.freq_upper_edge, other.freq_upper_edge)
|
Useful to quantify annotation size.
|
||||||
|
|
||||||
if freq_overlap_start >= freq_overlap_end or sample_overlap_start >= sample_overlap_end:
|
:returns: sample length multiplied by bandwidth."""
|
||||||
return 0
|
|
||||||
else:
|
return self.sample_count * (self.freq_upper_edge - self.freq_lower_edge)
|
||||||
return (sample_overlap_end - sample_overlap_start) * (freq_overlap_end - freq_overlap_start)
|
|
||||||
|
def __eq__(self, other: Annotation) -> bool:
|
||||||
def area(self):
|
return self.__dict__ == other.__dict__
|
||||||
"""
|
|
||||||
The 'area' of the bounding box, samples*frequency.
|
def to_sigmf_format(self):
|
||||||
Useful to quantify annotation size.
|
"""
|
||||||
|
Returns a JSON dictionary representing this annotation formatted to be saved in a .sigmf-meta file.
|
||||||
:returns: sample length multiplied by bandwidth."""
|
"""
|
||||||
|
|
||||||
if self.freq_lower_edge is None or self.freq_upper_edge is None:
|
annotation_dict = {SigMFFile.START_INDEX_KEY: self.sample_start, SigMFFile.LENGTH_INDEX_KEY: self.sample_count}
|
||||||
return 0
|
|
||||||
return self.sample_count * (self.freq_upper_edge - self.freq_lower_edge)
|
annotation_dict["metadata"] = {
|
||||||
|
SigMFFile.LABEL_KEY: self.label,
|
||||||
def __eq__(self, other: Annotation) -> bool:
|
SigMFFile.COMMENT_KEY: self.comment,
|
||||||
return self.__dict__ == other.__dict__
|
SigMFFile.FHI_KEY: self.freq_upper_edge,
|
||||||
|
SigMFFile.FLO_KEY: self.freq_lower_edge,
|
||||||
def to_sigmf_format(self) -> dict:
|
"ria:detail": self.detail,
|
||||||
"""
|
}
|
||||||
Returns a JSON dictionary representation, formatted for saving in a ``.sigmf-meta`` file.
|
|
||||||
"""
|
if _is_jsonable(annotation_dict):
|
||||||
|
return annotation_dict
|
||||||
annotation_dict = {SigMFFile.START_INDEX_KEY: self.sample_start, SigMFFile.LENGTH_INDEX_KEY: self.sample_count}
|
else:
|
||||||
|
raise ValueError("Annotation dictionary was not json serializable.")
|
||||||
metadata = {
|
|
||||||
SigMFFile.LABEL_KEY: self.label,
|
|
||||||
SigMFFile.COMMENT_KEY: self.comment,
|
def _is_jsonable(x: Any) -> bool:
|
||||||
"ria:detail": self.detail,
|
"""
|
||||||
}
|
:return: True if x is JSON serializable, False otherwise.
|
||||||
if self.freq_upper_edge is not None:
|
"""
|
||||||
metadata[SigMFFile.FHI_KEY] = self.freq_upper_edge
|
try:
|
||||||
if self.freq_lower_edge is not None:
|
json.dumps(x)
|
||||||
metadata[SigMFFile.FLO_KEY] = self.freq_lower_edge
|
return True
|
||||||
annotation_dict["metadata"] = metadata
|
except (TypeError, OverflowError):
|
||||||
|
return False
|
||||||
if _is_jsonable(annotation_dict):
|
|
||||||
return annotation_dict
|
|
||||||
else:
|
|
||||||
raise ValueError("Annotation dictionary was not json serializable.")
|
|
||||||
|
|
||||||
|
|
||||||
def _is_jsonable(x: Any) -> bool:
|
|
||||||
"""
|
|
||||||
:return: True if ``x`` is JSON serializable, False otherwise.
|
|
||||||
:rtype: bool
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
json.dumps(x)
|
|
||||||
return True
|
|
||||||
except (TypeError, OverflowError):
|
|
||||||
return False
|
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load Diff
8
src/ria_toolkit_oss/datatypes/__init__.py
Normal file
8
src/ria_toolkit_oss/datatypes/__init__.py
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
"""
|
||||||
|
The datatypes package contains abstract data types tailored for radio machine learning.
|
||||||
|
"""
|
||||||
|
|
||||||
|
__all__ = ["Annotation", "Recording"]
|
||||||
|
|
||||||
|
from .annotation import Annotation
|
||||||
|
from .recording import Recording
|
||||||
129
src/ria_toolkit_oss/datatypes/annotation.py
Normal file
129
src/ria_toolkit_oss/datatypes/annotation.py
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
from sigmf import SigMFFile
|
||||||
|
|
||||||
|
|
||||||
|
class Annotation:
|
||||||
|
"""Signal annotations are labels or additional information associated with specific data points or segments within
|
||||||
|
a signal. These annotations could be used for tasks like supervised learning, where the goal is to train a model
|
||||||
|
to recognize patterns or characteristics in the signal associated with these annotations.
|
||||||
|
|
||||||
|
Annotations can be used to label interesting points in your recording.
|
||||||
|
|
||||||
|
:param sample_start: The index of the starting sample of the annotation.
|
||||||
|
:type sample_start: int
|
||||||
|
:param sample_count: The index of the ending sample of the annotation, inclusive.
|
||||||
|
:type sample_count: int
|
||||||
|
:param freq_lower_edge: The lower frequency of the annotation.
|
||||||
|
:type freq_lower_edge: float
|
||||||
|
:param freq_upper_edge: The upper frequency of the annotation.
|
||||||
|
:type freq_upper_edge: float
|
||||||
|
:param label: The label that will be displayed with the bounding box in compatible viewers including IQEngine.
|
||||||
|
Defaults to an emtpy string.
|
||||||
|
:type label: str, optional
|
||||||
|
:param comment: A human-readable comment. Defaults to an empty string.
|
||||||
|
:type comment: str, optional
|
||||||
|
:param detail: A dictionary of user defined annotation-specific metadata. Defaults to None.
|
||||||
|
:type detail: dict, optional
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
sample_start: int,
|
||||||
|
sample_count: int,
|
||||||
|
freq_lower_edge: float,
|
||||||
|
freq_upper_edge: float,
|
||||||
|
label: Optional[str] = "",
|
||||||
|
comment: Optional[str] = "",
|
||||||
|
detail: Optional[dict] = None,
|
||||||
|
):
|
||||||
|
"""Initialize a new Annotation instance."""
|
||||||
|
self.sample_start = int(sample_start)
|
||||||
|
self.sample_count = int(sample_count)
|
||||||
|
self.freq_lower_edge = float(freq_lower_edge)
|
||||||
|
self.freq_upper_edge = float(freq_upper_edge)
|
||||||
|
self.label = str(label)
|
||||||
|
self.comment = str(comment)
|
||||||
|
|
||||||
|
if detail is None:
|
||||||
|
self.detail = {}
|
||||||
|
elif not _is_jsonable(detail):
|
||||||
|
raise ValueError(f"Detail object is not json serializable: {detail}")
|
||||||
|
else:
|
||||||
|
self.detail = detail
|
||||||
|
|
||||||
|
def is_valid(self) -> bool:
|
||||||
|
"""
|
||||||
|
Verify ``sample_count > 0`` and the ``freq_lower_edge < freq_upper_edge``.
|
||||||
|
|
||||||
|
:returns: True if valid, False if not.
|
||||||
|
"""
|
||||||
|
|
||||||
|
return self.sample_count > 0 and self.freq_lower_edge < self.freq_upper_edge
|
||||||
|
|
||||||
|
def overlap(self, other):
|
||||||
|
"""
|
||||||
|
Quantify how much the bounding box in this annotation overlaps with another annotation.
|
||||||
|
|
||||||
|
:param other: The other annotation.
|
||||||
|
:type other: Annotation
|
||||||
|
|
||||||
|
:returns: The area of the overlap in samples*frequency, or 0 if they do not overlap."""
|
||||||
|
|
||||||
|
sample_overlap_start = max(self.sample_start, other.sample_start)
|
||||||
|
sample_overlap_end = min(self.sample_start + self.sample_count, other.sample_start + other.sample_count)
|
||||||
|
|
||||||
|
freq_overlap_start = max(self.freq_lower_edge, other.freq_lower_edge)
|
||||||
|
freq_overlap_end = min(self.freq_upper_edge, other.freq_upper_edge)
|
||||||
|
|
||||||
|
if freq_overlap_start >= freq_overlap_end or sample_overlap_start >= sample_overlap_end:
|
||||||
|
return 0
|
||||||
|
else:
|
||||||
|
return (sample_overlap_end - sample_overlap_start) * (freq_overlap_end - freq_overlap_start)
|
||||||
|
|
||||||
|
def area(self):
|
||||||
|
"""
|
||||||
|
The 'area' of the bounding box, samples*frequency.
|
||||||
|
Useful to quantify annotation size.
|
||||||
|
|
||||||
|
:returns: sample length multiplied by bandwidth."""
|
||||||
|
|
||||||
|
return self.sample_count * (self.freq_upper_edge - self.freq_lower_edge)
|
||||||
|
|
||||||
|
def __eq__(self, other: Annotation) -> bool:
|
||||||
|
return self.__dict__ == other.__dict__
|
||||||
|
|
||||||
|
def to_sigmf_format(self) -> dict:
|
||||||
|
"""
|
||||||
|
Returns a JSON dictionary representation, formatted for saving in a ``.sigmf-meta`` file.
|
||||||
|
"""
|
||||||
|
|
||||||
|
annotation_dict = {SigMFFile.START_INDEX_KEY: self.sample_start, SigMFFile.LENGTH_INDEX_KEY: self.sample_count}
|
||||||
|
|
||||||
|
annotation_dict["metadata"] = {
|
||||||
|
SigMFFile.LABEL_KEY: self.label,
|
||||||
|
SigMFFile.COMMENT_KEY: self.comment,
|
||||||
|
SigMFFile.FHI_KEY: self.freq_upper_edge,
|
||||||
|
SigMFFile.FLO_KEY: self.freq_lower_edge,
|
||||||
|
"ria:detail": self.detail,
|
||||||
|
}
|
||||||
|
|
||||||
|
if _is_jsonable(annotation_dict):
|
||||||
|
return annotation_dict
|
||||||
|
else:
|
||||||
|
raise ValueError("Annotation dictionary was not json serializable.")
|
||||||
|
|
||||||
|
|
||||||
|
def _is_jsonable(x: Any) -> bool:
|
||||||
|
"""
|
||||||
|
:return: True if ``x`` is JSON serializable, False otherwise.
|
||||||
|
:rtype: bool
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
json.dumps(x)
|
||||||
|
return True
|
||||||
|
except (TypeError, OverflowError):
|
||||||
|
return False
|
||||||
|
|
@ -7,8 +7,8 @@ from typing import Any, Optional
|
||||||
|
|
||||||
from packaging.version import Version
|
from packaging.version import Version
|
||||||
|
|
||||||
from ria_toolkit_oss.data.datasets.license.dataset_license import DatasetLicense
|
from ria_toolkit_oss.datatypes.datasets.license.dataset_license import DatasetLicense
|
||||||
from ria_toolkit_oss.data.datasets.radio_dataset import RadioDataset
|
from ria_toolkit_oss.datatypes.datasets.radio_dataset import RadioDataset
|
||||||
from ria_toolkit_oss.utils.abstract_attribute import abstract_attribute
|
from ria_toolkit_oss.utils.abstract_attribute import abstract_attribute
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -7,11 +7,11 @@ from typing import Optional
|
||||||
import h5py
|
import h5py
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data.datasets.h5helpers import (
|
from ria_toolkit_oss.datatypes.datasets.h5helpers import (
|
||||||
append_entry_inplace,
|
append_entry_inplace,
|
||||||
copy_dataset_entry_by_index,
|
copy_dataset_entry_by_index,
|
||||||
)
|
)
|
||||||
from ria_toolkit_oss.data.datasets.radio_dataset import RadioDataset
|
from ria_toolkit_oss.datatypes.datasets.radio_dataset import RadioDataset
|
||||||
|
|
||||||
|
|
||||||
class IQDataset(RadioDataset, ABC):
|
class IQDataset(RadioDataset, ABC):
|
||||||
|
|
@ -19,7 +19,7 @@ class IQDataset(RadioDataset, ABC):
|
||||||
radiofrequency (RF) signals represented as In-phase (I) and Quadrature (Q) samples.
|
radiofrequency (RF) signals represented as In-phase (I) and Quadrature (Q) samples.
|
||||||
|
|
||||||
For machine learning tasks that involve processing spectrograms, please use
|
For machine learning tasks that involve processing spectrograms, please use
|
||||||
ria_toolkit_oss.data.datasets.SpectDataset instead.
|
ria_toolkit_oss.datatypes.datasets.SpectDataset instead.
|
||||||
|
|
||||||
This is an abstract interface defining common properties and behaviour of IQDatasets. Therefore, this class
|
This is an abstract interface defining common properties and behaviour of IQDatasets. Therefore, this class
|
||||||
should not be instantiated directly. Instead, it is subclassed to define custom interfaces for specific machine
|
should not be instantiated directly. Instead, it is subclassed to define custom interfaces for specific machine
|
||||||
|
|
@ -12,7 +12,7 @@ import numpy as np
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
from numpy.typing import ArrayLike
|
from numpy.typing import ArrayLike
|
||||||
|
|
||||||
from ria_toolkit_oss.data.datasets.h5helpers import (
|
from ria_toolkit_oss.datatypes.datasets.h5helpers import (
|
||||||
append_entry_inplace,
|
append_entry_inplace,
|
||||||
copy_file,
|
copy_file,
|
||||||
copy_over_example,
|
copy_over_example,
|
||||||
|
|
@ -29,7 +29,7 @@ class RadioDataset(ABC):
|
||||||
|
|
||||||
This is an abstract interface defining common properties and behavior of radio datasets. Therefore, this class
|
This is an abstract interface defining common properties and behavior of radio datasets. Therefore, this class
|
||||||
should not be instantiated directly. Instead, it should be subclassed to define specific interfaces for different
|
should not be instantiated directly. Instead, it should be subclassed to define specific interfaces for different
|
||||||
types of radio datasets. For example, see ria_toolkit_oss.data.datasets.IQDataset, which is a radio dataset
|
types of radio datasets. For example, see ria_toolkit_oss.datatypes.datasets.IQDataset, which is a radio dataset
|
||||||
subclass tailored for tasks involving the processing of radio signals represented as IQ (In-phase and Quadrature)
|
subclass tailored for tasks involving the processing of radio signals represented as IQ (In-phase and Quadrature)
|
||||||
samples.
|
samples.
|
||||||
|
|
||||||
|
|
@ -3,7 +3,7 @@ from __future__ import annotations
|
||||||
import os
|
import os
|
||||||
from abc import ABC
|
from abc import ABC
|
||||||
|
|
||||||
from ria_toolkit_oss.data.datasets.radio_dataset import RadioDataset
|
from ria_toolkit_oss.datatypes.datasets.radio_dataset import RadioDataset
|
||||||
|
|
||||||
|
|
||||||
class SpectDataset(RadioDataset, ABC):
|
class SpectDataset(RadioDataset, ABC):
|
||||||
|
|
@ -13,7 +13,7 @@ class SpectDataset(RadioDataset, ABC):
|
||||||
radio signal spectrograms.
|
radio signal spectrograms.
|
||||||
|
|
||||||
For machine learning tasks that involve processing on IQ samples, please use
|
For machine learning tasks that involve processing on IQ samples, please use
|
||||||
ria_toolkit_oss.data.datasets.IQDataset instead.
|
ria_toolkit_oss.datatypes.datasets.IQDataset instead.
|
||||||
|
|
||||||
This is an abstract interface defining common properties and behaviour of IQDatasets. Therefore, this class
|
This is an abstract interface defining common properties and behaviour of IQDatasets. Therefore, this class
|
||||||
should not be instantiated directly. Instead, it is subclassed to define custom interfaces for specific machine
|
should not be instantiated directly. Instead, it is subclassed to define custom interfaces for specific machine
|
||||||
|
|
@ -6,8 +6,11 @@ from typing import Optional
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from numpy.random import Generator
|
from numpy.random import Generator
|
||||||
|
|
||||||
from ria_toolkit_oss.data.datasets import RadioDataset
|
from ria_toolkit_oss.datatypes.datasets import RadioDataset
|
||||||
from ria_toolkit_oss.data.datasets.h5helpers import copy_over_example, make_empty_clone
|
from ria_toolkit_oss.datatypes.datasets.h5helpers import (
|
||||||
|
copy_over_example,
|
||||||
|
make_empty_clone,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def split(dataset: RadioDataset, lengths: list[int | float]) -> list[RadioDataset]:
|
def split(dataset: RadioDataset, lengths: list[int | float]) -> list[RadioDataset]:
|
||||||
|
|
@ -28,7 +31,7 @@ def split(dataset: RadioDataset, lengths: list[int | float]) -> list[RadioDatase
|
||||||
cases.
|
cases.
|
||||||
|
|
||||||
This function is deterministic, meaning it will always produce the same split. For a random split, see
|
This function is deterministic, meaning it will always produce the same split. For a random split, see
|
||||||
ria_toolkit_oss.data.datasets.random_split.
|
ria_toolkit_oss.datatypes.datasets.random_split.
|
||||||
|
|
||||||
:param dataset: Dataset to be split.
|
:param dataset: Dataset to be split.
|
||||||
:type dataset: RadioDataset
|
:type dataset: RadioDataset
|
||||||
|
|
@ -47,7 +50,7 @@ def split(dataset: RadioDataset, lengths: list[int | float]) -> list[RadioDatase
|
||||||
>>> import string
|
>>> import string
|
||||||
>>> import numpy as np
|
>>> import numpy as np
|
||||||
>>> import pandas as pd
|
>>> import pandas as pd
|
||||||
>>> from ria_toolkit_oss.data.datasets import split
|
>>> from ria_toolkit_oss.datatypes.datasets import split
|
||||||
|
|
||||||
First, let's generate some random data:
|
First, let's generate some random data:
|
||||||
|
|
||||||
|
|
@ -123,7 +126,7 @@ def random_split(
|
||||||
training and test datasets.
|
training and test datasets.
|
||||||
|
|
||||||
This restriction makes it unlikely that a random split will produce datasets with the exact lengths specified.
|
This restriction makes it unlikely that a random split will produce datasets with the exact lengths specified.
|
||||||
If it is important to ensure the closest possible split, consider using ria_toolkit_oss.data.datasets.split
|
If it is important to ensure the closest possible split, consider using ria_toolkit_oss.datatypes.datasets.split
|
||||||
instead.
|
instead.
|
||||||
|
|
||||||
:param dataset: Dataset to be split.
|
:param dataset: Dataset to be split.
|
||||||
|
|
@ -141,7 +144,7 @@ def random_split(
|
||||||
:rtype: list of RadioDataset
|
:rtype: list of RadioDataset
|
||||||
|
|
||||||
See Also:
|
See Also:
|
||||||
ria_toolkit_oss.data.datasets.split: Usage is the same as for ``random_split()``.
|
ria_toolkit_oss.datatypes.datasets.split: Usage is the same as for ``random_split()``.
|
||||||
"""
|
"""
|
||||||
if not isinstance(dataset, RadioDataset):
|
if not isinstance(dataset, RadioDataset):
|
||||||
raise ValueError(f"'dataset' must be RadioDataset or one of its subclasses, got {type(dataset)}.")
|
raise ValueError(f"'dataset' must be RadioDataset or one of its subclasses, got {type(dataset)}.")
|
||||||
855
src/ria_toolkit_oss/datatypes/recording.py
Normal file
855
src/ria_toolkit_oss/datatypes/recording.py
Normal file
|
|
@ -0,0 +1,855 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import copy
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
import warnings
|
||||||
|
from typing import Any, Iterator, Optional
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
from numpy.typing import ArrayLike
|
||||||
|
|
||||||
|
from ria_toolkit_oss.datatypes.annotation import Annotation
|
||||||
|
|
||||||
|
PROTECTED_KEYS = ["rec_id", "timestamp"]
|
||||||
|
|
||||||
|
|
||||||
|
class Recording:
|
||||||
|
"""Tape of complex IQ (in-phase and quadrature) samples with associated metadata and annotations.
|
||||||
|
|
||||||
|
Recording data is a complex array of shape C x N, where C is the number of channels
|
||||||
|
and N is the number of samples in each channel.
|
||||||
|
|
||||||
|
Metadata is stored in a dictionary of key value pairs,
|
||||||
|
to include information such as sample_rate and center_frequency.
|
||||||
|
|
||||||
|
Annotations are a list of :class:`~ria_toolkit_oss.datatypes.Annotation`,
|
||||||
|
defining bounding boxes in time and frequency with labels and metadata.
|
||||||
|
|
||||||
|
Here, signal data is represented as a NumPy array. This class is then extended in the RIA Backends to provide
|
||||||
|
support for different data structures, such as Tensors.
|
||||||
|
|
||||||
|
Recordings are long-form tapes can be obtained either from a software-defined radio (SDR) or generated
|
||||||
|
synthetically. Then, machine learning datasets are curated from collection of recordings by segmenting these
|
||||||
|
longer-form tapes into shorter units called slices.
|
||||||
|
|
||||||
|
All recordings are assigned a unique 64-character recording ID, ``rec_id``. If this field is missing from the
|
||||||
|
provided metadata, a new ID will be generated upon object instantiation.
|
||||||
|
|
||||||
|
:param data: Signal data as a tape IQ samples, either C x N complex, where C is the number of
|
||||||
|
channels and N is number of samples in the signal. If data is a one-dimensional array of complex samples with
|
||||||
|
length N, it will be reshaped to a two-dimensional array with dimensions 1 x N.
|
||||||
|
:type data: array_like
|
||||||
|
|
||||||
|
:param metadata: Additional information associated with the recording.
|
||||||
|
:type metadata: dict, optional
|
||||||
|
:param annotations: A collection of :class:`~ria_toolkit_oss.datatypes.Annotation` objects defining bounding boxes.
|
||||||
|
:type annotations: list of Annotations, optional
|
||||||
|
|
||||||
|
:param dtype: Explicitly specify the data-type of the complex samples. Must be a complex NumPy type, such as
|
||||||
|
``np.complex64`` or ``np.complex128``. Default is None, in which case the type is determined implicitly. If
|
||||||
|
``data`` is a NumPy array, the Recording will use the dtype of ``data`` directly without any conversion.
|
||||||
|
:type dtype: numpy dtype object, optional
|
||||||
|
:param timestamp: The timestamp when the recording data was generated. If provided, it should be a float or integer
|
||||||
|
representing the time in seconds since epoch (e.g., ``time.time()``). Only used if the `timestamp` field is not
|
||||||
|
present in the provided metadata.
|
||||||
|
:type dtype: float or int, optional
|
||||||
|
|
||||||
|
:raises ValueError: If data is not complex 1xN or CxN.
|
||||||
|
:raises ValueError: If metadata is not a python dict.
|
||||||
|
:raises ValueError: If metadata is not json serializable.
|
||||||
|
:raises ValueError: If annotations is not a list of valid annotation objects.
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from ria_toolkit_oss.datatypes import Recording, Annotation
|
||||||
|
|
||||||
|
>>> # Create an array of complex samples, just 1s in this case.
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
|
||||||
|
>>> # Create a dictionary of relevant metadata.
|
||||||
|
>>> sample_rate = 1e6
|
||||||
|
>>> center_frequency = 2.44e9
|
||||||
|
>>> metadata = {
|
||||||
|
... "sample_rate": sample_rate,
|
||||||
|
... "center_frequency": center_frequency,
|
||||||
|
... "author": "me",
|
||||||
|
... }
|
||||||
|
|
||||||
|
>>> # Create an annotation for the annotations list.
|
||||||
|
>>> annotations = [
|
||||||
|
... Annotation(
|
||||||
|
... sample_start=0,
|
||||||
|
... sample_count=1000,
|
||||||
|
... freq_lower_edge=center_frequency - (sample_rate / 2),
|
||||||
|
... freq_upper_edge=center_frequency + (sample_rate / 2),
|
||||||
|
... label="example",
|
||||||
|
... )
|
||||||
|
... ]
|
||||||
|
|
||||||
|
>>> # Store samples, metadata, and annotations together in a convenient object.
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata, annotations=annotations)
|
||||||
|
>>> print(recording.metadata)
|
||||||
|
{'sample_rate': 1000000.0, 'center_frequency': 2440000000.0, 'author': 'me'}
|
||||||
|
>>> print(recording.annotations[0].label)
|
||||||
|
'example'
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__( # noqa C901
|
||||||
|
self,
|
||||||
|
data: ArrayLike | list[list],
|
||||||
|
metadata: Optional[dict[str, any]] = None,
|
||||||
|
dtype: Optional[np.dtype] = None,
|
||||||
|
timestamp: Optional[float | int] = None,
|
||||||
|
annotations: Optional[list[Annotation]] = None,
|
||||||
|
):
|
||||||
|
|
||||||
|
data_arr = np.asarray(data)
|
||||||
|
|
||||||
|
if np.iscomplexobj(data_arr):
|
||||||
|
# Expect C x N
|
||||||
|
if data_arr.ndim == 1:
|
||||||
|
self._data = np.expand_dims(data_arr, axis=0) # N -> 1 x N
|
||||||
|
elif data_arr.ndim == 2:
|
||||||
|
self._data = data_arr
|
||||||
|
else:
|
||||||
|
raise ValueError("Complex data must be C x N.")
|
||||||
|
|
||||||
|
else:
|
||||||
|
raise ValueError("Input data must be complex.")
|
||||||
|
|
||||||
|
if dtype is not None:
|
||||||
|
self._data = self._data.astype(dtype)
|
||||||
|
|
||||||
|
assert np.iscomplexobj(self._data)
|
||||||
|
|
||||||
|
if metadata is None:
|
||||||
|
self._metadata = {}
|
||||||
|
elif isinstance(metadata, dict):
|
||||||
|
self._metadata = metadata
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Metadata must be a python dict, but was {type(metadata)}.")
|
||||||
|
|
||||||
|
if not _is_jsonable(metadata):
|
||||||
|
raise ValueError("Value must be JSON serializable.")
|
||||||
|
|
||||||
|
if "timestamp" not in self.metadata:
|
||||||
|
if timestamp is not None:
|
||||||
|
if not isinstance(timestamp, (int, float)):
|
||||||
|
raise ValueError(f"timestamp must be int or float, not {type(timestamp)}")
|
||||||
|
self._metadata["timestamp"] = timestamp
|
||||||
|
else:
|
||||||
|
self._metadata["timestamp"] = time.time()
|
||||||
|
else:
|
||||||
|
if not isinstance(self._metadata["timestamp"], (int, float)):
|
||||||
|
raise ValueError(f"timestamp must be int or float, not {type(self._metadata['timestamp'])}")
|
||||||
|
|
||||||
|
if "rec_id" not in self.metadata:
|
||||||
|
self._metadata["rec_id"] = generate_recording_id(data=self.data, timestamp=self._metadata["timestamp"])
|
||||||
|
|
||||||
|
if annotations is None:
|
||||||
|
self._annotations = []
|
||||||
|
elif isinstance(annotations, list):
|
||||||
|
self._annotations = annotations
|
||||||
|
else:
|
||||||
|
raise ValueError("Annotations must be a list or None.")
|
||||||
|
|
||||||
|
if not all(isinstance(annotation, Annotation) for annotation in self._annotations):
|
||||||
|
raise ValueError("All elements in self._annotations must be of type Annotation.")
|
||||||
|
|
||||||
|
self._index = 0
|
||||||
|
|
||||||
|
@property
|
||||||
|
def data(self) -> np.ndarray:
|
||||||
|
"""
|
||||||
|
:return: Recording data, as a complex array.
|
||||||
|
:type: np.ndarray
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
For recordings with more than 1,024 samples, this property returns a read-only view of the data.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
To access specific samples, consider indexing the object directly with ``rec[c, n]``.
|
||||||
|
"""
|
||||||
|
if self._data.size > 1024:
|
||||||
|
# Returning a read-only view prevents mutation at a distance while maintaining performance.
|
||||||
|
v = self._data.view()
|
||||||
|
v.setflags(write=False)
|
||||||
|
return v
|
||||||
|
else:
|
||||||
|
return self._data.copy()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def metadata(self) -> dict:
|
||||||
|
"""
|
||||||
|
:return: Dictionary of recording metadata.
|
||||||
|
:type: dict
|
||||||
|
"""
|
||||||
|
return self._metadata.copy()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def annotations(self) -> list[Annotation]:
|
||||||
|
"""
|
||||||
|
:return: List of recording annotations
|
||||||
|
:type: list of Annotation objects
|
||||||
|
"""
|
||||||
|
return self._annotations.copy()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def shape(self) -> tuple[int]:
|
||||||
|
"""
|
||||||
|
:return: The shape of the data array.
|
||||||
|
:type: tuple of ints
|
||||||
|
"""
|
||||||
|
return np.shape(self.data)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def n_chan(self) -> int:
|
||||||
|
"""
|
||||||
|
:return: The number of channels in the recording.
|
||||||
|
:type: int
|
||||||
|
"""
|
||||||
|
return self.shape[0]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def rec_id(self) -> str:
|
||||||
|
"""
|
||||||
|
:return: Recording ID.
|
||||||
|
:type: str
|
||||||
|
"""
|
||||||
|
return self.metadata["rec_id"]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def dtype(self) -> str:
|
||||||
|
"""
|
||||||
|
:return: Data-type of the data array's elements.
|
||||||
|
:type: numpy dtype object
|
||||||
|
"""
|
||||||
|
return self.data.dtype
|
||||||
|
|
||||||
|
@property
|
||||||
|
def timestamp(self) -> float | int:
|
||||||
|
"""
|
||||||
|
:return: Recording timestamp (time in seconds since epoch).
|
||||||
|
:type: float or int
|
||||||
|
"""
|
||||||
|
return self.metadata["timestamp"]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def sample_rate(self) -> float | None:
|
||||||
|
"""
|
||||||
|
:return: Sample rate of the recording, or None is 'sample_rate' is not in metadata.
|
||||||
|
:type: str
|
||||||
|
"""
|
||||||
|
return self.metadata.get("sample_rate")
|
||||||
|
|
||||||
|
@sample_rate.setter
|
||||||
|
def sample_rate(self, sample_rate: float | int) -> None:
|
||||||
|
"""Set the sample rate of the recording.
|
||||||
|
|
||||||
|
:param sample_rate: The sample rate of the recording.
|
||||||
|
:type sample_rate: float or int
|
||||||
|
|
||||||
|
:return: None
|
||||||
|
"""
|
||||||
|
self.add_to_metadata(key="sample_rate", value=sample_rate)
|
||||||
|
|
||||||
|
def astype(self, dtype: np.dtype) -> Recording:
|
||||||
|
"""Copy of the recording, data cast to a specified type.
|
||||||
|
|
||||||
|
.. todo: This method is not yet implemented.
|
||||||
|
|
||||||
|
:param dtype: Data-type to which the array is cast. Must be a complex scalar type, such as ``np.complex64`` or
|
||||||
|
``np.complex128``.
|
||||||
|
:type dtype: NumPy data type, optional
|
||||||
|
|
||||||
|
.. note: Casting to a data type with less precision can risk losing data by truncating or rounding values,
|
||||||
|
potentially resulting in a loss of accuracy and significant information.
|
||||||
|
|
||||||
|
:return: A new recording with the same metadata and data, with dtype.
|
||||||
|
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
.. todo::
|
||||||
|
|
||||||
|
Usage examples coming soon!
|
||||||
|
|
||||||
|
"""
|
||||||
|
# Rather than check for a valid datatype, let's cast and check the result. This makes it easier to provide
|
||||||
|
# cross-platform support where the types are aliased across platforms.
|
||||||
|
with warnings.catch_warnings():
|
||||||
|
warnings.simplefilter("ignore") # Casting may generate user warnings. E.g., complex -> real
|
||||||
|
data = self.data.astype(dtype)
|
||||||
|
|
||||||
|
if np.iscomplexobj(data):
|
||||||
|
return Recording(data=data, metadata=self.metadata, annotations=self.annotations)
|
||||||
|
else:
|
||||||
|
raise ValueError("dtype must be a complex number scalar type.")
|
||||||
|
|
||||||
|
def add_to_metadata(self, key: str, value: Any) -> None:
|
||||||
|
"""Add a new key-value pair to the recording metadata.
|
||||||
|
|
||||||
|
:param key: New metadata key, must be snake_case.
|
||||||
|
:type key: str
|
||||||
|
:param value: Corresponding metadata value.
|
||||||
|
:type value: any
|
||||||
|
|
||||||
|
:raises ValueError: If key is already in metadata or if key is not a valid metadata key.
|
||||||
|
:raises ValueError: If value is not JSON serializable.
|
||||||
|
|
||||||
|
:return: None.
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and add metadata:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from ria_toolkit_oss.datatypes import Recording
|
||||||
|
>>>
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
>>> metadata = {
|
||||||
|
>>> "sample_rate": 1e6,
|
||||||
|
>>> "center_frequency": 2.44e9,
|
||||||
|
>>> }
|
||||||
|
>>>
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> print(recording.metadata)
|
||||||
|
{'sample_rate': 1000000.0,
|
||||||
|
'center_frequency': 2440000000.0,
|
||||||
|
'timestamp': 17369...,
|
||||||
|
'rec_id': 'fda0f41...'}
|
||||||
|
>>>
|
||||||
|
>>> recording.add_to_metadata(key="author", value="me")
|
||||||
|
>>> print(recording.metadata)
|
||||||
|
{'sample_rate': 1000000.0,
|
||||||
|
'center_frequency': 2440000000.0,
|
||||||
|
'author': 'me',
|
||||||
|
'timestamp': 17369...,
|
||||||
|
'rec_id': 'fda0f41...'}
|
||||||
|
"""
|
||||||
|
if key in self.metadata:
|
||||||
|
raise ValueError(
|
||||||
|
f"Key {key} already in metadata. Use Recording.update_metadata() to modify existing fields."
|
||||||
|
)
|
||||||
|
|
||||||
|
if not _is_valid_metadata_key(key):
|
||||||
|
raise ValueError(f"Invalid metadata key: {key}.")
|
||||||
|
|
||||||
|
if not _is_jsonable(value):
|
||||||
|
raise ValueError("Value must be JSON serializable.")
|
||||||
|
|
||||||
|
self._metadata[key] = value
|
||||||
|
|
||||||
|
def update_metadata(self, key: str, value: Any) -> None:
|
||||||
|
"""Update the value of an existing metadata key,
|
||||||
|
or add the key value pair if it does not already exist.
|
||||||
|
|
||||||
|
:param key: Existing metadata key.
|
||||||
|
:type key: str
|
||||||
|
:param value: New value to enter at key.
|
||||||
|
:type value: any
|
||||||
|
|
||||||
|
:raises ValueError: If value is not JSON serializable
|
||||||
|
:raises ValueError: If key is protected.
|
||||||
|
|
||||||
|
:return: None.
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and update metadata:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
>>> metadata = {
|
||||||
|
>>> "sample_rate": 1e6,
|
||||||
|
>>> "center_frequency": 2.44e9,
|
||||||
|
>>> "author": "me"
|
||||||
|
>>> }
|
||||||
|
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> print(recording.metadata)
|
||||||
|
{'sample_rate': 1000000.0,
|
||||||
|
'center_frequency': 2440000000.0,
|
||||||
|
'author': "me",
|
||||||
|
'timestamp': 17369...
|
||||||
|
'rec_id': 'fda0f41...'}
|
||||||
|
|
||||||
|
>>> recording.update_metadata(key="author", value=you")
|
||||||
|
>>> print(recording.metadata)
|
||||||
|
{'sample_rate': 1000000.0,
|
||||||
|
'center_frequency': 2440000000.0,
|
||||||
|
'author': "you",
|
||||||
|
'timestamp': 17369...
|
||||||
|
'rec_id': 'fda0f41...'}
|
||||||
|
"""
|
||||||
|
if key not in self.metadata:
|
||||||
|
self.add_to_metadata(key=key, value=value)
|
||||||
|
return
|
||||||
|
|
||||||
|
if not _is_jsonable(value):
|
||||||
|
raise ValueError("Value must be JSON serializable.")
|
||||||
|
|
||||||
|
if key in PROTECTED_KEYS: # Check protected keys.
|
||||||
|
raise ValueError(f"Key {key} is protected and cannot be modified or removed.")
|
||||||
|
|
||||||
|
else:
|
||||||
|
self._metadata[key] = value
|
||||||
|
|
||||||
|
def remove_from_metadata(self, key: str):
|
||||||
|
"""
|
||||||
|
Remove a key from the recording metadata.
|
||||||
|
Does not remove key if it is protected.
|
||||||
|
|
||||||
|
:param key: The key to remove.
|
||||||
|
:type key: str
|
||||||
|
|
||||||
|
:raises ValueError: If key is protected.
|
||||||
|
|
||||||
|
:return: None.
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and add metadata:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
>>> metadata = {
|
||||||
|
... "sample_rate": 1e6,
|
||||||
|
... "center_frequency": 2.44e9,
|
||||||
|
... }
|
||||||
|
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> print(recording.metadata)
|
||||||
|
{'sample_rate': 1000000.0,
|
||||||
|
'center_frequency': 2440000000.0,
|
||||||
|
'timestamp': 17369..., # Example value
|
||||||
|
'rec_id': 'fda0f41...'} # Example value
|
||||||
|
|
||||||
|
>>> recording.add_to_metadata(key="author", value="me")
|
||||||
|
>>> print(recording.metadata)
|
||||||
|
{'sample_rate': 1000000.0,
|
||||||
|
'center_frequency': 2440000000.0,
|
||||||
|
'author': 'me',
|
||||||
|
'timestamp': 17369..., # Example value
|
||||||
|
'rec_id': 'fda0f41...'} # Example value
|
||||||
|
"""
|
||||||
|
if key not in PROTECTED_KEYS:
|
||||||
|
self._metadata.pop(key, None)
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Key {key} is protected and cannot be modified or removed.")
|
||||||
|
|
||||||
|
def view(self, output_path: Optional[str] = "images/signal.png", **kwargs) -> None:
|
||||||
|
"""Create a plot of various signal visualizations as a PNG image.
|
||||||
|
|
||||||
|
:param output_path: The output image path. Defaults to "images/signal.png".
|
||||||
|
:type output_path: str, optional
|
||||||
|
:param kwargs: Keyword arguments passed on to utils.view.view_sig.
|
||||||
|
:type: dict of keyword arguments
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and view it as a plot in a .png image:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from utils.data import Recording
|
||||||
|
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
>>> metadata = {
|
||||||
|
>>> "sample_rate": 1e6,
|
||||||
|
>>> "center_frequency": 2.44e9,
|
||||||
|
>>> }
|
||||||
|
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> recording.view()
|
||||||
|
"""
|
||||||
|
from ria_toolkit_oss.view.view_signal import view_sig
|
||||||
|
|
||||||
|
view_sig(recording=self, output_path=output_path, **kwargs)
|
||||||
|
|
||||||
|
def simple_view(self, **kwargs) -> None:
|
||||||
|
"""Create a plot of various signal visualizations as a PNG or SVG image.
|
||||||
|
|
||||||
|
:param kwargs: Keyword arguments passed on to utils.view.view_signal_simple.create_plots.
|
||||||
|
:type: dict of keyword arguments
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and view it as a plot in a .png image:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from utils.data import Recording
|
||||||
|
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
>>> metadata = {
|
||||||
|
>>> "sample_rate": 1e6,
|
||||||
|
>>> "center_frequency": 2.44e9,
|
||||||
|
>>> }
|
||||||
|
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> recording.simple_view()
|
||||||
|
"""
|
||||||
|
from ria_toolkit_oss.view.view_signal_simple import view_simple_sig
|
||||||
|
|
||||||
|
view_simple_sig(recording=self, **kwargs)
|
||||||
|
|
||||||
|
def to_sigmf(
|
||||||
|
self, filename: Optional[str] = None, path: Optional[os.PathLike | str] = None, overwrite: bool = False
|
||||||
|
) -> None:
|
||||||
|
"""Write recording to a set of SigMF files.
|
||||||
|
|
||||||
|
The SigMF io format is defined by the `SigMF Specification Project <https://github.com/sigmf/SigMF>`_
|
||||||
|
|
||||||
|
:param recording: The recording to be written to file.
|
||||||
|
:type recording: ria_toolkit_oss.datatypes.Recording
|
||||||
|
:param filename: The name of the file where the recording is to be saved. Defaults to auto generated filename.
|
||||||
|
:type filename: os.PathLike or str, optional
|
||||||
|
:param path: The directory path to where the recording is to be saved. Defaults to recordings/.
|
||||||
|
:type path: os.PathLike or str, optional
|
||||||
|
|
||||||
|
:raises IOError: If there is an issue encountered during the file writing process.
|
||||||
|
|
||||||
|
:return: None
|
||||||
|
"""
|
||||||
|
from ria_toolkit_oss.io.recording import to_sigmf
|
||||||
|
|
||||||
|
to_sigmf(filename=filename, path=path, recording=self, overwrite=overwrite)
|
||||||
|
|
||||||
|
def to_npy(
|
||||||
|
self, filename: Optional[str] = None, path: Optional[os.PathLike | str] = None, overwrite: bool = False
|
||||||
|
) -> str:
|
||||||
|
"""Write recording to ``.npy`` binary file.
|
||||||
|
|
||||||
|
:param filename: The name of the file where the recording is to be saved. Defaults to auto generated filename.
|
||||||
|
:type filename: os.PathLike or str, optional
|
||||||
|
:param path: The directory path to where the recording is to be saved. Defaults to recordings/.
|
||||||
|
:type path: os.PathLike or str, optional
|
||||||
|
|
||||||
|
:raises IOError: If there is an issue encountered during the file writing process.
|
||||||
|
|
||||||
|
:return: Path where the file was saved.
|
||||||
|
:rtype: str
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and save it to a .npy file:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
>>> metadata = {
|
||||||
|
>>> "sample_rate": 1e6,
|
||||||
|
>>> "center_frequency": 2.44e9,
|
||||||
|
>>> }
|
||||||
|
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> recording.to_npy()
|
||||||
|
"""
|
||||||
|
from ria_toolkit_oss.io.recording import to_npy
|
||||||
|
|
||||||
|
to_npy(recording=self, filename=filename, path=path, overwrite=overwrite)
|
||||||
|
|
||||||
|
def to_wav(
|
||||||
|
self,
|
||||||
|
filename: Optional[str] = None,
|
||||||
|
path: Optional[os.PathLike | str] = None,
|
||||||
|
target_sample_rate: Optional[int] = 48000,
|
||||||
|
bits_per_sample: int = 32,
|
||||||
|
overwrite: bool = False,
|
||||||
|
) -> str:
|
||||||
|
"""Write recording to WAV file with embedded YAML metadata.
|
||||||
|
|
||||||
|
WAV format uses stereo audio with I (in-phase) in left channel and Q (quadrature) in right channel.
|
||||||
|
Metadata is stored in standard LIST INFO chunks with RF-specific metadata encoded as YAML
|
||||||
|
in the ICMT (comment) field for human readability.
|
||||||
|
|
||||||
|
:param filename: The name of the file where the recording is to be saved. Defaults to auto generated filename.
|
||||||
|
:type filename: os.PathLike or str, optional
|
||||||
|
:param path: The directory path to where the recording is to be saved. Defaults to recordings/.
|
||||||
|
:type path: os.PathLike or str, optional
|
||||||
|
:param target_sample_rate: Sample rate stored in the WAV header when no sample_rate metadata
|
||||||
|
is present. IQ samples are written without decimation or interpolation. Default is 48000 Hz.
|
||||||
|
:type target_sample_rate: int, optional
|
||||||
|
:param bits_per_sample: Bits per sample (32 for float32, 16 for int16). Default is 32.
|
||||||
|
:type bits_per_sample: int, optional
|
||||||
|
:param overwrite: Whether to overwrite existing files. Default is False.
|
||||||
|
:type overwrite: bool, optional
|
||||||
|
|
||||||
|
:raises IOError: If there is an issue encountered during the file writing process.
|
||||||
|
|
||||||
|
:return: Path where the file was saved.
|
||||||
|
:rtype: str
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and save it to a .wav file:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from utils.data import Recording
|
||||||
|
>>> samples = numpy.exp(1j * 2 * numpy.pi * 0.1 * numpy.arange(10000))
|
||||||
|
>>> metadata = {"sample_rate": 1e6, "center_frequency": 915e6}
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> recording.to_wav()
|
||||||
|
"""
|
||||||
|
from ria_toolkit_oss.io.recording import to_wav
|
||||||
|
|
||||||
|
return to_wav(
|
||||||
|
recording=self,
|
||||||
|
filename=filename,
|
||||||
|
path=path,
|
||||||
|
target_sample_rate=target_sample_rate,
|
||||||
|
bits_per_sample=bits_per_sample,
|
||||||
|
overwrite=overwrite,
|
||||||
|
)
|
||||||
|
|
||||||
|
def to_blue(
|
||||||
|
self,
|
||||||
|
filename: Optional[str] = None,
|
||||||
|
path: Optional[os.PathLike | str] = None,
|
||||||
|
data_format: str = "CI",
|
||||||
|
overwrite: bool = False,
|
||||||
|
) -> str:
|
||||||
|
"""Write recording to MIDAS Blue file format.
|
||||||
|
|
||||||
|
MIDAS Blue is a legacy RF file format with a 512-byte binary header.
|
||||||
|
Commonly used with X-Midas and other RF/radar signal processing tools.
|
||||||
|
|
||||||
|
:param filename: The name of the file where the recording is to be saved. Defaults to auto generated filename.
|
||||||
|
:type filename: os.PathLike or str, optional
|
||||||
|
:param path: The directory path to where the recording is to be saved. Defaults to recordings/.
|
||||||
|
:type path: os.PathLike or str, optional
|
||||||
|
:param data_format: Format code (default 'CI' = complex int16).
|
||||||
|
Common formats: 'CI' (complex int16), 'CF' (complex float32), 'CD' (complex float64).
|
||||||
|
Integer formats require the IQ samples to already be scaled within [-1, 1).
|
||||||
|
:type data_format: str, optional
|
||||||
|
:param overwrite: Whether to overwrite existing files. Default is False.
|
||||||
|
:type overwrite: bool, optional
|
||||||
|
|
||||||
|
:raises IOError: If there is an issue encountered during the file writing process.
|
||||||
|
|
||||||
|
:return: Path where the file was saved.
|
||||||
|
:rtype: str
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and save it to a .blue file:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from utils.data import Recording
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
>>> metadata = {"sample_rate": 1e6, "center_frequency": 2.44e9}
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> recording.to_blue()
|
||||||
|
"""
|
||||||
|
from ria_toolkit_oss.io.recording import to_blue
|
||||||
|
|
||||||
|
return to_blue(recording=self, filename=filename, path=path, data_format=data_format, overwrite=overwrite)
|
||||||
|
|
||||||
|
def trim(self, num_samples: int, start_sample: Optional[int] = 0) -> Recording:
|
||||||
|
"""Trim Recording samples to a desired length, shifting annotations to maintain alignment.
|
||||||
|
|
||||||
|
:param start_sample: The start index of the desired trimmed recording. Defaults to 0.
|
||||||
|
:type start_sample: int, optional
|
||||||
|
:param num_samples: The number of samples that the output trimmed recording will have.
|
||||||
|
:type num_samples: int
|
||||||
|
:raises IndexError: If start_sample + num_samples is greater than the length of the recording.
|
||||||
|
:raises IndexError: If sample_start < 0 or num_samples < 0.
|
||||||
|
|
||||||
|
:return: The trimmed Recording.
|
||||||
|
:rtype: Recording
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording and trim it:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64)
|
||||||
|
>>> metadata = {
|
||||||
|
... "sample_rate": 1e6,
|
||||||
|
... "center_frequency": 2.44e9,
|
||||||
|
... }
|
||||||
|
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> print(len(recording))
|
||||||
|
10000
|
||||||
|
|
||||||
|
>>> trimmed_recording = recording.trim(start_sample=1000, num_samples=1000)
|
||||||
|
>>> print(len(trimmed_recording))
|
||||||
|
1000
|
||||||
|
"""
|
||||||
|
|
||||||
|
if start_sample < 0:
|
||||||
|
raise IndexError("start_sample cannot be < 0.")
|
||||||
|
elif start_sample + num_samples > len(self):
|
||||||
|
raise IndexError(
|
||||||
|
f"start_sample {start_sample} + num_samples {num_samples} > recording length {len(self)}."
|
||||||
|
)
|
||||||
|
|
||||||
|
end_sample = start_sample + num_samples
|
||||||
|
|
||||||
|
data = self.data[:, start_sample:end_sample]
|
||||||
|
|
||||||
|
new_annotations = copy.deepcopy(self.annotations)
|
||||||
|
trimmed_annotations = []
|
||||||
|
for annotation in new_annotations:
|
||||||
|
# skip annotations entirely outside the trim window
|
||||||
|
if annotation.sample_start + annotation.sample_count <= start_sample:
|
||||||
|
continue
|
||||||
|
if annotation.sample_start >= end_sample:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# trim annotation if it goes outside the trim boundaries
|
||||||
|
if annotation.sample_start < start_sample:
|
||||||
|
annotation.sample_count = annotation.sample_count - (start_sample - annotation.sample_start)
|
||||||
|
annotation.sample_start = start_sample
|
||||||
|
|
||||||
|
if annotation.sample_start + annotation.sample_count > end_sample:
|
||||||
|
annotation.sample_count = end_sample - annotation.sample_start
|
||||||
|
|
||||||
|
# shift annotation to align with the new start point
|
||||||
|
annotation.sample_start = annotation.sample_start - start_sample
|
||||||
|
trimmed_annotations.append(annotation)
|
||||||
|
|
||||||
|
return Recording(data=data, metadata=self.metadata, annotations=trimmed_annotations)
|
||||||
|
|
||||||
|
def normalize(self) -> Recording:
|
||||||
|
"""Scale the recording data, relative to its maximum value, so that the magnitude of the maximum sample is 1.
|
||||||
|
|
||||||
|
:return: Recording where the maximum sample amplitude is 1.
|
||||||
|
:rtype: Recording
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
Create a recording with maximum amplitude 0.5 and normalize to a maximum amplitude of 1:
|
||||||
|
|
||||||
|
>>> import numpy
|
||||||
|
>>> from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
>>> samples = numpy.ones(10000, dtype=numpy.complex64) * 0.5
|
||||||
|
>>> metadata = {
|
||||||
|
... "sample_rate": 1e6,
|
||||||
|
... "center_frequency": 2.44e9,
|
||||||
|
... }
|
||||||
|
|
||||||
|
>>> recording = Recording(data=samples, metadata=metadata)
|
||||||
|
>>> print(numpy.max(numpy.abs(recording.data)))
|
||||||
|
0.5
|
||||||
|
|
||||||
|
>>> normalized_recording = recording.normalize()
|
||||||
|
>>> print(numpy.max(numpy.abs(normalized_recording.data)))
|
||||||
|
1
|
||||||
|
"""
|
||||||
|
max_val = np.max(abs(self.data))
|
||||||
|
if max_val == 0:
|
||||||
|
raise ValueError("Cannot normalize a recording with all-zero data.")
|
||||||
|
scaled_data = self.data / max_val
|
||||||
|
return Recording(data=scaled_data, metadata=self.metadata, annotations=self.annotations)
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
"""The length of a recording is defined by the number of complex samples in each channel of the recording."""
|
||||||
|
return self.shape[1]
|
||||||
|
|
||||||
|
def __eq__(self, other: Recording) -> bool:
|
||||||
|
"""Two Recordings are equal if all data, metadata, and annotations are the same."""
|
||||||
|
|
||||||
|
# counter used to allow for differently ordered annotation lists
|
||||||
|
return (
|
||||||
|
np.array_equal(self.data, other.data)
|
||||||
|
and self.metadata == other.metadata
|
||||||
|
and self.annotations == other.annotations
|
||||||
|
)
|
||||||
|
|
||||||
|
def __ne__(self, other: Recording) -> bool:
|
||||||
|
"""Two Recordings are equal if all data, and metadata, and annotations are the same."""
|
||||||
|
return not self.__eq__(other=other)
|
||||||
|
|
||||||
|
def __iter__(self) -> Iterator:
|
||||||
|
self._index = 0
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __next__(self) -> np.ndarray:
|
||||||
|
if self._index < self.n_chan:
|
||||||
|
to_ret = self.data[self._index]
|
||||||
|
self._index += 1
|
||||||
|
return to_ret
|
||||||
|
else:
|
||||||
|
raise StopIteration
|
||||||
|
|
||||||
|
def __getitem__(self, key: int | tuple[int] | slice) -> np.ndarray | np.complexfloating:
|
||||||
|
"""If key is an integer, tuple of integers, or a slice, return the corresponding samples.
|
||||||
|
|
||||||
|
For arrays with 1,024 or fewer samples, return a copy of the recording data. For larger arrays, return a
|
||||||
|
read-only view. This prevents mutation at a distance while maintaining performance.
|
||||||
|
"""
|
||||||
|
if isinstance(key, (int, tuple, slice)):
|
||||||
|
v = self._data[key]
|
||||||
|
if isinstance(v, np.complexfloating):
|
||||||
|
return v
|
||||||
|
elif v.size > 1024:
|
||||||
|
v.setflags(write=False) # Make view read-only.
|
||||||
|
return v
|
||||||
|
else:
|
||||||
|
return v.copy()
|
||||||
|
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Key must be an integer, tuple, or slice but was {type(key)}.")
|
||||||
|
|
||||||
|
def __setitem__(self, *args, **kwargs) -> None:
|
||||||
|
"""Raise an error if an attempt is made to assign to the recording."""
|
||||||
|
raise ValueError("Assignment to Recording is not allowed.")
|
||||||
|
|
||||||
|
|
||||||
|
def generate_recording_id(data: np.ndarray, timestamp: Optional[float | int] = None) -> str:
|
||||||
|
"""Generate unique 64-character recording ID. The recording ID is generated by hashing the recording data with
|
||||||
|
the datetime that the recording data was generated. If no datatime is provided, the current datatime is used.
|
||||||
|
|
||||||
|
:param data: Tape of IQ samples, as a NumPy array.
|
||||||
|
:type data: np.ndarray
|
||||||
|
:param timestamp: Unix timestamp in seconds. Defaults to None.
|
||||||
|
:type timestamp: float or int, optional
|
||||||
|
|
||||||
|
:return: 256-character hash, to be used as the recording ID.
|
||||||
|
:rtype: str
|
||||||
|
"""
|
||||||
|
if timestamp is None:
|
||||||
|
timestamp = time.time()
|
||||||
|
|
||||||
|
byte_sequence = data.tobytes() + str(timestamp).encode("utf-8")
|
||||||
|
sha256_hash = hashlib.sha256(byte_sequence)
|
||||||
|
|
||||||
|
return sha256_hash.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _is_jsonable(x: Any) -> bool:
|
||||||
|
"""
|
||||||
|
:return: True if x is JSON serializable, False otherwise.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
json.dumps(x)
|
||||||
|
return True
|
||||||
|
except (TypeError, OverflowError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _is_valid_metadata_key(key: Any) -> bool:
|
||||||
|
"""
|
||||||
|
:return: True if key is a valid metadata key, False otherwise.
|
||||||
|
"""
|
||||||
|
if isinstance(key, str) and key.islower() and re.match(pattern=r"^[a-z_]+$", string=key) is not None:
|
||||||
|
return True
|
||||||
|
|
||||||
|
else:
|
||||||
|
return False
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
"""
|
"""
|
||||||
Utilities for input/output operations on the ria_toolkit_oss.data.Recording object.
|
Utilities for input/output operations on the ria_toolkit_oss.datatypes.Recording object.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import datetime
|
import datetime
|
||||||
|
|
@ -19,8 +19,8 @@ from quantiphy import Quantity
|
||||||
from sigmf import SigMFFile, sigmffile
|
from sigmf import SigMFFile, sigmffile
|
||||||
from sigmf.utils import get_data_type_str
|
from sigmf.utils import get_data_type_str
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Annotation
|
from ria_toolkit_oss.datatypes import Annotation
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
|
|
||||||
_BLUE_META_PREFIX = "META_"
|
_BLUE_META_PREFIX = "META_"
|
||||||
_BLUE_META_TAG_MAX_LEN = 60
|
_BLUE_META_TAG_MAX_LEN = 60
|
||||||
|
|
@ -64,7 +64,7 @@ def to_npy(
|
||||||
"""Write recording to ``.npy`` binary file.
|
"""Write recording to ``.npy`` binary file.
|
||||||
|
|
||||||
:param recording: The recording to be written to file.
|
:param recording: The recording to be written to file.
|
||||||
:type recording: ria_toolkit_oss.data.Recording
|
:type recording: ria_toolkit_oss.datatypes.Recording
|
||||||
:param filename: The name of the file where the recording is to be saved. Defaults to auto generated filename.
|
:param filename: The name of the file where the recording is to be saved. Defaults to auto generated filename.
|
||||||
:type filename: os.PathLike or str, optional
|
:type filename: os.PathLike or str, optional
|
||||||
:param path: The directory path to where the recording is to be saved. Defaults to recordings/.
|
:param path: The directory path to where the recording is to be saved. Defaults to recordings/.
|
||||||
|
|
@ -135,7 +135,7 @@ def from_npy(file: os.PathLike | str, legacy: bool = False) -> Recording:
|
||||||
:raises IOError: If there is an issue encountered during the file reading process.
|
:raises IOError: If there is an issue encountered during the file reading process.
|
||||||
|
|
||||||
:return: The recording, as initialized from the ``.npy`` file.
|
:return: The recording, as initialized from the ``.npy`` file.
|
||||||
:rtype: ria_toolkit_oss.data.Recording
|
:rtype: ria_toolkit_oss.datatypes.Recording
|
||||||
"""
|
"""
|
||||||
|
|
||||||
filename, extension = os.path.splitext(file)
|
filename, extension = os.path.splitext(file)
|
||||||
|
|
@ -161,7 +161,7 @@ def from_npy(file: os.PathLike | str, legacy: bool = False) -> Recording:
|
||||||
try:
|
try:
|
||||||
raw_ann = np.load(f, allow_pickle=False)
|
raw_ann = np.load(f, allow_pickle=False)
|
||||||
ann_list = json.loads(raw_ann.tobytes().decode())
|
ann_list = json.loads(raw_ann.tobytes().decode())
|
||||||
from ria_toolkit_oss.data.annotation import Annotation
|
from ria_toolkit_oss.datatypes.annotation import Annotation
|
||||||
|
|
||||||
annotations = [Annotation(**a) for a in ann_list]
|
annotations = [Annotation(**a) for a in ann_list]
|
||||||
except EOFError:
|
except EOFError:
|
||||||
|
|
@ -175,15 +175,6 @@ def from_npy(file: os.PathLike | str, legacy: bool = False) -> Recording:
|
||||||
)
|
)
|
||||||
data = first # already loaded without pickle (numeric array)
|
data = first # already loaded without pickle (numeric array)
|
||||||
metadata = np.load(f, allow_pickle=True).tolist()
|
metadata = np.load(f, allow_pickle=True).tolist()
|
||||||
# Normalize namespaced keys (e.g. "BlockGenerator:Foo:sample_rate") to
|
|
||||||
# their bare equivalents so downstream code can find them reliably.
|
|
||||||
_STANDARD_KEYS = {"sample_rate", "center_frequency", "bandwidth"}
|
|
||||||
if isinstance(metadata, dict):
|
|
||||||
for k in list(metadata):
|
|
||||||
if ":" in k:
|
|
||||||
bare = k.rsplit(":", 1)[-1]
|
|
||||||
if bare in _STANDARD_KEYS and bare not in metadata:
|
|
||||||
metadata[bare] = metadata[k]
|
|
||||||
try:
|
try:
|
||||||
annotations = list(np.load(f, allow_pickle=True))
|
annotations = list(np.load(f, allow_pickle=True))
|
||||||
except EOFError:
|
except EOFError:
|
||||||
|
|
@ -207,7 +198,7 @@ def from_npy_legacy(file: os.PathLike | str) -> Recording:
|
||||||
:raises IOError: If there is an issue encountered during the file reading process.
|
:raises IOError: If there is an issue encountered during the file reading process.
|
||||||
|
|
||||||
:return: The recording, as initialized from the legacy ``.npy`` file.
|
:return: The recording, as initialized from the legacy ``.npy`` file.
|
||||||
:rtype: ria_toolkit_oss.data.Recording
|
:rtype: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
**Examples:**
|
**Examples:**
|
||||||
|
|
||||||
|
|
@ -279,7 +270,7 @@ def to_sigmf(
|
||||||
The SigMF io format is defined by the `SigMF Specification Project <https://github.com/sigmf/SigMF>`_
|
The SigMF io format is defined by the `SigMF Specification Project <https://github.com/sigmf/SigMF>`_
|
||||||
|
|
||||||
:param recording: The recording to be written to file.
|
:param recording: The recording to be written to file.
|
||||||
:type recording: ria_toolkit_oss.data.Recording
|
:type recording: ria_toolkit_oss.datatypes.Recording
|
||||||
:param filename: The name of the file where the recording is to be saved. Defaults to auto generated filename.
|
:param filename: The name of the file where the recording is to be saved. Defaults to auto generated filename.
|
||||||
:type filename: os.PathLike or str, optional
|
:type filename: os.PathLike or str, optional
|
||||||
:param path: The directory path to where the recording is to be saved. Defaults to recordings/.
|
:param path: The directory path to where the recording is to be saved. Defaults to recordings/.
|
||||||
|
|
@ -390,7 +381,7 @@ def from_sigmf(file: os.PathLike | str) -> Recording:
|
||||||
:raises IOError: If there is an issue encountered during the file reading process.
|
:raises IOError: If there is an issue encountered during the file reading process.
|
||||||
|
|
||||||
:return: The recording, as initialized from the SigMF files.
|
:return: The recording, as initialized from the SigMF files.
|
||||||
:rtype: ria_toolkit_oss.data.Recording
|
:rtype: ria_toolkit_oss.datatypes.Recording
|
||||||
"""
|
"""
|
||||||
|
|
||||||
file = str(file)
|
file = str(file)
|
||||||
|
|
@ -452,7 +443,7 @@ def to_wav(
|
||||||
in the ICMT (comment) field for human readability.
|
in the ICMT (comment) field for human readability.
|
||||||
|
|
||||||
:param recording: The recording to be written to file.
|
:param recording: The recording to be written to file.
|
||||||
:type recording: ria_toolkit_oss.data.Recording
|
:type recording: ria_toolkit_oss.datatypes.Recording
|
||||||
:param filename: The name of the file where the recording is to be saved.
|
:param filename: The name of the file where the recording is to be saved.
|
||||||
Defaults to auto-generated filename.
|
Defaults to auto-generated filename.
|
||||||
:type filename: str, optional
|
:type filename: str, optional
|
||||||
|
|
@ -562,7 +553,7 @@ def from_wav(file: os.PathLike | str) -> Recording:
|
||||||
:raises ValueError: If file is not stereo or has unsupported format.
|
:raises ValueError: If file is not stereo or has unsupported format.
|
||||||
|
|
||||||
:return: The recording, as initialized from the WAV file.
|
:return: The recording, as initialized from the WAV file.
|
||||||
:rtype: ria_toolkit_oss.data.Recording
|
:rtype: ria_toolkit_oss.datatypes.Recording
|
||||||
"""
|
"""
|
||||||
import wave
|
import wave
|
||||||
|
|
||||||
|
|
@ -644,7 +635,7 @@ def to_blue(
|
||||||
Commonly used with X-Midas and other RF/radar signal processing tools.
|
Commonly used with X-Midas and other RF/radar signal processing tools.
|
||||||
|
|
||||||
:param recording: The recording to be written to file.
|
:param recording: The recording to be written to file.
|
||||||
:type recording: ria_toolkit_oss.data.Recording
|
:type recording: ria_toolkit_oss.datatypes.Recording
|
||||||
:param filename: The name of the file where the recording is to be saved.
|
:param filename: The name of the file where the recording is to be saved.
|
||||||
Defaults to auto-generated filename.
|
Defaults to auto-generated filename.
|
||||||
:type filename: str, optional
|
:type filename: str, optional
|
||||||
|
|
@ -801,7 +792,7 @@ def from_blue(file: os.PathLike | str) -> Recording:
|
||||||
:raises ValueError: If file format is not valid or unsupported.
|
:raises ValueError: If file format is not valid or unsupported.
|
||||||
|
|
||||||
:return: The recording, as initialized from the Blue file.
|
:return: The recording, as initialized from the Blue file.
|
||||||
:rtype: ria_toolkit_oss.data.Recording
|
:rtype: ria_toolkit_oss.datatypes.Recording
|
||||||
"""
|
"""
|
||||||
filename = str(file)
|
filename = str(file)
|
||||||
if not filename.endswith(".blue"):
|
if not filename.endswith(".blue"):
|
||||||
|
|
@ -926,7 +917,7 @@ def load_recording(file: os.PathLike) -> Recording:
|
||||||
:raises ValueError: If the inferred file extension is not supported.
|
:raises ValueError: If the inferred file extension is not supported.
|
||||||
|
|
||||||
:return: The recording, as initialized from file(s).
|
:return: The recording, as initialized from file(s).
|
||||||
:rtype: ria_toolkit_oss.data.Recording
|
:rtype: ria_toolkit_oss.datatypes.Recording
|
||||||
"""
|
"""
|
||||||
_, extension = os.path.splitext(file)
|
_, extension = os.path.splitext(file)
|
||||||
extension = extension.lstrip(".")
|
extension = extension.lstrip(".")
|
||||||
|
|
|
||||||
|
|
@ -233,9 +233,6 @@ class TransmitterConfig:
|
||||||
# For sdr_remote control — keys: host, ssh_user, ssh_key_path, device_type, device_id, zmq_port
|
# For sdr_remote control — keys: host, ssh_user, ssh_key_path, device_type, device_id, zmq_port
|
||||||
sdr_remote: Optional[dict] = None
|
sdr_remote: Optional[dict] = None
|
||||||
|
|
||||||
# For sdr_agent control — keys: modulation, order, symbol_rate, center_frequency, filter, rolloff
|
|
||||||
sdr_agent: Optional[dict] = None
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_dict(cls, d: dict) -> "TransmitterConfig":
|
def from_dict(cls, d: dict) -> "TransmitterConfig":
|
||||||
schedule = [CaptureStep.from_dict(s) for s in d.get("schedule", [])]
|
schedule = [CaptureStep.from_dict(s) for s in d.get("schedule", [])]
|
||||||
|
|
@ -247,7 +244,6 @@ class TransmitterConfig:
|
||||||
script=d.get("script"),
|
script=d.get("script"),
|
||||||
device=d.get("device"),
|
device=d.get("device"),
|
||||||
sdr_remote=d.get("sdr_remote"),
|
sdr_remote=d.get("sdr_remote"),
|
||||||
sdr_agent=d.get("sdr_agent"),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -276,7 +272,6 @@ class OutputConfig:
|
||||||
path: str = "recordings"
|
path: str = "recordings"
|
||||||
device_id: Optional[str] = None # for device-profile campaigns
|
device_id: Optional[str] = None # for device-profile campaigns
|
||||||
repo: Optional[str] = None
|
repo: Optional[str] = None
|
||||||
folder: Optional[str] = None # repo subfolder: None = use campaign name, "" = no subfolder, str = custom
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_dict(cls, d: dict) -> "OutputConfig":
|
def from_dict(cls, d: dict) -> "OutputConfig":
|
||||||
|
|
@ -285,7 +280,6 @@ class OutputConfig:
|
||||||
path=str(d.get("path", "recordings")),
|
path=str(d.get("path", "recordings")),
|
||||||
device_id=d.get("device_id"),
|
device_id=d.get("device_id"),
|
||||||
repo=d.get("repo"),
|
repo=d.get("repo"),
|
||||||
folder=d.get("folder"),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -299,7 +293,6 @@ class CampaignConfig:
|
||||||
qa: QAConfig = field(default_factory=QAConfig)
|
qa: QAConfig = field(default_factory=QAConfig)
|
||||||
output: OutputConfig = field(default_factory=OutputConfig)
|
output: OutputConfig = field(default_factory=OutputConfig)
|
||||||
mode: str = "controlled_testbed"
|
mode: str = "controlled_testbed"
|
||||||
loops: int = 1 # repeat full schedule this many times; labels get _run{N:02d} suffix
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Loaders
|
# Loaders
|
||||||
|
|
@ -327,7 +320,6 @@ class CampaignConfig:
|
||||||
return cls(
|
return cls(
|
||||||
name=safe_name,
|
name=safe_name,
|
||||||
mode=str(campaign_meta.get("mode", "controlled_testbed")),
|
mode=str(campaign_meta.get("mode", "controlled_testbed")),
|
||||||
loops=max(1, int(campaign_meta.get("loops", 1))),
|
|
||||||
recorder=RecorderConfig.from_dict(raw["recorder"]),
|
recorder=RecorderConfig.from_dict(raw["recorder"]),
|
||||||
transmitters=transmitters,
|
transmitters=transmitters,
|
||||||
qa=QAConfig.from_dict(raw.get("qa", {})),
|
qa=QAConfig.from_dict(raw.get("qa", {})),
|
||||||
|
|
@ -392,7 +384,6 @@ class CampaignConfig:
|
||||||
return cls(
|
return cls(
|
||||||
name=safe_name,
|
name=safe_name,
|
||||||
mode=str(campaign_meta.get("mode", "controlled_testbed")),
|
mode=str(campaign_meta.get("mode", "controlled_testbed")),
|
||||||
loops=max(1, int(campaign_meta.get("loops", 1))),
|
|
||||||
recorder=RecorderConfig.from_dict(raw["recorder"]),
|
recorder=RecorderConfig.from_dict(raw["recorder"]),
|
||||||
transmitters=transmitters,
|
transmitters=transmitters,
|
||||||
qa=QAConfig.from_dict(raw.get("qa", {})),
|
qa=QAConfig.from_dict(raw.get("qa", {})),
|
||||||
|
|
@ -495,9 +486,9 @@ class CampaignConfig:
|
||||||
)
|
)
|
||||||
|
|
||||||
def total_capture_time_s(self) -> float:
|
def total_capture_time_s(self) -> float:
|
||||||
"""Sum of all step durations across all transmitters and loops."""
|
"""Sum of all step durations across all transmitters."""
|
||||||
return sum(step.duration for tx in self.transmitters for step in tx.schedule) * self.loops
|
return sum(step.duration for tx in self.transmitters for step in tx.schedule)
|
||||||
|
|
||||||
def total_steps(self) -> int:
|
def total_steps(self) -> int:
|
||||||
"""Total number of capture steps across all transmitters and loops."""
|
"""Total number of capture steps across all transmitters."""
|
||||||
return sum(len(tx.schedule) for tx in self.transmitters) * self.loops
|
return sum(len(tx.schedule) for tx in self.transmitters)
|
||||||
|
|
|
||||||
|
|
@ -5,19 +5,17 @@ from __future__ import annotations
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
import subprocess
|
import subprocess
|
||||||
import threading
|
|
||||||
import time
|
import time
|
||||||
from dataclasses import dataclass, field, replace
|
from dataclasses import dataclass, field
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Callable, Optional
|
from typing import Callable, Optional
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.io.recording import to_sigmf
|
from ria_toolkit_oss.io.recording import to_sigmf
|
||||||
|
|
||||||
from .campaign import CampaignConfig, CaptureStep, TransmitterConfig
|
from .campaign import CampaignConfig, CaptureStep, TransmitterConfig
|
||||||
from .labeler import build_output_filename, label_recording
|
from .labeler import build_output_filename, label_recording
|
||||||
from .qa import QAResult, check_recording
|
from .qa import QAResult, check_recording
|
||||||
from .tx_executor import TxExecutor
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
@ -171,21 +169,6 @@ def _run_script(script: str, *args: str, timeout: float = 15.0) -> str:
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _extract_tx_params(transmitter: TransmitterConfig) -> dict | None:
|
|
||||||
"""Build a tx_params dict from a transmitter's signal config for SigMF labeling.
|
|
||||||
|
|
||||||
For sdr_agent transmitters, returns the synthetic generation parameters
|
|
||||||
(modulation, order, symbol_rate, etc.) so recordings capture what was
|
|
||||||
transmitted. Returns None for control methods without signal-level params.
|
|
||||||
"""
|
|
||||||
sdr_agent_cfg = getattr(transmitter, "sdr_agent", None)
|
|
||||||
if not sdr_agent_cfg:
|
|
||||||
return None
|
|
||||||
# Extract known signal-level fields; ignore infra fields
|
|
||||||
_INFRA_KEYS = {"node_id", "session_code"}
|
|
||||||
return {k: v for k, v in sdr_agent_cfg.items() if k not in _INFRA_KEYS and v is not None}
|
|
||||||
|
|
||||||
|
|
||||||
class CampaignExecutor:
|
class CampaignExecutor:
|
||||||
"""Executes a :class:`CampaignConfig` end-to-end.
|
"""Executes a :class:`CampaignConfig` end-to-end.
|
||||||
|
|
||||||
|
|
@ -209,14 +192,11 @@ class CampaignExecutor:
|
||||||
config: CampaignConfig,
|
config: CampaignConfig,
|
||||||
progress_cb: Optional[Callable[[int, int, StepResult], None]] = None,
|
progress_cb: Optional[Callable[[int, int, StepResult], None]] = None,
|
||||||
verbose: bool = False,
|
verbose: bool = False,
|
||||||
skip_local_tx: bool = False,
|
|
||||||
):
|
):
|
||||||
self.config = config
|
self.config = config
|
||||||
self.progress_cb = progress_cb
|
self.progress_cb = progress_cb
|
||||||
self.skip_local_tx = skip_local_tx
|
|
||||||
self._sdr = None
|
self._sdr = None
|
||||||
self._remote_tx_controllers: dict = {}
|
self._remote_tx_controllers: dict = {}
|
||||||
self._tx_executors: dict[str, tuple] = {} # tx_id → (TxExecutor, stop_event, thread)
|
|
||||||
|
|
||||||
if verbose:
|
if verbose:
|
||||||
logging.basicConfig(level=logging.DEBUG)
|
logging.basicConfig(level=logging.DEBUG)
|
||||||
|
|
@ -236,12 +216,10 @@ class CampaignExecutor:
|
||||||
"""
|
"""
|
||||||
result = CampaignResult(campaign_name=self.config.name)
|
result = CampaignResult(campaign_name=self.config.name)
|
||||||
|
|
||||||
loops = self.config.loops
|
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Starting campaign '{self.config.name}': "
|
f"Starting campaign '{self.config.name}': "
|
||||||
f"{self.config.total_steps()} steps"
|
f"{self.config.total_steps()} steps, "
|
||||||
+ (f" ({self.config.total_steps() // loops} × {loops} loops)" if loops > 1 else "")
|
f"~{self.config.total_capture_time_s():.0f}s capture time"
|
||||||
+ f", ~{self.config.total_capture_time_s():.0f}s capture time"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
self._init_sdr()
|
self._init_sdr()
|
||||||
|
|
@ -250,36 +228,29 @@ class CampaignExecutor:
|
||||||
total = self.config.total_steps()
|
total = self.config.total_steps()
|
||||||
step_index = 0
|
step_index = 0
|
||||||
|
|
||||||
for loop_idx in range(loops):
|
for transmitter in self.config.transmitters:
|
||||||
if loops > 1:
|
logger.info(f"Transmitter: {transmitter.id} ({len(transmitter.schedule)} steps)")
|
||||||
logger.info(f"Loop {loop_idx + 1}/{loops}")
|
for step in transmitter.schedule:
|
||||||
for transmitter in self.config.transmitters:
|
step_result = self._execute_step(transmitter, step)
|
||||||
logger.info(f"Transmitter: {transmitter.id} ({len(transmitter.schedule)} steps)")
|
result.steps.append(step_result)
|
||||||
for step in transmitter.schedule:
|
step_index += 1
|
||||||
looped_step = replace(step, label=f"{step.label}_run{loop_idx + 1:02d}") if loops > 1 else step
|
|
||||||
step_result = self._execute_step(transmitter, looped_step)
|
|
||||||
result.steps.append(step_result)
|
|
||||||
step_index += 1
|
|
||||||
|
|
||||||
if self.progress_cb:
|
if self.progress_cb:
|
||||||
self.progress_cb(step_index, total, step_result)
|
self.progress_cb(step_index, total, step_result)
|
||||||
|
|
||||||
if step_result.error:
|
if step_result.error:
|
||||||
logger.warning(f"Step '{looped_step.label}' error: {step_result.error}")
|
logger.warning(f"Step '{step.label}' error: {step_result.error}")
|
||||||
elif step_result.qa.flagged:
|
elif step_result.qa.flagged:
|
||||||
logger.warning(
|
logger.warning(f"Step '{step.label}' flagged for review: " + "; ".join(step_result.qa.issues))
|
||||||
f"Step '{looped_step.label}' flagged for review: " + "; ".join(step_result.qa.issues)
|
else:
|
||||||
)
|
logger.info(
|
||||||
else:
|
f"Step '{step.label}' OK "
|
||||||
logger.info(
|
f"(SNR {step_result.qa.snr_db:.1f} dB, "
|
||||||
f"Step '{looped_step.label}' OK "
|
f"{step_result.qa.duration_s:.1f}s)"
|
||||||
f"(SNR {step_result.qa.snr_db:.1f} dB, "
|
)
|
||||||
f"{step_result.qa.duration_s:.1f}s)"
|
|
||||||
)
|
|
||||||
finally:
|
finally:
|
||||||
self._close_sdr()
|
self._close_sdr()
|
||||||
self._close_remote_tx_controllers()
|
self._close_remote_tx_controllers()
|
||||||
self._close_tx_executors()
|
|
||||||
|
|
||||||
result.end_time = time.time()
|
result.end_time = time.time()
|
||||||
logger.info(
|
logger.info(
|
||||||
|
|
@ -354,12 +325,6 @@ class CampaignExecutor:
|
||||||
logger.warning(f"Error closing remote Tx controller {tx_id}: {exc}")
|
logger.warning(f"Error closing remote Tx controller {tx_id}: {exc}")
|
||||||
self._remote_tx_controllers.clear()
|
self._remote_tx_controllers.clear()
|
||||||
|
|
||||||
def _close_tx_executors(self) -> None:
|
|
||||||
for tx_id, (_, stop_event, t) in list(self._tx_executors.items()):
|
|
||||||
stop_event.set()
|
|
||||||
t.join(timeout=5.0)
|
|
||||||
self._tx_executors.clear()
|
|
||||||
|
|
||||||
def _record(self, duration_s: float) -> Recording:
|
def _record(self, duration_s: float) -> Recording:
|
||||||
"""Capture ``duration_s`` seconds of IQ samples."""
|
"""Capture ``duration_s`` seconds of IQ samples."""
|
||||||
num_samples = int(duration_s * self.config.recorder.sample_rate)
|
num_samples = int(duration_s * self.config.recorder.sample_rate)
|
||||||
|
|
@ -404,7 +369,6 @@ class CampaignExecutor:
|
||||||
step=step,
|
step=step,
|
||||||
capture_timestamp=capture_timestamp,
|
capture_timestamp=capture_timestamp,
|
||||||
campaign_name=self.config.name,
|
campaign_name=self.config.name,
|
||||||
tx_params=_extract_tx_params(transmitter),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# QA
|
# QA
|
||||||
|
|
@ -473,30 +437,6 @@ class CampaignExecutor:
|
||||||
# Start transmission in background; _record() runs concurrently
|
# Start transmission in background; _record() runs concurrently
|
||||||
ctrl.transmit_async(step.duration + 1.0)
|
ctrl.transmit_async(step.duration + 1.0)
|
||||||
|
|
||||||
elif transmitter.control_method == "sdr_agent":
|
|
||||||
if self.skip_local_tx:
|
|
||||||
logger.debug(f"skip_local_tx — TX for '{transmitter.id}' delegated to TX agent node")
|
|
||||||
return
|
|
||||||
if not transmitter.sdr_agent:
|
|
||||||
logger.warning(f"Transmitter '{transmitter.id}' has no sdr_agent config — skipping")
|
|
||||||
return
|
|
||||||
step_dict: dict = {"label": step.label, "duration": step.duration + 1.0}
|
|
||||||
if step.power_dbm is not None:
|
|
||||||
step_dict["power_dbm"] = step.power_dbm
|
|
||||||
tx_config = {
|
|
||||||
"id": transmitter.id,
|
|
||||||
"sdr_agent": transmitter.sdr_agent,
|
|
||||||
"schedule": [step_dict],
|
|
||||||
}
|
|
||||||
rec = self.config.recorder
|
|
||||||
tx_device = transmitter.device or rec.device
|
|
||||||
sdr_device = _DEVICE_ALIASES.get(tx_device.lower(), tx_device.lower())
|
|
||||||
stop_event = threading.Event()
|
|
||||||
executor = TxExecutor(tx_config, sdr_device=sdr_device, stop_event=stop_event)
|
|
||||||
t = threading.Thread(target=executor.run, daemon=True, name=f"tx-{transmitter.id}")
|
|
||||||
self._tx_executors[transmitter.id] = (executor, stop_event, t)
|
|
||||||
t.start()
|
|
||||||
|
|
||||||
else:
|
else:
|
||||||
logger.warning(f"Unknown control method '{transmitter.control_method}' — skipping")
|
logger.warning(f"Unknown control method '{transmitter.control_method}' — skipping")
|
||||||
|
|
||||||
|
|
@ -519,13 +459,6 @@ class CampaignExecutor:
|
||||||
if ctrl is not None:
|
if ctrl is not None:
|
||||||
ctrl.wait_transmit(timeout=step.duration + 10.0)
|
ctrl.wait_transmit(timeout=step.duration + 10.0)
|
||||||
|
|
||||||
elif transmitter.control_method == "sdr_agent":
|
|
||||||
entry = self._tx_executors.pop(transmitter.id, None)
|
|
||||||
if entry is not None:
|
|
||||||
_, stop_event, t = entry
|
|
||||||
stop_event.set()
|
|
||||||
t.join(timeout=step.duration + 10.0)
|
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _step_params_json(transmitter: TransmitterConfig, step: CaptureStep) -> str:
|
def _step_params_json(transmitter: TransmitterConfig, step: CaptureStep) -> str:
|
||||||
"""Serialise step parameters to a JSON string for the control script."""
|
"""Serialise step parameters to a JSON string for the control script."""
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ from __future__ import annotations
|
||||||
|
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
|
|
||||||
from .campaign import CaptureStep
|
from .campaign import CaptureStep
|
||||||
|
|
||||||
|
|
@ -15,7 +15,6 @@ def label_recording(
|
||||||
step: CaptureStep,
|
step: CaptureStep,
|
||||||
capture_timestamp: float,
|
capture_timestamp: float,
|
||||||
campaign_name: Optional[str] = None,
|
campaign_name: Optional[str] = None,
|
||||||
tx_params: Optional[dict] = None,
|
|
||||||
) -> Recording:
|
) -> Recording:
|
||||||
"""Apply device identity and capture configuration labels to a recording's metadata.
|
"""Apply device identity and capture configuration labels to a recording's metadata.
|
||||||
|
|
||||||
|
|
@ -28,9 +27,6 @@ def label_recording(
|
||||||
step: The capture step that was active during this recording.
|
step: The capture step that was active during this recording.
|
||||||
capture_timestamp: Unix timestamp (float) of when capture started.
|
capture_timestamp: Unix timestamp (float) of when capture started.
|
||||||
campaign_name: Optional campaign name for cross-recording reference.
|
campaign_name: Optional campaign name for cross-recording reference.
|
||||||
tx_params: Optional dict of transmitter signal parameters (e.g. modulation,
|
|
||||||
order, symbol_rate) written as ``ria:tx_<key>`` fields so downstream
|
|
||||||
training pipelines know what was transmitted into the recording.
|
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
The same recording with updated metadata.
|
The same recording with updated metadata.
|
||||||
|
|
@ -61,11 +57,6 @@ def label_recording(
|
||||||
if step.power_dbm is not None:
|
if step.power_dbm is not None:
|
||||||
recording.update_metadata("tx_power_dbm", step.power_dbm)
|
recording.update_metadata("tx_power_dbm", step.power_dbm)
|
||||||
|
|
||||||
# Transmitter signal parameters (e.g. from sdr_agent synthetic generation)
|
|
||||||
if tx_params:
|
|
||||||
for key, value in tx_params.items():
|
|
||||||
recording.update_metadata(f"tx_{key}", value)
|
|
||||||
|
|
||||||
return recording
|
return recording
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ from dataclasses import dataclass, field
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
|
|
||||||
from .campaign import QAConfig
|
from .campaign import QAConfig
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,299 +0,0 @@
|
||||||
"""TX campaign executor — synthesises and transmits signals via a local SDR.
|
|
||||||
|
|
||||||
The TxExecutor receives a transmitter config dict (matching the
|
|
||||||
``sdr_agent`` control method's schema) and a step schedule, then for each
|
|
||||||
step builds a signal chain with the block generator and transmits it via
|
|
||||||
the local SDR device.
|
|
||||||
|
|
||||||
Supported modulations (``modulation`` field in config):
|
|
||||||
BPSK, QPSK, 8PSK, 16QAM, 64QAM, 256QAM, FSK, OOK, GMSK, OQPSK
|
|
||||||
|
|
||||||
Example config dict (matches CampaignConfig transmitter with
|
|
||||||
``control_method: sdr_agent``)::
|
|
||||||
|
|
||||||
{
|
|
||||||
"id": "synthetic-tx",
|
|
||||||
"type": "sdr",
|
|
||||||
"control_method": "sdr_agent",
|
|
||||||
"sdr_agent": {
|
|
||||||
"modulation": "QPSK",
|
|
||||||
"order": 4,
|
|
||||||
"symbol_rate": 1000000,
|
|
||||||
"center_frequency": 0.0,
|
|
||||||
"filter": "rrc",
|
|
||||||
"rolloff": 0.35
|
|
||||||
},
|
|
||||||
"schedule": [
|
|
||||||
{"label": "step1", "duration": 10, "power_dbm": -10}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
import threading
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_hz(val: object) -> float:
|
|
||||||
"""Parse a frequency value that may be a float (Hz) or a string like '2.45GHz'."""
|
|
||||||
if isinstance(val, (int, float)):
|
|
||||||
return float(val)
|
|
||||||
s = str(val).strip()
|
|
||||||
for suffix, mult in (("GHz", 1e9), ("MHz", 1e6), ("kHz", 1e3), ("Hz", 1.0)):
|
|
||||||
if s.endswith(suffix):
|
|
||||||
return float(s[: -len(suffix)]) * mult
|
|
||||||
return float(s)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_seconds(val: object) -> float:
|
|
||||||
"""Parse a duration value that may be a float (seconds) or a string like '5s'."""
|
|
||||||
if isinstance(val, (int, float)):
|
|
||||||
return float(val)
|
|
||||||
s = str(val).strip()
|
|
||||||
return float(s[:-1]) if s.endswith("s") else float(s)
|
|
||||||
|
|
||||||
|
|
||||||
# Mapping from modulation name → (PSK/QAM order, generator_type)
|
|
||||||
# 'psk' uses PSKGenerator, 'qam' uses QAMGenerator
|
|
||||||
_MOD_TABLE: dict[str, tuple[int, str]] = {
|
|
||||||
"BPSK": (1, "psk"),
|
|
||||||
"QPSK": (2, "psk"),
|
|
||||||
"8PSK": (3, "psk"),
|
|
||||||
"16QAM": (4, "qam"),
|
|
||||||
"64QAM": (6, "qam"),
|
|
||||||
"256QAM": (8, "qam"),
|
|
||||||
}
|
|
||||||
|
|
||||||
_SPECIAL_MODS = {"FSK", "OOK", "GMSK", "OQPSK"}
|
|
||||||
|
|
||||||
# usrp-uhd-client's tx_recording() streams 2 000-sample chunks and loops the
|
|
||||||
# source buffer for the full tx_time, so only this many samples ever need to
|
|
||||||
# be in RAM regardless of step duration or sample rate.
|
|
||||||
# 50 000 complex64 samples ≈ 400 kB — enough spectral diversity for looping.
|
|
||||||
_SYNTH_BLOCK_SAMPLES = 50_000
|
|
||||||
|
|
||||||
|
|
||||||
class TxExecutor:
|
|
||||||
"""Synthesise and transmit a signal campaign via a local SDR.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
config: Transmitter config dict (must have ``sdr_agent`` sub-dict with
|
|
||||||
modulation params, and ``schedule`` list of step dicts).
|
|
||||||
sdr_device: SDR device name to open in TX mode (e.g. "pluto", "usrp").
|
|
||||||
stop_event: External event that aborts the TX loop mid-step.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
config: dict,
|
|
||||||
sdr_device: str = "unknown",
|
|
||||||
stop_event: threading.Event | None = None,
|
|
||||||
) -> None:
|
|
||||||
self.config = config
|
|
||||||
self.sdr_device = sdr_device
|
|
||||||
self.stop_event = stop_event or threading.Event()
|
|
||||||
self._sdr: Any = None
|
|
||||||
|
|
||||||
def run(self) -> None:
|
|
||||||
"""Execute all steps in the schedule, transmitting for each step duration."""
|
|
||||||
agent_cfg: dict = self.config.get("sdr_agent") or {}
|
|
||||||
schedule: list[dict] = self.config.get("schedule") or []
|
|
||||||
|
|
||||||
if not schedule:
|
|
||||||
logger.warning("TxExecutor: no schedule steps — nothing to transmit")
|
|
||||||
return
|
|
||||||
|
|
||||||
modulation: str = agent_cfg.get("modulation", "QPSK").upper()
|
|
||||||
symbol_rate: float = float(agent_cfg.get("symbol_rate", 1e6))
|
|
||||||
center_freq: float = _parse_hz(agent_cfg.get("center_frequency", 0.0))
|
|
||||||
filter_type: str = agent_cfg.get("filter", "rrc").lower()
|
|
||||||
rolloff: float = float(agent_cfg.get("rolloff", 0.35))
|
|
||||||
loops: int = max(1, int(self.config.get("loops", 1)))
|
|
||||||
|
|
||||||
# Upsampling factor: samples_per_symbol, fixed at 8 for SDR compatibility.
|
|
||||||
sps = 8
|
|
||||||
sample_rate = symbol_rate * sps
|
|
||||||
|
|
||||||
self._init_sdr(sample_rate, center_freq)
|
|
||||||
try:
|
|
||||||
for loop_idx in range(loops):
|
|
||||||
if self.stop_event.is_set():
|
|
||||||
break
|
|
||||||
if loops > 1:
|
|
||||||
logger.info("TX loop %d/%d", loop_idx + 1, loops)
|
|
||||||
for step in schedule:
|
|
||||||
if self.stop_event.is_set():
|
|
||||||
break
|
|
||||||
looped_step = (
|
|
||||||
{**step, "label": f"{step.get('label', 'step')}_run{loop_idx + 1:02d}"} if loops > 1 else step
|
|
||||||
)
|
|
||||||
self._execute_step(looped_step, modulation, sps, symbol_rate, filter_type, rolloff)
|
|
||||||
finally:
|
|
||||||
self._close_sdr()
|
|
||||||
|
|
||||||
def _execute_step(
|
|
||||||
self,
|
|
||||||
step: dict,
|
|
||||||
modulation: str,
|
|
||||||
sps: int,
|
|
||||||
symbol_rate: float,
|
|
||||||
filter_type: str,
|
|
||||||
rolloff: float,
|
|
||||||
) -> None:
|
|
||||||
duration: float = _parse_seconds(step.get("duration", 10.0))
|
|
||||||
label: str = step.get("label", "step")
|
|
||||||
gain: float = float(step.get("power_dbm") or 0.0)
|
|
||||||
sample_rate = symbol_rate * sps
|
|
||||||
|
|
||||||
logger.info(
|
|
||||||
"TX step '%s': %.0f s, %s @ %.3f MHz (sps=%d, filter=%s)",
|
|
||||||
label,
|
|
||||||
duration,
|
|
||||||
modulation,
|
|
||||||
symbol_rate / 1e6,
|
|
||||||
sps,
|
|
||||||
filter_type,
|
|
||||||
)
|
|
||||||
|
|
||||||
num_samples = int(duration * sample_rate)
|
|
||||||
|
|
||||||
# Synthesise a short representative block. tx_recording() loops this
|
|
||||||
# buffer for the full tx_time using a 2 000-sample streaming callback,
|
|
||||||
# so peak memory is O(_SYNTH_BLOCK_SAMPLES) regardless of duration.
|
|
||||||
block_size = min(num_samples, _SYNTH_BLOCK_SAMPLES)
|
|
||||||
signal = self._synthesise(modulation, sps, block_size, filter_type, rolloff)
|
|
||||||
|
|
||||||
if self._sdr is not None:
|
|
||||||
try:
|
|
||||||
# Apply gain update if SDR supports it
|
|
||||||
if hasattr(self._sdr, "set_tx_gain"):
|
|
||||||
self._sdr.set_tx_gain(gain)
|
|
||||||
self._sdr.tx_recording(signal, tx_time=duration)
|
|
||||||
except Exception as exc:
|
|
||||||
logger.error("TX step '%s' SDR error: %s", label, exc)
|
|
||||||
else:
|
|
||||||
# No SDR available — simulate by sleeping for the step duration.
|
|
||||||
logger.warning("TX step '%s': no SDR — simulating %.0f s delay", label, duration)
|
|
||||||
self.stop_event.wait(timeout=duration)
|
|
||||||
|
|
||||||
def _synthesise(
|
|
||||||
self,
|
|
||||||
modulation: str,
|
|
||||||
sps: int,
|
|
||||||
num_samples: int,
|
|
||||||
filter_type: str,
|
|
||||||
rolloff: float,
|
|
||||||
):
|
|
||||||
"""Build a block-generator chain and return IQ samples as a numpy array."""
|
|
||||||
try:
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
from ria_toolkit_oss.signal.block_generator import (
|
|
||||||
BinarySource,
|
|
||||||
GMSKModulator,
|
|
||||||
Mapper,
|
|
||||||
OOKModulator,
|
|
||||||
OQPSKModulator,
|
|
||||||
RaisedCosineFilter,
|
|
||||||
RootRaisedCosineFilter,
|
|
||||||
Upsampling,
|
|
||||||
)
|
|
||||||
from ria_toolkit_oss.signal.block_generator.continuous_modulation.fsk_modulator import (
|
|
||||||
FSKModulator,
|
|
||||||
)
|
|
||||||
except ImportError as exc:
|
|
||||||
raise RuntimeError(f"ria_toolkit_oss block generator not available: {exc}") from exc
|
|
||||||
|
|
||||||
# ── Special modulations with their own source-connected modulator ──
|
|
||||||
if modulation in ("OOK", "GMSK", "OQPSK"):
|
|
||||||
src = BinarySource()
|
|
||||||
if modulation == "OOK":
|
|
||||||
mod = OOKModulator(src, samples_per_symbol=sps)
|
|
||||||
elif modulation == "GMSK":
|
|
||||||
mod = GMSKModulator(src, samples_per_symbol=sps)
|
|
||||||
else:
|
|
||||||
mod = OQPSKModulator(src, samples_per_symbol=sps)
|
|
||||||
recording = mod.record(num_samples)
|
|
||||||
flat = np.asarray(recording.data).flatten().astype(np.complex64)
|
|
||||||
if len(flat) < num_samples:
|
|
||||||
flat = np.tile(flat, num_samples // len(flat) + 1)
|
|
||||||
return flat[:num_samples]
|
|
||||||
|
|
||||||
if modulation == "FSK":
|
|
||||||
symbol_rate = num_samples / sps
|
|
||||||
bits_per_sym = 1 # 2-FSK
|
|
||||||
num_bits = max(num_samples // sps, 128) * bits_per_sym
|
|
||||||
bits = BinarySource()((1, num_bits))
|
|
||||||
mod = FSKModulator(
|
|
||||||
num_bits_per_symbol=bits_per_sym,
|
|
||||||
frequency_spacing=symbol_rate * 0.5,
|
|
||||||
symbol_duration=1.0 / max(symbol_rate, 1.0),
|
|
||||||
sampling_frequency=symbol_rate * sps,
|
|
||||||
)
|
|
||||||
flat = np.asarray(mod(bits)).flatten().astype(np.complex64)
|
|
||||||
if len(flat) < num_samples:
|
|
||||||
flat = np.tile(flat, num_samples // len(flat) + 1)
|
|
||||||
return flat[:num_samples]
|
|
||||||
|
|
||||||
# ── PSK / QAM via Mapper → Upsampling → pulse filter ──────────────
|
|
||||||
if modulation not in _MOD_TABLE:
|
|
||||||
logger.warning("Unknown modulation %r — defaulting to QPSK", modulation)
|
|
||||||
modulation = "QPSK"
|
|
||||||
|
|
||||||
bits_per_sym, gen_type = _MOD_TABLE[modulation]
|
|
||||||
mod_family = "QAM" if gen_type == "qam" else "PSK"
|
|
||||||
|
|
||||||
source = BinarySource()
|
|
||||||
mapper = Mapper(constellation_type=mod_family, num_bits_per_symbol=bits_per_sym)
|
|
||||||
upsampler = Upsampling(factor=sps)
|
|
||||||
|
|
||||||
mapper.connect_input([source])
|
|
||||||
upsampler.connect_input([mapper])
|
|
||||||
|
|
||||||
if filter_type in ("rrc",):
|
|
||||||
pulse_filter = RootRaisedCosineFilter(span_in_symbols=6, upsampling_factor=sps, beta=rolloff)
|
|
||||||
pulse_filter.connect_input([upsampler])
|
|
||||||
recording = pulse_filter.record(num_samples)
|
|
||||||
elif filter_type in ("rc",):
|
|
||||||
pulse_filter = RaisedCosineFilter(span_in_symbols=6, upsampling_factor=sps, beta=rolloff)
|
|
||||||
pulse_filter.connect_input([upsampler])
|
|
||||||
recording = pulse_filter.record(num_samples)
|
|
||||||
else:
|
|
||||||
# "none", "rect", "gaussian" — use upsampler output directly
|
|
||||||
recording = upsampler.record(num_samples)
|
|
||||||
|
|
||||||
flat = np.asarray(recording.data).flatten().astype(np.complex64)
|
|
||||||
if len(flat) < num_samples:
|
|
||||||
flat = np.tile(flat, num_samples // len(flat) + 1)
|
|
||||||
return flat[:num_samples]
|
|
||||||
|
|
||||||
def _init_sdr(self, sample_rate: float, center_freq: float) -> None:
|
|
||||||
try:
|
|
||||||
from ria_toolkit_oss.sdr import get_sdr_device
|
|
||||||
|
|
||||||
self._sdr = get_sdr_device(self.sdr_device)
|
|
||||||
self._sdr.init_tx(
|
|
||||||
sample_rate=sample_rate,
|
|
||||||
center_frequency=center_freq,
|
|
||||||
gain=0,
|
|
||||||
channel=0,
|
|
||||||
gain_mode="manual",
|
|
||||||
)
|
|
||||||
logger.info(
|
|
||||||
"TX SDR initialised: %s @ %.3f MHz, %.1f Msps", self.sdr_device, center_freq / 1e6, sample_rate / 1e6
|
|
||||||
)
|
|
||||||
except Exception as exc:
|
|
||||||
logger.warning("TX SDR init failed (%s) — will simulate: %s", self.sdr_device, exc)
|
|
||||||
self._sdr = None
|
|
||||||
|
|
||||||
def _close_sdr(self) -> None:
|
|
||||||
if self._sdr is not None:
|
|
||||||
try:
|
|
||||||
self._sdr.close()
|
|
||||||
except Exception as exc:
|
|
||||||
logger.debug("TX SDR close error: %s", exc)
|
|
||||||
self._sdr = None
|
|
||||||
|
|
@ -5,7 +5,7 @@ from typing import Optional
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from bladerf import _bladerf
|
from bladerf import _bladerf
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
from ria_toolkit_oss.sdr import SDR, SDRError, SDRParameterError
|
from ria_toolkit_oss.sdr import SDR, SDRError, SDRParameterError
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ from typing import Optional
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.sdr._external.libhackrf import HackRF as hrf
|
from ria_toolkit_oss.sdr._external.libhackrf import HackRF as hrf
|
||||||
from ria_toolkit_oss.sdr.sdr import SDR, SDRParameterError
|
from ria_toolkit_oss.sdr.sdr import SDR, SDRParameterError
|
||||||
|
|
||||||
|
|
@ -58,7 +58,7 @@ class HackRF(SDR):
|
||||||
:param channel: The channel the HackRF is set to. (Not actually used)
|
:param channel: The channel the HackRF is set to. (Not actually used)
|
||||||
:type channel: int
|
:type channel: int
|
||||||
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
||||||
'relative' means that gain should be a negative value, and it will be subtracted from the max gain (40).
|
'relative' means that gain should be a negative value, and it will be subtracted from the max gain (40).
|
||||||
:type gain_mode: str
|
:type gain_mode: str
|
||||||
"""
|
"""
|
||||||
print("Initializing RX")
|
print("Initializing RX")
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@ from typing import Optional
|
||||||
import adi
|
import adi
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.sdr.sdr import (
|
from ria_toolkit_oss.sdr.sdr import (
|
||||||
SDR,
|
SDR,
|
||||||
SDRError,
|
SDRError,
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ try:
|
||||||
except ImportError as exc: # pragma: no cover - dependency provided by end user
|
except ImportError as exc: # pragma: no cover - dependency provided by end user
|
||||||
raise ImportError("pyrtlsdr is required to use the RTLSDR class") from exc
|
raise ImportError("pyrtlsdr is required to use the RTLSDR class") from exc
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.sdr.sdr import SDR, SDRParameterError
|
from ria_toolkit_oss.sdr.sdr import SDR, SDRParameterError
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,7 @@ from typing import Optional
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import zmq
|
import zmq
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
|
|
||||||
|
|
||||||
class SDR(ABC):
|
class SDR(ABC):
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ from typing import Optional
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import uhd
|
import uhd
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.sdr.sdr import SDR, SDRParameterError
|
from ria_toolkit_oss.sdr.sdr import SDR, SDRParameterError
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -54,7 +54,7 @@ class USRP(SDR):
|
||||||
:param channel: The channel the USRP is set to.
|
:param channel: The channel the USRP is set to.
|
||||||
:type channel: int
|
:type channel: int
|
||||||
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
||||||
'relative' means that gain should be a negative value, and it will be subtracted from the max gain.
|
'relative' means that gain should be a negative value, and it will be subtracted from the max gain.
|
||||||
:type gain_mode: str
|
:type gain_mode: str
|
||||||
:param rx_buffer_size: Internal buffer size for receiving samples. Defaults to 960000.
|
:param rx_buffer_size: Internal buffer size for receiving samples. Defaults to 960000.
|
||||||
:type rx_buffer_size: int
|
:type rx_buffer_size: int
|
||||||
|
|
@ -285,7 +285,7 @@ class USRP(SDR):
|
||||||
:param channel: The channel the USRP is set to.
|
:param channel: The channel the USRP is set to.
|
||||||
:type channel: int
|
:type channel: int
|
||||||
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
||||||
'relative' means that gain should be a negative value, and it will be subtracted from the max gain.
|
'relative' means that gain should be a negative value, and it will be subtracted from the max gain.
|
||||||
:type gain_mode: str
|
:type gain_mode: str
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@
|
||||||
from fastapi import Depends, FastAPI
|
from fastapi import Depends, FastAPI
|
||||||
|
|
||||||
from .auth import require_api_key
|
from .auth import require_api_key
|
||||||
from .routers import conductor, inference
|
from .routers import inference, orchestrator
|
||||||
|
|
||||||
|
|
||||||
def create_app(api_key: str = "") -> FastAPI:
|
def create_app(api_key: str = "") -> FastAPI:
|
||||||
|
|
@ -28,9 +28,9 @@ def create_app(api_key: str = "") -> FastAPI:
|
||||||
app.state.api_key = api_key
|
app.state.api_key = api_key
|
||||||
|
|
||||||
app.include_router(
|
app.include_router(
|
||||||
conductor.router,
|
orchestrator.router,
|
||||||
prefix="/conductor",
|
prefix="/orchestrator",
|
||||||
tags=["Conductor"],
|
tags=["Orchestrator"],
|
||||||
dependencies=[Depends(require_api_key)],
|
dependencies=[Depends(require_api_key)],
|
||||||
)
|
)
|
||||||
app.include_router(
|
app.include_router(
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@ from pathlib import Path
|
||||||
from pydantic import BaseModel, field_validator
|
from pydantic import BaseModel, field_validator
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Conductor
|
# Orchestrator
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
"""Conductor routes: campaign deployment, status, and cancellation."""
|
"""Orchestrator routes: campaign deployment, status, and cancellation."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
|
@ -11,7 +11,7 @@ from scipy.signal import butter
|
||||||
from scipy.signal import chirp as sci_chirp
|
from scipy.signal import chirp as sci_chirp
|
||||||
from scipy.signal import hilbert, lfilter
|
from scipy.signal import hilbert, lfilter
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
|
|
||||||
|
|
||||||
def sine(
|
def sine(
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.signal.block_generator.generators.signal_generator import (
|
from ria_toolkit_oss.signal.block_generator.generators.signal_generator import (
|
||||||
SignalGenerator,
|
SignalGenerator,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.signal.block_generator.generators.signal_generator import (
|
from ria_toolkit_oss.signal.block_generator.generators.signal_generator import (
|
||||||
SignalGenerator,
|
SignalGenerator,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.signal.block_generator.generators.signal_generator import (
|
from ria_toolkit_oss.signal.block_generator.generators.signal_generator import (
|
||||||
SignalGenerator,
|
SignalGenerator,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
from ria_toolkit_oss.signal import Recordable
|
from ria_toolkit_oss.signal import Recordable
|
||||||
from ria_toolkit_oss.signal.block_generator.block import Block
|
from ria_toolkit_oss.signal.block_generator.block import Block
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ from datetime import datetime
|
||||||
import click
|
import click
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.signal.block_generator.mapping.mapper import Mapper
|
from ria_toolkit_oss.signal.block_generator.mapping.mapper import Mapper
|
||||||
from ria_toolkit_oss.signal.block_generator.multirate.upsampling import Upsampling
|
from ria_toolkit_oss.signal.block_generator.multirate.upsampling import Upsampling
|
||||||
from ria_toolkit_oss.signal.block_generator.pulse_shaping.raised_cosine_filter import (
|
from ria_toolkit_oss.signal.block_generator.pulse_shaping.raised_cosine_filter import (
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
from ria_toolkit_oss.signal.block_generator.data_types import DataType
|
from ria_toolkit_oss.signal.block_generator.data_types import DataType
|
||||||
from ria_toolkit_oss.signal.block_generator.recordable_block import RecordableBlock
|
from ria_toolkit_oss.signal.block_generator.recordable_block import RecordableBlock
|
||||||
from ria_toolkit_oss.signal.block_generator.source_block import SourceBlock
|
from ria_toolkit_oss.signal.block_generator.source_block import SourceBlock
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
|
||||||
class Recordable(ABC):
|
class Recordable(ABC):
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,7 @@ from typing import Optional
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from numpy.typing import ArrayLike
|
from numpy.typing import ArrayLike
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.utils.array_conversion import convert_to_2xn
|
from ria_toolkit_oss.utils.array_conversion import convert_to_2xn
|
||||||
|
|
||||||
# TODO: For round 2 of index generation, should j be at min 2 spots away from where it was to prevent adjacent patches.
|
# TODO: For round 2 of index generation, should j be at min 2 spots away from where it was to prevent adjacent patches.
|
||||||
|
|
@ -29,7 +29,7 @@ def generate_awgn(signal: ArrayLike | Recording, snr: Optional[float] = 1) -> np
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param snr: The signal-to-noise ratio in dB. Default is 1.
|
:param snr: The signal-to-noise ratio in dB. Default is 1.
|
||||||
:type snr: float, optional
|
:type snr: float, optional
|
||||||
|
|
||||||
|
|
@ -37,7 +37,7 @@ def generate_awgn(signal: ArrayLike | Recording, snr: Optional[float] = 1) -> np
|
||||||
|
|
||||||
:return: A numpy array representing the generated noise which matches the SNR of `signal`. If `signal` is a
|
:return: A numpy array representing the generated noise which matches the SNR of `signal`. If `signal` is a
|
||||||
Recording, returns a Recording object with its `data` attribute containing the generated noise array.
|
Recording, returns a Recording object with its `data` attribute containing the generated noise array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[2 + 5j, 1 + 8j]])
|
>>> rec = Recording(data=[[2 + 5j, 1 + 8j]])
|
||||||
>>> new_rec = generate_awgn(rec)
|
>>> new_rec = generate_awgn(rec)
|
||||||
|
|
@ -80,14 +80,14 @@ def time_reversal(signal: ArrayLike | Recording) -> np.ndarray | Recording:
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:raises ValueError: If `signal` is not CxN complex.
|
:raises ValueError: If `signal` is not CxN complex.
|
||||||
|
|
||||||
:return: A numpy array containing the reversed I and Q data samples if `signal` is an array.
|
:return: A numpy array containing the reversed I and Q data samples if `signal` is an array.
|
||||||
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing the
|
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing the
|
||||||
reversed array.
|
reversed array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[1+2j, 3+4j, 5+6j]])
|
>>> rec = Recording(data=[[1+2j, 3+4j, 5+6j]])
|
||||||
>>> new_rec = time_reversal(rec)
|
>>> new_rec = time_reversal(rec)
|
||||||
|
|
@ -123,14 +123,14 @@ def spectral_inversion(signal: ArrayLike | Recording) -> np.ndarray | Recording:
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:raises ValueError: If `signal` is not CxN complex.
|
:raises ValueError: If `signal` is not CxN complex.
|
||||||
|
|
||||||
:return: A numpy array containing the original I and negated Q data samples if `signal` is an array.
|
:return: A numpy array containing the original I and negated Q data samples if `signal` is an array.
|
||||||
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing the
|
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing the
|
||||||
inverted array.
|
inverted array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[0+45j, 2-10j]])
|
>>> rec = Recording(data=[[0+45j, 2-10j]])
|
||||||
>>> new_rec = spectral_inversion(rec)
|
>>> new_rec = spectral_inversion(rec)
|
||||||
|
|
@ -165,14 +165,14 @@ def channel_swap(signal: ArrayLike | Recording) -> np.ndarray | Recording:
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:raises ValueError: If `signal` is not CxN complex.
|
:raises ValueError: If `signal` is not CxN complex.
|
||||||
|
|
||||||
:return: A numpy array containing the swapped I and Q data samples if `signal` is an array.
|
:return: A numpy array containing the swapped I and Q data samples if `signal` is an array.
|
||||||
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing the
|
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing the
|
||||||
swapped array.
|
swapped array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[10+20j, 7+35j]])
|
>>> rec = Recording(data=[[10+20j, 7+35j]])
|
||||||
>>> new_rec = channel_swap(rec)
|
>>> new_rec = channel_swap(rec)
|
||||||
|
|
@ -207,14 +207,14 @@ def amplitude_reversal(signal: ArrayLike | Recording) -> np.ndarray | Recording:
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:raises ValueError: If `signal` is not CxN complex.
|
:raises ValueError: If `signal` is not CxN complex.
|
||||||
|
|
||||||
:return: A numpy array containing the negated I and Q data samples if `signal` is an array.
|
:return: A numpy array containing the negated I and Q data samples if `signal` is an array.
|
||||||
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing the
|
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing the
|
||||||
negated array.
|
negated array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[4-3j, -5-2j, -9+1j]])
|
>>> rec = Recording(data=[[4-3j, -5-2j, -9+1j]])
|
||||||
>>> new_rec = amplitude_reversal(rec)
|
>>> new_rec = amplitude_reversal(rec)
|
||||||
|
|
@ -253,7 +253,7 @@ def drop_samples( # noqa: C901 # TODO: Simplify function
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param max_section_size: Maximum allowable size of the section to be dropped and replaced. Default is 2.
|
:param max_section_size: Maximum allowable size of the section to be dropped and replaced. Default is 2.
|
||||||
:type max_section_size: int, optional
|
:type max_section_size: int, optional
|
||||||
:param fill_type: Fill option used to replace dropped section of data (back-fill, front-fill, mean, zeros).
|
:param fill_type: Fill option used to replace dropped section of data (back-fill, front-fill, mean, zeros).
|
||||||
|
|
@ -275,7 +275,7 @@ def drop_samples( # noqa: C901 # TODO: Simplify function
|
||||||
:return: A numpy array containing the I and Q data samples with replaced subsections if
|
:return: A numpy array containing the I and Q data samples with replaced subsections if
|
||||||
`signal` is an array. If `signal` is a `Recording`, returns a `Recording` object with its `data`
|
`signal` is an array. If `signal` is a `Recording`, returns a `Recording` object with its `data`
|
||||||
attribute containing the array with dropped samples.
|
attribute containing the array with dropped samples.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
||||||
>>> new_rec = drop_samples(rec)
|
>>> new_rec = drop_samples(rec)
|
||||||
|
|
@ -346,7 +346,7 @@ def quantize_tape(
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param bin_number: The number of bins the signal should be divided into. Default is 4.
|
:param bin_number: The number of bins the signal should be divided into. Default is 4.
|
||||||
:type bin_number: int, optional
|
:type bin_number: int, optional
|
||||||
:param rounding_type: The type of rounding applied during processing. Default is "floor".
|
:param rounding_type: The type of rounding applied during processing. Default is "floor".
|
||||||
|
|
@ -362,7 +362,7 @@ def quantize_tape(
|
||||||
:return: A numpy array containing the quantized I and Q data samples if `signal` is an array.
|
:return: A numpy array containing the quantized I and Q data samples if `signal` is an array.
|
||||||
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing
|
If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing
|
||||||
the quantized array.
|
the quantized array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[1+1j, 4+4j, 1+2j, 1+4j]])
|
>>> rec = Recording(data=[[1+1j, 4+4j, 1+2j, 1+4j]])
|
||||||
>>> new_rec = quantize_tape(rec)
|
>>> new_rec = quantize_tape(rec)
|
||||||
|
|
@ -421,7 +421,7 @@ def quantize_parts(
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param max_section_size: Maximum allowable size of the section to be quantized. Default is 2.
|
:param max_section_size: Maximum allowable size of the section to be quantized. Default is 2.
|
||||||
:type max_section_size: int, optional
|
:type max_section_size: int, optional
|
||||||
:param bin_number: The number of bins the signal should be divided into. Default is 4.
|
:param bin_number: The number of bins the signal should be divided into. Default is 4.
|
||||||
|
|
@ -439,7 +439,7 @@ def quantize_parts(
|
||||||
:return: A numpy array containing the I and Q data samples with quantized subsections if `signal`
|
:return: A numpy array containing the I and Q data samples with quantized subsections if `signal`
|
||||||
is an array. If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute
|
is an array. If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute
|
||||||
containing the partially quantized array.
|
containing the partially quantized array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
||||||
>>> new_rec = quantize_parts(rec)
|
>>> new_rec = quantize_parts(rec)
|
||||||
|
|
@ -510,7 +510,7 @@ def magnitude_rescale(
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param starting_bounds: The bounds (inclusive) as indices in which the starting position of the rescaling occurs.
|
:param starting_bounds: The bounds (inclusive) as indices in which the starting position of the rescaling occurs.
|
||||||
Default is None, but if user does not assign any bounds, the bounds become (random index, N-1).
|
Default is None, but if user does not assign any bounds, the bounds become (random index, N-1).
|
||||||
:type starting_bounds: tuple, optional
|
:type starting_bounds: tuple, optional
|
||||||
|
|
@ -522,7 +522,7 @@ def magnitude_rescale(
|
||||||
:return: A numpy array containing the I and Q data samples with the rescaled magnitude after the random
|
:return: A numpy array containing the I and Q data samples with the rescaled magnitude after the random
|
||||||
starting point if `signal` is an array. If `signal` is a `Recording`, returns a `Recording`
|
starting point if `signal` is an array. If `signal` is a `Recording`, returns a `Recording`
|
||||||
object with its `data` attribute containing the rescaled array.
|
object with its `data` attribute containing the rescaled array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
||||||
>>> new_rec = magniute_rescale(rec)
|
>>> new_rec = magniute_rescale(rec)
|
||||||
|
|
@ -571,7 +571,7 @@ def cut_out( # noqa: C901 # TODO: Simplify function
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param max_section_size: Maximum allowable size of the section to be quantized. Default is 3.
|
:param max_section_size: Maximum allowable size of the section to be quantized. Default is 3.
|
||||||
:type max_section_size: int, optional
|
:type max_section_size: int, optional
|
||||||
:param fill_type: Fill option used to replace cutout section of data (zeros, ones, low-snr, avg-snr-1, avg-snr-2).
|
:param fill_type: Fill option used to replace cutout section of data (zeros, ones, low-snr, avg-snr-1, avg-snr-2).
|
||||||
|
|
@ -596,7 +596,7 @@ def cut_out( # noqa: C901 # TODO: Simplify function
|
||||||
:return: A numpy array containing the I and Q data samples with random sections cut out and replaced according to
|
:return: A numpy array containing the I and Q data samples with random sections cut out and replaced according to
|
||||||
`fill_type` if `signal` is an array. If `signal` is a `Recording`, returns a `Recording` object
|
`fill_type` if `signal` is an array. If `signal` is a `Recording`, returns a `Recording` object
|
||||||
with its `data` attribute containing the cut out and replaced array.
|
with its `data` attribute containing the cut out and replaced array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
||||||
>>> new_rec = cut_out(rec)
|
>>> new_rec = cut_out(rec)
|
||||||
|
|
@ -666,7 +666,7 @@ def patch_shuffle(signal: ArrayLike | Recording, max_patch_size: Optional[int] =
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param max_patch_size: Maximum allowable patch size of the data that can be shuffled. Default is 3.
|
:param max_patch_size: Maximum allowable patch size of the data that can be shuffled. Default is 3.
|
||||||
:type max_patch_size: int, optional
|
:type max_patch_size: int, optional
|
||||||
|
|
||||||
|
|
@ -676,7 +676,7 @@ def patch_shuffle(signal: ArrayLike | Recording, max_patch_size: Optional[int] =
|
||||||
:return: A numpy array containing the I and Q data samples with randomly shuffled regions if `signal` is
|
:return: A numpy array containing the I and Q data samples with randomly shuffled regions if `signal` is
|
||||||
an array. If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing
|
an array. If `signal` is a `Recording`, returns a `Recording` object with its `data` attribute containing
|
||||||
the shuffled array.
|
the shuffled array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
>>> rec = Recording(data=[[2+5j, 1+8j, 6+4j, 3+7j, 4+9j]])
|
||||||
>>> new_rec = patch_shuffle(rec)
|
>>> new_rec = patch_shuffle(rec)
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,7 @@ import numpy as np
|
||||||
from numpy.typing import ArrayLike
|
from numpy.typing import ArrayLike
|
||||||
from scipy.signal import resample_poly
|
from scipy.signal import resample_poly
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
from ria_toolkit_oss.transforms import iq_augmentations
|
from ria_toolkit_oss.transforms import iq_augmentations
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -31,7 +31,7 @@ def add_awgn_to_signal(signal: ArrayLike | Recording, snr: Optional[float] = 1)
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex ``C x N`` array or `Recording`, where ``C`` is the number of channels
|
:param signal: Input IQ data as a complex ``C x N`` array or `Recording`, where ``C`` is the number of channels
|
||||||
and ``N`` is the length of the IQ examples.
|
and ``N`` is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param snr: The signal-to-noise ratio in dB. Default is 1.
|
:param snr: The signal-to-noise ratio in dB. Default is 1.
|
||||||
:type snr: float, optional
|
:type snr: float, optional
|
||||||
|
|
||||||
|
|
@ -39,7 +39,7 @@ def add_awgn_to_signal(signal: ArrayLike | Recording, snr: Optional[float] = 1)
|
||||||
|
|
||||||
:return: A numpy array which is the sum of the noise (which matches the SNR) and the original signal. If `signal`
|
:return: A numpy array which is the sum of the noise (which matches the SNR) and the original signal. If `signal`
|
||||||
is a `Recording`, returns a `Recording object` with its `data` attribute containing the noisy signal array.
|
is a `Recording`, returns a `Recording object` with its `data` attribute containing the noisy signal array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[1+1j, 2+2j]])
|
>>> rec = Recording(data=[[1+1j, 2+2j]])
|
||||||
>>> new_rec = add_awgn_to_signal(rec)
|
>>> new_rec = add_awgn_to_signal(rec)
|
||||||
|
|
@ -71,7 +71,7 @@ def time_shift(signal: ArrayLike | Recording, shift: Optional[int] = 1) -> np.nd
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param shift: The number of indices to shift by. Default is 1.
|
:param shift: The number of indices to shift by. Default is 1.
|
||||||
:type shift: int, optional
|
:type shift: int, optional
|
||||||
|
|
||||||
|
|
@ -80,7 +80,7 @@ def time_shift(signal: ArrayLike | Recording, shift: Optional[int] = 1) -> np.nd
|
||||||
|
|
||||||
:return: A numpy array which represents the time-shifted signal. If `signal` is a `Recording`,
|
:return: A numpy array which represents the time-shifted signal. If `signal` is a `Recording`,
|
||||||
returns a `Recording object` with its `data` attribute containing the time-shifted array.
|
returns a `Recording object` with its `data` attribute containing the time-shifted array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[1+1j, 2+2j, 3+3j, 4+4j, 5+5j]])
|
>>> rec = Recording(data=[[1+1j, 2+2j, 3+3j, 4+4j, 5+5j]])
|
||||||
>>> new_rec = time_shift(rec, -2)
|
>>> new_rec = time_shift(rec, -2)
|
||||||
|
|
@ -134,7 +134,7 @@ def frequency_shift(signal: ArrayLike | Recording, shift: Optional[float] = 0.5)
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param shift: The frequency shift relative to the sample rate. Must be in the range ``[-0.5, 0.5]``.
|
:param shift: The frequency shift relative to the sample rate. Must be in the range ``[-0.5, 0.5]``.
|
||||||
Default is 0.5.
|
Default is 0.5.
|
||||||
:type shift: float, optional
|
:type shift: float, optional
|
||||||
|
|
@ -144,7 +144,7 @@ def frequency_shift(signal: ArrayLike | Recording, shift: Optional[float] = 0.5)
|
||||||
|
|
||||||
:return: A numpy array which represents the frequency-shifted signal. If `signal` is a `Recording`,
|
:return: A numpy array which represents the frequency-shifted signal. If `signal` is a `Recording`,
|
||||||
returns a `Recording object` with its `data` attribute containing the frequency-shifted array.
|
returns a `Recording object` with its `data` attribute containing the frequency-shifted array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[1+1j, 2+2j, 3+3j, 4+4j]])
|
>>> rec = Recording(data=[[1+1j, 2+2j, 3+3j, 4+4j]])
|
||||||
>>> new_rec = frequency_shift(rec, -0.4)
|
>>> new_rec = frequency_shift(rec, -0.4)
|
||||||
|
|
@ -189,7 +189,7 @@ def phase_shift(signal: ArrayLike | Recording, phase: Optional[float] = np.pi) -
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param phase: The phase angle by which to rotate the IQ samples, in radians. Must be in the range ``[-π, π]``.
|
:param phase: The phase angle by which to rotate the IQ samples, in radians. Must be in the range ``[-π, π]``.
|
||||||
Default is π.
|
Default is π.
|
||||||
:type phase: float, optional
|
:type phase: float, optional
|
||||||
|
|
@ -199,7 +199,7 @@ def phase_shift(signal: ArrayLike | Recording, phase: Optional[float] = np.pi) -
|
||||||
|
|
||||||
:return: A numpy array which represents the phase-shifted signal. If `signal` is a `Recording`,
|
:return: A numpy array which represents the phase-shifted signal. If `signal` is a `Recording`,
|
||||||
returns a `Recording object` with its `data` attribute containing the phase-shifted array.
|
returns a `Recording object` with its `data` attribute containing the phase-shifted array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[1+1j, 2+2j, 3+3j, 4+4j]])
|
>>> rec = Recording(data=[[1+1j, 2+2j, 3+3j, 4+4j]])
|
||||||
>>> new_rec = phase_shift(rec, np.pi/2)
|
>>> new_rec = phase_shift(rec, np.pi/2)
|
||||||
|
|
@ -246,7 +246,7 @@ def iq_imbalance(
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param amplitude_imbalance: The IQ amplitude imbalance to apply, in dB. Default is 1.5.
|
:param amplitude_imbalance: The IQ amplitude imbalance to apply, in dB. Default is 1.5.
|
||||||
:type amplitude_imbalance: float, optional
|
:type amplitude_imbalance: float, optional
|
||||||
:param phase_imbalance: The IQ phase imbalance to apply, in radians. Default is π.
|
:param phase_imbalance: The IQ phase imbalance to apply, in radians. Default is π.
|
||||||
|
|
@ -260,7 +260,7 @@ def iq_imbalance(
|
||||||
|
|
||||||
:return: A numpy array which is the original signal with an applied IQ imbalance. If `signal` is a `Recording`,
|
:return: A numpy array which is the original signal with an applied IQ imbalance. If `signal` is a `Recording`,
|
||||||
returns a `Recording object` with its `data` attribute containing the IQ imbalanced signal array.
|
returns a `Recording object` with its `data` attribute containing the IQ imbalanced signal array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[2+18j, -34+2j, 3+9j]])
|
>>> rec = Recording(data=[[2+18j, -34+2j, 3+9j]])
|
||||||
>>> new_rec = iq_imbalance(rec, 1, np.pi, 2)
|
>>> new_rec = iq_imbalance(rec, 1, np.pi, 2)
|
||||||
|
|
@ -315,7 +315,7 @@ def resample(signal: ArrayLike | Recording, up: Optional[int] = 4, down: Optiona
|
||||||
|
|
||||||
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
:param signal: Input IQ data as a complex CxN array or `Recording`, where C is the number of channels and N
|
||||||
is the length of the IQ examples.
|
is the length of the IQ examples.
|
||||||
:type signal: array_like or ria_toolkit_oss.data.Recording
|
:type signal: array_like or ria_toolkit_oss.datatypes.Recording
|
||||||
:param up: The upsampling factor. Default is 4.
|
:param up: The upsampling factor. Default is 4.
|
||||||
:type up: int, optional
|
:type up: int, optional
|
||||||
:param down: The downsampling factor. Default is 2.
|
:param down: The downsampling factor. Default is 2.
|
||||||
|
|
@ -325,7 +325,7 @@ def resample(signal: ArrayLike | Recording, up: Optional[int] = 4, down: Optiona
|
||||||
|
|
||||||
:return: A numpy array which represents the resampled signal If `signal` is a `Recording`,
|
:return: A numpy array which represents the resampled signal If `signal` is a `Recording`,
|
||||||
returns a `Recording object` with its `data` attribute containing the resampled array.
|
returns a `Recording object` with its `data` attribute containing the resampled array.
|
||||||
:rtype: np.ndarray or ria_toolkit_oss.data.Recording
|
:rtype: np.ndarray or ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
>>> rec = Recording(data=[[1+1j, 2+2j]])
|
>>> rec = Recording(data=[[1+1j, 2+2j]])
|
||||||
>>> new_rec = resample(rec, 2, 1)
|
>>> new_rec = resample(rec, 2, 1)
|
||||||
|
|
|
||||||
|
|
@ -4,14 +4,14 @@ import scipy.signal as signal
|
||||||
from plotly.graph_objs import Figure
|
from plotly.graph_objs import Figure
|
||||||
from scipy.fft import fft, fftshift
|
from scipy.fft import fft, fftshift
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
|
||||||
def spectrogram(rec: Recording, thumbnail: bool = False) -> Figure:
|
def spectrogram(rec: Recording, thumbnail: bool = False) -> Figure:
|
||||||
"""Create a spectrogram for the recording.
|
"""Create a spectrogram for the recording.
|
||||||
|
|
||||||
:param rec: Signal to plot.
|
:param rec: Signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
:param thumbnail: Whether to return a small thumbnail version or full plot.
|
:param thumbnail: Whether to return a small thumbnail version or full plot.
|
||||||
:type thumbnail: bool
|
:type thumbnail: bool
|
||||||
|
|
||||||
|
|
@ -95,7 +95,7 @@ def iq_time_series(rec: Recording) -> Figure:
|
||||||
"""Create a time series plot of the real and imaginary parts of signal.
|
"""Create a time series plot of the real and imaginary parts of signal.
|
||||||
|
|
||||||
:param rec: Signal to plot.
|
:param rec: Signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: Time series plot as a Plotly figure.
|
:return: Time series plot as a Plotly figure.
|
||||||
"""
|
"""
|
||||||
|
|
@ -125,7 +125,7 @@ def frequency_spectrum(rec: Recording) -> Figure:
|
||||||
"""Create a frequency spectrum plot from the recording.
|
"""Create a frequency spectrum plot from the recording.
|
||||||
|
|
||||||
:param rec: Input signal to plot.
|
:param rec: Input signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: Frequency spectrum as a Plotly figure.
|
:return: Frequency spectrum as a Plotly figure.
|
||||||
"""
|
"""
|
||||||
|
|
@ -160,7 +160,7 @@ def constellation(rec: Recording) -> Figure:
|
||||||
"""Create a constellation plot from the recording.
|
"""Create a constellation plot from the recording.
|
||||||
|
|
||||||
:param rec: Input signal to plot.
|
:param rec: Input signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: Constellation as a Plotly figure.
|
:return: Constellation as a Plotly figure.
|
||||||
"""
|
"""
|
||||||
|
|
|
||||||
|
|
@ -32,15 +32,16 @@ def extract_metadata_fields(metadata):
|
||||||
|
|
||||||
|
|
||||||
def set_path(output_path):
|
def set_path(output_path):
|
||||||
path = pathlib.Path(output_path)
|
split_path = output_path.split("/")
|
||||||
|
|
||||||
# If only filename provided (no directory), use default 'images' folder
|
if len(split_path) == 1:
|
||||||
if len(path.parts) == 1:
|
folder = "images"
|
||||||
folder = pathlib.Path("images")
|
file = split_path[0]
|
||||||
file = path.name
|
elif len(split_path) > 2:
|
||||||
|
file = split_path[-1]
|
||||||
|
folder = "/".join(split_path[:-1])
|
||||||
else:
|
else:
|
||||||
folder = path.parent
|
folder, file = split_path
|
||||||
file = path.name
|
|
||||||
|
|
||||||
split_file = file.split(".")
|
split_file = file.split(".")
|
||||||
if len(split_file) == 2:
|
if len(split_file) == 2:
|
||||||
|
|
@ -52,5 +53,5 @@ def set_path(output_path):
|
||||||
extension = "png"
|
extension = "png"
|
||||||
file = file + ".png"
|
file = file + ".png"
|
||||||
|
|
||||||
folder.mkdir(parents=True, exist_ok=True)
|
pathlib.Path(folder).mkdir(parents=True, exist_ok=True)
|
||||||
return str(folder / file), extension
|
return "/".join([folder, file]), extension
|
||||||
|
|
|
||||||
|
|
@ -3,17 +3,16 @@ import os
|
||||||
import textwrap
|
import textwrap
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
import matplotlib
|
|
||||||
import matplotlib.pyplot as plt
|
import matplotlib.pyplot as plt
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from matplotlib import gridspec, ticker
|
from matplotlib import gridspec
|
||||||
from matplotlib.patches import Patch
|
from matplotlib.patches import Patch
|
||||||
from PIL import Image, UnidentifiedImageError
|
from PIL import Image
|
||||||
from scipy.fft import fft, fftshift
|
from scipy.fft import fft, fftshift
|
||||||
from scipy.signal import spectrogram
|
from scipy.signal import spectrogram
|
||||||
from scipy.signal.windows import hann
|
from scipy.signal.windows import hann
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.view.tools import (
|
from ria_toolkit_oss.view.tools import (
|
||||||
COLORS,
|
COLORS,
|
||||||
decimate,
|
decimate,
|
||||||
|
|
@ -81,8 +80,6 @@ def view_annotations(
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
for annotation in sorted(annotations, key=_threshold_sort_key, reverse=True):
|
for annotation in sorted(annotations, key=_threshold_sort_key, reverse=True):
|
||||||
if annotation.freq_lower_edge is None or annotation.freq_upper_edge is None:
|
|
||||||
continue
|
|
||||||
t_start = annotation.sample_start / sample_rate
|
t_start = annotation.sample_start / sample_rate
|
||||||
t_width = annotation.sample_count / sample_rate
|
t_width = annotation.sample_count / sample_rate
|
||||||
f_start = annotation.freq_lower_edge
|
f_start = annotation.freq_lower_edge
|
||||||
|
|
@ -188,7 +185,7 @@ def view_sig(
|
||||||
logo: Optional[bool] = True,
|
logo: Optional[bool] = True,
|
||||||
dark: Optional[bool] = True,
|
dark: Optional[bool] = True,
|
||||||
spines: Optional[bool] = False,
|
spines: Optional[bool] = False,
|
||||||
title_fontsize: Optional[int] = 25,
|
title_fontsize: Optional[int] = 35,
|
||||||
subtitle_fontsize: Optional[int] = 15,
|
subtitle_fontsize: Optional[int] = 15,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""
|
"""
|
||||||
|
|
@ -233,26 +230,11 @@ def view_sig(
|
||||||
complex_signal = recording.data[0]
|
complex_signal = recording.data[0]
|
||||||
sample_rate, center_frequency, _ = extract_metadata_fields(recording.metadata)
|
sample_rate, center_frequency, _ = extract_metadata_fields(recording.metadata)
|
||||||
|
|
||||||
subplot_height = 3 * (plot_spectrogram) + 2 * (iq + frequency) + 3 * (constellation or metadata or logo)
|
subplot_height = 2 * (plot_spectrogram + iq + frequency) + 3 * (constellation or metadata or logo)
|
||||||
subplot_width = max((constellation + metadata or 1), logo * 3)
|
subplot_width = max((constellation + metadata or 1), logo * 3)
|
||||||
|
|
||||||
if dark:
|
if dark:
|
||||||
plt.style.use("dark_background")
|
plt.style.use("dark_background")
|
||||||
matplotlib.rcParams.update(
|
|
||||||
{
|
|
||||||
"figure.facecolor": "#161616",
|
|
||||||
"axes.facecolor": "#161616",
|
|
||||||
"savefig.facecolor": "#161616",
|
|
||||||
"savefig.edgecolor": "#161616",
|
|
||||||
"font.size": 10,
|
|
||||||
"axes.titlesize": 15,
|
|
||||||
"axes.labelsize": 10,
|
|
||||||
"xtick.labelsize": 10,
|
|
||||||
"ytick.labelsize": 10,
|
|
||||||
"legend.frameon": False,
|
|
||||||
"legend.facecolor": "none",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
logo_path = os.path.dirname(__file__) + "/graphics/Qoherent-logo-white-transparent.png"
|
logo_path = os.path.dirname(__file__) + "/graphics/Qoherent-logo-white-transparent.png"
|
||||||
else:
|
else:
|
||||||
plt.style.use("default")
|
plt.style.use("default")
|
||||||
|
|
@ -270,8 +252,8 @@ def view_sig(
|
||||||
plot_x_indx = 0
|
plot_x_indx = 0
|
||||||
|
|
||||||
if plot_spectrogram:
|
if plot_spectrogram:
|
||||||
spec_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 3, :])
|
spec_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
|
||||||
plot_y_indx = plot_y_indx + 3
|
plot_y_indx = plot_y_indx + 2
|
||||||
fft_size = get_fft_size(plot_length=plot_length)
|
fft_size = get_fft_size(plot_length=plot_length)
|
||||||
|
|
||||||
_, t_spec, Sxx = spectrogram(
|
_, t_spec, Sxx = spectrogram(
|
||||||
|
|
@ -298,10 +280,7 @@ def view_sig(
|
||||||
)
|
)
|
||||||
|
|
||||||
set_spines(spec_ax, spines)
|
set_spines(spec_ax, spines)
|
||||||
spec_ax.set_title("Spectrogram", loc="left", fontsize=subtitle_fontsize)
|
spec_ax.set_title("Spectrogram", loc="center", fontsize=subtitle_fontsize)
|
||||||
spec_ax.set_xlabel("Time (s)")
|
|
||||||
spec_ax.set_ylabel("Frequency (MHz)")
|
|
||||||
spec_ax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
|
||||||
|
|
||||||
if iq:
|
if iq:
|
||||||
iq_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
|
iq_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
|
||||||
|
|
@ -312,13 +291,12 @@ def view_sig(
|
||||||
|
|
||||||
iq_ax.plot(t, plot_iq.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
|
iq_ax.plot(t, plot_iq.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
|
||||||
iq_ax.plot(t, plot_iq.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
|
iq_ax.plot(t, plot_iq.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
|
||||||
iq_ax.grid(True, alpha=0.2, linewidth=0.5)
|
iq_ax.grid(False)
|
||||||
|
|
||||||
iq_ax.set_ylabel("Amplitude")
|
iq_ax.set_ylabel("Amplitude")
|
||||||
iq_ax.set_xlim([min(t), max(t)])
|
iq_ax.set_xlim([min(t), max(t)])
|
||||||
iq_ax.set_xlabel("Time (s)")
|
iq_ax.set_xlabel("Time (s)")
|
||||||
iq_ax.set_title("IQ Sample Plot", loc="left", fontsize=subtitle_fontsize)
|
iq_ax.set_title("IQ Sample Plot", fontsize=subtitle_fontsize)
|
||||||
iq_ax.legend(loc="upper right", fontsize=10)
|
|
||||||
set_spines(iq_ax, spines)
|
set_spines(iq_ax, spines)
|
||||||
|
|
||||||
if frequency:
|
if frequency:
|
||||||
|
|
@ -332,14 +310,10 @@ def view_sig(
|
||||||
# Convert to dB
|
# Convert to dB
|
||||||
spectrum_db = 20 * np.log10(spectrum + 1e-12) # 20*log for magnitude
|
spectrum_db = 20 * np.log10(spectrum + 1e-12) # 20*log for magnitude
|
||||||
|
|
||||||
freqs = (
|
freqs = np.linspace(-sample_rate / 2, sample_rate / 2, len(complex_signal[:plot_length])) + center_frequency
|
||||||
np.linspace(-sample_rate / 2, sample_rate / 2, len(complex_signal[:plot_length])) + center_frequency
|
|
||||||
) / 1e6
|
|
||||||
freq_ax.plot(freqs, spectrum_db, color=COLORS["accent"], linewidth=0.8)
|
freq_ax.plot(freqs, spectrum_db, color=COLORS["accent"], linewidth=0.8)
|
||||||
freq_ax.set_xlabel("Frequency (MHz)")
|
|
||||||
freq_ax.set_ylabel("Magnitude (dB)")
|
freq_ax.set_ylabel("Magnitude (dB)")
|
||||||
freq_ax.grid(True, alpha=0.2, linewidth=0.5)
|
freq_ax.set_title("Frequency Spectrum (Windowed FFT)", fontsize=subtitle_fontsize)
|
||||||
freq_ax.set_title("Frequency Spectrum (Windowed FFT)", loc="left", fontsize=subtitle_fontsize)
|
|
||||||
set_spines(freq_ax, spines)
|
set_spines(freq_ax, spines)
|
||||||
|
|
||||||
if constellation:
|
if constellation:
|
||||||
|
|
@ -352,7 +326,7 @@ def view_sig(
|
||||||
const_ax.set_ylim([-1 * dimension, dimension])
|
const_ax.set_ylim([-1 * dimension, dimension])
|
||||||
const_ax.set_xlabel("In-phase (I)")
|
const_ax.set_xlabel("In-phase (I)")
|
||||||
const_ax.set_ylabel("Quadrature (Q)")
|
const_ax.set_ylabel("Quadrature (Q)")
|
||||||
const_ax.set_title("Constellation", loc="left", fontsize=subtitle_fontsize)
|
const_ax.set_title("Constellation", fontsize=subtitle_fontsize)
|
||||||
const_ax.set_aspect("equal")
|
const_ax.set_aspect("equal")
|
||||||
|
|
||||||
if not spines:
|
if not spines:
|
||||||
|
|
@ -401,8 +375,8 @@ def view_sig(
|
||||||
image = Image.open(logo_path) # Open the PNG image using PIL
|
image = Image.open(logo_path) # Open the PNG image using PIL
|
||||||
logo_ax.imshow(image)
|
logo_ax.imshow(image)
|
||||||
|
|
||||||
except (FileNotFoundError, UnidentifiedImageError, OSError) as exc:
|
except FileNotFoundError:
|
||||||
print(f"Warning, could not load logo image: {logo_path}. Reason: {exc}")
|
print(f"Warning, {logo_path} not found.")
|
||||||
|
|
||||||
fig.subplots_adjust(
|
fig.subplots_adjust(
|
||||||
left=0.1, # Left margin
|
left=0.1, # Left margin
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,7 @@ import numpy as np
|
||||||
from scipy.fft import fft, fftshift
|
from scipy.fft import fft, fftshift
|
||||||
from scipy.signal.windows import hann
|
from scipy.signal.windows import hann
|
||||||
|
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.view.tools import (
|
from ria_toolkit_oss.view.tools import (
|
||||||
COLORS,
|
COLORS,
|
||||||
decimate,
|
decimate,
|
||||||
|
|
@ -119,19 +119,24 @@ def setup_style(*, labels_mode: bool = False, compact_mode: bool = False) -> Non
|
||||||
label_font = 14
|
label_font = 14
|
||||||
else:
|
else:
|
||||||
base_font = 10
|
base_font = 10
|
||||||
title_font = 15
|
title_font = 12
|
||||||
label_font = 10
|
label_font = 10
|
||||||
|
|
||||||
matplotlib.rcParams.update(
|
matplotlib.rcParams.update(
|
||||||
{
|
{
|
||||||
"figure.facecolor": "#161616",
|
"figure.facecolor": "#0f172a",
|
||||||
"axes.facecolor": "#161616",
|
"axes.facecolor": "#1e293b",
|
||||||
"savefig.facecolor": "#161616",
|
"axes.edgecolor": COLORS["muted"],
|
||||||
"savefig.edgecolor": "#161616",
|
"axes.labelcolor": COLORS["light"],
|
||||||
|
"text.color": COLORS["light"],
|
||||||
|
"xtick.color": COLORS["muted"],
|
||||||
|
"ytick.color": COLORS["muted"],
|
||||||
|
"grid.color": COLORS["muted"],
|
||||||
|
"grid.alpha": 0.3,
|
||||||
"font.size": base_font,
|
"font.size": base_font,
|
||||||
"axes.titlesize": title_font,
|
"axes.titlesize": title_font,
|
||||||
"axes.labelsize": label_font,
|
"axes.labelsize": label_font,
|
||||||
"figure.titlesize": title_font + 4,
|
"figure.titlesize": title_font + 2,
|
||||||
"legend.frameon": False,
|
"legend.frameon": False,
|
||||||
"legend.facecolor": "none",
|
"legend.facecolor": "none",
|
||||||
"xtick.labelsize": base_font,
|
"xtick.labelsize": base_font,
|
||||||
|
|
@ -189,7 +194,7 @@ def view_simple_sig(
|
||||||
constellation_mode: Optional[bool] = False,
|
constellation_mode: Optional[bool] = False,
|
||||||
labels_mode: Optional[bool] = False,
|
labels_mode: Optional[bool] = False,
|
||||||
slice: Optional[tuple] = None,
|
slice: Optional[tuple] = None,
|
||||||
title: Optional[str] = "Signal Plot",
|
title: Optional[str] = "Signal",
|
||||||
):
|
):
|
||||||
"""
|
"""
|
||||||
Create a simple plot of various signal visualizations as a png or svg image.
|
Create a simple plot of various signal visualizations as a png or svg image.
|
||||||
|
|
@ -232,7 +237,7 @@ def view_simple_sig(
|
||||||
spec_signal = signal
|
spec_signal = signal
|
||||||
|
|
||||||
if compact_mode:
|
if compact_mode:
|
||||||
fig, (ax2, ax1) = plt.subplots(2, 1, figsize=(12, 6), gridspec_kw={"height_ratios": [5, 1]})
|
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 6), gridspec_kw={"height_ratios": [1, 5]})
|
||||||
show_title = False
|
show_title = False
|
||||||
show_labels = False
|
show_labels = False
|
||||||
ax_constellation = ax_psd = None
|
ax_constellation = ax_psd = None
|
||||||
|
|
@ -248,24 +253,25 @@ def view_simple_sig(
|
||||||
ax_psd = None
|
ax_psd = None
|
||||||
else:
|
else:
|
||||||
if constellation_mode:
|
if constellation_mode:
|
||||||
fig, ((ax2, ax1), (ax3, ax4)) = plt.subplots(2, 2, figsize=(16, 12))
|
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(16, 12))
|
||||||
ax_constellation, ax_psd = ax3, ax4
|
ax_constellation, ax_psd = ax3, ax4
|
||||||
else:
|
else:
|
||||||
fig, (ax2, ax1) = plt.subplots(2, 1, figsize=(14, 10))
|
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 10))
|
||||||
ax_constellation = ax_psd = None
|
ax_constellation = ax_psd = None
|
||||||
show_title = True
|
show_title = True
|
||||||
show_labels = labels_mode
|
show_labels = labels_mode
|
||||||
|
|
||||||
if show_title:
|
if show_title:
|
||||||
fig.suptitle(title, fontsize=25)
|
fig.suptitle(title, fontsize=16, color=COLORS["light"], y=0.96)
|
||||||
fig.patch.set_facecolor(matplotlib.rcParams["figure.facecolor"])
|
fig.patch.set_facecolor("#0f172a")
|
||||||
|
|
||||||
total_duration_s = len(signal) / sample_rate_hz if sample_rate_hz else 0.0
|
total_duration_s = len(signal) / sample_rate_hz if sample_rate_hz else 0.0
|
||||||
t_s = np.linspace(0, total_duration_s, len(display_signal)) if len(display_signal) else np.array([])
|
t_s = np.linspace(0, total_duration_s, len(display_signal)) if len(display_signal) else np.array([])
|
||||||
|
|
||||||
ax1.plot(t_s, display_signal.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
|
ax1.plot(t_s, display_signal.real, color=COLORS["purple"], linewidth=0.8, alpha=0.8, label="I")
|
||||||
ax1.plot(t_s, display_signal.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
|
ax1.plot(t_s, display_signal.imag, color=COLORS["magenta"], linewidth=0.8, alpha=0.8, label="Q")
|
||||||
ax1.grid(True, alpha=0.2, linewidth=0.5)
|
ax1.set_xlim(0, total_duration_s)
|
||||||
|
ax1.grid(True, alpha=0.3)
|
||||||
|
|
||||||
nfft, overlap = _get_nfft_size(signal=signal, fast_mode=fast_mode)
|
nfft, overlap = _get_nfft_size(signal=signal, fast_mode=fast_mode)
|
||||||
|
|
||||||
|
|
@ -279,7 +285,7 @@ def view_simple_sig(
|
||||||
)
|
)
|
||||||
|
|
||||||
ax2.set_ylim(center_freq_hz - sample_rate_hz / 2, center_freq_hz + sample_rate_hz / 2)
|
ax2.set_ylim(center_freq_hz - sample_rate_hz / 2, center_freq_hz + sample_rate_hz / 2)
|
||||||
ax1.set_xlim(ax2.get_xlim())
|
ax2.set_xlim(0, total_duration_s)
|
||||||
|
|
||||||
if show_labels:
|
if show_labels:
|
||||||
if horizontal_mode:
|
if horizontal_mode:
|
||||||
|
|
@ -288,25 +294,20 @@ def view_simple_sig(
|
||||||
ax2.set_xlabel("Time (s)")
|
ax2.set_xlabel("Time (s)")
|
||||||
|
|
||||||
ax1.set_ylabel("Amplitude")
|
ax1.set_ylabel("Amplitude")
|
||||||
ax1.set_title(f"IQ Sample Plot - {sdr} SDR", loc="left", pad=10, fontsize=15)
|
ax1.set_title(f"Time Series - {sdr} SDR", loc="left", pad=10)
|
||||||
ax1.legend(loc="upper right", fontsize=10)
|
ax1.legend(loc="upper right")
|
||||||
|
|
||||||
ax2.set_ylabel("Frequency (MHz)")
|
ax2.set_ylabel("Frequency (Hz)")
|
||||||
ax2.set_title(
|
ax2.set_title(
|
||||||
f"Spectrogram - {center_freq_hz / 1e6:.1f} MHz ± {sample_rate_hz / 2e6:.1f} MHz",
|
f"Spectrogram - {center_freq_hz / 1e6:.1f} MHz ± {sample_rate_hz / 2e6:.1f} MHz", loc="left", pad=10
|
||||||
loc="left",
|
|
||||||
pad=10,
|
|
||||||
fontsize=15,
|
|
||||||
)
|
)
|
||||||
ax2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
yticks = ax2.get_yticks()
|
||||||
|
ax2.set_yticklabels([f"{y / 1e6:.1f}" for y in yticks])
|
||||||
elif not compact_mode:
|
elif not compact_mode:
|
||||||
ax1.set_title("IQ Sample Plot", loc="left", pad=10, fontsize=15)
|
ax1.set_title("Time Series", loc="left", pad=10)
|
||||||
ax1.legend(loc="upper right", fontsize=10)
|
ax1.legend(loc="upper right", fontsize=8)
|
||||||
|
|
||||||
ax2.set_xlabel("Time (s)")
|
ax2.set_title("Spectrogram", loc="left", pad=10)
|
||||||
ax2.set_ylabel("Frequency (MHz)")
|
|
||||||
ax2.set_title("Spectrogram", loc="left", pad=10, fontsize=15)
|
|
||||||
ax2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
|
||||||
|
|
||||||
_add_annotations(
|
_add_annotations(
|
||||||
annotations=annotations,
|
annotations=annotations,
|
||||||
|
|
@ -338,8 +339,8 @@ def view_simple_sig(
|
||||||
)
|
)
|
||||||
ax_constellation.set_xlabel("In-phase (I)")
|
ax_constellation.set_xlabel("In-phase (I)")
|
||||||
ax_constellation.set_ylabel("Quadrature (Q)")
|
ax_constellation.set_ylabel("Quadrature (Q)")
|
||||||
ax_constellation.set_title("Constellation", loc="left", fontsize=15)
|
ax_constellation.set_title("Constellation")
|
||||||
ax_constellation.grid(True, alpha=0.2, linewidth=0.5)
|
ax_constellation.grid(True, alpha=0.3)
|
||||||
ax_constellation.set_aspect("equal")
|
ax_constellation.set_aspect("equal")
|
||||||
|
|
||||||
if ax_psd is not None:
|
if ax_psd is not None:
|
||||||
|
|
@ -350,11 +351,11 @@ def view_simple_sig(
|
||||||
freqs = freqs + center_freq_hz
|
freqs = freqs + center_freq_hz
|
||||||
spectrum_db = 10 * np.log10(spectrum + 1e-12)
|
spectrum_db = 10 * np.log10(spectrum + 1e-12)
|
||||||
|
|
||||||
ax_psd.plot(freqs / 1e6, spectrum_db, color=COLORS["accent"], linewidth=0.8)
|
ax_psd.plot(freqs / 1e6, spectrum_db, color=COLORS["accent"], linewidth=1.0)
|
||||||
ax_psd.set_xlabel("Frequency (MHz)")
|
ax_psd.set_xlabel("Frequency (MHz)")
|
||||||
ax_psd.set_ylabel("Power (dB)")
|
ax_psd.set_ylabel("Power (dB)")
|
||||||
ax_psd.set_title("Power Spectral Density", loc="left", fontsize=15)
|
ax_psd.set_title("Power Spectral Density")
|
||||||
ax_psd.grid(True, alpha=0.2, linewidth=0.5)
|
ax_psd.grid(True, alpha=0.3)
|
||||||
|
|
||||||
if compact_mode:
|
if compact_mode:
|
||||||
ax1.set_xticks([])
|
ax1.set_xticks([])
|
||||||
|
|
@ -366,20 +367,13 @@ def view_simple_sig(
|
||||||
else:
|
else:
|
||||||
plt.tight_layout()
|
plt.tight_layout()
|
||||||
if show_title:
|
if show_title:
|
||||||
plt.subplots_adjust(top=0.9)
|
plt.subplots_adjust(top=0.92)
|
||||||
|
|
||||||
if saveplot:
|
if saveplot:
|
||||||
output_path, extension = set_path(output_path=output_path)
|
output_path, extension = set_path(output_path=output_path)
|
||||||
dpi_value = _set_dpi(fast_mode=fast_mode, labels_mode=labels_mode, extension=extension)
|
dpi_value = _set_dpi(fast_mode=fast_mode, labels_mode=labels_mode, extension=extension)
|
||||||
|
|
||||||
plt.savefig(
|
plt.savefig(output_path, dpi=dpi_value, bbox_inches="tight", facecolor="#0f172a", edgecolor="none")
|
||||||
output_path,
|
|
||||||
dpi=dpi_value,
|
|
||||||
bbox_inches="tight",
|
|
||||||
pad_inches=0.3,
|
|
||||||
facecolor=matplotlib.rcParams["savefig.facecolor"],
|
|
||||||
edgecolor=matplotlib.rcParams["savefig.edgecolor"],
|
|
||||||
)
|
|
||||||
print(f"Saved signal plot to {output_path}")
|
print(f"Saved signal plot to {output_path}")
|
||||||
return output_path
|
return output_path
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,14 +4,14 @@ import scipy.signal as signal
|
||||||
from plotly.graph_objs import Figure
|
from plotly.graph_objs import Figure
|
||||||
from scipy.fft import fft, fftshift
|
from scipy.fft import fft, fftshift
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
|
|
||||||
|
|
||||||
def spectrogram(rec: Recording, thumbnail: bool = False) -> Figure:
|
def spectrogram(rec: Recording, thumbnail: bool = False) -> Figure:
|
||||||
"""Create a spectrogram for the recording.
|
"""Create a spectrogram for the recording.
|
||||||
|
|
||||||
:param rec: Signal to plot.
|
:param rec: Signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
:param thumbnail: Whether to return a small thumbnail version or full plot.
|
:param thumbnail: Whether to return a small thumbnail version or full plot.
|
||||||
:type thumbnail: bool
|
:type thumbnail: bool
|
||||||
|
|
||||||
|
|
@ -107,7 +107,7 @@ def iq_time_series(rec: Recording) -> Figure:
|
||||||
"""Create a time series plot of the real and imaginary parts of signal.
|
"""Create a time series plot of the real and imaginary parts of signal.
|
||||||
|
|
||||||
:param rec: Signal to plot.
|
:param rec: Signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: Time series plot, as a Plotly Figure.
|
:return: Time series plot, as a Plotly Figure.
|
||||||
"""
|
"""
|
||||||
|
|
@ -145,7 +145,7 @@ def frequency_spectrum(rec: Recording) -> Figure:
|
||||||
"""Create a frequency spectrum plot from the recording.
|
"""Create a frequency spectrum plot from the recording.
|
||||||
|
|
||||||
:param rec: Input signal to plot.
|
:param rec: Input signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: Frequency spectrum, as a Plotly figure.
|
:return: Frequency spectrum, as a Plotly figure.
|
||||||
"""
|
"""
|
||||||
|
|
@ -187,7 +187,7 @@ def constellation(rec: Recording) -> Figure:
|
||||||
"""Create a constellation plot from the recording.
|
"""Create a constellation plot from the recording.
|
||||||
|
|
||||||
:param rec: Input signal to plot.
|
:param rec: Input signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: Constellation, as a Plotly Figure.
|
:return: Constellation, as a Plotly Figure.
|
||||||
"""
|
"""
|
||||||
|
|
@ -222,7 +222,7 @@ def power_spectral_density(rec: Recording) -> Figure:
|
||||||
"""Create a Power Spectral Density (PSD) plot from the recording.
|
"""Create a Power Spectral Density (PSD) plot from the recording.
|
||||||
|
|
||||||
:param rec: Input signal to plot.
|
:param rec: Input signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: PSD plot, as a Plotly Figure.
|
:return: PSD plot, as a Plotly Figure.
|
||||||
"""
|
"""
|
||||||
|
|
@ -268,7 +268,7 @@ def fft_plot(rec: Recording) -> Figure:
|
||||||
"""Create an FFT magnitude plot from the recording.
|
"""Create an FFT magnitude plot from the recording.
|
||||||
|
|
||||||
:param rec: Input signal to plot.
|
:param rec: Input signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: FFT plot, as a Plotly Figure.
|
:return: FFT plot, as a Plotly Figure.
|
||||||
"""
|
"""
|
||||||
|
|
@ -312,7 +312,7 @@ def spectrogram_3d(rec: Recording) -> Figure:
|
||||||
"""Create a 3D spectrogram plot from the recording.
|
"""Create a 3D spectrogram plot from the recording.
|
||||||
|
|
||||||
:param rec: Input signal to plot.
|
:param rec: Input signal to plot.
|
||||||
:type rec: ria_toolkit_oss.data.Recording
|
:type rec: ria_toolkit_oss.datatypes.Recording
|
||||||
|
|
||||||
:return: 3D Spectrogram, as a Plotly Figure.
|
:return: 3D Spectrogram, as a Plotly Figure.
|
||||||
"""
|
"""
|
||||||
|
|
|
||||||
|
|
@ -2,57 +2,15 @@
|
||||||
This module contains the main group for the ria toolkit oss CLI.
|
This module contains the main group for the ria toolkit oss CLI.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import subprocess
|
import click
|
||||||
import sys
|
|
||||||
import warnings
|
|
||||||
|
|
||||||
warnings.filterwarnings(
|
from ria_toolkit_oss_cli.ria_toolkit_oss import commands
|
||||||
"ignore",
|
|
||||||
message="Unable to import Axes3D",
|
|
||||||
category=UserWarning,
|
|
||||||
module="matplotlib",
|
|
||||||
)
|
|
||||||
|
|
||||||
import click # noqa: E402
|
|
||||||
|
|
||||||
from ria_toolkit_oss_cli.ria_toolkit_oss import commands # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def _git_lfs_installed() -> bool:
|
@click.group()
|
||||||
"""Return True if git-lfs is available on PATH."""
|
|
||||||
try:
|
|
||||||
return (
|
|
||||||
subprocess.run(
|
|
||||||
["git", "lfs", "version"],
|
|
||||||
capture_output=True,
|
|
||||||
).returncode
|
|
||||||
== 0
|
|
||||||
)
|
|
||||||
except FileNotFoundError:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
@click.group(invoke_without_command=True)
|
|
||||||
@click.option("-v", "--verbose", is_flag=True, type=bool, help="Increase verbosity, especially useful for debugging.")
|
@click.option("-v", "--verbose", is_flag=True, type=bool, help="Increase verbosity, especially useful for debugging.")
|
||||||
@click.pass_context
|
def cli(verbose):
|
||||||
def cli(ctx, verbose):
|
pass
|
||||||
lfs_missing = not _git_lfs_installed()
|
|
||||||
if lfs_missing:
|
|
||||||
click.echo(
|
|
||||||
"Warning: git-lfs is not installed. RIA Hub projects require git-lfs to\n"
|
|
||||||
"track large binary files (models, recordings, datasets).\n"
|
|
||||||
"\n"
|
|
||||||
" Linux: sudo apt-get install git-lfs\n"
|
|
||||||
" macOS: brew install git-lfs\n"
|
|
||||||
" Other platforms: https://git-lfs.com\n"
|
|
||||||
"\n"
|
|
||||||
"After installing, run: git lfs install",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
if ctx.invoked_subcommand is None:
|
|
||||||
if lfs_missing and sys.stdin.isatty():
|
|
||||||
click.pause(info="\nPress Enter to continue...", err=True)
|
|
||||||
click.echo(ctx.get_help())
|
|
||||||
|
|
||||||
|
|
||||||
# Loop through project commands, binding them all to the CLI.
|
# Loop through project commands, binding them all to the CLI.
|
||||||
|
|
|
||||||
|
|
@ -1,97 +0,0 @@
|
||||||
"""Shared authentication and security helpers for RIA Hub API calls."""
|
|
||||||
|
|
||||||
import base64
|
|
||||||
import subprocess
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
DEFAULT_HUB = "https://riahub.ai"
|
|
||||||
|
|
||||||
|
|
||||||
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
|
|
||||||
"""Block redirects on authenticated requests to prevent credential exfiltration.
|
|
||||||
|
|
||||||
urllib re-sends the Authorization header on same-host redirects by default.
|
|
||||||
A malicious server could redirect a POST to a different host to harvest
|
|
||||||
credentials. We refuse all redirects — API clients should not encounter them
|
|
||||||
in normal operation.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
|
||||||
raise urllib.error.URLError(f"Unexpected redirect ({code}) to {newurl} — aborting to protect credentials")
|
|
||||||
|
|
||||||
|
|
||||||
def hub_opener() -> urllib.request.OpenerDirector:
|
|
||||||
"""Return a urllib opener that blocks redirects."""
|
|
||||||
return urllib.request.build_opener(_NoRedirectHandler)
|
|
||||||
|
|
||||||
|
|
||||||
def warn_if_insecure(hub: str) -> None:
|
|
||||||
"""Warn when credentials would be sent over plain HTTP to a non-localhost host."""
|
|
||||||
parsed = urllib.parse.urlparse(hub)
|
|
||||||
if parsed.scheme == "http":
|
|
||||||
host = parsed.hostname or ""
|
|
||||||
if host not in ("localhost", "127.0.0.1", "::1"):
|
|
||||||
click.echo(
|
|
||||||
f"Warning: sending credentials over plain HTTP to {host}. " "Use HTTPS in production.",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def basic_auth(username: str, password: str) -> str:
|
|
||||||
return "Basic " + base64.b64encode(f"{username}:{password}".encode()).decode()
|
|
||||||
|
|
||||||
|
|
||||||
def get_stored_credentials(hub_url: str) -> tuple[str | None, str | None]:
|
|
||||||
"""Ask git credential fill for stored creds. Returns (username, password) or (None, None)."""
|
|
||||||
parsed = urllib.parse.urlparse(hub_url)
|
|
||||||
payload = f"protocol={parsed.scheme}\nhost={parsed.netloc}\n\n"
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
["git", "credential", "fill"],
|
|
||||||
input=payload,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=5,
|
|
||||||
)
|
|
||||||
creds = {}
|
|
||||||
for line in result.stdout.splitlines():
|
|
||||||
# Partition on the FIRST '=' only so passwords containing '=' are preserved.
|
|
||||||
k, sep, v = line.partition("=")
|
|
||||||
if sep:
|
|
||||||
creds[k.strip()] = v # keep value verbatim
|
|
||||||
return creds.get("username"), creds.get("password")
|
|
||||||
except Exception:
|
|
||||||
return None, None
|
|
||||||
|
|
||||||
|
|
||||||
def store_credentials(hub_url: str, username: str, password: str) -> None:
|
|
||||||
"""Cache credentials via git credential approve (uses the system keychain/store)."""
|
|
||||||
parsed = urllib.parse.urlparse(hub_url)
|
|
||||||
payload = (
|
|
||||||
f"protocol={parsed.scheme}\n" f"host={parsed.netloc}\n" f"username={username}\n" f"password={password}\n\n"
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
subprocess.run(
|
|
||||||
["git", "credential", "approve"],
|
|
||||||
input=payload,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=5,
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
pass # non-fatal — next push just prompts again
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_credentials(hub: str) -> tuple[str, str]:
|
|
||||||
"""Return (username, password), prompting interactively if not cached."""
|
|
||||||
username, password = get_stored_credentials(hub)
|
|
||||||
if username and password:
|
|
||||||
return username, password
|
|
||||||
click.echo(f"No stored credentials found for {hub}.")
|
|
||||||
username = click.prompt("RIA Hub username")
|
|
||||||
password = click.prompt("Password / personal access token", hide_input=True)
|
|
||||||
return username, password
|
|
||||||
|
|
@ -11,8 +11,8 @@ from ria_toolkit_oss.annotations import (
|
||||||
split_recording_annotations,
|
split_recording_annotations,
|
||||||
threshold_qualifier,
|
threshold_qualifier,
|
||||||
)
|
)
|
||||||
from ria_toolkit_oss.data import Annotation
|
from ria_toolkit_oss.datatypes import Annotation
|
||||||
from ria_toolkit_oss.data.recording import Recording
|
from ria_toolkit_oss.datatypes.recording import Recording
|
||||||
from ria_toolkit_oss.io import load_recording, to_blue, to_npy, to_sigmf, to_wav
|
from ria_toolkit_oss.io import load_recording, to_blue, to_npy, to_sigmf, to_wav
|
||||||
from ria_toolkit_oss_cli.ria_toolkit_oss.common import (
|
from ria_toolkit_oss_cli.ria_toolkit_oss.common import (
|
||||||
format_frequency,
|
format_frequency,
|
||||||
|
|
@ -51,7 +51,7 @@ def detect_input_format(filepath):
|
||||||
raise click.ClickException(f"Unknown format for '{filepath}'. Supported: .sigmf, .npy, .wav, .blue")
|
raise click.ClickException(f"Unknown format for '{filepath}'. Supported: .sigmf, .npy, .wav, .blue")
|
||||||
|
|
||||||
|
|
||||||
def determine_output_path(input_path, output_path, fmt, overwrite):
|
def determine_output_path(input_path, output_path, fmt, quiet, overwrite):
|
||||||
input_path = Path(input_path)
|
input_path = Path(input_path)
|
||||||
input_is_annotated = input_path.stem.endswith("_annotated")
|
input_is_annotated = input_path.stem.endswith("_annotated")
|
||||||
|
|
||||||
|
|
@ -63,20 +63,24 @@ def determine_output_path(input_path, output_path, fmt, overwrite):
|
||||||
else:
|
else:
|
||||||
target = input_path.with_name(f"{input_path.stem}_annotated{input_path.suffix}")
|
target = input_path.with_name(f"{input_path.stem}_annotated{input_path.suffix}")
|
||||||
|
|
||||||
final_path = normalize_sigmf_path(target) if fmt == "sigmf" else target
|
if fmt == "sigmf":
|
||||||
|
final_path = normalize_sigmf_path(target)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(f"Saving SigMF metadata to: {final_path}")
|
||||||
|
else:
|
||||||
|
final_path = target
|
||||||
|
if not quiet:
|
||||||
|
click.echo(f"Saving to: {final_path}")
|
||||||
|
|
||||||
if final_path.exists() and not overwrite:
|
# Always allow writing to _annotated files; guard against overwriting originals
|
||||||
raise click.ClickException(f"{final_path} already exists. Use --overwrite to replace it.")
|
target_is_annotated = final_path.stem.endswith("_annotated")
|
||||||
|
if final_path.exists() and not target_is_annotated and final_path != input_path:
|
||||||
|
click.echo(f"Error: {final_path} is not an annotated file and cannot be overwritten.", err=True)
|
||||||
|
return None
|
||||||
|
|
||||||
return final_path
|
return final_path
|
||||||
|
|
||||||
|
|
||||||
def check_output_available(input_path, output_path, overwrite):
|
|
||||||
"""Raise ClickException before any work begins if the output file already exists."""
|
|
||||||
fmt = detect_input_format(Path(input_path))
|
|
||||||
determine_output_path(input_path=input_path, output_path=output_path, fmt=fmt, overwrite=overwrite)
|
|
||||||
|
|
||||||
|
|
||||||
def save_recording_auto(recording, output_path, input_path, quiet=False, overwrite=False):
|
def save_recording_auto(recording, output_path, input_path, quiet=False, overwrite=False):
|
||||||
"""Save recording, auto-detecting format from extension.
|
"""Save recording, auto-detecting format from extension.
|
||||||
|
|
||||||
|
|
@ -86,13 +90,10 @@ def save_recording_auto(recording, output_path, input_path, quiet=False, overwri
|
||||||
input_path = Path(input_path)
|
input_path = Path(input_path)
|
||||||
fmt = detect_input_format(input_path)
|
fmt = detect_input_format(input_path)
|
||||||
|
|
||||||
output_path = determine_output_path(input_path=input_path, output_path=output_path, fmt=fmt, overwrite=overwrite)
|
# Determine output path
|
||||||
|
output_path = determine_output_path(
|
||||||
if not quiet:
|
input_path=input_path, output_path=output_path, fmt=fmt, quiet=quiet, overwrite=overwrite
|
||||||
if fmt == "sigmf":
|
)
|
||||||
click.echo(f"Saving SigMF metadata to: {output_path}")
|
|
||||||
else:
|
|
||||||
click.echo(f"Saving to: {output_path}")
|
|
||||||
|
|
||||||
if fmt == "sigmf":
|
if fmt == "sigmf":
|
||||||
# Normalize path for SigMF
|
# Normalize path for SigMF
|
||||||
|
|
@ -256,11 +257,7 @@ def list(input, verbose):
|
||||||
user_comment = ann.comment or ""
|
user_comment = ann.comment or ""
|
||||||
|
|
||||||
# Basic info
|
# Basic info
|
||||||
freq_range = (
|
freq_range = f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
||||||
f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
|
||||||
if ann.freq_lower_edge is not None and ann.freq_upper_edge is not None
|
|
||||||
else "N/A"
|
|
||||||
)
|
|
||||||
click.echo(
|
click.echo(
|
||||||
f" [{i}] Samples {format_sample_count(ann.sample_start)}-"
|
f" [{i}] Samples {format_sample_count(ann.sample_start)}-"
|
||||||
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {ann.label}"
|
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {ann.label}"
|
||||||
|
|
@ -315,8 +312,6 @@ def add(input, start, count, label, freq_lower, freq_upper, comment, annotation_
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
# Validate sample range
|
# Validate sample range
|
||||||
n_samples = len(recording.data[0])
|
n_samples = len(recording.data[0])
|
||||||
if start < 0:
|
if start < 0:
|
||||||
|
|
@ -368,9 +363,12 @@ def add(input, start, count, label, freq_lower, freq_upper, comment, annotation_
|
||||||
if comment:
|
if comment:
|
||||||
click.echo(f" Comment: {comment}")
|
click.echo(f" Comment: {comment}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
try:
|
||||||
if not quiet:
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
click.echo(" ✓ Saved")
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
|
except Exception as e:
|
||||||
|
raise click.ClickException(f"Failed to save: {e}")
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
@ -468,6 +466,8 @@ def clear(input, output, overwrite, force, quiet):
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo(f"\nCleared {count_before} annotation(s)")
|
click.echo(f"\nCleared {count_before} annotation(s)")
|
||||||
|
|
||||||
|
recording._annotations = []
|
||||||
|
|
||||||
try:
|
try:
|
||||||
save_recording_auto(recording, output_path=input, input_path=input, quiet=quiet, overwrite=True)
|
save_recording_auto(recording, output_path=input, input_path=input, quiet=quiet, overwrite=True)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -503,9 +503,6 @@ def clear(input, output, overwrite, force, quiet):
|
||||||
default="standalone",
|
default="standalone",
|
||||||
help="Annotation type",
|
help="Annotation type",
|
||||||
)
|
)
|
||||||
@click.option(
|
|
||||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
|
||||||
)
|
|
||||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||||
|
|
@ -520,7 +517,6 @@ def energy(
|
||||||
nfft,
|
nfft,
|
||||||
obw_power,
|
obw_power,
|
||||||
annotation_type,
|
annotation_type,
|
||||||
sample_rate,
|
|
||||||
output,
|
output,
|
||||||
overwrite,
|
overwrite,
|
||||||
quiet,
|
quiet,
|
||||||
|
|
@ -543,11 +539,8 @@ def energy(
|
||||||
ria annotate energy signal.npy --threshold 1.5 --min-distance 10000
|
ria annotate energy signal.npy --threshold 1.5 --min-distance 10000
|
||||||
ria annotate energy signal.sigmf-data --freq-method obw
|
ria annotate energy signal.sigmf-data --freq-method obw
|
||||||
ria annotate energy signal.sigmf-data --freq-method full-detected
|
ria annotate energy signal.sigmf-data --freq-method full-detected
|
||||||
ria annotate energy signal.npy --sample-rate 1e6
|
|
||||||
|
|
||||||
"""
|
"""
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
recording = load_recording(input)
|
recording = load_recording(input)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -555,15 +548,6 @@ def energy(
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
if sample_rate is not None:
|
|
||||||
recording.sample_rate = sample_rate
|
|
||||||
|
|
||||||
if recording.sample_rate is None:
|
|
||||||
raise click.ClickException(
|
|
||||||
"Recording metadata does not contain a sample rate. "
|
|
||||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
|
||||||
)
|
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo("\nDetecting signals using energy-based method...")
|
click.echo("\nDetecting signals using energy-based method...")
|
||||||
click.echo(" Time detection:")
|
click.echo(" Time detection:")
|
||||||
|
|
@ -591,13 +575,13 @@ def energy(
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo(f" ✓ Added {added} annotation(s)")
|
click.echo(f" ✓ Added {added} annotation(s)")
|
||||||
|
|
||||||
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Energy detection failed: {e}")
|
raise click.ClickException(f"Energy detection failed: {e}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
|
||||||
if not quiet:
|
|
||||||
click.echo(" ✓ Saved")
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# CUSUM detection subcommand
|
# CUSUM detection subcommand
|
||||||
|
|
@ -617,13 +601,10 @@ def energy(
|
||||||
default="standalone",
|
default="standalone",
|
||||||
help="Annotation type",
|
help="Annotation type",
|
||||||
)
|
)
|
||||||
@click.option(
|
|
||||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
|
||||||
)
|
|
||||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||||
def cusum(input, label, min_duration, window_size, tolerance, annotation_type, sample_rate, output, overwrite, quiet):
|
def cusum(input, label, min_duration, window_size, tolerance, annotation_type, output, overwrite, quiet):
|
||||||
"""Auto-detect segments using CUSUM method.
|
"""Auto-detect segments using CUSUM method.
|
||||||
|
|
||||||
Detects signal state changes (on/off, amplitude transitions). Best for
|
Detects signal state changes (on/off, amplitude transitions). Best for
|
||||||
|
|
@ -635,10 +616,7 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
||||||
Examples:
|
Examples:
|
||||||
ria annotate cusum signal.sigmf-data --min-duration 5.0
|
ria annotate cusum signal.sigmf-data --min-duration 5.0
|
||||||
ria annotate cusum data.npy --min-duration 10.0 --label state
|
ria annotate cusum data.npy --min-duration 10.0 --label state
|
||||||
ria annotate cusum data.npy --sample-rate 1e6
|
|
||||||
"""
|
"""
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
recording = load_recording(input)
|
recording = load_recording(input)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -646,15 +624,6 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
if sample_rate is not None:
|
|
||||||
recording.sample_rate = sample_rate
|
|
||||||
|
|
||||||
if recording.sample_rate is None:
|
|
||||||
raise click.ClickException(
|
|
||||||
"Recording metadata does not contain a sample rate. "
|
|
||||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
|
||||||
)
|
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo("\nDetecting segments using CUSUM...")
|
click.echo("\nDetecting segments using CUSUM...")
|
||||||
click.echo(f" Min duration: {min_duration} ms")
|
click.echo(f" Min duration: {min_duration} ms")
|
||||||
|
|
@ -675,13 +644,13 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo(f" ✓ Added {added} annotation(s)")
|
click.echo(f" ✓ Added {added} annotation(s)")
|
||||||
|
|
||||||
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"CUSUM detection failed: {e}")
|
raise click.ClickException(f"CUSUM detection failed: {e}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
|
||||||
if not quiet:
|
|
||||||
click.echo(" ✓ Saved")
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Threshold detection subcommand
|
# Threshold detection subcommand
|
||||||
|
|
@ -706,13 +675,10 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
||||||
help="Annotation type",
|
help="Annotation type",
|
||||||
)
|
)
|
||||||
@click.option("--channel", type=int, default=0, help="Channel index to annotate (default: 0)")
|
@click.option("--channel", type=int, default=0, help="Channel index to annotate (default: 0)")
|
||||||
@click.option(
|
|
||||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
|
||||||
)
|
|
||||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||||
def threshold(input, threshold, label, window_size, annotation_type, channel, sample_rate, output, overwrite, quiet):
|
def threshold(input, threshold, label, window_size, annotation_type, channel, output, overwrite, quiet):
|
||||||
"""Auto-detect signals using threshold method.
|
"""Auto-detect signals using threshold method.
|
||||||
|
|
||||||
Detects samples above a percentage of maximum magnitude. Best for simple
|
Detects samples above a percentage of maximum magnitude. Best for simple
|
||||||
|
|
@ -722,13 +688,10 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
||||||
Examples:
|
Examples:
|
||||||
ria annotate threshold signal.sigmf-data --threshold 0.7 --label wifi
|
ria annotate threshold signal.sigmf-data --threshold 0.7 --label wifi
|
||||||
ria annotate threshold data.npy --threshold 0.5 --window-size 2048
|
ria annotate threshold data.npy --threshold 0.5 --window-size 2048
|
||||||
ria annotate threshold data.npy --threshold 0.4 --sample-rate 1e6
|
|
||||||
"""
|
"""
|
||||||
if not (0.0 <= threshold <= 1.0):
|
if not (0.0 <= threshold <= 1.0):
|
||||||
raise click.ClickException(f"--threshold must be between 0.0 and 1.0, got {threshold}")
|
raise click.ClickException(f"--threshold must be between 0.0 and 1.0, got {threshold}")
|
||||||
|
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
recording = load_recording(input)
|
recording = load_recording(input)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -736,21 +699,11 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
if sample_rate is not None:
|
|
||||||
recording.sample_rate = sample_rate
|
|
||||||
|
|
||||||
if recording.sample_rate is None:
|
|
||||||
raise click.ClickException(
|
|
||||||
"Recording metadata does not contain a sample rate. "
|
|
||||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
|
||||||
)
|
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo("\nDetecting signals using threshold qualifier...")
|
click.echo("\nDetecting signals using threshold qualifier...")
|
||||||
click.echo(f" Threshold: {threshold * 100:.1f}% of max magnitude")
|
click.echo(f" Threshold: {threshold * 100:.1f}% of max magnitude")
|
||||||
click.echo(f" Window size: {'auto (1ms)' if window_size is None else f'{window_size} samples'}")
|
click.echo(f" Window size: {'auto (1ms)' if window_size is None else f'{window_size} samples'}")
|
||||||
click.echo(f" Channel: {channel}")
|
click.echo(f" Channel: {channel}")
|
||||||
click.echo(f" Sample rate: {recording.sample_rate:.0f} Hz")
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
initial_count = len(recording.annotations)
|
initial_count = len(recording.annotations)
|
||||||
|
|
@ -766,13 +719,13 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo(f" ✓ Added {added} annotation(s)")
|
click.echo(f" ✓ Added {added} annotation(s)")
|
||||||
|
|
||||||
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Threshold detection failed: {e}")
|
raise click.ClickException(f"Threshold detection failed: {e}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
|
||||||
if not quiet:
|
|
||||||
click.echo(" ✓ Saved")
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Separate subcommand (Phase 2: Parallel signal separation)
|
# Separate subcommand (Phase 2: Parallel signal separation)
|
||||||
|
|
@ -785,30 +738,11 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
||||||
@click.option("--nfft", type=int, default=65536, help="FFT size for spectral analysis")
|
@click.option("--nfft", type=int, default=65536, help="FFT size for spectral analysis")
|
||||||
@click.option("--noise-threshold-db", type=float, help="Noise floor threshold in dB (auto-estimated if not specified)")
|
@click.option("--noise-threshold-db", type=float, help="Noise floor threshold in dB (auto-estimated if not specified)")
|
||||||
@click.option("--min-component-bw", type=float, default=50e3, help="Min component bandwidth in Hz")
|
@click.option("--min-component-bw", type=float, default=50e3, help="Min component bandwidth in Hz")
|
||||||
@click.option(
|
|
||||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
|
||||||
)
|
|
||||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||||
@click.option("--verbose", is_flag=True, help="Verbose output (show detected components)")
|
@click.option("--verbose", is_flag=True, help="Verbose output (show detected components)")
|
||||||
def _log_separate_start(quiet, recording, indices_list, nfft, noise_threshold_db, min_component_bw):
|
def separate(input, indices, nfft, noise_threshold_db, min_component_bw, output, overwrite, quiet, verbose):
|
||||||
if not quiet:
|
|
||||||
click.echo("\nSplitting annotations by frequency components...")
|
|
||||||
click.echo(f" Input annotations: {len(recording.annotations)}")
|
|
||||||
if indices_list:
|
|
||||||
click.echo(f" Splitting indices: {indices_list}")
|
|
||||||
click.echo(f" FFT size: {nfft}")
|
|
||||||
if noise_threshold_db is not None:
|
|
||||||
click.echo(f" Noise threshold: {noise_threshold_db} dB")
|
|
||||||
else:
|
|
||||||
click.echo(" Noise threshold: auto-estimated")
|
|
||||||
click.echo(f" Min component BW: {format_frequency(min_component_bw)}")
|
|
||||||
|
|
||||||
|
|
||||||
def separate(
|
|
||||||
input, indices, nfft, noise_threshold_db, min_component_bw, sample_rate, output, overwrite, quiet, verbose
|
|
||||||
):
|
|
||||||
"""
|
"""
|
||||||
Auto-detect parallel frequency-offset signals and split into sub-bands.
|
Auto-detect parallel frequency-offset signals and split into sub-bands.
|
||||||
|
|
||||||
|
|
@ -834,8 +768,6 @@ def separate(
|
||||||
ria annotate separate signal.npy --min-component-bw 100000
|
ria annotate separate signal.npy --min-component-bw 100000
|
||||||
|
|
||||||
"""
|
"""
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
recording = load_recording(input)
|
recording = load_recording(input)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -843,15 +775,6 @@ def separate(
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
if sample_rate is not None:
|
|
||||||
recording.sample_rate = sample_rate
|
|
||||||
|
|
||||||
if recording.sample_rate is None:
|
|
||||||
raise click.ClickException(
|
|
||||||
"Recording metadata does not contain a sample rate. "
|
|
||||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Parse indices if specified
|
# Parse indices if specified
|
||||||
indices_list = get_indices_list(indices=indices, recording=recording)
|
indices_list = get_indices_list(indices=indices, recording=recording)
|
||||||
|
|
||||||
|
|
@ -860,7 +783,17 @@ def separate(
|
||||||
click.echo("No annotations to split")
|
click.echo("No annotations to split")
|
||||||
return
|
return
|
||||||
|
|
||||||
_log_separate_start(quiet, recording, indices_list, nfft, noise_threshold_db, min_component_bw)
|
if not quiet:
|
||||||
|
click.echo("\nSplitting annotations by frequency components...")
|
||||||
|
click.echo(f" Input annotations: {len(recording.annotations)}")
|
||||||
|
if indices_list:
|
||||||
|
click.echo(f" Splitting indices: {indices_list}")
|
||||||
|
click.echo(f" FFT size: {nfft}")
|
||||||
|
if noise_threshold_db is not None:
|
||||||
|
click.echo(f" Noise threshold: {noise_threshold_db} dB")
|
||||||
|
else:
|
||||||
|
click.echo(" Noise threshold: auto-estimated")
|
||||||
|
click.echo(f" Min component BW: {format_frequency(min_component_bw)}")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
initial_count = len(recording.annotations)
|
initial_count = len(recording.annotations)
|
||||||
|
|
@ -882,19 +815,14 @@ def separate(
|
||||||
click.echo("\n Details:")
|
click.echo("\n Details:")
|
||||||
for i in range(initial_count, final_count):
|
for i in range(initial_count, final_count):
|
||||||
ann = recording.annotations[i]
|
ann = recording.annotations[i]
|
||||||
freq_range = (
|
freq_range = f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
||||||
f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
|
||||||
if ann.freq_lower_edge is not None and ann.freq_upper_edge is not None
|
|
||||||
else "N/A"
|
|
||||||
)
|
|
||||||
click.echo(
|
click.echo(
|
||||||
f" [{i}] samples {format_sample_count(ann.sample_start)}-"
|
f" [{i}] samples {format_sample_count(ann.sample_start)}-"
|
||||||
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {freq_range}"
|
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {freq_range}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Spectral separation failed: {e}")
|
raise click.ClickException(f"Spectral separation failed: {e}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
|
||||||
if not quiet:
|
|
||||||
click.echo(" ✓ Saved")
|
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user