Compare commits

..

No commits in common. "main" and "zfp-oss" have entirely different histories.

54 changed files with 1761 additions and 3848 deletions

3
.gitignore vendored
View File

@ -99,5 +99,4 @@ cython_debug/
*.sigmf-meta *.sigmf-meta
*.blue *.blue
*.wav *.wav
/images/ images/
!docs/source/images/**

View File

@ -1,24 +1,5 @@
# Changelog # Changelog
## [0.1.7] - 2026-05-26
### Added
- **Human-readable agent names**`ria-agent register` now generates a default `adjective-colour-animal` name (e.g. `swift-teal-falcon`) via the new `namegen` module when `--name` is omitted, instead of registering with an empty string.
- **Structured registration error messages**`ria-agent register` translates hub responses into actionable English for the known failure reasons (`invalid_key`, `expired`, `revoked`, `already_consumed`) and rate-limit (`HTTP 429`) responses, instead of surfacing raw `HTTP 4xx` text.
### Changed
- **`ria-agent register` `--api-key` help** — now describes the personal `ria_reg_*` registration key flow (minted from **Settings → RIA Agents** on the hub, shown once at mint time). The legacy shared `[wac] API_KEY` is still accepted by the hub for back-compat, but the CLI documents the per-user flow as preferred.
- **`ria-agent register` success output** — now prints both the hub-assigned agent ID and the chosen name: `Registered agent: <id> (<name>)`.
### Fixed
- **`ria-agent register` blocked by Cloudflare on hubs behind it** — set an explicit `User-Agent` (`ria-agent/<package-version> (+https://riahub.ai/qoherent/ria-toolkit-oss)`) so the request isn't rejected as `Python-urllib/<ver>` (Cloudflare Browser Integrity Check returns HTTP 403, edge error code 1010). Version is read from package metadata so it tracks releases automatically.
- **`ria-agent register` could hang indefinitely** — added a 15-second timeout to the hub request; previously `urllib`'s default of no timeout meant a stuck hub would block the CLI forever.
---
## [0.1.0] - 2026-02-20 ## [0.1.0] - 2026-02-20
### Added ### Added

View File

@ -185,7 +185,7 @@ RIA Toolkit OSS is developed and maintained by [Qoherent](https://qoherent.ai/),
If you are doing research with RIA Toolkit OSS, please cite the project: If you are doing research with RIA Toolkit OSS, please cite the project:
``` ```
[1] Qoherent Inc., "Radio Intelligence Apps Toolkit OSS," 2026. [Online]. Available: https://riahub.ai/qoherent/ria-toolkit-oss [1] Qoherent Inc., "Radio Intelligence Apps Toolkit OSS," 2025. [Online]. Available: https://riahub.ai/qoherent/ria-toolkit-oss
``` ```
If you like what we're doing, don't forget to give the project a star! ⭐ If you like what we're doing, don't forget to give the project a star! ⭐

View File

@ -6,10 +6,10 @@ This is a practical reference for the ``ria`` CLI from ``ria-toolkit-oss``.
**Scope of this guide:** **Scope of this guide:**
* **Installation and SDR driver prerequisites** — how to install RIA Toolkit OSS and configure the system drivers your hardware requires * Installation and setup
* **End-to-end CLI workflow** — a step-by-step walkthrough from hardware discovery through capture, annotation, and processing * End-to-end CLI workflows
* **Full command reference** — options, flags, and examples for every ``ria`` command * Full command reference for CLI features
* **Python scripting preview** — using the toolkit API directly without the CLI * Brief scripting section
**Official resources:** **Official resources:**
@ -18,15 +18,76 @@ This is a practical reference for the ``ria`` CLI from ``ria-toolkit-oss``.
* `PyPI package <https://pypi.org/project/ria-toolkit-oss/>`_ * `PyPI package <https://pypi.org/project/ria-toolkit-oss/>`_
* `RIA Hub Conda package <https://riahub.ai/qoherent/-/packages/conda/ria-toolkit-oss>`_ * `RIA Hub Conda package <https://riahub.ai/qoherent/-/packages/conda/ria-toolkit-oss>`_
.. contents:: Contents
:local:
:depth: 2
:backlinks: none
1) Installation and Setup 1) Installation and Setup
========================== ==========================
Before using the ``ria`` CLI, follow the :doc:`Installation <installation>` guide to 1.1 Installation with Conda
install RIA Toolkit OSS and any SDR drivers required for your hardware. ----------------------------
RIA Toolkit OSS is available as a Conda package on RIA Hub. This is typically the easiest
path when using SDR tooling that depends on native/system libraries.
.. code-block:: bash
conda update --force conda
conda config --add channels https://riahub.ai/api/packages/qoherent/conda
conda activate base
conda install ria-toolkit-oss
Verify:
.. code-block:: bash
conda list | grep ria-toolkit-oss
1.1 SDR driver prerequisites 1.2 Installation with pip
--------------------------
Use pip unless you specifically need to edit toolkit source.
.. code-block:: bash
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install ria-toolkit-oss
Verify CLI entrypoint:
.. code-block:: bash
ria --help
``pyproject.toml`` defines two script entry points:
* ``ria``
* ``ria-tools``
Both point to the same CLI module (``ria_toolkit_oss_cli.cli:cli``).
1.3 Optional install from source
----------------------------------
Use this for local development or testing unreleased changes.
.. code-block:: bash
git clone https://riahub.ai/qoherent/ria-toolkit-oss.git
cd ria-toolkit-oss
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
1.4 SDR driver prerequisites
----------------------------- -----------------------------
Toolkit package install does not install all system SDR drivers. Install vendor/runtime Toolkit package install does not install all system SDR drivers. Install vendor/runtime
@ -34,22 +95,11 @@ dependencies for the hardware you use.
Examples (depends on device and OS): Examples (depends on device and OS):
.. list-table:: * USRP: UHD drivers
:widths: 25 75 * Pluto: libiio / IIO utilities
:header-rows: 1 * BladeRF: libbladeRF
* HackRF: libhackrf
* - Device * RTL-SDR: librtlsdr
- Driver Package
* - USRP
- UHD drivers
* - Pluto
- libiio / IIO utilities
* - BladeRF
- libbladeRF
* - HackRF
- libhackrf
* - RTL-SDR
- librtlsdr
See repo docs under ``docs/source/sdr_guides/*`` and your OS package instructions. See repo docs under ``docs/source/sdr_guides/*`` and your OS package instructions.
@ -69,34 +119,18 @@ Top-level CLI follows this model:
**Top-level commands:** **Top-level commands:**
.. list-table:: * ``discover``
:widths: 25 75 * ``init``
:header-rows: 1 * ``capture``
* ``view``
* - Command * ``annotate`` (group)
- Purpose * ``convert``
* - :ref:`discover <cmd-discover>` * ``split``
- Probe SDR drivers and enumerate attached hardware * ``combine``
* - :ref:`init <cmd-init>` * ``generate`` (group)
- Create and manage user metadata defaults * ``transform`` (group)
* - :ref:`capture <cmd-capture>` * ``transmit``
- Record IQ samples from a connected SDR * ``synth`` (alias of ``generate`` in command bindings)
* - :ref:`view <cmd-view>`
- Generate visualizations from IQ files
* - :ref:`annotate <cmd-annotate>`
- Label signal regions manually or with auto-detection (group)
* - :ref:`convert <cmd-convert>`
- Convert between IQ file formats
* - :ref:`split <cmd-split>`
- Split, trim, or extract recordings
* - :ref:`combine <cmd-combine>`
- Merge multiple recordings by concatenation or addition
* - :ref:`generate / synth <cmd-generate>`
- Generate synthetic IQ signals (group; ``synth`` is an alias)
* - :ref:`transform <cmd-transform>`
- Apply augmentations or impairments to recordings (group)
* - :ref:`transmit <cmd-transmit>`
- Transmit IQ through a TX-capable SDR
3) Quick End-to-End Workflow 3) Quick End-to-End Workflow
@ -124,8 +158,10 @@ provenance fields.
.. code-block:: bash .. code-block:: bash
ria init ria init
ria init --author "Jane Doe" --project "rf-campaign-1" --location "Lab-A" # non-interactive # or non-interactive
ria init --show # show config ria init --author "Jane Doe" --project "rf-campaign-1" --location "Lab-A"
# show config
ria init --show
3.3 Capture IQ 3.3 Capture IQ
@ -191,14 +227,13 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
.. code-block:: bash .. code-block:: bash
ria transmit -d hackrf -f 2.44G -s 2e6 --input capture.sigmf-data ria transmit -d hackrf -f 2.44G -s 2e6 --input capture.sigmf-data
ria transmit -d hackrf --generate lfm --continuous # generated waveform # or generated waveform
ria transmit -d hackrf --generate lfm --continuous
4) Command Reference 4) Command Reference
===================== =====================
.. _cmd-discover:
4.1 ``discover`` 4.1 ``discover``
----------------- -----------------
@ -228,8 +263,6 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
hidden in default output. hidden in default output.
.. _cmd-init:
4.2 ``init`` 4.2 ``init``
------------- -------------
@ -276,8 +309,6 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
generate metadata, and YAML config loading paths). generate metadata, and YAML config loading paths).
.. _cmd-capture:
4.3 ``capture`` 4.3 ``capture``
---------------- ----------------
@ -351,8 +382,6 @@ Device selection (``--device``) is optional if only one device is detected. Exac
ria capture -c capture_config.yaml ria capture -c capture_config.yaml
.. _cmd-view:
4.4 ``view`` 4.4 ``view``
------------- -------------
@ -413,21 +442,7 @@ Device selection (``--device``) is optional if only one device is detected. Exac
ria view capture.npy --type full --title "Test Capture" --format pdf ria view capture.npy --type full --title "Test Capture" --format pdf
ria view capture.npy --show --no-save ria view capture.npy --show --no-save
ria view old.npy --legacy --type simple ria view old.npy --legacy --type simple
ria view recordings\qam64_35.npy --type simple
ria view recordings\qam64_35.npy --type full
.. figure:: ../images/recordings/qam64_35.png
:alt: Example output of ria view recordings\qam64_35.npy --type simple
Output of ``ria view recordings\qam64_35.npy --type simple``
.. figure:: ../images/recordings/qam64_35-full.png
:alt: Example output of ria view recordings\qam64_35.npy --type full
Output of ``ria view recordings\qam64_35.npy --type full``
.. _cmd-annotate:
4.5 ``annotate`` group 4.5 ``annotate`` group
----------------------- -----------------------
@ -444,30 +459,8 @@ Device selection (``--device``) is optional if only one device is detected. Exac
ria annotate <subcommand> ... ria annotate <subcommand> ...
**Subcommands:** **Subcommands:** ``list``, ``add``, ``remove``, ``clear``, ``energy``, ``cusum``,
``threshold``, ``separate``
.. list-table::
:widths: 25 75
:header-rows: 1
* - Subcommand
- Purpose
* - ``list``
- Inspect all annotations on a recording
* - ``add``
- Add one annotation with explicit sample-domain bounds
* - ``remove``
- Remove one annotation by index
* - ``clear``
- Remove all annotations from a recording
* - ``energy``
- Auto-detect regions above the estimated noise floor
* - ``cusum``
- Auto-detect regime changes using change-point detection
* - ``threshold``
- Auto-detect regions using normalized magnitude thresholding
* - ``separate``
- Decompose annotations into narrower spectral components
**General behavior:** **General behavior:**
@ -594,16 +587,8 @@ annotations.
ria annotate add capture.sigmf-data --start 10000 --count 5000 --label burst ria annotate add capture.sigmf-data --start 10000 --count 5000 --label burst
ria annotate energy capture.sigmf-data --label signal --threshold 1.3 ria annotate energy capture.sigmf-data --label signal --threshold 1.3
ria annotate cusum capture.sigmf-data --min-duration 5 ria annotate cusum capture.sigmf-data --min-duration 5
ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%
ria annotate separate capture.sigmf-data --indices 0,1 --verbose ria annotate separate capture.sigmf-data --indices 0,1 --verbose
.. figure:: ../images/recordings/sample_recording3_annotated.png
:alt: Example output of ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%
Output of ``ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%``
.. _cmd-convert:
4.6 ``convert`` 4.6 ``convert``
---------------- ----------------
@ -644,8 +629,6 @@ inferred from the output file extension.
ria convert old.npy --format sigmf --legacy --overwrite ria convert old.npy --format sigmf --legacy --overwrite
.. _cmd-split:
4.7 ``split`` 4.7 ``split``
-------------- --------------
@ -687,8 +670,6 @@ Choose exactly one operation per invocation:
ria split annotated.sigmf-data --extract-annotations --annotation-label payload ria split annotated.sigmf-data --extract-annotations --annotation-label payload
.. _cmd-combine:
4.8 ``combine`` 4.8 ``combine``
---------------- ----------------
@ -736,8 +717,6 @@ Choose exactly one operation per invocation:
ria combine signal.npy pattern.npy out.npy --mode add --align-mode repeat-spaced --repeat-spacing 10000 ria combine signal.npy pattern.npy out.npy --mode add --align-mode repeat-spaced --repeat-spacing 10000
.. _cmd-generate:
4.9 ``generate`` group (and ``synth`` alias) 4.9 ``generate`` group (and ``synth`` alias)
--------------------------------------------- ---------------------------------------------
@ -749,34 +728,15 @@ Choose exactly one operation per invocation:
``ria synth ...`` is an alias for ``ria generate ...``. ``ria synth ...`` is an alias for ``ria generate ...``.
**Usage:** **Shape:**
.. code-block:: bash .. code-block:: bash
ria generate <subcommand> [subcommand options] [common options] ria generate <subcommand> [subcommand options] [common options]
**Available subcommands:** **Available subcommands:**
``tone``, ``noise``, ``chirp``, ``square``, ``sawtooth``, ``qam``, ``apsk``, ``pam``,
.. list-table:: ``fsk``, ``ook``, ``oqpsk``, ``gmsk``, ``psk``
:widths: 30 70
:header-rows: 1
* - Subcommand(s)
- Description
* - ``tone``
- Clean sinusoidal calibration/reference source
* - ``noise``
- Baseline noise floor data or controlled additive-noise synthesis
* - ``chirp``
- Sweep-based radar/sonar-style signals and bandwidth occupancy tests
* - ``square``, ``sawtooth``
- Periodic waveform primitives
* - ``qam``, ``apsk``, ``pam``, ``psk``
- Digital modulation families with pulse-shaping filter support
* - ``fsk``
- Frequency-shift keying with configurable tone spacing
* - ``ook``, ``oqpsk``, ``gmsk``
- On-off keying and continuous-phase modulation schemes
**Common options shared across all generators:** **Common options shared across all generators:**
@ -800,16 +760,22 @@ Multipath and IQ imbalance flags apply impairment-style post-processing during g
Options: ``--frequency``, ``--amplitude``, ``--phase`` Options: ``--frequency``, ``--amplitude``, ``--phase``
Clean sinusoidal calibration/reference source.
``noise`` ``noise``
~~~~~~~~~~ ~~~~~~~~~~
Options: ``--noise-type {gaussian,uniform}``, ``--power`` Options: ``--noise-type {gaussian,uniform}``, ``--power``
Baseline noise floor data or controlled additive-noise synthesis.
``chirp`` ``chirp``
~~~~~~~~~~ ~~~~~~~~~~
Options: ``--bandwidth`` (required), ``--period`` (required), ``--type {up,down,up_down}`` Options: ``--bandwidth`` (required), ``--period`` (required), ``--type {up,down,up_down}``
Sweep-based radar/sonar-style signals and bandwidth occupancy tests.
``square`` ``square``
~~~~~~~~~~~ ~~~~~~~~~~~
@ -860,8 +826,6 @@ symbol transition sharpness).
ria synth psk -s 2e6 -r 100e3 -M 8 -N 8000 -o psk8.npy ria synth psk -s 2e6 -r 100e3 -M 8 -N 8000 -o psk8.npy
.. _cmd-transform:
4.10 ``transform`` group 4.10 ``transform`` group
------------------------- -------------------------
@ -870,7 +834,7 @@ symbol transition sharpness).
* Apply algorithmic transforms to existing recordings. * Apply algorithmic transforms to existing recordings.
* Run reusable augmentations/impairments for dataset diversity and robustness testing. * Run reusable augmentations/impairments for dataset diversity and robustness testing.
**Usage:** **Shape:**
.. code-block:: bash .. code-block:: bash
@ -931,8 +895,6 @@ inspect parameter hints. ``--view`` writes a PNG preview alongside transform out
ria transform custom my_filter in.npy out.npy --transform-dir ./my_transforms --params cutoff=0.2 ria transform custom my_filter in.npy out.npy --transform-dir ./my_transforms --params cutoff=0.2
.. _cmd-transmit:
4.11 ``transmit`` 4.11 ``transmit``
------------------ ------------------
@ -1031,7 +993,17 @@ experiment-specific fields on the CLI.
ria generate noise --config generate.yaml ria generate noise --config generate.yaml
6) Version Notes 6) Practical Tips and Safety
=============================
* Use ``ria discover`` before capture/transmit sessions.
* Keep TX gain conservative first; validate with attenuators/dummy loads when needed.
* Prefer SigMF for interoperable metadata and annotations.
* For long workflows, keep outputs organized by campaign directories and consistent prefixes.
* Use ``--verbose`` when debugging device init or driver issues.
7) Version Notes
================= =================
These notes are based on the current implementation and should be re-validated against future These notes are based on the current implementation and should be re-validated against future
@ -1044,19 +1016,18 @@ releases.
3. Multiple non-CLI modules still import ``utils.*``, which can create runtime dependency 3. Multiple non-CLI modules still import ``utils.*``, which can create runtime dependency
coupling when using only ``ria-toolkit-oss`` in isolation. coupling when using only ``ria-toolkit-oss`` in isolation.
.. tip::
If you observe unexpected import errors after install, check the package version and If you observe unexpected import errors after install, check the package version and
changelog, then test ``ria --help`` in a clean virtual environment. changelog, then test ``ria --help`` in a clean virtual environment.
7) Brief Scripting (Python) Preview 8) Brief Scripting (Python) Preview
===================================== =====================================
For quick non-CLI use: For quick non-CLI use:
.. code-block:: python .. code-block:: python
from ria_toolkit_oss.data import Recording from ria_toolkit_oss.datatypes import Recording
from ria_toolkit_oss.io import load_recording, to_sigmf from ria_toolkit_oss.io import load_recording, to_sigmf
from ria_toolkit_oss.transforms import iq_augmentations, iq_impairments from ria_toolkit_oss.transforms import iq_augmentations, iq_impairments
@ -1066,3 +1037,47 @@ For quick non-CLI use:
to_sigmf(imp, filename="capture_awgn", path=".") to_sigmf(imp, filename="capture_awgn", path=".")
You can also call annotation algorithms and block-generator primitives from Python directly. You can also call annotation algorithms and block-generator primitives from Python directly.
9) Cheat Sheet
===============
.. code-block:: bash
# Install
pip install ria-toolkit-oss
# Discover
ria discover -v
# Init defaults
ria init --author "Jane" --project "rf1" --location "Lab-A"
# Capture
ria capture -d pluto -f 2.44G -s 2e6 -n 1000000 -o cap.sigmf-data
# View
ria view cap.sigmf-data --type simple
# Annotate
ria annotate energy cap.sigmf-data --threshold 1.2
ria annotate list cap.sigmf-data --verbose
# Convert
ria convert cap.sigmf-data cap.npy
# Split
ria split cap.sigmf-data --split-every 100000 --output-dir chunks
# Combine
ria combine chunks/a.npy chunks/b.npy merged.npy
# Generate
ria generate qam -s 2e6 -r 100e3 -M 16 -N 5000 -o qam16.npy
# Transform
ria transform augment channel_swap cap.npy
ria transform impair add_awgn_to_signal cap.npy --params snr=10
# Transmit
ria transmit -d hackrf --input cap.sigmf-data -f 2.44G -s 2e6

View File

@ -1,7 +1,6 @@
/* Change the hex values below to customize heading colours */ /* Change the hex values below to customize heading colours */
.rst-content { color: #e0e0e0; } .rst-content h1 { color: #2c3e50; }
.rst-content h1 { color: #ffffff; }
.rst-content h2, .rst-content h2,
.rst-content h2 a { color: #ffffff !important; font-size: 22px !important; } .rst-content h2 a { color: #ffffff !important; font-size: 22px !important; }
@ -23,20 +22,8 @@
.rst-content .admonition.warning p { .rst-content .admonition.warning p {
color: #ffffff !important; color: #ffffff !important;
} }
.rst-content h4 { color: #cccccc; } .rst-content h4 { color: #404040; }
.highlight * { color: #ffffff !important; } .highlight * { color: #ffffff !important; }
.ria-cmd { color: #2980b9 !important; } .ria-cmd { color: #2980b9 !important; }
/* Table header text */
.rst-content table.docutils th {
color: #ffffff !important;
}
/* Remove alternating row background colors from tables */
.rst-content table.docutils td,
.rst-content table.docutils tr:nth-child(2n-1) td {
background-color: transparent !important;
}

View File

@ -12,9 +12,9 @@ sys.path.insert(0, os.path.abspath(os.path.join('..', '..')))
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = 'ria-toolkit-oss' project = 'ria-toolkit-oss'
copyright = '2026, Qoherent Inc' copyright = '2025, Qoherent Inc'
author = 'Qoherent Inc.' author = 'Qoherent Inc.'
release = '0.1.7' release = '0.1.5'
# -- General configuration --------------------------------------------------- # -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

View File

@ -1,4 +1,4 @@
.. _sdr_examples: .. _examples:
############ ############
SDR Examples SDR Examples

Binary file not shown.

BIN
docs/source/images/recordings/qam64_35.png (Stored with Git LFS)

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

File diff suppressed because it is too large Load Diff

View File

@ -4,26 +4,7 @@ Installation
RIA Hub Toolkit OSS can be installed either as a Conda package or as a standard Python package. RIA Hub Toolkit OSS can be installed either as a Conda package or as a standard Python package.
Please note that SDR drivers must be installed separately. Refer to the relevant guide in the Please note that SDR drivers must be installed separately. Refer to the relevant guide in the
:ref:`SDR Guides <sdr_guides>` section of the documentation for additional setup instructions. :ref:`SDR Guides <sdr_guides>` section of the documentation for addition setup instructions.
Common driver packages by device (exact package names depend on your OS):
.. list-table::
:widths: 25 75
:header-rows: 1
* - Device
- Driver Package
* - USRP
- UHD drivers
* - Pluto
- libiio / IIO utilities
* - BladeRF
- libbladeRF
* - HackRF
- libhackrf
* - RTL-SDR
- librtlsdr
We want your experience with RIA Toolkit OSS to be as smooth and frictionless as possible. If you run into any We want your experience with RIA Toolkit OSS to be as smooth and frictionless as possible. If you run into any
issues during installation, please reach out to our support team: ``support@qoherent.ai``. issues during installation, please reach out to our support team: ``support@qoherent.ai``.
@ -103,22 +84,12 @@ Please follow the steps below to install RIA Toolkit OSS using pip:
python -m venv venv python -m venv venv
venv\Scripts\activate venv\Scripts\activate
2. Upgrade pip and install RIA Toolkit OSS: 2. Install RIA Toolkit OSS from PyPI with pip:
.. code-block:: bash .. code-block:: bash
pip install --upgrade pip
pip install ria-toolkit-oss pip install ria-toolkit-oss
3. Verify the CLI is available:
.. code-block:: bash
ria --help
A successful install prints the top-level help text. ``pyproject.toml`` registers two
entrypoints — ``ria`` and ``ria-tools`` — that both point to the same CLI module.
RIA Toolkit OSS can also be installed from RIA Hub. However, RIA Hub does not yet support a proxy or cache for public packages. RIA Toolkit OSS can also be installed from RIA Hub. However, RIA Hub does not yet support a proxy or cache for public packages.
We intend to add this missing functionality soon. In the meantime, please use the ``--no-deps`` option with pip to skip automatic We intend to add this missing functionality soon. In the meantime, please use the ``--no-deps`` option with pip to skip automatic
dependency installation, and then manually install each dependency afterward. dependency installation, and then manually install each dependency afterward.
@ -148,6 +119,3 @@ Follow the steps below to install RIA Toolkit OSS from source:
.. code-block:: bash .. code-block:: bash
pip install . pip install .
For local development, use ``pip install -e .`` instead to install in editable mode
so local changes take effect immediately without reinstalling.

View File

@ -1,5 +1,5 @@
Data Package (ria_toolkit_oss.data) Datatypes Package (ria_toolkit_oss.data)
======================================= =============================================
.. |br| raw:: html .. |br| raw:: html

View File

@ -40,36 +40,26 @@ Limitations
- USB 3.0 connectivity is required for optimal performance; using USB 2.0 will significantly limit data - USB 3.0 connectivity is required for optimal performance; using USB 2.0 will significantly limit data
transfer rates. transfer rates.
Set up instructions (Linux) Set up instructions (Linux, Radioconda)
--------------------------- ---------------------------------------
No additional Python packages are required for BladeRF beyond the base RIA Toolkit OSS installation. 1. Activate your Radioconda environment.
1. Install the system library:
.. code-block:: bash .. code-block:: bash
sudo apt install libbladerf-dev conda activate <your-env-name>
For a more complete installation including CLI tools and FPGA images, use the Nuand PPA: 2. Install the base dependencies and drivers (*Easy method*):
.. code-block:: bash .. code-block:: bash
sudo add-apt-repository ppa:nuandllc/bladerf sudo add-apt-repository ppa:nuandllc/bladerf
sudo apt-get update sudo apt-get update
sudo apt-get install bladerf libbladerf-dev sudo apt-get install bladerf
sudo apt-get install bladerf-fpga-hostedxa4 # Necessary for BladeRF 2.0 Micro xA4 sudo apt-get install libbladerf-dev
sudo apt-get install bladerf-fpga-hostedxa4 # Necessary for installation of bladeRF 2.0 Micro A4.
2. Install udev rules: 3. Install a ``udev`` rule by creating a link into your Radioconda installation:
For most users:
.. code-block:: bash
sudo udevadm control --reload
sudo udevadm trigger
For **Radioconda** users, create symlinks from your conda environment instead:
.. code-block:: bash .. code-block:: bash

View File

@ -39,28 +39,23 @@ Limitations
- Bandwidth is limited to 20 MHz. - Bandwidth is limited to 20 MHz.
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs. - USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
Set up instructions (Linux) Set up instructions (Linux, Radioconda)
--------------------------- ---------------------------------------
HackRF is supported out of the box after installing RIA Toolkit OSS. 1. Activate your Radioconda environment:
1. Ensure ``libhackrf`` is installed at the system level. On most Ubuntu installations this is already
present. If not:
.. code-block:: bash .. code-block:: bash
sudo apt install libhackrf-dev conda activate <your-env-name>
2. Install udev rules to allow non-root device access: 2. Install the System Package (Ubuntu / Debian):
For most users:
.. code-block:: bash .. code-block:: bash
sudo udevadm control --reload sudo apt-get update
sudo udevadm trigger sudo apt-get install hackrf
For **Radioconda** users, create a symlink from your conda environment instead: 3. Install a ``udev`` rule by creating a link into your Radioconda installation:
.. code-block:: bash .. code-block:: bash
@ -68,7 +63,7 @@ HackRF is supported out of the box after installing RIA Toolkit OSS.
sudo udevadm control --reload sudo udevadm control --reload
sudo udevadm trigger sudo udevadm trigger
Make sure your user account belongs to the ``plugdev`` group in order to access your device: Make sure your user account belongs to the plugdev group in order to access your device:
.. code-block:: bash .. code-block:: bash
@ -76,7 +71,7 @@ HackRF is supported out of the box after installing RIA Toolkit OSS.
.. note:: .. note::
You may have to restart your system for group membership changes to take effect. You may have to restart your system for changes to take effect.
Further information Further information
------------------- -------------------

View File

@ -43,34 +43,34 @@ Limitations
affect stability. affect stability.
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs. - USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
Set up instructions (Linux) Set up instructions (Linux, Radioconda)
--------------------------- ---------------------------------------
The PlutoSDR is supported out of the box after installing RIA Toolkit OSS. The required Python package 1. Activate your Radioconda environment:
(``pyadi-iio``) is included in the toolkit's dependencies.
1. Ensure ``libiio`` is installed at the system level. On most Ubuntu installations this is already present.
If not:
.. code-block:: bash .. code-block:: bash
sudo apt install libiio-dev libiio-utils libiio0 conda activate <your-env-name>
.. note:: 2. Install system dependencies:
PlutoSDR devices are discoverable over both USB and network (mDNS). Network discovery uses Avahi — if
``avahi-daemon`` is not running, network discovery will be skipped but USB discovery still works.
2. Install a ``udev`` rule to allow non-root device access:
For most users:
.. code-block:: bash .. code-block:: bash
sudo udevadm control --reload sudo apt-get update
sudo udevadm trigger sudo apt-get install -y \
build-essential \
git \
libxml2-dev \
bison \
flex \
libcdk5-dev \
cmake \
libusb-1.0-0-dev \
libavahi-client-dev \
libavahi-common-dev \
libaio-dev
For **Radioconda** users, create a symlink from your conda environment instead: 3. Install a ``udev`` rule by creating a link into your Radioconda installation:
.. code-block:: bash .. code-block:: bash
@ -78,18 +78,11 @@ The PlutoSDR is supported out of the box after installing RIA Toolkit OSS. The r
sudo udevadm control --reload sudo udevadm control --reload
sudo udevadm trigger sudo udevadm trigger
Once you can communicate with the hardware, you may want to perform the post-install steps detailed on Once you can talk to the hardware, you may want to perform the post-install steps detailed on the `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_.
the `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_.
3. (Optional) Building ``libiio`` or ``libad9361-iio`` from source: 4. (Optional) Building ``libiio`` or ``libad9361-iio`` from source:
This step is only required if you need a version not available via ``apt``. First install build This step is only required if you want the latest version of these libraries not provided in Radioconda.
dependencies:
.. code-block:: bash
sudo apt-get install -y build-essential git libxml2-dev bison flex libcdk5-dev cmake \
libusb-1.0-0-dev libavahi-client-dev libavahi-common-dev libaio-dev
.. code-block:: bash .. code-block:: bash

View File

@ -30,10 +30,18 @@ Limitations
- Sensitivity and performance can vary depending on the specific model and components. - Sensitivity and performance can vary depending on the specific model and components.
- Requires external software for signal processing and analysis. - Requires external software for signal processing and analysis.
Set up instructions (Linux) Set up instructions (Linux, Radioconda)
--------------------------- ---------------------------------------
1. If you previously had RTL-SDR drivers installed, purge them first: 1. Activate your Radioconda environment:
.. code-block:: bash
conda activate <your-env-name>
2. Purge drivers:
If you already have other drivers installed, purge them from your system.
.. code-block:: bash .. code-block:: bash
@ -45,95 +53,47 @@ Set up instructions (Linux)
sudo rm -rvf /usr/local/include/rtl_* sudo rm -rvf /usr/local/include/rtl_*
sudo rm -rvf /usr/local/bin/rtl_* sudo rm -rvf /usr/local/bin/rtl_*
2. Install build dependencies: 3. Install RTL-SDR Blog drivers:
.. code-block:: bash .. code-block:: bash
sudo apt install libusb-1.0-0-dev git cmake pkg-config build-essential sudo apt-get install libusb-1.0-0-dev git cmake pkg-config build-essential
git clone https://github.com/osmocom/rtl-sdr
3. Build ``librtlsdr`` from source: cd rtl-sdr
mkdir build
The standard ``librtlsdr`` package available via ``apt`` is missing symbols required by the Python cd build
bindings. Build from the **rtl-sdr-blog fork**: cmake ../ -DINSTALL_UDEV_RULES=ON
.. code-block:: bash
git clone https://github.com/rtlsdrblog/rtl-sdr-blog.git
cd rtl-sdr-blog
mkdir build && cd build
cmake .. -DINSTALL_UDEV_RULES=ON
make make
sudo make install sudo make install
sudo cp ../rtl-sdr.rules /etc/udev/rules.d/ sudo cp ../rtl-sdr.rules /etc/udev/rules.d/
sudo ldconfig sudo ldconfig
.. important:: 4. Blacklist the DVB-T modules that would otherwise claim the device:
Do not use the osmocom ``rtl-sdr`` repository or the Ubuntu ``librtlsdr-dev`` apt package. Neither
provides the ``rtlsdr_set_dithering`` symbol that the Python bindings require.
4. Blacklist the kernel DVB driver:
The kernel DVB-T driver (``dvb_usb_rtl28xxu``) claims the RTL-SDR device and prevents ``librtlsdr``
from accessing it.
For most users:
.. code-block:: bash .. code-block:: bash
echo 'blacklist dvb_usb_rtl28xxu' | sudo tee /etc/modprobe.d/blacklist-rtlsdr.conf
sudo modprobe -r dvb_usb_rtl28xxu
For **Radioconda** users, a blacklist configuration is already provided in your conda environment:
.. code-block:: bash
sudo ln -s $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf /etc/modprobe.d/radioconda-rtl-sdr-blacklist.conf sudo ln -s $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf /etc/modprobe.d/radioconda-rtl-sdr-blacklist.conf
sudo modprobe -r $(cat $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf | sed -n -e 's/^blacklist //p') sudo modprobe -r $(cat $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf | sed -n -e 's/^blacklist //p')
If ``modprobe -r`` fails with "Module is in use", unplug the RTL-SDR dongle, run the command again,
then plug it back in. Alternatively, reboot — the blacklist takes effect on next boot.
.. note:: .. note::
Some systems also require blacklisting additional DVB-T modules. Add these entries to your In addition to the Radioconda blacklist file, some systems also require
blacklist configuration if needed: manually blacklisting the following DVB-T modules to prevent them from
claiming the device:
- ``dvb_usb_rtl28xxu``
- ``rtl2832`` - ``rtl2832``
- ``rtl2830`` - ``rtl2830``
5. Reload udev rules: Add these entries to ``rtlsdr.conf`` (or create the file at
``/etc/modprobe.d/rtlsdr.conf``) if they are not already present.
For most users (rules are installed by the build step above): 5. Install a udev rule by creating a link into your radioconda installation:
.. code-block:: bash .. code-block:: bash
sudo udevadm control --reload
sudo udevadm trigger
For **Radioconda** users, create a symlink from your conda environment instead:
.. code-block:: bash
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/rtl-sdr.rules /etc/udev/rules.d/radioconda-rtl-sdr.rules sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/rtl-sdr.rules /etc/udev/rules.d/radioconda-rtl-sdr.rules
sudo udevadm control --reload sudo udevadm control --reload
sudo udevadm trigger sudo udevadm trigger
6. Install Python packages:
.. code-block:: bash
pip install pyrtlsdr==0.3.0
pip install setuptools==69.5.1
.. note::
``pyrtlsdr`` 0.4.0 references a ``rtlsdr_set_dithering`` symbol not present in standard
``librtlsdr`` builds. Version 0.3.0 works correctly.
``pyrtlsdr`` 0.3.0 depends on ``pkg_resources``, which was removed in ``setuptools`` >= 82.
Pinning to 69.5.1 ensures ``pkg_resources`` is available.
Further Information Further Information
------------------- -------------------
- `RTL-SDR Official Website <https://www.rtl-sdr.com/>`_ - `RTL-SDR Official Website <https://www.rtl-sdr.com/>`_

View File

@ -39,48 +39,18 @@ Limitations
Set up instructions (Linux) Set up instructions (Linux)
--------------------------------- ---------------------------------
ThinkRF devices require the ``pyrf`` package, which is written in Python 2 syntax and must be patched Install PyRF
after installation to work with Python 3.
.. note::
``lib2to3`` was fully removed in Python 3.13. ThinkRF support is currently limited to
**Python 3.12 and below**.
1. Install ``lib2to3``:
On some distributions (including Ubuntu 24.04+), ``lib2to3`` is not included by default:
.. code-block:: bash .. code-block:: bash
sudo apt install python3-lib2to3 pip install 'pyrf>=2.8.0'
2. Install ``pyrf``: Convert PyRF scripts to Python 3
.. code-block:: bash .. code-block:: bash
pip install pyrf cd ../scripts
./convert_pyrf_to_python3.sh
3. Patch ``pyrf`` for Python 3:
The ``pyrf`` package contains Python 2 syntax throughout (e.g., ``dict.iteritems()``, ``print``
statements). Run the following to automatically convert the entire package to Python 3:
.. code-block:: bash
python -c "
from lib2to3.refactor import RefactoringTool, get_fixers_from_package
import pyrf, os
pyrf_path = os.path.dirname(pyrf.__file__)
fixers = get_fixers_from_package('lib2to3.fixes')
tool = RefactoringTool(fixers)
tool.refactor_dir(pyrf_path, write=True)
print('Done')
"
.. note::
This patches the entire ``pyrf`` package in place, which is required for the driver to fully load.
Further Information Further Information
------------------- -------------------

View File

@ -41,97 +41,34 @@ Limitations
- Compatibility with certain software tools may vary depending on the version of the UHD. - Compatibility with certain software tools may vary depending on the version of the UHD.
- Price range can be a consideration, especially for high-end models. - Price range can be a consideration, especially for high-end models.
Set up instructions (Linux) Set up instructions (Linux, Radioconda)
--------------------------- ---------------------------------------
USRP devices require the UHD (USRP Hardware Driver) library with Python bindings. There is no pip-installable 1. Activate your Radioconda environment:
UHD package — it must either be installed via conda or built from source.
**Option A: Install via conda (recommended for conda environments)** .. code-block:: bash
conda activate <your-env-name>
2. Install UHD and Python bindings:
.. code-block:: bash .. code-block:: bash
conda install conda-forge::uhd conda install conda-forge::uhd
**Option B: Build from source (required for pip/venv environments)** 3. Download UHD images:
The Python bindings must target the same Python version used in your virtual environment.
1. Install build dependencies:
.. code-block:: bash
sudo apt install cmake build-essential libboost-all-dev libusb-1.0-0-dev \
python3-dev python3-numpy libncurses-dev
2. Install the Mako template library into your virtual environment (used by UHD's build system):
.. code-block:: bash
pip install mako
3. Clone and build UHD with your virtual environment activated:
.. code-block:: bash
git clone https://github.com/EttusResearch/uhd.git
cd uhd
git checkout v4.7.0.0
cd host
mkdir build && cd build
cmake -DENABLE_PYTHON_API=ON -DPYTHON_EXECUTABLE=$(which python3) ..
make -j$(nproc)
sudo make install
sudo ldconfig
.. important::
Run the ``cmake`` command with your virtual environment activated so ``$(which python3)`` points
to the correct interpreter. Before running ``make``, verify the cmake output includes::
-- * LibUHD - Python API → must say "Enabling"
-- Python interpreter: .../your-venv/bin/python3
If "LibUHD - Python API" is not listed under enabled components, the Python bindings will not be
built. The build typically takes 1030 minutes.
4. Copy the Python bindings into your virtual environment if ``import uhd`` fails after installation:
.. code-block:: bash
cp -r ~/uhd/host/build/python/uhd ~/.venv/lib/python3.XX/site-packages/
Replace ``python3.XX`` with your Python version (e.g., ``python3.12``).
.. note::
If you have a pre-existing UHD installation built against a different Python version, you will see
a circular import error. The bindings must match the Python version in your virtual environment exactly.
**After either installation method:**
1. Download UHD FPGA/firmware images:
.. code-block:: bash .. code-block:: bash
uhd_images_downloader uhd_images_downloader
2. Verify device access: 4. Verify access to your device:
.. code-block:: bash .. code-block:: bash
uhd_find_devices uhd_find_devices
For USB devices (e.g. B-series), install a ``udev`` rule. For USB devices only (e.g. B series), install a ``udev`` rule by creating a link into your Radioconda installation.
For most users:
.. code-block:: bash
sudo udevadm control --reload
sudo udevadm trigger
For **Radioconda** users, create a symlink from your conda environment instead:
.. code-block:: bash .. code-block:: bash
@ -139,7 +76,7 @@ UHD package — it must either be installed via conda or built from source.
sudo udevadm control --reload sudo udevadm control --reload
sudo udevadm trigger sudo udevadm trigger
3. (Optional) Update firmware/FPGA images: 5. (Optional) Update firmware/FPGA images:
.. code-block:: bash .. code-block:: bash

1407
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,6 +1,6 @@
[project] [project]
name = "ria-toolkit-oss" name = "ria-toolkit-oss"
version = "0.1.7" version = "0.1.5"
description = "An open-source version of the RIA Toolkit, including the fundamental tools to get started developing, testing, and deploying radio intelligence applications" description = "An open-source version of the RIA Toolkit, including the fundamental tools to get started developing, testing, and deploying radio intelligence applications"
license = { text = "AGPL-3.0-only" } license = { text = "AGPL-3.0-only" }
readme = "README.md" readme = "README.md"

View File

@ -5,11 +5,8 @@ Subcommands:
- ``ria-agent run [legacy args]`` legacy long-poll NodeAgent (unchanged). - ``ria-agent run [legacy args]`` legacy long-poll NodeAgent (unchanged).
- ``ria-agent stream`` new WebSocket-based IQ streamer. - ``ria-agent stream`` new WebSocket-based IQ streamer.
- ``ria-agent detect`` print SDR drivers whose modules import cleanly. - ``ria-agent detect`` print SDR drivers whose modules import cleanly.
- ``ria-agent register --hub URL --api-key KEY`` register with the hub - ``ria-agent register --hub URL --api-key KEY`` register with the hub and
using a personal registration key (minted from **Settings RIA Agents** save credentials (and optional TX interlocks) to ``~/.ria/agent.json``.
on the hub, shown once at mint time) and save credentials (and optional
TX interlocks) to ``~/.ria/agent.json``. The hub also accepts the legacy
shared ``[wac] API_KEY`` for back-compat, but that path is deprecated.
Invoking ``ria-agent`` with no subcommand falls through to the legacy Invoking ``ria-agent`` with no subcommand falls through to the legacy
long-poll behavior for back-compatibility with existing deployments. long-poll behavior for back-compatibility with existing deployments.
@ -28,75 +25,9 @@ from .hardware import available_devices
from .legacy_executor import main as _legacy_main from .legacy_executor import main as _legacy_main
from .namegen import generate_agent_name from .namegen import generate_agent_name
DEFAULT_HUB_URL = "https://riahub.ai"
_LEGACY_ALIASES = {"--hub", "--key", "--name", "--device", "--insecure", "--log-level", "--config"} _LEGACY_ALIASES = {"--hub", "--key", "--name", "--device", "--insecure", "--log-level", "--config"}
def _user_agent() -> str:
"""Build the User-Agent header for hub requests.
Set explicitly so we don't fall back to Python's default `Python-urllib/<ver>`,
which is blocked by Cloudflare's Browser Integrity Check on `riahub.ai`
(HTTP 403 edge code 1010). Version is read from package metadata so it
tracks releases instead of going stale.
"""
from importlib.metadata import PackageNotFoundError, version
try:
pkg_version = version("ria-toolkit-oss")
except PackageNotFoundError:
pkg_version = "unknown"
return f"ria-agent/{pkg_version} (+https://riahub.ai/qoherent/ria-toolkit-oss)"
# How long to wait on the hub before giving up. The register endpoint is a
# small DB lookup + insert; anything past this is a stuck hub, not a slow one.
_REGISTER_TIMEOUT_S = 15
REGISTRATION_REASON_MESSAGES = {
"invalid_key": ("Registration key not recognized. Generate a fresh key from " "Settings → RIA Agents on the hub."),
"expired": ("This registration key has expired. Generate a new one from " "Settings → RIA Agents on the hub."),
"revoked": ("This registration key was revoked. Generate a new one from " "Settings → RIA Agents on the hub."),
"already_consumed": (
"This single-use registration key has already been used. "
"Generate a new one, or mint a reusable key instead."
),
}
def _explain_registration_failure(status: int, body: bytes) -> str:
"""Return a human-readable explanation for a failed register call."""
try:
parsed = json.loads(body) if body else None
except ValueError:
parsed = None
if status == 429:
# 429 carries a plain string detail, never a reason code.
if isinstance(parsed, dict) and parsed.get("detail"):
detail = parsed["detail"]
else:
detail = body.decode("utf-8", "replace") or "rate limited"
return f"Registration rate-limited by the hub: {detail}"
if not isinstance(parsed, dict):
text = body.decode("utf-8", "replace")
return f"HTTP {status}: {text or 'no body'}"
detail = parsed.get("detail")
if isinstance(detail, dict):
reason = detail.get("reason")
if reason in REGISTRATION_REASON_MESSAGES:
return REGISTRATION_REASON_MESSAGES[reason]
if reason:
return f"Registration rejected ({reason})"
if isinstance(detail, str) and detail:
return f"Registration rejected: {detail}"
return f"HTTP {status}: {parsed}"
def _cmd_detect(_args: argparse.Namespace) -> int: def _cmd_detect(_args: argparse.Namespace) -> int:
devices = available_devices() devices = available_devices()
if not devices: if not devices:
@ -108,7 +39,6 @@ def _cmd_detect(_args: argparse.Namespace) -> int:
def _cmd_register(args: argparse.Namespace) -> int: def _cmd_register(args: argparse.Namespace) -> int:
import urllib.error
import urllib.request import urllib.request
hub_url = args.hub.rstrip("/") hub_url = args.hub.rstrip("/")
@ -121,20 +51,11 @@ def _cmd_register(args: argparse.Namespace) -> int:
headers={ headers={
"Content-Type": "application/json", "Content-Type": "application/json",
"X-API-Key": args.api_key, "X-API-Key": args.api_key,
"User-Agent": _user_agent(),
}, },
) )
try: try:
with urllib.request.urlopen(req, timeout=_REGISTER_TIMEOUT_S) as resp: with urllib.request.urlopen(req) as resp:
data = json.loads(resp.read()) data = json.loads(resp.read())
except urllib.error.HTTPError as e:
try:
err_body = e.read()
except Exception:
err_body = b""
msg = _explain_registration_failure(e.code, err_body)
print(f"error: registration failed: {msg}", file=sys.stderr)
return 1
except Exception as e: except Exception as e:
print(f"error: registration failed: {e}", file=sys.stderr) print(f"error: registration failed: {e}", file=sys.stderr)
return 1 return 1
@ -159,7 +80,7 @@ def _cmd_register(args: argparse.Namespace) -> int:
cfg.tx_allowed_freq_ranges = [[float(lo), float(hi)] for lo, hi in freq_ranges] cfg.tx_allowed_freq_ranges = [[float(lo), float(hi)] for lo, hi in freq_ranges]
path = _config.save(cfg) path = _config.save(cfg)
print(f"Registered agent: {agent_id} ({name})") print(f"Registered agent: {agent_id}")
if cfg.tx_enabled: if cfg.tx_enabled:
caps: list[str] = [] caps: list[str] = []
if cfg.tx_max_gain_db is not None: if cfg.tx_max_gain_db is not None:
@ -219,17 +140,8 @@ def main() -> None:
sub.add_parser("detect", help="List available SDR drivers") sub.add_parser("detect", help="List available SDR drivers")
p_reg = sub.add_parser("register", help="Register agent with RIA Hub and save credentials") p_reg = sub.add_parser("register", help="Register agent with RIA Hub and save credentials")
p_reg.add_argument("--hub", default=DEFAULT_HUB_URL, help=f"RIA Hub URL (default: {DEFAULT_HUB_URL})") p_reg.add_argument("--hub", required=True, help="RIA Hub URL (e.g. http://whitehorse:3005)")
p_reg.add_argument( p_reg.add_argument("--api-key", dest="api_key", required=True, help="Hub API key")
"--api-key",
dest="api_key",
required=True,
help=(
"Personal registration key from the RIA Agents page on the hub "
"(format: ria_reg_...). Shown once when generated; save it then. "
"The legacy shared API key is also accepted but deprecated."
),
)
p_reg.add_argument("--name", default=None, help="Human-friendly agent name") p_reg.add_argument("--name", default=None, help="Human-friendly agent name")
p_reg.add_argument("--insecure", action="store_true", help="Skip TLS verification") p_reg.add_argument("--insecure", action="store_true", help="Skip TLS verification")
p_reg.add_argument( p_reg.add_argument(

View File

@ -45,14 +45,7 @@ def is_annotation_contained(inner: Annotation, outer: Annotation) -> bool:
outer_sample_stop = outer.sample_start + outer.sample_count outer_sample_stop = outer.sample_start + outer.sample_count
if inner.sample_start > outer.sample_start and inner_sample_stop < outer_sample_stop: if inner.sample_start > outer.sample_start and inner_sample_stop < outer_sample_stop:
if ( if inner.freq_lower_edge > outer.freq_lower_edge and inner.freq_upper_edge < outer.freq_upper_edge:
inner.freq_lower_edge is not None
and inner.freq_upper_edge is not None
and outer.freq_lower_edge is not None
and outer.freq_upper_edge is not None
and inner.freq_lower_edge > outer.freq_lower_edge
and inner.freq_upper_edge < outer.freq_upper_edge
):
return True return True
return False return False

View File

@ -38,13 +38,7 @@ def annotate_with_cusum(
:type annotation_type: str :type annotation_type: str
""" """
sample_rate = recording.metadata.get("sample_rate") sample_rate = recording.metadata["sample_rate"]
if sample_rate is None:
raise ValueError(
"Recording metadata does not contain 'sample_rate'. "
"Supply it with --sample-rate when using the CLI, or set "
"recording.sample_rate before calling this function."
)
center_frequency = recording.metadata.get("center_frequency", 0) center_frequency = recording.metadata.get("center_frequency", 0)
# Create an object of the time segmenter # Create an object of the time segmenter

View File

@ -6,7 +6,6 @@ and occupied bandwidth calculation following ITU-R SM.328 standard.
""" """
import json import json
import warnings
from typing import Tuple from typing import Tuple
import numpy as np import numpy as np
@ -120,17 +119,6 @@ def detect_signals_energy(
if active: if active:
boundaries.append((start, len(smoothed_power) - start)) boundaries.append((start, len(smoothed_power) - start))
if not boundaries and noise_floor > 0:
peak = float(np.max(smoothed_power))
dynamic_range = peak / noise_floor
if dynamic_range < threshold_factor:
warnings.warn(
f"detect_signals_energy: no signal boundaries found — dynamic range {dynamic_range:.2f}x is below "
f"the threshold factor {threshold_factor}x. The signal may be constant-envelope (e.g. CW or chirp). "
"If the entire recording is signal, use 'ria annotate cusum' to segment it as a single region.",
stacklevel=2,
)
# Merge boundaries that are closer than min_distance # Merge boundaries that are closer than min_distance
merged_boundaries = [] merged_boundaries = []
if boundaries: if boundaries:
@ -147,13 +135,7 @@ def detect_signals_energy(
merged_boundaries.append((start, length)) merged_boundaries.append((start, length))
# Create annotations from detected boundaries # Create annotations from detected boundaries
sample_rate = recording.metadata.get("sample_rate") sample_rate = recording.metadata["sample_rate"]
if sample_rate is None:
raise ValueError(
"Recording metadata does not contain 'sample_rate'. "
"Supply it with --sample-rate when using the CLI, or set "
"recording.sample_rate before calling this function."
)
center_frequency = recording.metadata.get("center_frequency", 0) center_frequency = recording.metadata.get("center_frequency", 0)
# Validate frequency method # Validate frequency method
@ -369,12 +351,7 @@ def annotate_with_obw(
>>> annotated = annotate_with_obw(recording, label="signal_obw") >>> annotated = annotate_with_obw(recording, label="signal_obw")
""" """
signal = recording.data[0] signal = recording.data[0]
sample_rate = recording.metadata.get("sample_rate") sample_rate = recording.metadata["sample_rate"]
if sample_rate is None:
raise ValueError(
"Recording metadata does not contain 'sample_rate'. "
"Set recording.sample_rate before calling this function."
)
center_freq = recording.metadata.get("center_frequency", 0) center_freq = recording.metadata.get("center_frequency", 0)
# Calculate OBW # Calculate OBW

View File

@ -49,7 +49,6 @@ allowing splitting of overlapping signals into separate training samples.
""" """
import json import json
import warnings
from typing import List, Optional, Tuple from typing import List, Optional, Tuple
import numpy as np import numpy as np
@ -402,13 +401,7 @@ def split_recording_annotations(
return recording return recording
signal = recording.data[0] signal = recording.data[0]
sample_rate = recording.metadata.get("sample_rate") sample_rate = recording.metadata["sample_rate"]
if sample_rate is None:
raise ValueError(
"Recording metadata does not contain 'sample_rate'. "
"Supply it with --sample-rate when using the CLI, or set "
"recording.sample_rate before calling this function."
)
center_frequency = recording.metadata.get("center_frequency", 0.0) center_frequency = recording.metadata.get("center_frequency", 0.0)
# Build new annotation list # Build new annotation list
@ -432,11 +425,8 @@ def split_recording_annotations(
else: else:
# No components found, keep original # No components found, keep original
new_annotations.append(anno) new_annotations.append(anno)
except Exception as e: except Exception:
warnings.warn( # Split failed for any reason, keep original
f"split_recording_annotations: failed to split annotation at index {i} ({e}); keeping original.",
stacklevel=2,
)
new_annotations.append(anno) new_annotations.append(anno)
else: else:
# Not in split list, keep as-is # Not in split list, keep as-is

View File

@ -24,7 +24,7 @@ def qualify_slice_from_annotations(recording: Recording, slice_length: int):
output_recordings = [] output_recordings = []
for i in range(len(recording.data[0]) // slice_length): for i in range((len(recording.data[0]) // slice_length) - 1):
start_index = slice_length * i start_index = slice_length * i
end_index = slice_length * (i + 1) end_index = slice_length * (i + 1)

View File

@ -35,24 +35,17 @@ def isolate_signal(recording: Recording, annotation: Annotation) -> Recording:
isolation_bw = anno_bw isolation_bw = anno_bw
sample_rate = recording.metadata.get("sample_rate")
if sample_rate is None:
raise ValueError(
"Recording metadata does not contain 'sample_rate'. "
"Set recording.sample_rate before calling isolate_signal."
)
# frequency shift the center of the box about zero # frequency shift the center of the box about zero
shifted_signal_slice = frequency_shift_iq_samples( shifted_signal_slice = frequency_shift_iq_samples(
iq_samples=signal_slice, iq_samples=signal_slice,
sample_rate=sample_rate, sample_rate=recording.metadata["sample_rate"],
shift_frequency=-1 * anno_base_center_freq, shift_frequency=-1 * anno_base_center_freq,
) )
# filter # filter
if isolation_bw < sample_rate - 1: if isolation_bw < recording.metadata["sample_rate"] - 1:
filtered_signal = apply_complex_lowpass_filter( filtered_signal = apply_complex_lowpass_filter(
signal=shifted_signal_slice, cutoff_frequency=isolation_bw, sample_rate=sample_rate signal=shifted_signal_slice, cutoff_frequency=isolation_bw, sample_rate=recording.metadata["sample_rate"]
) )
else: else:

View File

@ -42,7 +42,6 @@ classification or demodulation stages.
""" """
import json import json
import warnings
from typing import Optional from typing import Optional
import numpy as np import numpy as np
@ -217,22 +216,11 @@ def threshold_qualifier(
""" """
# Extract signal and metadata # Extract signal and metadata
sample_data = recording.data[channel] sample_data = recording.data[channel]
sample_rate = recording.metadata.get("sample_rate") sample_rate = recording.metadata["sample_rate"]
if sample_rate is None:
raise ValueError(
"Recording metadata does not contain 'sample_rate'. "
"Supply it with --sample-rate when using the CLI, or set "
"recording.sample_rate before calling this function."
)
center_frequency = recording.metadata.get("center_frequency", 0) center_frequency = recording.metadata.get("center_frequency", 0)
n_samples = len(sample_data)
if window_size is None: if window_size is None:
window_size = max(64, int(sample_rate * 0.001)) window_size = max(64, int(sample_rate * 0.001))
# Cap at 1% of signal length so short recordings aren't over-smoothed into
# a flat envelope that collapses the dynamic range below the early-exit guard.
window_size = min(window_size, max(64, n_samples // 100))
# --- 1. SIGNAL CONDITIONING --- # --- 1. SIGNAL CONDITIONING ---
# Convert to power (Magnitude squared) # Convert to power (Magnitude squared)
@ -249,12 +237,6 @@ def threshold_qualifier(
# Soft early exit: keep a guard for low-contrast noise, but compute it from # Soft early exit: keep a guard for low-contrast noise, but compute it from
# the quieter tail of the envelope so burst-heavy captures are not rejected. # the quieter tail of the envelope so burst-heavy captures are not rejected.
if dynamic_range_ratio < 1.5: if dynamic_range_ratio < 1.5:
warnings.warn(
f"threshold_qualifier: dynamic range ratio {dynamic_range_ratio:.2f} is below 1.5 — "
"the signal appears to be constant-envelope or pure noise, so no burst boundaries can be found. "
"If the entire recording is signal, use 'ria annotate cusum' to segment it as a single region.",
stacklevel=2,
)
return Recording(data=recording.data, metadata=recording.metadata, annotations=recording.annotations) return Recording(data=recording.data, metadata=recording.metadata, annotations=recording.annotations)
trigger_val = noise_floor + threshold * (max_power - noise_floor) trigger_val = noise_floor + threshold * (max_power - noise_floor)
@ -314,7 +296,7 @@ def threshold_qualifier(
# burst energy does not bleed through the long window into adjacent regions, # burst energy does not bleed through the long window into adjacent regions,
# which would inflate macro_residual_max and push the trigger above the # which would inflate macro_residual_max and push the trigger above the
# faint burst's average power. # faint burst's average power.
macro_window_size = min(max(window_size * 16, int(sample_rate * 0.02)), max(window_size * 2, n_samples // 4)) macro_window_size = max(window_size * 16, int(sample_rate * 0.02))
macro_kernel = np.ones(macro_window_size, dtype=np.float64) / macro_window_size macro_kernel = np.ones(macro_window_size, dtype=np.float64) / macro_window_size
# Expand each annotated range by half the macro window on both sides so that # Expand each annotated range by half the macro window on both sides so that
# the long convolution cannot "see" the leading/trailing edges of already- # the long convolution cannot "see" the leading/trailing edges of already-

View File

@ -17,10 +17,10 @@ class Annotation:
:type sample_start: int :type sample_start: int
:param sample_count: The index of the ending sample of the annotation, inclusive. :param sample_count: The index of the ending sample of the annotation, inclusive.
:type sample_count: int :type sample_count: int
:param freq_lower_edge: The lower frequency of the annotation. Optional; None if not specified in source. :param freq_lower_edge: The lower frequency of the annotation.
:type freq_lower_edge: float, optional :type freq_lower_edge: float
:param freq_upper_edge: The upper frequency of the annotation. Optional; None if not specified in source. :param freq_upper_edge: The upper frequency of the annotation.
:type freq_upper_edge: float, optional :type freq_upper_edge: float
:param label: The label that will be displayed with the bounding box in compatible viewers including IQEngine. :param label: The label that will be displayed with the bounding box in compatible viewers including IQEngine.
Defaults to an emtpy string. Defaults to an emtpy string.
:type label: str, optional :type label: str, optional
@ -34,8 +34,8 @@ class Annotation:
self, self,
sample_start: int, sample_start: int,
sample_count: int, sample_count: int,
freq_lower_edge: Optional[float] = None, freq_lower_edge: float,
freq_upper_edge: Optional[float] = None, freq_upper_edge: float,
label: Optional[str] = "", label: Optional[str] = "",
comment: Optional[str] = "", comment: Optional[str] = "",
detail: Optional[dict] = None, detail: Optional[dict] = None,
@ -43,8 +43,8 @@ class Annotation:
"""Initialize a new Annotation instance.""" """Initialize a new Annotation instance."""
self.sample_start = int(sample_start) self.sample_start = int(sample_start)
self.sample_count = int(sample_count) self.sample_count = int(sample_count)
self.freq_lower_edge = float(freq_lower_edge) if freq_lower_edge is not None else None self.freq_lower_edge = float(freq_lower_edge)
self.freq_upper_edge = float(freq_upper_edge) if freq_upper_edge is not None else None self.freq_upper_edge = float(freq_upper_edge)
self.label = str(label) self.label = str(label)
self.comment = str(comment) self.comment = str(comment)
@ -62,8 +62,6 @@ class Annotation:
:returns: True if valid, False if not. :returns: True if valid, False if not.
""" """
if self.freq_lower_edge is None or self.freq_upper_edge is None:
return self.sample_count > 0
return self.sample_count > 0 and self.freq_lower_edge < self.freq_upper_edge return self.sample_count > 0 and self.freq_lower_edge < self.freq_upper_edge
def overlap(self, other): def overlap(self, other):
@ -75,14 +73,6 @@ class Annotation:
:returns: The area of the overlap in samples*frequency, or 0 if they do not overlap.""" :returns: The area of the overlap in samples*frequency, or 0 if they do not overlap."""
if (
self.freq_lower_edge is None
or self.freq_upper_edge is None
or other.freq_lower_edge is None
or other.freq_upper_edge is None
):
return 0
sample_overlap_start = max(self.sample_start, other.sample_start) sample_overlap_start = max(self.sample_start, other.sample_start)
sample_overlap_end = min(self.sample_start + self.sample_count, other.sample_start + other.sample_count) sample_overlap_end = min(self.sample_start + self.sample_count, other.sample_start + other.sample_count)
@ -101,8 +91,6 @@ class Annotation:
:returns: sample length multiplied by bandwidth.""" :returns: sample length multiplied by bandwidth."""
if self.freq_lower_edge is None or self.freq_upper_edge is None:
return 0
return self.sample_count * (self.freq_upper_edge - self.freq_lower_edge) return self.sample_count * (self.freq_upper_edge - self.freq_lower_edge)
def __eq__(self, other: Annotation) -> bool: def __eq__(self, other: Annotation) -> bool:
@ -115,16 +103,13 @@ class Annotation:
annotation_dict = {SigMFFile.START_INDEX_KEY: self.sample_start, SigMFFile.LENGTH_INDEX_KEY: self.sample_count} annotation_dict = {SigMFFile.START_INDEX_KEY: self.sample_start, SigMFFile.LENGTH_INDEX_KEY: self.sample_count}
metadata = { annotation_dict["metadata"] = {
SigMFFile.LABEL_KEY: self.label, SigMFFile.LABEL_KEY: self.label,
SigMFFile.COMMENT_KEY: self.comment, SigMFFile.COMMENT_KEY: self.comment,
SigMFFile.FHI_KEY: self.freq_upper_edge,
SigMFFile.FLO_KEY: self.freq_lower_edge,
"ria:detail": self.detail, "ria:detail": self.detail,
} }
if self.freq_upper_edge is not None:
metadata[SigMFFile.FHI_KEY] = self.freq_upper_edge
if self.freq_lower_edge is not None:
metadata[SigMFFile.FLO_KEY] = self.freq_lower_edge
annotation_dict["metadata"] = metadata
if _is_jsonable(annotation_dict): if _is_jsonable(annotation_dict):
return annotation_dict return annotation_dict

View File

@ -175,15 +175,6 @@ def from_npy(file: os.PathLike | str, legacy: bool = False) -> Recording:
) )
data = first # already loaded without pickle (numeric array) data = first # already loaded without pickle (numeric array)
metadata = np.load(f, allow_pickle=True).tolist() metadata = np.load(f, allow_pickle=True).tolist()
# Normalize namespaced keys (e.g. "BlockGenerator:Foo:sample_rate") to
# their bare equivalents so downstream code can find them reliably.
_STANDARD_KEYS = {"sample_rate", "center_frequency", "bandwidth"}
if isinstance(metadata, dict):
for k in list(metadata):
if ":" in k:
bare = k.rsplit(":", 1)[-1]
if bare in _STANDARD_KEYS and bare not in metadata:
metadata[bare] = metadata[k]
try: try:
annotations = list(np.load(f, allow_pickle=True)) annotations = list(np.load(f, allow_pickle=True))
except EOFError: except EOFError:

View File

@ -32,15 +32,16 @@ def extract_metadata_fields(metadata):
def set_path(output_path): def set_path(output_path):
path = pathlib.Path(output_path) split_path = output_path.split("/")
# If only filename provided (no directory), use default 'images' folder if len(split_path) == 1:
if len(path.parts) == 1: folder = "images"
folder = pathlib.Path("images") file = split_path[0]
file = path.name elif len(split_path) > 2:
file = split_path[-1]
folder = "/".join(split_path[:-1])
else: else:
folder = path.parent folder, file = split_path
file = path.name
split_file = file.split(".") split_file = file.split(".")
if len(split_file) == 2: if len(split_file) == 2:
@ -52,5 +53,5 @@ def set_path(output_path):
extension = "png" extension = "png"
file = file + ".png" file = file + ".png"
folder.mkdir(parents=True, exist_ok=True) pathlib.Path(folder).mkdir(parents=True, exist_ok=True)
return str(folder / file), extension return "/".join([folder, file]), extension

View File

@ -3,12 +3,11 @@ import os
import textwrap import textwrap
from typing import Optional from typing import Optional
import matplotlib
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np import numpy as np
from matplotlib import gridspec, ticker from matplotlib import gridspec
from matplotlib.patches import Patch from matplotlib.patches import Patch
from PIL import Image, UnidentifiedImageError from PIL import Image
from scipy.fft import fft, fftshift from scipy.fft import fft, fftshift
from scipy.signal import spectrogram from scipy.signal import spectrogram
from scipy.signal.windows import hann from scipy.signal.windows import hann
@ -81,8 +80,6 @@ def view_annotations(
return 0 return 0
for annotation in sorted(annotations, key=_threshold_sort_key, reverse=True): for annotation in sorted(annotations, key=_threshold_sort_key, reverse=True):
if annotation.freq_lower_edge is None or annotation.freq_upper_edge is None:
continue
t_start = annotation.sample_start / sample_rate t_start = annotation.sample_start / sample_rate
t_width = annotation.sample_count / sample_rate t_width = annotation.sample_count / sample_rate
f_start = annotation.freq_lower_edge f_start = annotation.freq_lower_edge
@ -188,7 +185,7 @@ def view_sig(
logo: Optional[bool] = True, logo: Optional[bool] = True,
dark: Optional[bool] = True, dark: Optional[bool] = True,
spines: Optional[bool] = False, spines: Optional[bool] = False,
title_fontsize: Optional[int] = 25, title_fontsize: Optional[int] = 35,
subtitle_fontsize: Optional[int] = 15, subtitle_fontsize: Optional[int] = 15,
) -> None: ) -> None:
""" """
@ -233,26 +230,11 @@ def view_sig(
complex_signal = recording.data[0] complex_signal = recording.data[0]
sample_rate, center_frequency, _ = extract_metadata_fields(recording.metadata) sample_rate, center_frequency, _ = extract_metadata_fields(recording.metadata)
subplot_height = 3 * (plot_spectrogram) + 2 * (iq + frequency) + 3 * (constellation or metadata or logo) subplot_height = 2 * (plot_spectrogram + iq + frequency) + 3 * (constellation or metadata or logo)
subplot_width = max((constellation + metadata or 1), logo * 3) subplot_width = max((constellation + metadata or 1), logo * 3)
if dark: if dark:
plt.style.use("dark_background") plt.style.use("dark_background")
matplotlib.rcParams.update(
{
"figure.facecolor": "#161616",
"axes.facecolor": "#161616",
"savefig.facecolor": "#161616",
"savefig.edgecolor": "#161616",
"font.size": 10,
"axes.titlesize": 15,
"axes.labelsize": 10,
"xtick.labelsize": 10,
"ytick.labelsize": 10,
"legend.frameon": False,
"legend.facecolor": "none",
}
)
logo_path = os.path.dirname(__file__) + "/graphics/Qoherent-logo-white-transparent.png" logo_path = os.path.dirname(__file__) + "/graphics/Qoherent-logo-white-transparent.png"
else: else:
plt.style.use("default") plt.style.use("default")
@ -270,8 +252,8 @@ def view_sig(
plot_x_indx = 0 plot_x_indx = 0
if plot_spectrogram: if plot_spectrogram:
spec_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 3, :]) spec_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
plot_y_indx = plot_y_indx + 3 plot_y_indx = plot_y_indx + 2
fft_size = get_fft_size(plot_length=plot_length) fft_size = get_fft_size(plot_length=plot_length)
_, t_spec, Sxx = spectrogram( _, t_spec, Sxx = spectrogram(
@ -298,10 +280,7 @@ def view_sig(
) )
set_spines(spec_ax, spines) set_spines(spec_ax, spines)
spec_ax.set_title("Spectrogram", loc="left", fontsize=subtitle_fontsize) spec_ax.set_title("Spectrogram", loc="center", fontsize=subtitle_fontsize)
spec_ax.set_xlabel("Time (s)")
spec_ax.set_ylabel("Frequency (MHz)")
spec_ax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
if iq: if iq:
iq_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :]) iq_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
@ -312,13 +291,12 @@ def view_sig(
iq_ax.plot(t, plot_iq.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I") iq_ax.plot(t, plot_iq.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
iq_ax.plot(t, plot_iq.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q") iq_ax.plot(t, plot_iq.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
iq_ax.grid(True, alpha=0.2, linewidth=0.5) iq_ax.grid(False)
iq_ax.set_ylabel("Amplitude") iq_ax.set_ylabel("Amplitude")
iq_ax.set_xlim([min(t), max(t)]) iq_ax.set_xlim([min(t), max(t)])
iq_ax.set_xlabel("Time (s)") iq_ax.set_xlabel("Time (s)")
iq_ax.set_title("IQ Sample Plot", loc="left", fontsize=subtitle_fontsize) iq_ax.set_title("IQ Sample Plot", fontsize=subtitle_fontsize)
iq_ax.legend(loc="upper right", fontsize=10)
set_spines(iq_ax, spines) set_spines(iq_ax, spines)
if frequency: if frequency:
@ -332,14 +310,10 @@ def view_sig(
# Convert to dB # Convert to dB
spectrum_db = 20 * np.log10(spectrum + 1e-12) # 20*log for magnitude spectrum_db = 20 * np.log10(spectrum + 1e-12) # 20*log for magnitude
freqs = ( freqs = np.linspace(-sample_rate / 2, sample_rate / 2, len(complex_signal[:plot_length])) + center_frequency
np.linspace(-sample_rate / 2, sample_rate / 2, len(complex_signal[:plot_length])) + center_frequency
) / 1e6
freq_ax.plot(freqs, spectrum_db, color=COLORS["accent"], linewidth=0.8) freq_ax.plot(freqs, spectrum_db, color=COLORS["accent"], linewidth=0.8)
freq_ax.set_xlabel("Frequency (MHz)")
freq_ax.set_ylabel("Magnitude (dB)") freq_ax.set_ylabel("Magnitude (dB)")
freq_ax.grid(True, alpha=0.2, linewidth=0.5) freq_ax.set_title("Frequency Spectrum (Windowed FFT)", fontsize=subtitle_fontsize)
freq_ax.set_title("Frequency Spectrum (Windowed FFT)", loc="left", fontsize=subtitle_fontsize)
set_spines(freq_ax, spines) set_spines(freq_ax, spines)
if constellation: if constellation:
@ -352,7 +326,7 @@ def view_sig(
const_ax.set_ylim([-1 * dimension, dimension]) const_ax.set_ylim([-1 * dimension, dimension])
const_ax.set_xlabel("In-phase (I)") const_ax.set_xlabel("In-phase (I)")
const_ax.set_ylabel("Quadrature (Q)") const_ax.set_ylabel("Quadrature (Q)")
const_ax.set_title("Constellation", loc="left", fontsize=subtitle_fontsize) const_ax.set_title("Constellation", fontsize=subtitle_fontsize)
const_ax.set_aspect("equal") const_ax.set_aspect("equal")
if not spines: if not spines:
@ -401,8 +375,8 @@ def view_sig(
image = Image.open(logo_path) # Open the PNG image using PIL image = Image.open(logo_path) # Open the PNG image using PIL
logo_ax.imshow(image) logo_ax.imshow(image)
except (FileNotFoundError, UnidentifiedImageError, OSError) as exc: except FileNotFoundError:
print(f"Warning, could not load logo image: {logo_path}. Reason: {exc}") print(f"Warning, {logo_path} not found.")
fig.subplots_adjust( fig.subplots_adjust(
left=0.1, # Left margin left=0.1, # Left margin

View File

@ -119,19 +119,24 @@ def setup_style(*, labels_mode: bool = False, compact_mode: bool = False) -> Non
label_font = 14 label_font = 14
else: else:
base_font = 10 base_font = 10
title_font = 15 title_font = 12
label_font = 10 label_font = 10
matplotlib.rcParams.update( matplotlib.rcParams.update(
{ {
"figure.facecolor": "#161616", "figure.facecolor": "#0f172a",
"axes.facecolor": "#161616", "axes.facecolor": "#1e293b",
"savefig.facecolor": "#161616", "axes.edgecolor": COLORS["muted"],
"savefig.edgecolor": "#161616", "axes.labelcolor": COLORS["light"],
"text.color": COLORS["light"],
"xtick.color": COLORS["muted"],
"ytick.color": COLORS["muted"],
"grid.color": COLORS["muted"],
"grid.alpha": 0.3,
"font.size": base_font, "font.size": base_font,
"axes.titlesize": title_font, "axes.titlesize": title_font,
"axes.labelsize": label_font, "axes.labelsize": label_font,
"figure.titlesize": title_font + 4, "figure.titlesize": title_font + 2,
"legend.frameon": False, "legend.frameon": False,
"legend.facecolor": "none", "legend.facecolor": "none",
"xtick.labelsize": base_font, "xtick.labelsize": base_font,
@ -189,7 +194,7 @@ def view_simple_sig(
constellation_mode: Optional[bool] = False, constellation_mode: Optional[bool] = False,
labels_mode: Optional[bool] = False, labels_mode: Optional[bool] = False,
slice: Optional[tuple] = None, slice: Optional[tuple] = None,
title: Optional[str] = "Signal Plot", title: Optional[str] = "Signal",
): ):
""" """
Create a simple plot of various signal visualizations as a png or svg image. Create a simple plot of various signal visualizations as a png or svg image.
@ -232,7 +237,7 @@ def view_simple_sig(
spec_signal = signal spec_signal = signal
if compact_mode: if compact_mode:
fig, (ax2, ax1) = plt.subplots(2, 1, figsize=(12, 6), gridspec_kw={"height_ratios": [5, 1]}) fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 6), gridspec_kw={"height_ratios": [1, 5]})
show_title = False show_title = False
show_labels = False show_labels = False
ax_constellation = ax_psd = None ax_constellation = ax_psd = None
@ -248,24 +253,25 @@ def view_simple_sig(
ax_psd = None ax_psd = None
else: else:
if constellation_mode: if constellation_mode:
fig, ((ax2, ax1), (ax3, ax4)) = plt.subplots(2, 2, figsize=(16, 12)) fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(16, 12))
ax_constellation, ax_psd = ax3, ax4 ax_constellation, ax_psd = ax3, ax4
else: else:
fig, (ax2, ax1) = plt.subplots(2, 1, figsize=(14, 10)) fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 10))
ax_constellation = ax_psd = None ax_constellation = ax_psd = None
show_title = True show_title = True
show_labels = labels_mode show_labels = labels_mode
if show_title: if show_title:
fig.suptitle(title, fontsize=25) fig.suptitle(title, fontsize=16, color=COLORS["light"], y=0.96)
fig.patch.set_facecolor(matplotlib.rcParams["figure.facecolor"]) fig.patch.set_facecolor("#0f172a")
total_duration_s = len(signal) / sample_rate_hz if sample_rate_hz else 0.0 total_duration_s = len(signal) / sample_rate_hz if sample_rate_hz else 0.0
t_s = np.linspace(0, total_duration_s, len(display_signal)) if len(display_signal) else np.array([]) t_s = np.linspace(0, total_duration_s, len(display_signal)) if len(display_signal) else np.array([])
ax1.plot(t_s, display_signal.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I") ax1.plot(t_s, display_signal.real, color=COLORS["purple"], linewidth=0.8, alpha=0.8, label="I")
ax1.plot(t_s, display_signal.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q") ax1.plot(t_s, display_signal.imag, color=COLORS["magenta"], linewidth=0.8, alpha=0.8, label="Q")
ax1.grid(True, alpha=0.2, linewidth=0.5) ax1.set_xlim(0, total_duration_s)
ax1.grid(True, alpha=0.3)
nfft, overlap = _get_nfft_size(signal=signal, fast_mode=fast_mode) nfft, overlap = _get_nfft_size(signal=signal, fast_mode=fast_mode)
@ -279,7 +285,7 @@ def view_simple_sig(
) )
ax2.set_ylim(center_freq_hz - sample_rate_hz / 2, center_freq_hz + sample_rate_hz / 2) ax2.set_ylim(center_freq_hz - sample_rate_hz / 2, center_freq_hz + sample_rate_hz / 2)
ax1.set_xlim(ax2.get_xlim()) ax2.set_xlim(0, total_duration_s)
if show_labels: if show_labels:
if horizontal_mode: if horizontal_mode:
@ -288,25 +294,20 @@ def view_simple_sig(
ax2.set_xlabel("Time (s)") ax2.set_xlabel("Time (s)")
ax1.set_ylabel("Amplitude") ax1.set_ylabel("Amplitude")
ax1.set_title(f"IQ Sample Plot - {sdr} SDR", loc="left", pad=10, fontsize=15) ax1.set_title(f"Time Series - {sdr} SDR", loc="left", pad=10)
ax1.legend(loc="upper right", fontsize=10) ax1.legend(loc="upper right")
ax2.set_ylabel("Frequency (MHz)") ax2.set_ylabel("Frequency (Hz)")
ax2.set_title( ax2.set_title(
f"Spectrogram - {center_freq_hz / 1e6:.1f} MHz ± {sample_rate_hz / 2e6:.1f} MHz", f"Spectrogram - {center_freq_hz / 1e6:.1f} MHz ± {sample_rate_hz / 2e6:.1f} MHz", loc="left", pad=10
loc="left",
pad=10,
fontsize=15,
) )
ax2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}")) yticks = ax2.get_yticks()
ax2.set_yticklabels([f"{y / 1e6:.1f}" for y in yticks])
elif not compact_mode: elif not compact_mode:
ax1.set_title("IQ Sample Plot", loc="left", pad=10, fontsize=15) ax1.set_title("Time Series", loc="left", pad=10)
ax1.legend(loc="upper right", fontsize=10) ax1.legend(loc="upper right", fontsize=8)
ax2.set_xlabel("Time (s)") ax2.set_title("Spectrogram", loc="left", pad=10)
ax2.set_ylabel("Frequency (MHz)")
ax2.set_title("Spectrogram", loc="left", pad=10, fontsize=15)
ax2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
_add_annotations( _add_annotations(
annotations=annotations, annotations=annotations,
@ -338,8 +339,8 @@ def view_simple_sig(
) )
ax_constellation.set_xlabel("In-phase (I)") ax_constellation.set_xlabel("In-phase (I)")
ax_constellation.set_ylabel("Quadrature (Q)") ax_constellation.set_ylabel("Quadrature (Q)")
ax_constellation.set_title("Constellation", loc="left", fontsize=15) ax_constellation.set_title("Constellation")
ax_constellation.grid(True, alpha=0.2, linewidth=0.5) ax_constellation.grid(True, alpha=0.3)
ax_constellation.set_aspect("equal") ax_constellation.set_aspect("equal")
if ax_psd is not None: if ax_psd is not None:
@ -350,11 +351,11 @@ def view_simple_sig(
freqs = freqs + center_freq_hz freqs = freqs + center_freq_hz
spectrum_db = 10 * np.log10(spectrum + 1e-12) spectrum_db = 10 * np.log10(spectrum + 1e-12)
ax_psd.plot(freqs / 1e6, spectrum_db, color=COLORS["accent"], linewidth=0.8) ax_psd.plot(freqs / 1e6, spectrum_db, color=COLORS["accent"], linewidth=1.0)
ax_psd.set_xlabel("Frequency (MHz)") ax_psd.set_xlabel("Frequency (MHz)")
ax_psd.set_ylabel("Power (dB)") ax_psd.set_ylabel("Power (dB)")
ax_psd.set_title("Power Spectral Density", loc="left", fontsize=15) ax_psd.set_title("Power Spectral Density")
ax_psd.grid(True, alpha=0.2, linewidth=0.5) ax_psd.grid(True, alpha=0.3)
if compact_mode: if compact_mode:
ax1.set_xticks([]) ax1.set_xticks([])
@ -366,20 +367,13 @@ def view_simple_sig(
else: else:
plt.tight_layout() plt.tight_layout()
if show_title: if show_title:
plt.subplots_adjust(top=0.9) plt.subplots_adjust(top=0.92)
if saveplot: if saveplot:
output_path, extension = set_path(output_path=output_path) output_path, extension = set_path(output_path=output_path)
dpi_value = _set_dpi(fast_mode=fast_mode, labels_mode=labels_mode, extension=extension) dpi_value = _set_dpi(fast_mode=fast_mode, labels_mode=labels_mode, extension=extension)
plt.savefig( plt.savefig(output_path, dpi=dpi_value, bbox_inches="tight", facecolor="#0f172a", edgecolor="none")
output_path,
dpi=dpi_value,
bbox_inches="tight",
pad_inches=0.3,
facecolor=matplotlib.rcParams["savefig.facecolor"],
edgecolor=matplotlib.rcParams["savefig.edgecolor"],
)
print(f"Saved signal plot to {output_path}") print(f"Saved signal plot to {output_path}")
return output_path return output_path

View File

@ -2,57 +2,15 @@
This module contains the main group for the ria toolkit oss CLI. This module contains the main group for the ria toolkit oss CLI.
""" """
import subprocess import click
import sys
import warnings
warnings.filterwarnings( from ria_toolkit_oss_cli.ria_toolkit_oss import commands
"ignore",
message="Unable to import Axes3D",
category=UserWarning,
module="matplotlib",
)
import click # noqa: E402
from ria_toolkit_oss_cli.ria_toolkit_oss import commands # noqa: E402
def _git_lfs_installed() -> bool: @click.group()
"""Return True if git-lfs is available on PATH."""
try:
return (
subprocess.run(
["git", "lfs", "version"],
capture_output=True,
).returncode
== 0
)
except FileNotFoundError:
return False
@click.group(invoke_without_command=True)
@click.option("-v", "--verbose", is_flag=True, type=bool, help="Increase verbosity, especially useful for debugging.") @click.option("-v", "--verbose", is_flag=True, type=bool, help="Increase verbosity, especially useful for debugging.")
@click.pass_context def cli(verbose):
def cli(ctx, verbose): pass
lfs_missing = not _git_lfs_installed()
if lfs_missing:
click.echo(
"Warning: git-lfs is not installed. RIA Hub projects require git-lfs to\n"
"track large binary files (models, recordings, datasets).\n"
"\n"
" Linux: sudo apt-get install git-lfs\n"
" macOS: brew install git-lfs\n"
" Other platforms: https://git-lfs.com\n"
"\n"
"After installing, run: git lfs install",
err=True,
)
if ctx.invoked_subcommand is None:
if lfs_missing and sys.stdin.isatty():
click.pause(info="\nPress Enter to continue...", err=True)
click.echo(ctx.get_help())
# Loop through project commands, binding them all to the CLI. # Loop through project commands, binding them all to the CLI.

View File

@ -1,97 +0,0 @@
"""Shared authentication and security helpers for RIA Hub API calls."""
import base64
import subprocess
import urllib.error
import urllib.parse
import urllib.request
import click
DEFAULT_HUB = "https://riahub.ai"
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
"""Block redirects on authenticated requests to prevent credential exfiltration.
urllib re-sends the Authorization header on same-host redirects by default.
A malicious server could redirect a POST to a different host to harvest
credentials. We refuse all redirects API clients should not encounter them
in normal operation.
"""
def redirect_request(self, req, fp, code, msg, headers, newurl):
raise urllib.error.URLError(f"Unexpected redirect ({code}) to {newurl} — aborting to protect credentials")
def hub_opener() -> urllib.request.OpenerDirector:
"""Return a urllib opener that blocks redirects."""
return urllib.request.build_opener(_NoRedirectHandler)
def warn_if_insecure(hub: str) -> None:
"""Warn when credentials would be sent over plain HTTP to a non-localhost host."""
parsed = urllib.parse.urlparse(hub)
if parsed.scheme == "http":
host = parsed.hostname or ""
if host not in ("localhost", "127.0.0.1", "::1"):
click.echo(
f"Warning: sending credentials over plain HTTP to {host}. " "Use HTTPS in production.",
err=True,
)
def basic_auth(username: str, password: str) -> str:
return "Basic " + base64.b64encode(f"{username}:{password}".encode()).decode()
def get_stored_credentials(hub_url: str) -> tuple[str | None, str | None]:
"""Ask git credential fill for stored creds. Returns (username, password) or (None, None)."""
parsed = urllib.parse.urlparse(hub_url)
payload = f"protocol={parsed.scheme}\nhost={parsed.netloc}\n\n"
try:
result = subprocess.run(
["git", "credential", "fill"],
input=payload,
capture_output=True,
text=True,
timeout=5,
)
creds = {}
for line in result.stdout.splitlines():
# Partition on the FIRST '=' only so passwords containing '=' are preserved.
k, sep, v = line.partition("=")
if sep:
creds[k.strip()] = v # keep value verbatim
return creds.get("username"), creds.get("password")
except Exception:
return None, None
def store_credentials(hub_url: str, username: str, password: str) -> None:
"""Cache credentials via git credential approve (uses the system keychain/store)."""
parsed = urllib.parse.urlparse(hub_url)
payload = (
f"protocol={parsed.scheme}\n" f"host={parsed.netloc}\n" f"username={username}\n" f"password={password}\n\n"
)
try:
subprocess.run(
["git", "credential", "approve"],
input=payload,
capture_output=True,
text=True,
timeout=5,
)
except Exception:
pass # non-fatal — next push just prompts again
def resolve_credentials(hub: str) -> tuple[str, str]:
"""Return (username, password), prompting interactively if not cached."""
username, password = get_stored_credentials(hub)
if username and password:
return username, password
click.echo(f"No stored credentials found for {hub}.")
username = click.prompt("RIA Hub username")
password = click.prompt("Password / personal access token", hide_input=True)
return username, password

View File

@ -51,7 +51,7 @@ def detect_input_format(filepath):
raise click.ClickException(f"Unknown format for '{filepath}'. Supported: .sigmf, .npy, .wav, .blue") raise click.ClickException(f"Unknown format for '{filepath}'. Supported: .sigmf, .npy, .wav, .blue")
def determine_output_path(input_path, output_path, fmt, overwrite): def determine_output_path(input_path, output_path, fmt, quiet, overwrite):
input_path = Path(input_path) input_path = Path(input_path)
input_is_annotated = input_path.stem.endswith("_annotated") input_is_annotated = input_path.stem.endswith("_annotated")
@ -63,20 +63,24 @@ def determine_output_path(input_path, output_path, fmt, overwrite):
else: else:
target = input_path.with_name(f"{input_path.stem}_annotated{input_path.suffix}") target = input_path.with_name(f"{input_path.stem}_annotated{input_path.suffix}")
final_path = normalize_sigmf_path(target) if fmt == "sigmf" else target if fmt == "sigmf":
final_path = normalize_sigmf_path(target)
if not quiet:
click.echo(f"Saving SigMF metadata to: {final_path}")
else:
final_path = target
if not quiet:
click.echo(f"Saving to: {final_path}")
if final_path.exists() and not overwrite: # Always allow writing to _annotated files; guard against overwriting originals
raise click.ClickException(f"{final_path} already exists. Use --overwrite to replace it.") target_is_annotated = final_path.stem.endswith("_annotated")
if final_path.exists() and not target_is_annotated and final_path != input_path:
click.echo(f"Error: {final_path} is not an annotated file and cannot be overwritten.", err=True)
return None
return final_path return final_path
def check_output_available(input_path, output_path, overwrite):
"""Raise ClickException before any work begins if the output file already exists."""
fmt = detect_input_format(Path(input_path))
determine_output_path(input_path=input_path, output_path=output_path, fmt=fmt, overwrite=overwrite)
def save_recording_auto(recording, output_path, input_path, quiet=False, overwrite=False): def save_recording_auto(recording, output_path, input_path, quiet=False, overwrite=False):
"""Save recording, auto-detecting format from extension. """Save recording, auto-detecting format from extension.
@ -86,13 +90,10 @@ def save_recording_auto(recording, output_path, input_path, quiet=False, overwri
input_path = Path(input_path) input_path = Path(input_path)
fmt = detect_input_format(input_path) fmt = detect_input_format(input_path)
output_path = determine_output_path(input_path=input_path, output_path=output_path, fmt=fmt, overwrite=overwrite) # Determine output path
output_path = determine_output_path(
if not quiet: input_path=input_path, output_path=output_path, fmt=fmt, quiet=quiet, overwrite=overwrite
if fmt == "sigmf": )
click.echo(f"Saving SigMF metadata to: {output_path}")
else:
click.echo(f"Saving to: {output_path}")
if fmt == "sigmf": if fmt == "sigmf":
# Normalize path for SigMF # Normalize path for SigMF
@ -256,11 +257,7 @@ def list(input, verbose):
user_comment = ann.comment or "" user_comment = ann.comment or ""
# Basic info # Basic info
freq_range = ( freq_range = f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
if ann.freq_lower_edge is not None and ann.freq_upper_edge is not None
else "N/A"
)
click.echo( click.echo(
f" [{i}] Samples {format_sample_count(ann.sample_start)}-" f" [{i}] Samples {format_sample_count(ann.sample_start)}-"
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {ann.label}" f"{format_sample_count(ann.sample_start + ann.sample_count)}: {ann.label}"
@ -315,8 +312,6 @@ def add(input, start, count, label, freq_lower, freq_upper, comment, annotation_
except Exception as e: except Exception as e:
raise click.ClickException(f"Failed to load recording: {e}") raise click.ClickException(f"Failed to load recording: {e}")
check_output_available(input, output, overwrite)
# Validate sample range # Validate sample range
n_samples = len(recording.data[0]) n_samples = len(recording.data[0])
if start < 0: if start < 0:
@ -368,9 +363,12 @@ def add(input, start, count, label, freq_lower, freq_upper, comment, annotation_
if comment: if comment:
click.echo(f" Comment: {comment}") click.echo(f" Comment: {comment}")
try:
save_recording_auto(recording, output, input, quiet, overwrite) save_recording_auto(recording, output, input, quiet, overwrite)
if not quiet: if not quiet:
click.echo(" ✓ Saved") click.echo(" ✓ Saved")
except Exception as e:
raise click.ClickException(f"Failed to save: {e}")
# ============================================================================ # ============================================================================
@ -468,6 +466,8 @@ def clear(input, output, overwrite, force, quiet):
if not quiet: if not quiet:
click.echo(f"\nCleared {count_before} annotation(s)") click.echo(f"\nCleared {count_before} annotation(s)")
recording._annotations = []
try: try:
save_recording_auto(recording, output_path=input, input_path=input, quiet=quiet, overwrite=True) save_recording_auto(recording, output_path=input, input_path=input, quiet=quiet, overwrite=True)
if not quiet: if not quiet:
@ -503,9 +503,6 @@ def clear(input, output, overwrite, force, quiet):
default="standalone", default="standalone",
help="Annotation type", help="Annotation type",
) )
@click.option(
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
)
@click.option("--output", "-o", type=click.Path(), help="Output file path") @click.option("--output", "-o", type=click.Path(), help="Output file path")
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)") @click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
@click.option("--quiet", is_flag=True, help="Quiet mode") @click.option("--quiet", is_flag=True, help="Quiet mode")
@ -520,7 +517,6 @@ def energy(
nfft, nfft,
obw_power, obw_power,
annotation_type, annotation_type,
sample_rate,
output, output,
overwrite, overwrite,
quiet, quiet,
@ -543,11 +539,8 @@ def energy(
ria annotate energy signal.npy --threshold 1.5 --min-distance 10000 ria annotate energy signal.npy --threshold 1.5 --min-distance 10000
ria annotate energy signal.sigmf-data --freq-method obw ria annotate energy signal.sigmf-data --freq-method obw
ria annotate energy signal.sigmf-data --freq-method full-detected ria annotate energy signal.sigmf-data --freq-method full-detected
ria annotate energy signal.npy --sample-rate 1e6
""" """
check_output_available(input, output, overwrite)
try: try:
recording = load_recording(input) recording = load_recording(input)
if not quiet: if not quiet:
@ -555,15 +548,6 @@ def energy(
except Exception as e: except Exception as e:
raise click.ClickException(f"Failed to load recording: {e}") raise click.ClickException(f"Failed to load recording: {e}")
if sample_rate is not None:
recording.sample_rate = sample_rate
if recording.sample_rate is None:
raise click.ClickException(
"Recording metadata does not contain a sample rate. "
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
)
if not quiet: if not quiet:
click.echo("\nDetecting signals using energy-based method...") click.echo("\nDetecting signals using energy-based method...")
click.echo(" Time detection:") click.echo(" Time detection:")
@ -591,12 +575,12 @@ def energy(
if not quiet: if not quiet:
click.echo(f" ✓ Added {added} annotation(s)") click.echo(f" ✓ Added {added} annotation(s)")
except Exception as e:
raise click.ClickException(f"Energy detection failed: {e}")
save_recording_auto(recording, output, input, quiet, overwrite) save_recording_auto(recording, output, input, quiet, overwrite)
if not quiet: if not quiet:
click.echo(" ✓ Saved") click.echo(" ✓ Saved")
except Exception as e:
raise click.ClickException(f"Energy detection failed: {e}")
# ============================================================================ # ============================================================================
@ -617,13 +601,10 @@ def energy(
default="standalone", default="standalone",
help="Annotation type", help="Annotation type",
) )
@click.option(
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
)
@click.option("--output", "-o", type=click.Path(), help="Output file path") @click.option("--output", "-o", type=click.Path(), help="Output file path")
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)") @click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
@click.option("--quiet", is_flag=True, help="Quiet mode") @click.option("--quiet", is_flag=True, help="Quiet mode")
def cusum(input, label, min_duration, window_size, tolerance, annotation_type, sample_rate, output, overwrite, quiet): def cusum(input, label, min_duration, window_size, tolerance, annotation_type, output, overwrite, quiet):
"""Auto-detect segments using CUSUM method. """Auto-detect segments using CUSUM method.
Detects signal state changes (on/off, amplitude transitions). Best for Detects signal state changes (on/off, amplitude transitions). Best for
@ -635,10 +616,7 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
Examples: Examples:
ria annotate cusum signal.sigmf-data --min-duration 5.0 ria annotate cusum signal.sigmf-data --min-duration 5.0
ria annotate cusum data.npy --min-duration 10.0 --label state ria annotate cusum data.npy --min-duration 10.0 --label state
ria annotate cusum data.npy --sample-rate 1e6
""" """
check_output_available(input, output, overwrite)
try: try:
recording = load_recording(input) recording = load_recording(input)
if not quiet: if not quiet:
@ -646,15 +624,6 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
except Exception as e: except Exception as e:
raise click.ClickException(f"Failed to load recording: {e}") raise click.ClickException(f"Failed to load recording: {e}")
if sample_rate is not None:
recording.sample_rate = sample_rate
if recording.sample_rate is None:
raise click.ClickException(
"Recording metadata does not contain a sample rate. "
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
)
if not quiet: if not quiet:
click.echo("\nDetecting segments using CUSUM...") click.echo("\nDetecting segments using CUSUM...")
click.echo(f" Min duration: {min_duration} ms") click.echo(f" Min duration: {min_duration} ms")
@ -675,12 +644,12 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
if not quiet: if not quiet:
click.echo(f" ✓ Added {added} annotation(s)") click.echo(f" ✓ Added {added} annotation(s)")
except Exception as e:
raise click.ClickException(f"CUSUM detection failed: {e}")
save_recording_auto(recording, output, input, quiet, overwrite) save_recording_auto(recording, output, input, quiet, overwrite)
if not quiet: if not quiet:
click.echo(" ✓ Saved") click.echo(" ✓ Saved")
except Exception as e:
raise click.ClickException(f"CUSUM detection failed: {e}")
# ============================================================================ # ============================================================================
@ -706,13 +675,10 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
help="Annotation type", help="Annotation type",
) )
@click.option("--channel", type=int, default=0, help="Channel index to annotate (default: 0)") @click.option("--channel", type=int, default=0, help="Channel index to annotate (default: 0)")
@click.option(
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
)
@click.option("--output", "-o", type=click.Path(), help="Output file path") @click.option("--output", "-o", type=click.Path(), help="Output file path")
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)") @click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
@click.option("--quiet", is_flag=True, help="Quiet mode") @click.option("--quiet", is_flag=True, help="Quiet mode")
def threshold(input, threshold, label, window_size, annotation_type, channel, sample_rate, output, overwrite, quiet): def threshold(input, threshold, label, window_size, annotation_type, channel, output, overwrite, quiet):
"""Auto-detect signals using threshold method. """Auto-detect signals using threshold method.
Detects samples above a percentage of maximum magnitude. Best for simple Detects samples above a percentage of maximum magnitude. Best for simple
@ -722,13 +688,10 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
Examples: Examples:
ria annotate threshold signal.sigmf-data --threshold 0.7 --label wifi ria annotate threshold signal.sigmf-data --threshold 0.7 --label wifi
ria annotate threshold data.npy --threshold 0.5 --window-size 2048 ria annotate threshold data.npy --threshold 0.5 --window-size 2048
ria annotate threshold data.npy --threshold 0.4 --sample-rate 1e6
""" """
if not (0.0 <= threshold <= 1.0): if not (0.0 <= threshold <= 1.0):
raise click.ClickException(f"--threshold must be between 0.0 and 1.0, got {threshold}") raise click.ClickException(f"--threshold must be between 0.0 and 1.0, got {threshold}")
check_output_available(input, output, overwrite)
try: try:
recording = load_recording(input) recording = load_recording(input)
if not quiet: if not quiet:
@ -736,21 +699,11 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
except Exception as e: except Exception as e:
raise click.ClickException(f"Failed to load recording: {e}") raise click.ClickException(f"Failed to load recording: {e}")
if sample_rate is not None:
recording.sample_rate = sample_rate
if recording.sample_rate is None:
raise click.ClickException(
"Recording metadata does not contain a sample rate. "
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
)
if not quiet: if not quiet:
click.echo("\nDetecting signals using threshold qualifier...") click.echo("\nDetecting signals using threshold qualifier...")
click.echo(f" Threshold: {threshold * 100:.1f}% of max magnitude") click.echo(f" Threshold: {threshold * 100:.1f}% of max magnitude")
click.echo(f" Window size: {'auto (1ms)' if window_size is None else f'{window_size} samples'}") click.echo(f" Window size: {'auto (1ms)' if window_size is None else f'{window_size} samples'}")
click.echo(f" Channel: {channel}") click.echo(f" Channel: {channel}")
click.echo(f" Sample rate: {recording.sample_rate:.0f} Hz")
try: try:
initial_count = len(recording.annotations) initial_count = len(recording.annotations)
@ -766,12 +719,12 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
if not quiet: if not quiet:
click.echo(f" ✓ Added {added} annotation(s)") click.echo(f" ✓ Added {added} annotation(s)")
except Exception as e:
raise click.ClickException(f"Threshold detection failed: {e}")
save_recording_auto(recording, output, input, quiet, overwrite) save_recording_auto(recording, output, input, quiet, overwrite)
if not quiet: if not quiet:
click.echo(" ✓ Saved") click.echo(" ✓ Saved")
except Exception as e:
raise click.ClickException(f"Threshold detection failed: {e}")
# ============================================================================ # ============================================================================
@ -785,30 +738,11 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
@click.option("--nfft", type=int, default=65536, help="FFT size for spectral analysis") @click.option("--nfft", type=int, default=65536, help="FFT size for spectral analysis")
@click.option("--noise-threshold-db", type=float, help="Noise floor threshold in dB (auto-estimated if not specified)") @click.option("--noise-threshold-db", type=float, help="Noise floor threshold in dB (auto-estimated if not specified)")
@click.option("--min-component-bw", type=float, default=50e3, help="Min component bandwidth in Hz") @click.option("--min-component-bw", type=float, default=50e3, help="Min component bandwidth in Hz")
@click.option(
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
)
@click.option("--output", "-o", type=click.Path(), help="Output file path") @click.option("--output", "-o", type=click.Path(), help="Output file path")
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)") @click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
@click.option("--quiet", is_flag=True, help="Quiet mode") @click.option("--quiet", is_flag=True, help="Quiet mode")
@click.option("--verbose", is_flag=True, help="Verbose output (show detected components)") @click.option("--verbose", is_flag=True, help="Verbose output (show detected components)")
def _log_separate_start(quiet, recording, indices_list, nfft, noise_threshold_db, min_component_bw): def separate(input, indices, nfft, noise_threshold_db, min_component_bw, output, overwrite, quiet, verbose):
if not quiet:
click.echo("\nSplitting annotations by frequency components...")
click.echo(f" Input annotations: {len(recording.annotations)}")
if indices_list:
click.echo(f" Splitting indices: {indices_list}")
click.echo(f" FFT size: {nfft}")
if noise_threshold_db is not None:
click.echo(f" Noise threshold: {noise_threshold_db} dB")
else:
click.echo(" Noise threshold: auto-estimated")
click.echo(f" Min component BW: {format_frequency(min_component_bw)}")
def separate(
input, indices, nfft, noise_threshold_db, min_component_bw, sample_rate, output, overwrite, quiet, verbose
):
""" """
Auto-detect parallel frequency-offset signals and split into sub-bands. Auto-detect parallel frequency-offset signals and split into sub-bands.
@ -834,8 +768,6 @@ def separate(
ria annotate separate signal.npy --min-component-bw 100000 ria annotate separate signal.npy --min-component-bw 100000
""" """
check_output_available(input, output, overwrite)
try: try:
recording = load_recording(input) recording = load_recording(input)
if not quiet: if not quiet:
@ -843,15 +775,6 @@ def separate(
except Exception as e: except Exception as e:
raise click.ClickException(f"Failed to load recording: {e}") raise click.ClickException(f"Failed to load recording: {e}")
if sample_rate is not None:
recording.sample_rate = sample_rate
if recording.sample_rate is None:
raise click.ClickException(
"Recording metadata does not contain a sample rate. "
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
)
# Parse indices if specified # Parse indices if specified
indices_list = get_indices_list(indices=indices, recording=recording) indices_list = get_indices_list(indices=indices, recording=recording)
@ -860,7 +783,17 @@ def separate(
click.echo("No annotations to split") click.echo("No annotations to split")
return return
_log_separate_start(quiet, recording, indices_list, nfft, noise_threshold_db, min_component_bw) if not quiet:
click.echo("\nSplitting annotations by frequency components...")
click.echo(f" Input annotations: {len(recording.annotations)}")
if indices_list:
click.echo(f" Splitting indices: {indices_list}")
click.echo(f" FFT size: {nfft}")
if noise_threshold_db is not None:
click.echo(f" Noise threshold: {noise_threshold_db} dB")
else:
click.echo(" Noise threshold: auto-estimated")
click.echo(f" Min component BW: {format_frequency(min_component_bw)}")
try: try:
initial_count = len(recording.annotations) initial_count = len(recording.annotations)
@ -882,19 +815,14 @@ def separate(
click.echo("\n Details:") click.echo("\n Details:")
for i in range(initial_count, final_count): for i in range(initial_count, final_count):
ann = recording.annotations[i] ann = recording.annotations[i]
freq_range = ( freq_range = f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
if ann.freq_lower_edge is not None and ann.freq_upper_edge is not None
else "N/A"
)
click.echo( click.echo(
f" [{i}] samples {format_sample_count(ann.sample_start)}-" f" [{i}] samples {format_sample_count(ann.sample_start)}-"
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {freq_range}" f"{format_sample_count(ann.sample_start + ann.sample_count)}: {freq_range}"
) )
except Exception as e:
raise click.ClickException(f"Spectral separation failed: {e}")
save_recording_auto(recording, output, input, quiet, overwrite) save_recording_auto(recording, output, input, quiet, overwrite)
if not quiet: if not quiet:
click.echo(" ✓ Saved") click.echo(" ✓ Saved")
except Exception as e:
raise click.ClickException(f"Spectral separation failed: {e}")

View File

@ -16,11 +16,9 @@ from .generate import generate
# from .generate import generate # from .generate import generate
from .init import init from .init import init
from .serve import serve from .serve import serve
from .setup_repo import setup_repo
from .split import split from .split import split
from .transform import transform from .transform import transform
from .transmit import transmit from .transmit import transmit
from .upload import upload
from .view import view from .view import view
# Aliases # Aliases

View File

@ -1,401 +0,0 @@
"""ria setup_repo — create and configure a RIA Hub Project repo."""
import json
import os
import re
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request
import click
from ._hub_auth import (
DEFAULT_HUB,
_NoRedirectHandler,
basic_auth,
resolve_credentials,
store_credentials,
warn_if_insecure,
)
RIA_LFS_RULES = [
("*.pt", "filter=lfs diff=lfs merge=lfs -text"),
("*.pth", "filter=lfs diff=lfs merge=lfs -text"),
("*.onnx", "filter=lfs diff=lfs merge=lfs -text"),
("*.sigmf", "filter=lfs diff=lfs merge=lfs -text"),
("*.sigmf-data", "filter=lfs diff=lfs merge=lfs -text"),
("*.sigmf-meta", "filter=lfs diff=lfs merge=lfs -text"),
("*.npy", "filter=lfs diff=lfs merge=lfs -text"),
("*.npz", "filter=lfs diff=lfs merge=lfs -text"),
("*.h5", "filter=lfs diff=lfs merge=lfs -text"),
("*.hdf5", "filter=lfs diff=lfs merge=lfs -text"),
("*.bin", "filter=lfs diff=lfs merge=lfs -text"),
("*.pkl", "filter=lfs diff=lfs merge=lfs -text"),
]
# Repo names must be safe directory names and valid git remote path components.
_SAFE_NAME_RE = re.compile(r"^[A-Za-z0-9._-]{1,100}$")
# ---------------------------------------------------------------------------
# API helpers
# ---------------------------------------------------------------------------
def _api_request(
hub: str,
path: str,
method: str,
username: str,
password: str,
body: dict | None = None,
) -> tuple[dict, int]:
"""
Make an authenticated request to the RIA Hub API.
Returns (parsed_response_body, http_status_code).
Status 0 means a network/connection error.
Credentials are sent as HTTP Basic auth safe over HTTPS and localhost HTTP.
Redirects are blocked to prevent credential exfiltration.
"""
url = f"{hub.rstrip('/')}/api/v1{path}"
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(url, data=data, method=method)
req.add_header("Content-Type", "application/json")
req.add_header("Authorization", basic_auth(username, password))
opener = urllib.request.build_opener(_NoRedirectHandler)
try:
with opener.open(req, timeout=15) as resp:
return json.loads(resp.read() or b"{}"), resp.status
except urllib.error.HTTPError as e:
try:
resp_body = json.loads(e.read() or b"{}")
except Exception:
resp_body = {}
return resp_body, e.code
except urllib.error.URLError as e:
return {"message": str(e.reason)}, 0
def _get_authenticated_username(hub: str, username: str, password: str) -> str | None:
"""Return the login name of the authenticated user from GET /api/v1/user.
This is the canonical username for URL construction it may differ from
git config user.name which is a display name, not a login.
"""
body, status = _api_request(hub, "/user", "GET", username, password)
if status == 200:
return body.get("login")
return None
def _repo_exists(hub: str, owner: str, name: str, username: str, password: str) -> bool:
body, status = _api_request(
hub,
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(name, safe='')}",
"GET",
username,
password,
)
return status == 200
def _create_repo_on_hub(hub: str, name: str, username: str, password: str, private: bool) -> bool:
"""Create an RIA Hub Project repo via API.
Returns True if the repo was freshly created (server seeded README.md and
.gitattributes via auto_init + is_ria), False if the hub was unreachable
(local fallback needed). Exits on fatal errors (auth, quota, name taken).
"""
body, status = _api_request(
hub,
"/user/repos",
"POST",
username,
password,
{
"name": name,
"auto_init": True,
"is_ria": True,
"private": private,
"default_branch": "main",
},
)
if status == 201:
click.echo(f"Repository '{name}' created on RIA Hub.")
return True
if status == 0:
click.echo(
f"Warning: could not reach RIA Hub at {hub}: {body.get('message', 'connection failed')}",
err=True,
)
click.echo("Continuing with local setup only — create the repo manually on RIA Hub.", err=True)
return False
msg = body.get("message", "")
if status == 401:
click.echo("Error: authentication failed — check your username/password.", err=True)
sys.exit(1)
if status in (403, 413) or "quota" in msg.lower() or "limit" in msg.lower():
click.echo("Error: cannot create repository — storage quota or account limit reached.", err=True)
if msg:
click.echo(f" Server message: {msg}", err=True)
sys.exit(1)
if status == 422 or "already exist" in msg.lower():
click.echo(f"Repository '{name}' already exists on RIA Hub.")
return False
click.echo(f"Error creating repository (HTTP {status}): {msg}", err=True)
sys.exit(1)
# ---------------------------------------------------------------------------
# Local git helpers
# ---------------------------------------------------------------------------
def _tracked_patterns(ga_path: str) -> set:
if not os.path.exists(ga_path):
return set()
patterns = set()
with open(ga_path, encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line or line.startswith("#"):
continue
m = re.match(r"^(\S+)\s+", line)
if m:
patterns.add(m.group(1))
return patterns
def _write_local_ria_files(repo_path: str, repo_name: str) -> None:
"""Seed README.md and .gitattributes locally (used when hub is unreachable or --no-remote)."""
# README
for candidate in ("README.md", "README.rst", "README.txt", "README"):
if os.path.exists(os.path.join(repo_path, candidate)):
click.echo(f"README: {candidate} already exists, skipping")
break
else:
with open(os.path.join(repo_path, "README.md"), "w", encoding="utf-8") as f:
f.write(
f"# {repo_name}\n"
"\n"
"A RIA Hub project.\n"
"\n"
"## Description\n"
"\n"
"<!-- Add your project description here -->\n"
"\n"
"## Contents\n"
"\n"
"<!-- Describe the signals, models, or datasets in this repository -->\n"
)
click.echo("README.md: created")
# .gitattributes
ga_path = os.path.join(repo_path, ".gitattributes")
existing = _tracked_patterns(ga_path)
new_rules = [(p, a) for p, a in RIA_LFS_RULES if p not in existing]
if new_rules:
existing_content = ""
if os.path.exists(ga_path):
with open(ga_path, encoding="utf-8") as f:
existing_content = f.read()
separator = "" if (not existing_content or existing_content.endswith("\n")) else "\n"
addition = separator + "".join(f"{pattern} {attrs}\n" for pattern, attrs in new_rules)
with open(ga_path, "a", encoding="utf-8") as f:
f.write(addition)
click.echo(f".gitattributes: {len(new_rules)} rule(s) added")
else:
click.echo(".gitattributes: all RIA Hub rules are already present")
def _git(repo_path: str, *args: str, check: bool = True) -> subprocess.CompletedProcess:
return subprocess.run(
["git", "-C", repo_path, *args],
capture_output=True,
text=True,
check=check,
)
def _resolve_path_and_name(name: str | None, local_path: str | None) -> tuple[str, str]:
if local_path:
repo_path = os.path.abspath(local_path)
repo_name = name or os.path.basename(repo_path)
elif name:
repo_path = os.path.abspath(name)
repo_name = name
else:
repo_path = os.path.abspath(".")
repo_name = os.path.basename(repo_path)
return repo_path, repo_name
def _resolve_owner(hub: str, username: str | None, password: str | None, owner: str | None) -> str:
if not owner and username and password:
api_login = _get_authenticated_username(hub, username, password)
owner = api_login or username
return owner or "unknown"
def _git_init(repo_path: str) -> None:
if os.path.isdir(os.path.join(repo_path, ".git")):
return
result = _git(repo_path, "init", "-b", "main", check=False)
if result.returncode != 0:
# Older git (< 2.28) doesn't support -b; fall back and rename.
_git(repo_path, "init")
_git(repo_path, "symbolic-ref", "HEAD", "refs/heads/main")
click.echo("git init: done (branch: main)")
def _configure_remote(
repo_path: str, hub: str, resolved_owner: str, repo_name: str, username: str | None, no_remote: bool
) -> None:
if no_remote or not username:
click.echo(
f"Skipped remote setup. Add it manually:\n"
f" git -C {repo_path} remote add origin "
f"{hub.rstrip('/')}/{resolved_owner}/{repo_name}.git"
)
return
remote_url = f"{hub.rstrip('/')}/{resolved_owner}/{repo_name}.git"
existing = _git(repo_path, "remote", "get-url", "origin", check=False)
if existing.returncode == 0:
existing_url = existing.stdout.strip()
if existing_url == remote_url:
click.echo(f"remote origin: {remote_url} (already set)")
else:
click.echo(
f"remote 'origin' already points to {existing_url}.\n"
f" To update: git remote set-url origin {remote_url}"
)
else:
_git(repo_path, "remote", "add", "origin", remote_url)
click.echo(f"remote origin: {remote_url}")
# ---------------------------------------------------------------------------
# Command
# ---------------------------------------------------------------------------
@click.command("setup_repo")
@click.argument("name", required=False)
@click.option(
"--path", "local_path", default=None, help="Local directory (default: current dir, or created from NAME)."
)
@click.option("--hub", default=DEFAULT_HUB, show_default=True, metavar="URL", help="RIA Hub base URL.")
@click.option(
"--owner",
default=None,
metavar="USER",
help="RIA Hub login username (default: looked up from the API using your credentials).",
)
@click.option("--private", is_flag=True, default=False, help="Create the repository as private.")
@click.option(
"--no-remote", is_flag=True, default=False, help="Skip creating the repository on RIA Hub (local setup only)."
)
def setup_repo(
name: str | None,
local_path: str | None,
hub: str,
owner: str | None,
private: bool,
no_remote: bool,
) -> None:
"""Create and configure a RIA Hub Project repo.
NAME is the repository name. If the local directory does not exist or is
not a git repo, it will be initialised automatically. Credentials are
retrieved from git's credential store — no token setup required if you
have used RIA Hub with git before.
\b
Examples:
ria setup_repo my-dataset
ria setup_repo my-dataset --hub https://riahub.example.com
ria setup_repo --path ./existing-dir
ria setup_repo my-dataset --private
"""
repo_path, repo_name = _resolve_path_and_name(name, local_path)
if not _SAFE_NAME_RE.match(repo_name):
click.echo(
f"Error: '{repo_name}' is not a valid repository name.\n"
"Use only letters, numbers, hyphens, underscores, and dots (max 100 chars).",
err=True,
)
sys.exit(1)
if not no_remote:
warn_if_insecure(hub)
username, password = (None, None) if no_remote else resolve_credentials(hub)
resolved_owner = _resolve_owner(hub, username, password, owner)
# newly_created=True means the server ran auto_init+is_ria and seeded
# README.md + .gitattributes in the initial commit; local setup pulls
# those files via fetch rather than writing them from scratch.
newly_created = False
if not no_remote and username and password:
if _repo_exists(hub, resolved_owner, repo_name, username, password):
click.echo(f"Repository '{resolved_owner}/{repo_name}' already exists on RIA Hub.")
else:
newly_created = _create_repo_on_hub(hub, repo_name, username, password, private)
store_credentials(hub, username, password)
if not os.path.exists(repo_path):
os.makedirs(repo_path)
click.echo(f"Created directory: {repo_path}")
_git_init(repo_path)
if subprocess.run(["git", "lfs", "version"], capture_output=True).returncode != 0:
click.echo(
"Error: git-lfs is not installed.\n"
" Linux: sudo apt-get install git-lfs\n"
" macOS: brew install git-lfs\n"
" Other platforms: https://git-lfs.com",
err=True,
)
sys.exit(1)
_git(repo_path, "lfs", "install", "--local")
click.echo("git lfs install --local: done")
_configure_remote(repo_path, hub, resolved_owner, repo_name, username, no_remote)
if newly_created:
fetch = _git(repo_path, "fetch", "origin", check=False)
if fetch.returncode == 0:
_git(repo_path, "reset", "--hard", "origin/main")
click.echo("Pulled initial commit from RIA Hub (README.md + .gitattributes)")
else:
click.echo("Warning: fetch failed — falling back to local file setup.", err=True)
_write_local_ria_files(repo_path, repo_name)
else:
_write_local_ria_files(repo_path, repo_name)
if newly_created:
click.echo(f"\nRepo is ready. Push your work:\n cd {repo_path}\n git push -u origin main")
else:
click.echo(
f"\nRepo is ready. Commit and push:\n"
f" cd {repo_path}\n"
f" git add README.md .gitattributes\n"
f" git commit -m 'chore: initialise RIA Hub project'\n"
f" git push -u origin main"
)

View File

@ -1,392 +0,0 @@
"""ria upload — stream large files to a RIA Hub Project via the LFS API.
How it works
------------
1. The file is hashed locally (SHA-256 + size) this is the LFS object ID.
2. A single POST to the repo's LFS batch endpoint returns an upload URL
(and headers) for any object the server does not already have.
3. The file is streamed to that URL in fixed-size chunks nothing is ever
fully loaded into memory, so files of any size work.
4. A commit is created via the Gitea contents API that records the LFS
pointer (a small text file) so the file appears in the repo tree.
No server-side changes are required this uses the same authenticated LFS
protocol that `git lfs push` uses internally.
"""
import base64
import hashlib
import http.client
import json
import math
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
import click
from ._hub_auth import (
DEFAULT_HUB,
basic_auth,
hub_opener,
resolve_credentials,
warn_if_insecure,
)
# Read buffer for hashing and streaming — 8 MB keeps memory use flat
# for arbitrarily large files.
_CHUNK = 8 * 1024 * 1024
LFS_MEDIA_TYPE = "application/vnd.git-lfs+json"
# ---------------------------------------------------------------------------
# File helpers
# ---------------------------------------------------------------------------
def _hash_file(path: str) -> tuple[str, int]:
"""Return (sha256_hex, byte_size) by streaming the file."""
h = hashlib.sha256()
size = 0
with open(path, "rb") as f:
while True:
chunk = f.read(_CHUNK)
if not chunk:
break
h.update(chunk)
size += len(chunk)
return h.hexdigest(), size
def _lfs_pointer_text(oid: str, size: int) -> str:
return f"version https://git-lfs.github.com/spec/v1\noid sha256:{oid}\nsize {size}\n"
# ---------------------------------------------------------------------------
# LFS batch API
# ---------------------------------------------------------------------------
def _lfs_batch(
hub: str,
owner: str,
repo: str,
objects: list[dict],
username: str,
password: str,
) -> dict:
"""
POST to /{owner}/{repo}.git/info/lfs/objects/batch.
Returns the parsed JSON response.
Raises on HTTP error or JSON decode failure.
"""
url = (
f"{hub.rstrip('/')}"
f"/{urllib.parse.quote(owner, safe='')}"
f"/{urllib.parse.quote(repo, safe='')}"
f".git/info/lfs/objects/batch"
)
body = json.dumps(
{
"operation": "upload",
"transfers": ["basic"],
"objects": objects,
}
).encode()
req = urllib.request.Request(url, data=body, method="POST")
req.add_header("Content-Type", LFS_MEDIA_TYPE)
req.add_header("Accept", LFS_MEDIA_TYPE)
req.add_header("Authorization", basic_auth(username, password))
opener = hub_opener()
try:
with opener.open(req, timeout=30) as resp:
return json.loads(resp.read())
except urllib.error.HTTPError as e:
body_text = e.read().decode(errors="replace")
raise RuntimeError(f"LFS batch request failed (HTTP {e.code}): {body_text}") from e
# ---------------------------------------------------------------------------
# Gitea contents API — create / update a file to record the LFS pointer
# ---------------------------------------------------------------------------
def _get_file_sha(
hub: str,
owner: str,
repo: str,
path: str,
branch: str,
username: str,
password: str,
) -> str | None:
"""Return the blob SHA of an existing file, or None if it doesn't exist."""
url = (
f"{hub.rstrip('/')}/api/v1"
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(repo, safe='')}"
f"/contents/{urllib.parse.quote(path)}"
f"?ref={urllib.parse.quote(branch)}"
)
req = urllib.request.Request(url)
req.add_header("Authorization", basic_auth(username, password))
try:
with hub_opener().open(req, timeout=15) as resp:
return json.loads(resp.read()).get("sha")
except urllib.error.HTTPError as e:
if e.code == 404:
return None
raise
def _commit_lfs_pointer(
hub: str,
owner: str,
repo: str,
remote_path: str,
pointer_text: str,
branch: str,
message: str,
username: str,
password: str,
) -> None:
"""Create or update a file in the repo containing the LFS pointer."""
url = (
f"{hub.rstrip('/')}/api/v1"
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(repo, safe='')}"
f"/contents/{urllib.parse.quote(remote_path)}"
)
existing_sha = _get_file_sha(hub, owner, repo, remote_path, branch, username, password)
body: dict = {
"message": message,
"content": base64.b64encode(pointer_text.encode()).decode(),
"branch": branch,
}
if existing_sha:
body["sha"] = existing_sha
method = "PUT" if existing_sha else "POST"
req = urllib.request.Request(url, data=json.dumps(body).encode(), method=method)
req.add_header("Content-Type", "application/json")
req.add_header("Authorization", basic_auth(username, password))
try:
with hub_opener().open(req, timeout=30) as resp:
resp.read()
except urllib.error.HTTPError as e:
body_text = e.read().decode(errors="replace")
raise RuntimeError(f"Failed to commit LFS pointer for '{remote_path}' (HTTP {e.code}): {body_text}") from e
# ---------------------------------------------------------------------------
# Per-file upload logic
# ---------------------------------------------------------------------------
def _upload_single_file(
hub: str,
owner: str,
repo_name: str,
username: str,
password: str,
file_path: str,
remote_dir: str,
message: str | None,
branch: str,
) -> None:
"""Hash, upload (if needed), and commit the LFS pointer for one file."""
filename = os.path.basename(file_path)
file_size = os.path.getsize(file_path)
size_mb = file_size / (1024 * 1024)
click.echo(f"\n {filename} ({size_mb:.1f} MB)")
click.echo(" Hashing...", nl=False)
oid, size = _hash_file(file_path)
click.echo(f" sha256:{oid[:12]}...")
try:
batch = _lfs_batch(hub, owner, repo_name, [{"oid": oid, "size": size}], username, password)
except RuntimeError as e:
click.echo(f"\n Error: {e}", err=True)
sys.exit(1)
objects = batch.get("objects", [])
if not objects:
click.echo(" Already in LFS — skipping upload.")
else:
obj = objects[0]
if "error" in obj:
err_msg = obj["error"].get("message", "unknown error")
err_code = obj["error"].get("code", 0)
if err_code == 413 or "quota" in err_msg.lower() or "limit" in err_msg.lower():
click.echo(
f"\n Error: storage quota exceeded for this repo.\n Server: {err_msg}",
err=True,
)
else:
click.echo(f"\n Error from server: {err_msg}", err=True)
sys.exit(1)
upload_action = obj.get("actions", {}).get("upload")
if not upload_action:
click.echo(" Already in LFS — skipping upload.")
else:
href = upload_action["href"]
up_headers = upload_action.get("header", {})
chunks = math.ceil(size / _CHUNK)
click.echo(f" Uploading ({size_mb:.1f} MB, {chunks} chunk{'s' if chunks != 1 else ''})...")
try:
_stream_upload_progress(href, up_headers, file_path, size)
except RuntimeError as e:
click.echo(f"\n Upload failed: {e}", err=True)
sys.exit(1)
click.echo(" Upload complete.")
verify_action = obj.get("actions", {}).get("verify")
if verify_action:
try:
vreq = urllib.request.Request(
verify_action["href"],
data=json.dumps({"oid": oid, "size": size}).encode(),
method="POST",
)
vreq.add_header("Content-Type", LFS_MEDIA_TYPE)
vreq.add_header("Accept", LFS_MEDIA_TYPE)
for k, v in verify_action.get("header", {}).items():
vreq.add_header(k, v)
with urllib.request.urlopen(vreq, timeout=15):
pass
except Exception:
pass # verify is optional; non-fatal on failure
pointer = _lfs_pointer_text(oid, size)
remote_path = (f"{remote_dir.rstrip('/')}/{filename}").lstrip("/") if remote_dir else filename
commit_msg = message or f"chore: upload {filename} via ria"
click.echo(f" Committing pointer → {remote_path}...", nl=False)
try:
_commit_lfs_pointer(hub, owner, repo_name, remote_path, pointer, branch, commit_msg, username, password)
click.echo(" done.")
except RuntimeError as e:
click.echo(f"\n Error: {e}", err=True)
sys.exit(1)
def _stream_upload_progress(href: str, headers: dict, file_path: str, size: int) -> None:
"""Stream file_path to href with a click progress bar."""
parsed = urllib.parse.urlparse(href)
host = parsed.netloc
path_q = parsed.path + (f"?{parsed.query}" if parsed.query else "")
if parsed.scheme == "https":
conn = http.client.HTTPSConnection(host, timeout=300)
else:
conn = http.client.HTTPConnection(host, timeout=300)
all_headers = dict(headers)
all_headers.setdefault("Content-Type", "application/octet-stream")
all_headers["Content-Length"] = str(size)
try:
conn.connect()
conn.putrequest("PUT", path_q)
for k, v in all_headers.items():
conn.putheader(k, v)
conn.endheaders()
with click.progressbar(
length=size,
label=" ",
width=40,
show_eta=True,
show_percent=True,
fill_char="",
empty_char="",
) as bar:
with open(file_path, "rb") as f:
while True:
chunk = f.read(_CHUNK)
if not chunk:
break
conn.send(chunk)
bar.update(len(chunk))
resp = conn.getresponse()
resp.read()
if resp.status not in (200, 201):
raise RuntimeError(f"HTTP {resp.status}")
finally:
conn.close()
# ---------------------------------------------------------------------------
# Command
# ---------------------------------------------------------------------------
@click.command("upload")
@click.argument("files", nargs=-1, required=True)
@click.option(
"--repo", required=True, metavar="OWNER/NAME", help="Target repository on RIA Hub (e.g. benchinnery/my-dataset)."
)
@click.option("--hub", default=DEFAULT_HUB, show_default=True, metavar="URL", help="RIA Hub base URL.")
@click.option("--branch", default="main", show_default=True, help="Branch to commit the files to.")
@click.option(
"--path",
"remote_dir",
default="",
metavar="DIR",
help="Remote directory path inside the repo (default: repo root).",
)
@click.option("--message", "-m", default=None, help="Commit message (default: 'chore: upload <filename> via ria').")
def upload(
files: tuple[str],
repo: str,
hub: str,
branch: str,
remote_dir: str,
message: str | None,
) -> None:
"""Upload large files to a RIA Hub Project via Git LFS.
Files are streamed directly to the repo's LFS object store — nothing is
buffered into memory, so files of any size work. Each file creates one
commit recording the LFS pointer.
\b
Examples:
ria upload recording.sigmf-data --repo benchinnery/my-recordings
ria upload *.npy --repo benchinnery/my-recordings --branch main
ria upload big.pt --repo benchinnery/models --path weights/
"""
# Validate repo argument
if "/" not in repo:
click.echo("Error: --repo must be in the form OWNER/NAME.", err=True)
sys.exit(1)
owner, repo_name = repo.split("/", 1)
# Expand and validate files
resolved = []
for pattern in files:
if not os.path.isfile(pattern):
click.echo(f"Error: '{pattern}' is not a file or does not exist.", err=True)
sys.exit(1)
resolved.append(os.path.abspath(pattern))
hub = hub.rstrip("/")
warn_if_insecure(hub)
username, password = resolve_credentials(hub)
click.echo(f"Uploading {len(resolved)} file(s) to {owner}/{repo_name} on {hub}...")
for file_path in resolved:
_upload_single_file(hub, owner, repo_name, username, password, file_path, remote_dir, message, branch)
click.echo(f"\nAll done. {len(resolved)} file(s) uploaded to {owner}/{repo_name}.")

View File

@ -1,145 +0,0 @@
"""Structured error reporting for `ria-agent register` (T2)."""
from __future__ import annotations
import json
import sys
import urllib.error
from io import BytesIO
from unittest.mock import patch
import pytest
from ria_toolkit_oss.agent import cli as agent_cli
def _structured(reason: str) -> bytes:
return json.dumps({"detail": {"reason": reason}}).encode()
@pytest.mark.parametrize(
"reason",
["invalid_key", "expired", "revoked", "already_consumed"],
)
def test_explain_maps_known_reasons(reason):
msg = agent_cli._explain_registration_failure(403, _structured(reason))
assert msg == agent_cli.REGISTRATION_REASON_MESSAGES[reason]
def test_explain_unknown_reason_falls_through_with_code():
msg = agent_cli._explain_registration_failure(403, _structured("brand_new_thing"))
assert "brand_new_thing" in msg
assert "rejected" in msg.lower()
def test_explain_string_detail():
body = json.dumps({"detail": "Forbidden"}).encode()
msg = agent_cli._explain_registration_failure(403, body)
assert msg == "Registration rejected: Forbidden"
def test_explain_429_with_string_detail():
body = json.dumps({"detail": "Too many attempts; try again shortly"}).encode()
msg = agent_cli._explain_registration_failure(429, body)
assert "rate-limited" in msg
assert "Too many attempts" in msg
def test_explain_429_with_no_body():
msg = agent_cli._explain_registration_failure(429, b"")
assert "rate-limited" in msg
def test_explain_malformed_json():
msg = agent_cli._explain_registration_failure(500, b"<html>boom</html>")
assert msg.startswith("HTTP 500")
assert "boom" in msg
def test_explain_empty_body():
msg = agent_cli._explain_registration_failure(502, b"")
assert msg == "HTTP 502: no body"
def _http_error(status: int, body: bytes) -> urllib.error.HTTPError:
return urllib.error.HTTPError(
url="http://hub/screens/agents/register",
code=status,
msg="",
hdrs=None, # type: ignore[arg-type]
fp=BytesIO(body),
)
def test_user_agent_is_set_and_not_python_default():
"""Cloudflare on `riahub.ai` returns 403 code 1010 to `Python-urllib/*`.
Guarding the UA explicitly is the entire point of the register-flow fix;
if this test ever breaks, the production bug is back.
"""
ua = agent_cli._user_agent()
assert ua, "User-Agent must not be empty"
assert not ua.lower().startswith(
"python-urllib"
), f"User-Agent must not be Python's default (got {ua!r}) — Cloudflare blocks it"
assert ua.startswith("ria-agent/")
def test_register_request_carries_explicit_user_agent(tmp_path):
"""Capture the outbound urllib Request and verify the UA header is set."""
cfg_path = tmp_path / "agent.json"
captured: dict = {}
def _fake_urlopen(req, *args, **kwargs):
# urllib normalizes header names; get_header takes the title-cased form.
captured["ua"] = req.get_header("User-agent")
captured["api_key"] = req.get_header("X-api-key")
captured["timeout"] = kwargs.get("timeout")
raise urllib.error.HTTPError(
url=req.full_url,
code=403,
msg="",
hdrs=None, # type: ignore[arg-type]
fp=BytesIO(_structured("invalid_key")),
)
with (
patch.dict("os.environ", {"RIA_AGENT_CONFIG": str(cfg_path)}, clear=False),
patch("urllib.request.urlopen", side_effect=_fake_urlopen),
patch.object(
sys,
"argv",
["ria-agent", "register", "--hub", "http://hub:3005", "--api-key", "ria_reg_x"],
),
):
with pytest.raises(SystemExit):
agent_cli.main()
assert captured["ua"], "User-Agent header was not sent"
assert not captured["ua"].lower().startswith("python-urllib")
assert captured["api_key"] == "ria_reg_x"
assert captured["timeout"] is not None, "register must pass a timeout to urlopen"
def test_register_surfaces_reason_on_http_error(tmp_path, capsys):
cfg_path = tmp_path / "agent.json"
err = _http_error(403, _structured("revoked"))
with (
patch.dict("os.environ", {"RIA_AGENT_CONFIG": str(cfg_path)}, clear=False),
patch("urllib.request.urlopen", side_effect=err),
patch.object(
sys,
"argv",
["ria-agent", "register", "--hub", "http://hub:3005", "--api-key", "ria_reg_x"],
),
):
with pytest.raises(SystemExit) as exc:
agent_cli.main()
assert exc.value.code == 1
captured = capsys.readouterr()
assert "revoked" in captured.err.lower()
assert "Settings → RIA Agents" in captured.err
# Config must NOT be written on failure.
assert not cfg_path.exists()

View File

@ -199,44 +199,3 @@ def test_annotation_to_sigmf_format_values():
values = list(result.values()) values = list(result.values())
assert 50 in values or ann.sample_start in values assert 50 in values or ann.sample_start in values
assert 100 in values or ann.sample_count in values assert 100 in values or ann.sample_count in values
# ---------------------------------------------------------------------------
# None freq-edge regression tests (SigMF optional fields)
# ---------------------------------------------------------------------------
def test_annotation_no_freq_edges():
ann = Annotation(sample_start=0, sample_count=10, label="burst")
assert ann.freq_lower_edge is None
assert ann.freq_upper_edge is None
def test_annotation_is_valid_no_freq_edges():
ann = Annotation(sample_start=0, sample_count=10, label="burst")
assert ann.is_valid() is True
ann_zero = Annotation(sample_start=0, sample_count=0, label="burst")
assert ann_zero.is_valid() is False
def test_annotation_overlap_none_edges_returns_zero():
ann1 = Annotation(sample_start=0, sample_count=10)
ann2 = Annotation(sample_start=0, sample_count=10, freq_lower_edge=0, freq_upper_edge=100)
assert ann1.overlap(ann2) == 0
assert ann2.overlap(ann1) == 0
def test_annotation_area_none_edges_returns_zero():
ann = Annotation(sample_start=0, sample_count=10, label="burst")
assert ann.area() == 0
def test_annotation_to_sigmf_omits_freq_keys_when_none():
from sigmf import SigMFFile
ann = Annotation(sample_start=0, sample_count=10, label="burst")
result = ann.to_sigmf_format()
metadata = result["metadata"]
assert SigMFFile.FLO_KEY not in metadata
assert SigMFFile.FHI_KEY not in metadata

View File

@ -189,21 +189,3 @@ def test_sigmf_3(tmp_path):
to_sigmf(recording=recording1, path=tmp_path, filename=filename.name) to_sigmf(recording=recording1, path=tmp_path, filename=filename.name)
except IOError as e: except IOError as e:
assert str(e) == "File already exists" assert str(e) == "File already exists"
def test_sigmf_annotation_without_freq_edges(tmp_path):
# Regression: annotations that omit the optional SigMF freq edge fields must
# load without error; edges should be None and the annotation still valid.
ann = Annotation(sample_start=0, sample_count=5, label="burst")
recording1 = Recording(data=complex_data_1, metadata=sample_metadata, annotations=[ann])
filename = tmp_path / "test"
to_sigmf(recording=recording1, path=tmp_path, filename=filename.name, overwrite=True)
recording2 = from_sigmf(filename)
assert len(recording2.annotations) == 1
loaded = recording2.annotations[0]
assert loaded.freq_lower_edge is None
assert loaded.freq_upper_edge is None
assert loaded.is_valid() is True
assert loaded.label == "burst"