Compare commits
No commits in common. "main" and "zfp-oss" have entirely different histories.
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -99,5 +99,4 @@ cython_debug/
|
||||||
*.sigmf-meta
|
*.sigmf-meta
|
||||||
*.blue
|
*.blue
|
||||||
*.wav
|
*.wav
|
||||||
/images/
|
images/
|
||||||
!docs/source/images/**
|
|
||||||
|
|
|
||||||
19
CHANGELOG.md
19
CHANGELOG.md
|
|
@ -1,24 +1,5 @@
|
||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
## [0.1.7] - 2026-05-26
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- **Human-readable agent names** — `ria-agent register` now generates a default `adjective-colour-animal` name (e.g. `swift-teal-falcon`) via the new `namegen` module when `--name` is omitted, instead of registering with an empty string.
|
|
||||||
- **Structured registration error messages** — `ria-agent register` translates hub responses into actionable English for the known failure reasons (`invalid_key`, `expired`, `revoked`, `already_consumed`) and rate-limit (`HTTP 429`) responses, instead of surfacing raw `HTTP 4xx` text.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- **`ria-agent register` `--api-key` help** — now describes the personal `ria_reg_*` registration key flow (minted from **Settings → RIA Agents** on the hub, shown once at mint time). The legacy shared `[wac] API_KEY` is still accepted by the hub for back-compat, but the CLI documents the per-user flow as preferred.
|
|
||||||
- **`ria-agent register` success output** — now prints both the hub-assigned agent ID and the chosen name: `Registered agent: <id> (<name>)`.
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- **`ria-agent register` blocked by Cloudflare on hubs behind it** — set an explicit `User-Agent` (`ria-agent/<package-version> (+https://riahub.ai/qoherent/ria-toolkit-oss)`) so the request isn't rejected as `Python-urllib/<ver>` (Cloudflare Browser Integrity Check returns HTTP 403, edge error code 1010). Version is read from package metadata so it tracks releases automatically.
|
|
||||||
- **`ria-agent register` could hang indefinitely** — added a 15-second timeout to the hub request; previously `urllib`'s default of no timeout meant a stuck hub would block the CLI forever.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## [0.1.0] - 2026-02-20
|
## [0.1.0] - 2026-02-20
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
|
||||||
|
|
@ -185,7 +185,7 @@ RIA Toolkit OSS is developed and maintained by [Qoherent](https://qoherent.ai/),
|
||||||
If you are doing research with RIA Toolkit OSS, please cite the project:
|
If you are doing research with RIA Toolkit OSS, please cite the project:
|
||||||
|
|
||||||
```
|
```
|
||||||
[1] Qoherent Inc., "Radio Intelligence Apps Toolkit OSS," 2026. [Online]. Available: https://riahub.ai/qoherent/ria-toolkit-oss
|
[1] Qoherent Inc., "Radio Intelligence Apps Toolkit OSS," 2025. [Online]. Available: https://riahub.ai/qoherent/ria-toolkit-oss
|
||||||
```
|
```
|
||||||
|
|
||||||
If you like what we're doing, don't forget to give the project a star! ⭐
|
If you like what we're doing, don't forget to give the project a star! ⭐
|
||||||
|
|
|
||||||
|
|
@ -6,10 +6,10 @@ This is a practical reference for the ``ria`` CLI from ``ria-toolkit-oss``.
|
||||||
|
|
||||||
**Scope of this guide:**
|
**Scope of this guide:**
|
||||||
|
|
||||||
* **Installation and SDR driver prerequisites** — how to install RIA Toolkit OSS and configure the system drivers your hardware requires
|
* Installation and setup
|
||||||
* **End-to-end CLI workflow** — a step-by-step walkthrough from hardware discovery through capture, annotation, and processing
|
* End-to-end CLI workflows
|
||||||
* **Full command reference** — options, flags, and examples for every ``ria`` command
|
* Full command reference for CLI features
|
||||||
* **Python scripting preview** — using the toolkit API directly without the CLI
|
* Brief scripting section
|
||||||
|
|
||||||
**Official resources:**
|
**Official resources:**
|
||||||
|
|
||||||
|
|
@ -18,15 +18,76 @@ This is a practical reference for the ``ria`` CLI from ``ria-toolkit-oss``.
|
||||||
* `PyPI package <https://pypi.org/project/ria-toolkit-oss/>`_
|
* `PyPI package <https://pypi.org/project/ria-toolkit-oss/>`_
|
||||||
* `RIA Hub Conda package <https://riahub.ai/qoherent/-/packages/conda/ria-toolkit-oss>`_
|
* `RIA Hub Conda package <https://riahub.ai/qoherent/-/packages/conda/ria-toolkit-oss>`_
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
:depth: 2
|
||||||
|
:backlinks: none
|
||||||
|
|
||||||
|
|
||||||
1) Installation and Setup
|
1) Installation and Setup
|
||||||
==========================
|
==========================
|
||||||
|
|
||||||
Before using the ``ria`` CLI, follow the :doc:`Installation <installation>` guide to
|
1.1 Installation with Conda
|
||||||
install RIA Toolkit OSS and any SDR drivers required for your hardware.
|
----------------------------
|
||||||
|
|
||||||
|
RIA Toolkit OSS is available as a Conda package on RIA Hub. This is typically the easiest
|
||||||
|
path when using SDR tooling that depends on native/system libraries.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
conda update --force conda
|
||||||
|
conda config --add channels https://riahub.ai/api/packages/qoherent/conda
|
||||||
|
conda activate base
|
||||||
|
conda install ria-toolkit-oss
|
||||||
|
|
||||||
|
Verify:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
conda list | grep ria-toolkit-oss
|
||||||
|
|
||||||
|
|
||||||
1.1 SDR driver prerequisites
|
1.2 Installation with pip
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
Use pip unless you specifically need to edit toolkit source.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
python3 -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install --upgrade pip
|
||||||
|
pip install ria-toolkit-oss
|
||||||
|
|
||||||
|
Verify CLI entrypoint:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
ria --help
|
||||||
|
|
||||||
|
``pyproject.toml`` defines two script entry points:
|
||||||
|
|
||||||
|
* ``ria``
|
||||||
|
* ``ria-tools``
|
||||||
|
|
||||||
|
Both point to the same CLI module (``ria_toolkit_oss_cli.cli:cli``).
|
||||||
|
|
||||||
|
|
||||||
|
1.3 Optional install from source
|
||||||
|
----------------------------------
|
||||||
|
|
||||||
|
Use this for local development or testing unreleased changes.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
git clone https://riahub.ai/qoherent/ria-toolkit-oss.git
|
||||||
|
cd ria-toolkit-oss
|
||||||
|
python3 -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install -e .
|
||||||
|
|
||||||
|
|
||||||
|
1.4 SDR driver prerequisites
|
||||||
-----------------------------
|
-----------------------------
|
||||||
|
|
||||||
Toolkit package install does not install all system SDR drivers. Install vendor/runtime
|
Toolkit package install does not install all system SDR drivers. Install vendor/runtime
|
||||||
|
|
@ -34,22 +95,11 @@ dependencies for the hardware you use.
|
||||||
|
|
||||||
Examples (depends on device and OS):
|
Examples (depends on device and OS):
|
||||||
|
|
||||||
.. list-table::
|
* USRP: UHD drivers
|
||||||
:widths: 25 75
|
* Pluto: libiio / IIO utilities
|
||||||
:header-rows: 1
|
* BladeRF: libbladeRF
|
||||||
|
* HackRF: libhackrf
|
||||||
* - Device
|
* RTL-SDR: librtlsdr
|
||||||
- Driver Package
|
|
||||||
* - USRP
|
|
||||||
- UHD drivers
|
|
||||||
* - Pluto
|
|
||||||
- libiio / IIO utilities
|
|
||||||
* - BladeRF
|
|
||||||
- libbladeRF
|
|
||||||
* - HackRF
|
|
||||||
- libhackrf
|
|
||||||
* - RTL-SDR
|
|
||||||
- librtlsdr
|
|
||||||
|
|
||||||
See repo docs under ``docs/source/sdr_guides/*`` and your OS package instructions.
|
See repo docs under ``docs/source/sdr_guides/*`` and your OS package instructions.
|
||||||
|
|
||||||
|
|
@ -69,34 +119,18 @@ Top-level CLI follows this model:
|
||||||
|
|
||||||
**Top-level commands:**
|
**Top-level commands:**
|
||||||
|
|
||||||
.. list-table::
|
* ``discover``
|
||||||
:widths: 25 75
|
* ``init``
|
||||||
:header-rows: 1
|
* ``capture``
|
||||||
|
* ``view``
|
||||||
* - Command
|
* ``annotate`` (group)
|
||||||
- Purpose
|
* ``convert``
|
||||||
* - :ref:`discover <cmd-discover>`
|
* ``split``
|
||||||
- Probe SDR drivers and enumerate attached hardware
|
* ``combine``
|
||||||
* - :ref:`init <cmd-init>`
|
* ``generate`` (group)
|
||||||
- Create and manage user metadata defaults
|
* ``transform`` (group)
|
||||||
* - :ref:`capture <cmd-capture>`
|
* ``transmit``
|
||||||
- Record IQ samples from a connected SDR
|
* ``synth`` (alias of ``generate`` in command bindings)
|
||||||
* - :ref:`view <cmd-view>`
|
|
||||||
- Generate visualizations from IQ files
|
|
||||||
* - :ref:`annotate <cmd-annotate>`
|
|
||||||
- Label signal regions manually or with auto-detection (group)
|
|
||||||
* - :ref:`convert <cmd-convert>`
|
|
||||||
- Convert between IQ file formats
|
|
||||||
* - :ref:`split <cmd-split>`
|
|
||||||
- Split, trim, or extract recordings
|
|
||||||
* - :ref:`combine <cmd-combine>`
|
|
||||||
- Merge multiple recordings by concatenation or addition
|
|
||||||
* - :ref:`generate / synth <cmd-generate>`
|
|
||||||
- Generate synthetic IQ signals (group; ``synth`` is an alias)
|
|
||||||
* - :ref:`transform <cmd-transform>`
|
|
||||||
- Apply augmentations or impairments to recordings (group)
|
|
||||||
* - :ref:`transmit <cmd-transmit>`
|
|
||||||
- Transmit IQ through a TX-capable SDR
|
|
||||||
|
|
||||||
|
|
||||||
3) Quick End-to-End Workflow
|
3) Quick End-to-End Workflow
|
||||||
|
|
@ -124,8 +158,10 @@ provenance fields.
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
ria init
|
ria init
|
||||||
ria init --author "Jane Doe" --project "rf-campaign-1" --location "Lab-A" # non-interactive
|
# or non-interactive
|
||||||
ria init --show # show config
|
ria init --author "Jane Doe" --project "rf-campaign-1" --location "Lab-A"
|
||||||
|
# show config
|
||||||
|
ria init --show
|
||||||
|
|
||||||
|
|
||||||
3.3 Capture IQ
|
3.3 Capture IQ
|
||||||
|
|
@ -191,14 +227,13 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
ria transmit -d hackrf -f 2.44G -s 2e6 --input capture.sigmf-data
|
ria transmit -d hackrf -f 2.44G -s 2e6 --input capture.sigmf-data
|
||||||
ria transmit -d hackrf --generate lfm --continuous # generated waveform
|
# or generated waveform
|
||||||
|
ria transmit -d hackrf --generate lfm --continuous
|
||||||
|
|
||||||
|
|
||||||
4) Command Reference
|
4) Command Reference
|
||||||
=====================
|
=====================
|
||||||
|
|
||||||
.. _cmd-discover:
|
|
||||||
|
|
||||||
4.1 ``discover``
|
4.1 ``discover``
|
||||||
-----------------
|
-----------------
|
||||||
|
|
||||||
|
|
@ -228,8 +263,6 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
||||||
hidden in default output.
|
hidden in default output.
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-init:
|
|
||||||
|
|
||||||
4.2 ``init``
|
4.2 ``init``
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
|
|
@ -276,8 +309,6 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
||||||
generate metadata, and YAML config loading paths).
|
generate metadata, and YAML config loading paths).
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-capture:
|
|
||||||
|
|
||||||
4.3 ``capture``
|
4.3 ``capture``
|
||||||
----------------
|
----------------
|
||||||
|
|
||||||
|
|
@ -351,8 +382,6 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
||||||
ria capture -c capture_config.yaml
|
ria capture -c capture_config.yaml
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-view:
|
|
||||||
|
|
||||||
4.4 ``view``
|
4.4 ``view``
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
|
|
@ -413,21 +442,7 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
||||||
ria view capture.npy --type full --title "Test Capture" --format pdf
|
ria view capture.npy --type full --title "Test Capture" --format pdf
|
||||||
ria view capture.npy --show --no-save
|
ria view capture.npy --show --no-save
|
||||||
ria view old.npy --legacy --type simple
|
ria view old.npy --legacy --type simple
|
||||||
ria view recordings\qam64_35.npy --type simple
|
|
||||||
ria view recordings\qam64_35.npy --type full
|
|
||||||
|
|
||||||
.. figure:: ../images/recordings/qam64_35.png
|
|
||||||
:alt: Example output of ria view recordings\qam64_35.npy --type simple
|
|
||||||
|
|
||||||
Output of ``ria view recordings\qam64_35.npy --type simple``
|
|
||||||
|
|
||||||
.. figure:: ../images/recordings/qam64_35-full.png
|
|
||||||
:alt: Example output of ria view recordings\qam64_35.npy --type full
|
|
||||||
|
|
||||||
Output of ``ria view recordings\qam64_35.npy --type full``
|
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-annotate:
|
|
||||||
|
|
||||||
4.5 ``annotate`` group
|
4.5 ``annotate`` group
|
||||||
-----------------------
|
-----------------------
|
||||||
|
|
@ -444,30 +459,8 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
||||||
|
|
||||||
ria annotate <subcommand> ...
|
ria annotate <subcommand> ...
|
||||||
|
|
||||||
**Subcommands:**
|
**Subcommands:** ``list``, ``add``, ``remove``, ``clear``, ``energy``, ``cusum``,
|
||||||
|
``threshold``, ``separate``
|
||||||
.. list-table::
|
|
||||||
:widths: 25 75
|
|
||||||
:header-rows: 1
|
|
||||||
|
|
||||||
* - Subcommand
|
|
||||||
- Purpose
|
|
||||||
* - ``list``
|
|
||||||
- Inspect all annotations on a recording
|
|
||||||
* - ``add``
|
|
||||||
- Add one annotation with explicit sample-domain bounds
|
|
||||||
* - ``remove``
|
|
||||||
- Remove one annotation by index
|
|
||||||
* - ``clear``
|
|
||||||
- Remove all annotations from a recording
|
|
||||||
* - ``energy``
|
|
||||||
- Auto-detect regions above the estimated noise floor
|
|
||||||
* - ``cusum``
|
|
||||||
- Auto-detect regime changes using change-point detection
|
|
||||||
* - ``threshold``
|
|
||||||
- Auto-detect regions using normalized magnitude thresholding
|
|
||||||
* - ``separate``
|
|
||||||
- Decompose annotations into narrower spectral components
|
|
||||||
|
|
||||||
**General behavior:**
|
**General behavior:**
|
||||||
|
|
||||||
|
|
@ -594,16 +587,8 @@ annotations.
|
||||||
ria annotate add capture.sigmf-data --start 10000 --count 5000 --label burst
|
ria annotate add capture.sigmf-data --start 10000 --count 5000 --label burst
|
||||||
ria annotate energy capture.sigmf-data --label signal --threshold 1.3
|
ria annotate energy capture.sigmf-data --label signal --threshold 1.3
|
||||||
ria annotate cusum capture.sigmf-data --min-duration 5
|
ria annotate cusum capture.sigmf-data --min-duration 5
|
||||||
ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%
|
|
||||||
ria annotate separate capture.sigmf-data --indices 0,1 --verbose
|
ria annotate separate capture.sigmf-data --indices 0,1 --verbose
|
||||||
|
|
||||||
.. figure:: ../images/recordings/sample_recording3_annotated.png
|
|
||||||
:alt: Example output of ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%
|
|
||||||
|
|
||||||
Output of ``ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%``
|
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-convert:
|
|
||||||
|
|
||||||
4.6 ``convert``
|
4.6 ``convert``
|
||||||
----------------
|
----------------
|
||||||
|
|
@ -644,8 +629,6 @@ inferred from the output file extension.
|
||||||
ria convert old.npy --format sigmf --legacy --overwrite
|
ria convert old.npy --format sigmf --legacy --overwrite
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-split:
|
|
||||||
|
|
||||||
4.7 ``split``
|
4.7 ``split``
|
||||||
--------------
|
--------------
|
||||||
|
|
||||||
|
|
@ -687,8 +670,6 @@ Choose exactly one operation per invocation:
|
||||||
ria split annotated.sigmf-data --extract-annotations --annotation-label payload
|
ria split annotated.sigmf-data --extract-annotations --annotation-label payload
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-combine:
|
|
||||||
|
|
||||||
4.8 ``combine``
|
4.8 ``combine``
|
||||||
----------------
|
----------------
|
||||||
|
|
||||||
|
|
@ -736,8 +717,6 @@ Choose exactly one operation per invocation:
|
||||||
ria combine signal.npy pattern.npy out.npy --mode add --align-mode repeat-spaced --repeat-spacing 10000
|
ria combine signal.npy pattern.npy out.npy --mode add --align-mode repeat-spaced --repeat-spacing 10000
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-generate:
|
|
||||||
|
|
||||||
4.9 ``generate`` group (and ``synth`` alias)
|
4.9 ``generate`` group (and ``synth`` alias)
|
||||||
---------------------------------------------
|
---------------------------------------------
|
||||||
|
|
||||||
|
|
@ -749,34 +728,15 @@ Choose exactly one operation per invocation:
|
||||||
|
|
||||||
``ria synth ...`` is an alias for ``ria generate ...``.
|
``ria synth ...`` is an alias for ``ria generate ...``.
|
||||||
|
|
||||||
**Usage:**
|
**Shape:**
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
ria generate <subcommand> [subcommand options] [common options]
|
ria generate <subcommand> [subcommand options] [common options]
|
||||||
|
|
||||||
**Available subcommands:**
|
**Available subcommands:**
|
||||||
|
``tone``, ``noise``, ``chirp``, ``square``, ``sawtooth``, ``qam``, ``apsk``, ``pam``,
|
||||||
.. list-table::
|
``fsk``, ``ook``, ``oqpsk``, ``gmsk``, ``psk``
|
||||||
:widths: 30 70
|
|
||||||
:header-rows: 1
|
|
||||||
|
|
||||||
* - Subcommand(s)
|
|
||||||
- Description
|
|
||||||
* - ``tone``
|
|
||||||
- Clean sinusoidal calibration/reference source
|
|
||||||
* - ``noise``
|
|
||||||
- Baseline noise floor data or controlled additive-noise synthesis
|
|
||||||
* - ``chirp``
|
|
||||||
- Sweep-based radar/sonar-style signals and bandwidth occupancy tests
|
|
||||||
* - ``square``, ``sawtooth``
|
|
||||||
- Periodic waveform primitives
|
|
||||||
* - ``qam``, ``apsk``, ``pam``, ``psk``
|
|
||||||
- Digital modulation families with pulse-shaping filter support
|
|
||||||
* - ``fsk``
|
|
||||||
- Frequency-shift keying with configurable tone spacing
|
|
||||||
* - ``ook``, ``oqpsk``, ``gmsk``
|
|
||||||
- On-off keying and continuous-phase modulation schemes
|
|
||||||
|
|
||||||
**Common options shared across all generators:**
|
**Common options shared across all generators:**
|
||||||
|
|
||||||
|
|
@ -800,16 +760,22 @@ Multipath and IQ imbalance flags apply impairment-style post-processing during g
|
||||||
|
|
||||||
Options: ``--frequency``, ``--amplitude``, ``--phase``
|
Options: ``--frequency``, ``--amplitude``, ``--phase``
|
||||||
|
|
||||||
|
Clean sinusoidal calibration/reference source.
|
||||||
|
|
||||||
``noise``
|
``noise``
|
||||||
~~~~~~~~~~
|
~~~~~~~~~~
|
||||||
|
|
||||||
Options: ``--noise-type {gaussian,uniform}``, ``--power``
|
Options: ``--noise-type {gaussian,uniform}``, ``--power``
|
||||||
|
|
||||||
|
Baseline noise floor data or controlled additive-noise synthesis.
|
||||||
|
|
||||||
``chirp``
|
``chirp``
|
||||||
~~~~~~~~~~
|
~~~~~~~~~~
|
||||||
|
|
||||||
Options: ``--bandwidth`` (required), ``--period`` (required), ``--type {up,down,up_down}``
|
Options: ``--bandwidth`` (required), ``--period`` (required), ``--type {up,down,up_down}``
|
||||||
|
|
||||||
|
Sweep-based radar/sonar-style signals and bandwidth occupancy tests.
|
||||||
|
|
||||||
``square``
|
``square``
|
||||||
~~~~~~~~~~~
|
~~~~~~~~~~~
|
||||||
|
|
||||||
|
|
@ -860,8 +826,6 @@ symbol transition sharpness).
|
||||||
ria synth psk -s 2e6 -r 100e3 -M 8 -N 8000 -o psk8.npy
|
ria synth psk -s 2e6 -r 100e3 -M 8 -N 8000 -o psk8.npy
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-transform:
|
|
||||||
|
|
||||||
4.10 ``transform`` group
|
4.10 ``transform`` group
|
||||||
-------------------------
|
-------------------------
|
||||||
|
|
||||||
|
|
@ -870,7 +834,7 @@ symbol transition sharpness).
|
||||||
* Apply algorithmic transforms to existing recordings.
|
* Apply algorithmic transforms to existing recordings.
|
||||||
* Run reusable augmentations/impairments for dataset diversity and robustness testing.
|
* Run reusable augmentations/impairments for dataset diversity and robustness testing.
|
||||||
|
|
||||||
**Usage:**
|
**Shape:**
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
|
|
@ -931,8 +895,6 @@ inspect parameter hints. ``--view`` writes a PNG preview alongside transform out
|
||||||
ria transform custom my_filter in.npy out.npy --transform-dir ./my_transforms --params cutoff=0.2
|
ria transform custom my_filter in.npy out.npy --transform-dir ./my_transforms --params cutoff=0.2
|
||||||
|
|
||||||
|
|
||||||
.. _cmd-transmit:
|
|
||||||
|
|
||||||
4.11 ``transmit``
|
4.11 ``transmit``
|
||||||
------------------
|
------------------
|
||||||
|
|
||||||
|
|
@ -1031,7 +993,17 @@ experiment-specific fields on the CLI.
|
||||||
ria generate noise --config generate.yaml
|
ria generate noise --config generate.yaml
|
||||||
|
|
||||||
|
|
||||||
6) Version Notes
|
6) Practical Tips and Safety
|
||||||
|
=============================
|
||||||
|
|
||||||
|
* Use ``ria discover`` before capture/transmit sessions.
|
||||||
|
* Keep TX gain conservative first; validate with attenuators/dummy loads when needed.
|
||||||
|
* Prefer SigMF for interoperable metadata and annotations.
|
||||||
|
* For long workflows, keep outputs organized by campaign directories and consistent prefixes.
|
||||||
|
* Use ``--verbose`` when debugging device init or driver issues.
|
||||||
|
|
||||||
|
|
||||||
|
7) Version Notes
|
||||||
=================
|
=================
|
||||||
|
|
||||||
These notes are based on the current implementation and should be re-validated against future
|
These notes are based on the current implementation and should be re-validated against future
|
||||||
|
|
@ -1044,19 +1016,18 @@ releases.
|
||||||
3. Multiple non-CLI modules still import ``utils.*``, which can create runtime dependency
|
3. Multiple non-CLI modules still import ``utils.*``, which can create runtime dependency
|
||||||
coupling when using only ``ria-toolkit-oss`` in isolation.
|
coupling when using only ``ria-toolkit-oss`` in isolation.
|
||||||
|
|
||||||
.. tip::
|
If you observe unexpected import errors after install, check the package version and
|
||||||
If you observe unexpected import errors after install, check the package version and
|
changelog, then test ``ria --help`` in a clean virtual environment.
|
||||||
changelog, then test ``ria --help`` in a clean virtual environment.
|
|
||||||
|
|
||||||
|
|
||||||
7) Brief Scripting (Python) Preview
|
8) Brief Scripting (Python) Preview
|
||||||
=====================================
|
=====================================
|
||||||
|
|
||||||
For quick non-CLI use:
|
For quick non-CLI use:
|
||||||
|
|
||||||
.. code-block:: python
|
.. code-block:: python
|
||||||
|
|
||||||
from ria_toolkit_oss.data import Recording
|
from ria_toolkit_oss.datatypes import Recording
|
||||||
from ria_toolkit_oss.io import load_recording, to_sigmf
|
from ria_toolkit_oss.io import load_recording, to_sigmf
|
||||||
from ria_toolkit_oss.transforms import iq_augmentations, iq_impairments
|
from ria_toolkit_oss.transforms import iq_augmentations, iq_impairments
|
||||||
|
|
||||||
|
|
@ -1066,3 +1037,47 @@ For quick non-CLI use:
|
||||||
to_sigmf(imp, filename="capture_awgn", path=".")
|
to_sigmf(imp, filename="capture_awgn", path=".")
|
||||||
|
|
||||||
You can also call annotation algorithms and block-generator primitives from Python directly.
|
You can also call annotation algorithms and block-generator primitives from Python directly.
|
||||||
|
|
||||||
|
|
||||||
|
9) Cheat Sheet
|
||||||
|
===============
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
# Install
|
||||||
|
pip install ria-toolkit-oss
|
||||||
|
|
||||||
|
# Discover
|
||||||
|
ria discover -v
|
||||||
|
|
||||||
|
# Init defaults
|
||||||
|
ria init --author "Jane" --project "rf1" --location "Lab-A"
|
||||||
|
|
||||||
|
# Capture
|
||||||
|
ria capture -d pluto -f 2.44G -s 2e6 -n 1000000 -o cap.sigmf-data
|
||||||
|
|
||||||
|
# View
|
||||||
|
ria view cap.sigmf-data --type simple
|
||||||
|
|
||||||
|
# Annotate
|
||||||
|
ria annotate energy cap.sigmf-data --threshold 1.2
|
||||||
|
ria annotate list cap.sigmf-data --verbose
|
||||||
|
|
||||||
|
# Convert
|
||||||
|
ria convert cap.sigmf-data cap.npy
|
||||||
|
|
||||||
|
# Split
|
||||||
|
ria split cap.sigmf-data --split-every 100000 --output-dir chunks
|
||||||
|
|
||||||
|
# Combine
|
||||||
|
ria combine chunks/a.npy chunks/b.npy merged.npy
|
||||||
|
|
||||||
|
# Generate
|
||||||
|
ria generate qam -s 2e6 -r 100e3 -M 16 -N 5000 -o qam16.npy
|
||||||
|
|
||||||
|
# Transform
|
||||||
|
ria transform augment channel_swap cap.npy
|
||||||
|
ria transform impair add_awgn_to_signal cap.npy --params snr=10
|
||||||
|
|
||||||
|
# Transmit
|
||||||
|
ria transmit -d hackrf --input cap.sigmf-data -f 2.44G -s 2e6
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,6 @@
|
||||||
/* Change the hex values below to customize heading colours */
|
/* Change the hex values below to customize heading colours */
|
||||||
|
|
||||||
.rst-content { color: #e0e0e0; }
|
.rst-content h1 { color: #2c3e50; }
|
||||||
.rst-content h1 { color: #ffffff; }
|
|
||||||
.rst-content h2,
|
.rst-content h2,
|
||||||
.rst-content h2 a { color: #ffffff !important; font-size: 22px !important; }
|
.rst-content h2 a { color: #ffffff !important; font-size: 22px !important; }
|
||||||
|
|
||||||
|
|
@ -23,20 +22,8 @@
|
||||||
.rst-content .admonition.warning p {
|
.rst-content .admonition.warning p {
|
||||||
color: #ffffff !important;
|
color: #ffffff !important;
|
||||||
}
|
}
|
||||||
.rst-content h4 { color: #cccccc; }
|
.rst-content h4 { color: #404040; }
|
||||||
|
|
||||||
.highlight * { color: #ffffff !important; }
|
.highlight * { color: #ffffff !important; }
|
||||||
|
|
||||||
.ria-cmd { color: #2980b9 !important; }
|
.ria-cmd { color: #2980b9 !important; }
|
||||||
|
|
||||||
|
|
||||||
/* Table header text */
|
|
||||||
.rst-content table.docutils th {
|
|
||||||
color: #ffffff !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Remove alternating row background colors from tables */
|
|
||||||
.rst-content table.docutils td,
|
|
||||||
.rst-content table.docutils tr:nth-child(2n-1) td {
|
|
||||||
background-color: transparent !important;
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -12,9 +12,9 @@ sys.path.insert(0, os.path.abspath(os.path.join('..', '..')))
|
||||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
|
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
|
||||||
|
|
||||||
project = 'ria-toolkit-oss'
|
project = 'ria-toolkit-oss'
|
||||||
copyright = '2026, Qoherent Inc'
|
copyright = '2025, Qoherent Inc'
|
||||||
author = 'Qoherent Inc.'
|
author = 'Qoherent Inc.'
|
||||||
release = '0.1.7'
|
release = '0.1.5'
|
||||||
|
|
||||||
# -- General configuration ---------------------------------------------------
|
# -- General configuration ---------------------------------------------------
|
||||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
|
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
.. _sdr_examples:
|
.. _examples:
|
||||||
|
|
||||||
############
|
############
|
||||||
SDR Examples
|
SDR Examples
|
||||||
|
|
|
||||||
BIN
docs/source/images/recordings/qam64_35-full.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/qam64_35-full.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/qam64_35.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/qam64_35.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3-full.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3-full.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_annotated.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_annotated.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_awgn.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_awgn.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_cusum.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_cusum.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_threshold.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_threshold.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording5_after_separate.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording5_after_separate.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording5_before_separate.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording5_before_separate.png
(Stored with Git LFS)
Binary file not shown.
File diff suppressed because it is too large
Load Diff
|
|
@ -4,26 +4,7 @@ Installation
|
||||||
RIA Hub Toolkit OSS can be installed either as a Conda package or as a standard Python package.
|
RIA Hub Toolkit OSS can be installed either as a Conda package or as a standard Python package.
|
||||||
|
|
||||||
Please note that SDR drivers must be installed separately. Refer to the relevant guide in the
|
Please note that SDR drivers must be installed separately. Refer to the relevant guide in the
|
||||||
:ref:`SDR Guides <sdr_guides>` section of the documentation for additional setup instructions.
|
:ref:`SDR Guides <sdr_guides>` section of the documentation for addition setup instructions.
|
||||||
|
|
||||||
Common driver packages by device (exact package names depend on your OS):
|
|
||||||
|
|
||||||
.. list-table::
|
|
||||||
:widths: 25 75
|
|
||||||
:header-rows: 1
|
|
||||||
|
|
||||||
* - Device
|
|
||||||
- Driver Package
|
|
||||||
* - USRP
|
|
||||||
- UHD drivers
|
|
||||||
* - Pluto
|
|
||||||
- libiio / IIO utilities
|
|
||||||
* - BladeRF
|
|
||||||
- libbladeRF
|
|
||||||
* - HackRF
|
|
||||||
- libhackrf
|
|
||||||
* - RTL-SDR
|
|
||||||
- librtlsdr
|
|
||||||
|
|
||||||
We want your experience with RIA Toolkit OSS to be as smooth and frictionless as possible. If you run into any
|
We want your experience with RIA Toolkit OSS to be as smooth and frictionless as possible. If you run into any
|
||||||
issues during installation, please reach out to our support team: ``support@qoherent.ai``.
|
issues during installation, please reach out to our support team: ``support@qoherent.ai``.
|
||||||
|
|
@ -103,22 +84,12 @@ Please follow the steps below to install RIA Toolkit OSS using pip:
|
||||||
python -m venv venv
|
python -m venv venv
|
||||||
venv\Scripts\activate
|
venv\Scripts\activate
|
||||||
|
|
||||||
2. Upgrade pip and install RIA Toolkit OSS:
|
2. Install RIA Toolkit OSS from PyPI with pip:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
pip install --upgrade pip
|
|
||||||
pip install ria-toolkit-oss
|
pip install ria-toolkit-oss
|
||||||
|
|
||||||
3. Verify the CLI is available:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
ria --help
|
|
||||||
|
|
||||||
A successful install prints the top-level help text. ``pyproject.toml`` registers two
|
|
||||||
entrypoints — ``ria`` and ``ria-tools`` — that both point to the same CLI module.
|
|
||||||
|
|
||||||
RIA Toolkit OSS can also be installed from RIA Hub. However, RIA Hub does not yet support a proxy or cache for public packages.
|
RIA Toolkit OSS can also be installed from RIA Hub. However, RIA Hub does not yet support a proxy or cache for public packages.
|
||||||
We intend to add this missing functionality soon. In the meantime, please use the ``--no-deps`` option with pip to skip automatic
|
We intend to add this missing functionality soon. In the meantime, please use the ``--no-deps`` option with pip to skip automatic
|
||||||
dependency installation, and then manually install each dependency afterward.
|
dependency installation, and then manually install each dependency afterward.
|
||||||
|
|
@ -148,6 +119,3 @@ Follow the steps below to install RIA Toolkit OSS from source:
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
pip install .
|
pip install .
|
||||||
|
|
||||||
For local development, use ``pip install -e .`` instead to install in editable mode
|
|
||||||
so local changes take effect immediately without reinstalling.
|
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
Data Package (ria_toolkit_oss.data)
|
Datatypes Package (ria_toolkit_oss.data)
|
||||||
=======================================
|
=============================================
|
||||||
|
|
||||||
.. |br| raw:: html
|
.. |br| raw:: html
|
||||||
|
|
||||||
|
|
@ -40,44 +40,34 @@ Limitations
|
||||||
- USB 3.0 connectivity is required for optimal performance; using USB 2.0 will significantly limit data
|
- USB 3.0 connectivity is required for optimal performance; using USB 2.0 will significantly limit data
|
||||||
transfer rates.
|
transfer rates.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
No additional Python packages are required for BladeRF beyond the base RIA Toolkit OSS installation.
|
1. Activate your Radioconda environment.
|
||||||
|
|
||||||
1. Install the system library:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo apt install libbladerf-dev
|
conda activate <your-env-name>
|
||||||
|
|
||||||
For a more complete installation including CLI tools and FPGA images, use the Nuand PPA:
|
2. Install the base dependencies and drivers (*Easy method*):
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo add-apt-repository ppa:nuandllc/bladerf
|
sudo add-apt-repository ppa:nuandllc/bladerf
|
||||||
sudo apt-get update
|
sudo apt-get update
|
||||||
sudo apt-get install bladerf libbladerf-dev
|
sudo apt-get install bladerf
|
||||||
sudo apt-get install bladerf-fpga-hostedxa4 # Necessary for BladeRF 2.0 Micro xA4
|
sudo apt-get install libbladerf-dev
|
||||||
|
sudo apt-get install bladerf-fpga-hostedxa4 # Necessary for installation of bladeRF 2.0 Micro A4.
|
||||||
|
|
||||||
2. Install udev rules:
|
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||||
|
|
||||||
For most users:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo udevadm control --reload
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bladerf1.rules /etc/udev/rules.d/88-radioconda-nuand-bladerf1.rules
|
||||||
sudo udevadm trigger
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bladerf2.rules /etc/udev/rules.d/88-radioconda-nuand-bladerf2.rules
|
||||||
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bootloader.rules /etc/udev/rules.d/88-radioconda-nuand-bootloader.rules
|
||||||
For **Radioconda** users, create symlinks from your conda environment instead:
|
sudo udevadm control --reload
|
||||||
|
sudo udevadm trigger
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bladerf1.rules /etc/udev/rules.d/88-radioconda-nuand-bladerf1.rules
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bladerf2.rules /etc/udev/rules.d/88-radioconda-nuand-bladerf2.rules
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/88-nuand-bootloader.rules /etc/udev/rules.d/88-radioconda-nuand-bootloader.rules
|
|
||||||
sudo udevadm control --reload
|
|
||||||
sudo udevadm trigger
|
|
||||||
|
|
||||||
Further Information
|
Further Information
|
||||||
-------------------
|
-------------------
|
||||||
|
|
|
||||||
|
|
@ -39,44 +39,39 @@ Limitations
|
||||||
- Bandwidth is limited to 20 MHz.
|
- Bandwidth is limited to 20 MHz.
|
||||||
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
HackRF is supported out of the box after installing RIA Toolkit OSS.
|
1. Activate your Radioconda environment:
|
||||||
|
|
||||||
1. Ensure ``libhackrf`` is installed at the system level. On most Ubuntu installations this is already
|
|
||||||
present. If not:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo apt install libhackrf-dev
|
conda activate <your-env-name>
|
||||||
|
|
||||||
2. Install udev rules to allow non-root device access:
|
2. Install the System Package (Ubuntu / Debian):
|
||||||
|
|
||||||
For most users:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo udevadm control --reload
|
sudo apt-get update
|
||||||
sudo udevadm trigger
|
sudo apt-get install hackrf
|
||||||
|
|
||||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/53-hackrf.rules /etc/udev/rules.d/53-radioconda-hackrf.rules
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/53-hackrf.rules /etc/udev/rules.d/53-radioconda-hackrf.rules
|
||||||
sudo udevadm control --reload
|
sudo udevadm control --reload
|
||||||
sudo udevadm trigger
|
sudo udevadm trigger
|
||||||
|
|
||||||
Make sure your user account belongs to the ``plugdev`` group in order to access your device:
|
Make sure your user account belongs to the plugdev group in order to access your device:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo usermod -a -G plugdev <user>
|
sudo usermod -a -G plugdev <user>
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
You may have to restart your system for group membership changes to take effect.
|
You may have to restart your system for changes to take effect.
|
||||||
|
|
||||||
Further information
|
Further information
|
||||||
-------------------
|
-------------------
|
||||||
|
|
|
||||||
|
|
@ -43,34 +43,34 @@ Limitations
|
||||||
affect stability.
|
affect stability.
|
||||||
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
The PlutoSDR is supported out of the box after installing RIA Toolkit OSS. The required Python package
|
1. Activate your Radioconda environment:
|
||||||
(``pyadi-iio``) is included in the toolkit's dependencies.
|
|
||||||
|
|
||||||
1. Ensure ``libiio`` is installed at the system level. On most Ubuntu installations this is already present.
|
|
||||||
If not:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo apt install libiio-dev libiio-utils libiio0
|
conda activate <your-env-name>
|
||||||
|
|
||||||
.. note::
|
2. Install system dependencies:
|
||||||
|
|
||||||
PlutoSDR devices are discoverable over both USB and network (mDNS). Network discovery uses Avahi — if
|
|
||||||
``avahi-daemon`` is not running, network discovery will be skipped but USB discovery still works.
|
|
||||||
|
|
||||||
2. Install a ``udev`` rule to allow non-root device access:
|
|
||||||
|
|
||||||
For most users:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo udevadm control --reload
|
sudo apt-get update
|
||||||
sudo udevadm trigger
|
sudo apt-get install -y \
|
||||||
|
build-essential \
|
||||||
|
git \
|
||||||
|
libxml2-dev \
|
||||||
|
bison \
|
||||||
|
flex \
|
||||||
|
libcdk5-dev \
|
||||||
|
cmake \
|
||||||
|
libusb-1.0-0-dev \
|
||||||
|
libavahi-client-dev \
|
||||||
|
libavahi-common-dev \
|
||||||
|
libaio-dev
|
||||||
|
|
||||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
|
|
@ -78,18 +78,11 @@ The PlutoSDR is supported out of the box after installing RIA Toolkit OSS. The r
|
||||||
sudo udevadm control --reload
|
sudo udevadm control --reload
|
||||||
sudo udevadm trigger
|
sudo udevadm trigger
|
||||||
|
|
||||||
Once you can communicate with the hardware, you may want to perform the post-install steps detailed on
|
Once you can talk to the hardware, you may want to perform the post-install steps detailed on the `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_.
|
||||||
the `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_.
|
|
||||||
|
|
||||||
3. (Optional) Building ``libiio`` or ``libad9361-iio`` from source:
|
4. (Optional) Building ``libiio`` or ``libad9361-iio`` from source:
|
||||||
|
|
||||||
This step is only required if you need a version not available via ``apt``. First install build
|
This step is only required if you want the latest version of these libraries not provided in Radioconda.
|
||||||
dependencies:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
sudo apt-get install -y build-essential git libxml2-dev bison flex libcdk5-dev cmake \
|
|
||||||
libusb-1.0-0-dev libavahi-client-dev libavahi-common-dev libaio-dev
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -30,10 +30,18 @@ Limitations
|
||||||
- Sensitivity and performance can vary depending on the specific model and components.
|
- Sensitivity and performance can vary depending on the specific model and components.
|
||||||
- Requires external software for signal processing and analysis.
|
- Requires external software for signal processing and analysis.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
1. If you previously had RTL-SDR drivers installed, purge them first:
|
1. Activate your Radioconda environment:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
conda activate <your-env-name>
|
||||||
|
|
||||||
|
2. Purge drivers:
|
||||||
|
|
||||||
|
If you already have other drivers installed, purge them from your system.
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
|
|
@ -45,95 +53,47 @@ Set up instructions (Linux)
|
||||||
sudo rm -rvf /usr/local/include/rtl_*
|
sudo rm -rvf /usr/local/include/rtl_*
|
||||||
sudo rm -rvf /usr/local/bin/rtl_*
|
sudo rm -rvf /usr/local/bin/rtl_*
|
||||||
|
|
||||||
2. Install build dependencies:
|
3. Install RTL-SDR Blog drivers:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo apt install libusb-1.0-0-dev git cmake pkg-config build-essential
|
sudo apt-get install libusb-1.0-0-dev git cmake pkg-config build-essential
|
||||||
|
git clone https://github.com/osmocom/rtl-sdr
|
||||||
3. Build ``librtlsdr`` from source:
|
cd rtl-sdr
|
||||||
|
mkdir build
|
||||||
The standard ``librtlsdr`` package available via ``apt`` is missing symbols required by the Python
|
cd build
|
||||||
bindings. Build from the **rtl-sdr-blog fork**:
|
cmake ../ -DINSTALL_UDEV_RULES=ON
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
git clone https://github.com/rtlsdrblog/rtl-sdr-blog.git
|
|
||||||
cd rtl-sdr-blog
|
|
||||||
mkdir build && cd build
|
|
||||||
cmake .. -DINSTALL_UDEV_RULES=ON
|
|
||||||
make
|
make
|
||||||
sudo make install
|
sudo make install
|
||||||
sudo cp ../rtl-sdr.rules /etc/udev/rules.d/
|
sudo cp ../rtl-sdr.rules /etc/udev/rules.d/
|
||||||
sudo ldconfig
|
sudo ldconfig
|
||||||
|
|
||||||
.. important::
|
4. Blacklist the DVB-T modules that would otherwise claim the device:
|
||||||
|
|
||||||
Do not use the osmocom ``rtl-sdr`` repository or the Ubuntu ``librtlsdr-dev`` apt package. Neither
|
|
||||||
provides the ``rtlsdr_set_dithering`` symbol that the Python bindings require.
|
|
||||||
|
|
||||||
4. Blacklist the kernel DVB driver:
|
|
||||||
|
|
||||||
The kernel DVB-T driver (``dvb_usb_rtl28xxu``) claims the RTL-SDR device and prevents ``librtlsdr``
|
|
||||||
from accessing it.
|
|
||||||
|
|
||||||
For most users:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
echo 'blacklist dvb_usb_rtl28xxu' | sudo tee /etc/modprobe.d/blacklist-rtlsdr.conf
|
|
||||||
sudo modprobe -r dvb_usb_rtl28xxu
|
|
||||||
|
|
||||||
For **Radioconda** users, a blacklist configuration is already provided in your conda environment:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf /etc/modprobe.d/radioconda-rtl-sdr-blacklist.conf
|
sudo ln -s $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf /etc/modprobe.d/radioconda-rtl-sdr-blacklist.conf
|
||||||
sudo modprobe -r $(cat $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf | sed -n -e 's/^blacklist //p')
|
sudo modprobe -r $(cat $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf | sed -n -e 's/^blacklist //p')
|
||||||
|
|
||||||
If ``modprobe -r`` fails with "Module is in use", unplug the RTL-SDR dongle, run the command again,
|
.. note::
|
||||||
then plug it back in. Alternatively, reboot — the blacklist takes effect on next boot.
|
|
||||||
|
|
||||||
.. note::
|
In addition to the Radioconda blacklist file, some systems also require
|
||||||
|
manually blacklisting the following DVB-T modules to prevent them from
|
||||||
|
claiming the device:
|
||||||
|
|
||||||
Some systems also require blacklisting additional DVB-T modules. Add these entries to your
|
- ``dvb_usb_rtl28xxu``
|
||||||
blacklist configuration if needed:
|
- ``rtl2832``
|
||||||
|
- ``rtl2830``
|
||||||
|
|
||||||
- ``rtl2832``
|
Add these entries to ``rtlsdr.conf`` (or create the file at
|
||||||
- ``rtl2830``
|
``/etc/modprobe.d/rtlsdr.conf``) if they are not already present.
|
||||||
|
|
||||||
5. Reload udev rules:
|
5. Install a udev rule by creating a link into your radioconda installation:
|
||||||
|
|
||||||
For most users (rules are installed by the build step above):
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo udevadm control --reload
|
|
||||||
sudo udevadm trigger
|
|
||||||
|
|
||||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/rtl-sdr.rules /etc/udev/rules.d/radioconda-rtl-sdr.rules
|
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/rtl-sdr.rules /etc/udev/rules.d/radioconda-rtl-sdr.rules
|
||||||
sudo udevadm control --reload
|
sudo udevadm control --reload
|
||||||
sudo udevadm trigger
|
sudo udevadm trigger
|
||||||
|
|
||||||
6. Install Python packages:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
pip install pyrtlsdr==0.3.0
|
|
||||||
pip install setuptools==69.5.1
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
``pyrtlsdr`` 0.4.0 references a ``rtlsdr_set_dithering`` symbol not present in standard
|
|
||||||
``librtlsdr`` builds. Version 0.3.0 works correctly.
|
|
||||||
|
|
||||||
``pyrtlsdr`` 0.3.0 depends on ``pkg_resources``, which was removed in ``setuptools`` >= 82.
|
|
||||||
Pinning to 69.5.1 ensures ``pkg_resources`` is available.
|
|
||||||
|
|
||||||
Further Information
|
Further Information
|
||||||
-------------------
|
-------------------
|
||||||
- `RTL-SDR Official Website <https://www.rtl-sdr.com/>`_
|
- `RTL-SDR Official Website <https://www.rtl-sdr.com/>`_
|
||||||
|
|
|
||||||
|
|
@ -39,48 +39,18 @@ Limitations
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux)
|
||||||
---------------------------------
|
---------------------------------
|
||||||
|
|
||||||
ThinkRF devices require the ``pyrf`` package, which is written in Python 2 syntax and must be patched
|
Install PyRF
|
||||||
after installation to work with Python 3.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
``lib2to3`` was fully removed in Python 3.13. ThinkRF support is currently limited to
|
|
||||||
**Python 3.12 and below**.
|
|
||||||
|
|
||||||
1. Install ``lib2to3``:
|
|
||||||
|
|
||||||
On some distributions (including Ubuntu 24.04+), ``lib2to3`` is not included by default:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
sudo apt install python3-lib2to3
|
pip install 'pyrf>=2.8.0'
|
||||||
|
|
||||||
2. Install ``pyrf``:
|
Convert PyRF scripts to Python 3
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
pip install pyrf
|
cd ../scripts
|
||||||
|
./convert_pyrf_to_python3.sh
|
||||||
3. Patch ``pyrf`` for Python 3:
|
|
||||||
|
|
||||||
The ``pyrf`` package contains Python 2 syntax throughout (e.g., ``dict.iteritems()``, ``print``
|
|
||||||
statements). Run the following to automatically convert the entire package to Python 3:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
python -c "
|
|
||||||
from lib2to3.refactor import RefactoringTool, get_fixers_from_package
|
|
||||||
import pyrf, os
|
|
||||||
pyrf_path = os.path.dirname(pyrf.__file__)
|
|
||||||
fixers = get_fixers_from_package('lib2to3.fixes')
|
|
||||||
tool = RefactoringTool(fixers)
|
|
||||||
tool.refactor_dir(pyrf_path, write=True)
|
|
||||||
print('Done')
|
|
||||||
"
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
This patches the entire ``pyrf`` package in place, which is required for the driver to fully load.
|
|
||||||
|
|
||||||
Further Information
|
Further Information
|
||||||
-------------------
|
-------------------
|
||||||
|
|
|
||||||
|
|
@ -41,111 +41,48 @@ Limitations
|
||||||
- Compatibility with certain software tools may vary depending on the version of the UHD.
|
- Compatibility with certain software tools may vary depending on the version of the UHD.
|
||||||
- Price range can be a consideration, especially for high-end models.
|
- Price range can be a consideration, especially for high-end models.
|
||||||
|
|
||||||
Set up instructions (Linux)
|
Set up instructions (Linux, Radioconda)
|
||||||
---------------------------
|
---------------------------------------
|
||||||
|
|
||||||
USRP devices require the UHD (USRP Hardware Driver) library with Python bindings. There is no pip-installable
|
1. Activate your Radioconda environment:
|
||||||
UHD package — it must either be installed via conda or built from source.
|
|
||||||
|
|
||||||
**Option A: Install via conda (recommended for conda environments)**
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
conda install conda-forge::uhd
|
conda activate <your-env-name>
|
||||||
|
|
||||||
**Option B: Build from source (required for pip/venv environments)**
|
2. Install UHD and Python bindings:
|
||||||
|
|
||||||
The Python bindings must target the same Python version used in your virtual environment.
|
.. code-block:: bash
|
||||||
|
|
||||||
1. Install build dependencies:
|
conda install conda-forge::uhd
|
||||||
|
|
||||||
.. code-block:: bash
|
3. Download UHD images:
|
||||||
|
|
||||||
sudo apt install cmake build-essential libboost-all-dev libusb-1.0-0-dev \
|
|
||||||
python3-dev python3-numpy libncurses-dev
|
|
||||||
|
|
||||||
2. Install the Mako template library into your virtual environment (used by UHD's build system):
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
pip install mako
|
|
||||||
|
|
||||||
3. Clone and build UHD with your virtual environment activated:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
git clone https://github.com/EttusResearch/uhd.git
|
|
||||||
cd uhd
|
|
||||||
git checkout v4.7.0.0
|
|
||||||
cd host
|
|
||||||
mkdir build && cd build
|
|
||||||
cmake -DENABLE_PYTHON_API=ON -DPYTHON_EXECUTABLE=$(which python3) ..
|
|
||||||
make -j$(nproc)
|
|
||||||
sudo make install
|
|
||||||
sudo ldconfig
|
|
||||||
|
|
||||||
.. important::
|
|
||||||
|
|
||||||
Run the ``cmake`` command with your virtual environment activated so ``$(which python3)`` points
|
|
||||||
to the correct interpreter. Before running ``make``, verify the cmake output includes::
|
|
||||||
|
|
||||||
-- * LibUHD - Python API → must say "Enabling"
|
|
||||||
-- Python interpreter: .../your-venv/bin/python3
|
|
||||||
|
|
||||||
If "LibUHD - Python API" is not listed under enabled components, the Python bindings will not be
|
|
||||||
built. The build typically takes 10–30 minutes.
|
|
||||||
|
|
||||||
4. Copy the Python bindings into your virtual environment if ``import uhd`` fails after installation:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
cp -r ~/uhd/host/build/python/uhd ~/.venv/lib/python3.XX/site-packages/
|
|
||||||
|
|
||||||
Replace ``python3.XX`` with your Python version (e.g., ``python3.12``).
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
If you have a pre-existing UHD installation built against a different Python version, you will see
|
|
||||||
a circular import error. The bindings must match the Python version in your virtual environment exactly.
|
|
||||||
|
|
||||||
**After either installation method:**
|
|
||||||
|
|
||||||
1. Download UHD FPGA/firmware images:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
uhd_images_downloader
|
uhd_images_downloader
|
||||||
|
|
||||||
2. Verify device access:
|
4. Verify access to your device:
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
uhd_find_devices
|
uhd_find_devices
|
||||||
|
|
||||||
For USB devices (e.g. B-series), install a ``udev`` rule.
|
For USB devices only (e.g. B series), install a ``udev`` rule by creating a link into your Radioconda installation.
|
||||||
|
|
||||||
For most users:
|
.. code-block:: bash
|
||||||
|
|
||||||
.. code-block:: bash
|
sudo ln -s $CONDA_PREFIX/lib/uhd/utils/uhd-usrp.rules /etc/udev/rules.d/radioconda-uhd-usrp.rules
|
||||||
|
sudo udevadm control --reload
|
||||||
|
sudo udevadm trigger
|
||||||
|
|
||||||
sudo udevadm control --reload
|
5. (Optional) Update firmware/FPGA images:
|
||||||
sudo udevadm trigger
|
|
||||||
|
|
||||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
.. code-block:: bash
|
||||||
|
|
||||||
.. code-block:: bash
|
uhd_usrp_probe
|
||||||
|
|
||||||
sudo ln -s $CONDA_PREFIX/lib/uhd/utils/uhd-usrp.rules /etc/udev/rules.d/radioconda-uhd-usrp.rules
|
This will ensure your device is running the latest firmware and FPGA versions.
|
||||||
sudo udevadm control --reload
|
|
||||||
sudo udevadm trigger
|
|
||||||
|
|
||||||
3. (Optional) Update firmware/FPGA images:
|
|
||||||
|
|
||||||
.. code-block:: bash
|
|
||||||
|
|
||||||
uhd_usrp_probe
|
|
||||||
|
|
||||||
This will ensure your device is running the latest firmware and FPGA versions.
|
|
||||||
|
|
||||||
Further information
|
Further information
|
||||||
-------------------
|
-------------------
|
||||||
|
|
|
||||||
1407
poetry.lock
generated
1407
poetry.lock
generated
File diff suppressed because it is too large
Load Diff
|
|
@ -1,6 +1,6 @@
|
||||||
[project]
|
[project]
|
||||||
name = "ria-toolkit-oss"
|
name = "ria-toolkit-oss"
|
||||||
version = "0.1.7"
|
version = "0.1.5"
|
||||||
description = "An open-source version of the RIA Toolkit, including the fundamental tools to get started developing, testing, and deploying radio intelligence applications"
|
description = "An open-source version of the RIA Toolkit, including the fundamental tools to get started developing, testing, and deploying radio intelligence applications"
|
||||||
license = { text = "AGPL-3.0-only" }
|
license = { text = "AGPL-3.0-only" }
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
|
|
|
||||||
|
|
@ -5,11 +5,8 @@ Subcommands:
|
||||||
- ``ria-agent run [legacy args]`` — legacy long-poll NodeAgent (unchanged).
|
- ``ria-agent run [legacy args]`` — legacy long-poll NodeAgent (unchanged).
|
||||||
- ``ria-agent stream`` — new WebSocket-based IQ streamer.
|
- ``ria-agent stream`` — new WebSocket-based IQ streamer.
|
||||||
- ``ria-agent detect`` — print SDR drivers whose modules import cleanly.
|
- ``ria-agent detect`` — print SDR drivers whose modules import cleanly.
|
||||||
- ``ria-agent register --hub URL --api-key KEY`` — register with the hub
|
- ``ria-agent register --hub URL --api-key KEY`` — register with the hub and
|
||||||
using a personal registration key (minted from **Settings → RIA Agents**
|
save credentials (and optional TX interlocks) to ``~/.ria/agent.json``.
|
||||||
on the hub, shown once at mint time) and save credentials (and optional
|
|
||||||
TX interlocks) to ``~/.ria/agent.json``. The hub also accepts the legacy
|
|
||||||
shared ``[wac] API_KEY`` for back-compat, but that path is deprecated.
|
|
||||||
|
|
||||||
Invoking ``ria-agent`` with no subcommand falls through to the legacy
|
Invoking ``ria-agent`` with no subcommand falls through to the legacy
|
||||||
long-poll behavior for back-compatibility with existing deployments.
|
long-poll behavior for back-compatibility with existing deployments.
|
||||||
|
|
@ -28,75 +25,9 @@ from .hardware import available_devices
|
||||||
from .legacy_executor import main as _legacy_main
|
from .legacy_executor import main as _legacy_main
|
||||||
from .namegen import generate_agent_name
|
from .namegen import generate_agent_name
|
||||||
|
|
||||||
DEFAULT_HUB_URL = "https://riahub.ai"
|
|
||||||
|
|
||||||
_LEGACY_ALIASES = {"--hub", "--key", "--name", "--device", "--insecure", "--log-level", "--config"}
|
_LEGACY_ALIASES = {"--hub", "--key", "--name", "--device", "--insecure", "--log-level", "--config"}
|
||||||
|
|
||||||
|
|
||||||
def _user_agent() -> str:
|
|
||||||
"""Build the User-Agent header for hub requests.
|
|
||||||
|
|
||||||
Set explicitly so we don't fall back to Python's default `Python-urllib/<ver>`,
|
|
||||||
which is blocked by Cloudflare's Browser Integrity Check on `riahub.ai`
|
|
||||||
(HTTP 403 edge code 1010). Version is read from package metadata so it
|
|
||||||
tracks releases instead of going stale.
|
|
||||||
"""
|
|
||||||
from importlib.metadata import PackageNotFoundError, version
|
|
||||||
|
|
||||||
try:
|
|
||||||
pkg_version = version("ria-toolkit-oss")
|
|
||||||
except PackageNotFoundError:
|
|
||||||
pkg_version = "unknown"
|
|
||||||
return f"ria-agent/{pkg_version} (+https://riahub.ai/qoherent/ria-toolkit-oss)"
|
|
||||||
|
|
||||||
|
|
||||||
# How long to wait on the hub before giving up. The register endpoint is a
|
|
||||||
# small DB lookup + insert; anything past this is a stuck hub, not a slow one.
|
|
||||||
_REGISTER_TIMEOUT_S = 15
|
|
||||||
|
|
||||||
|
|
||||||
REGISTRATION_REASON_MESSAGES = {
|
|
||||||
"invalid_key": ("Registration key not recognized. Generate a fresh key from " "Settings → RIA Agents on the hub."),
|
|
||||||
"expired": ("This registration key has expired. Generate a new one from " "Settings → RIA Agents on the hub."),
|
|
||||||
"revoked": ("This registration key was revoked. Generate a new one from " "Settings → RIA Agents on the hub."),
|
|
||||||
"already_consumed": (
|
|
||||||
"This single-use registration key has already been used. "
|
|
||||||
"Generate a new one, or mint a reusable key instead."
|
|
||||||
),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _explain_registration_failure(status: int, body: bytes) -> str:
|
|
||||||
"""Return a human-readable explanation for a failed register call."""
|
|
||||||
try:
|
|
||||||
parsed = json.loads(body) if body else None
|
|
||||||
except ValueError:
|
|
||||||
parsed = None
|
|
||||||
|
|
||||||
if status == 429:
|
|
||||||
# 429 carries a plain string detail, never a reason code.
|
|
||||||
if isinstance(parsed, dict) and parsed.get("detail"):
|
|
||||||
detail = parsed["detail"]
|
|
||||||
else:
|
|
||||||
detail = body.decode("utf-8", "replace") or "rate limited"
|
|
||||||
return f"Registration rate-limited by the hub: {detail}"
|
|
||||||
|
|
||||||
if not isinstance(parsed, dict):
|
|
||||||
text = body.decode("utf-8", "replace")
|
|
||||||
return f"HTTP {status}: {text or 'no body'}"
|
|
||||||
|
|
||||||
detail = parsed.get("detail")
|
|
||||||
if isinstance(detail, dict):
|
|
||||||
reason = detail.get("reason")
|
|
||||||
if reason in REGISTRATION_REASON_MESSAGES:
|
|
||||||
return REGISTRATION_REASON_MESSAGES[reason]
|
|
||||||
if reason:
|
|
||||||
return f"Registration rejected ({reason})"
|
|
||||||
if isinstance(detail, str) and detail:
|
|
||||||
return f"Registration rejected: {detail}"
|
|
||||||
return f"HTTP {status}: {parsed}"
|
|
||||||
|
|
||||||
|
|
||||||
def _cmd_detect(_args: argparse.Namespace) -> int:
|
def _cmd_detect(_args: argparse.Namespace) -> int:
|
||||||
devices = available_devices()
|
devices = available_devices()
|
||||||
if not devices:
|
if not devices:
|
||||||
|
|
@ -108,7 +39,6 @@ def _cmd_detect(_args: argparse.Namespace) -> int:
|
||||||
|
|
||||||
|
|
||||||
def _cmd_register(args: argparse.Namespace) -> int:
|
def _cmd_register(args: argparse.Namespace) -> int:
|
||||||
import urllib.error
|
|
||||||
import urllib.request
|
import urllib.request
|
||||||
|
|
||||||
hub_url = args.hub.rstrip("/")
|
hub_url = args.hub.rstrip("/")
|
||||||
|
|
@ -121,20 +51,11 @@ def _cmd_register(args: argparse.Namespace) -> int:
|
||||||
headers={
|
headers={
|
||||||
"Content-Type": "application/json",
|
"Content-Type": "application/json",
|
||||||
"X-API-Key": args.api_key,
|
"X-API-Key": args.api_key,
|
||||||
"User-Agent": _user_agent(),
|
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
with urllib.request.urlopen(req, timeout=_REGISTER_TIMEOUT_S) as resp:
|
with urllib.request.urlopen(req) as resp:
|
||||||
data = json.loads(resp.read())
|
data = json.loads(resp.read())
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
try:
|
|
||||||
err_body = e.read()
|
|
||||||
except Exception:
|
|
||||||
err_body = b""
|
|
||||||
msg = _explain_registration_failure(e.code, err_body)
|
|
||||||
print(f"error: registration failed: {msg}", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"error: registration failed: {e}", file=sys.stderr)
|
print(f"error: registration failed: {e}", file=sys.stderr)
|
||||||
return 1
|
return 1
|
||||||
|
|
@ -159,7 +80,7 @@ def _cmd_register(args: argparse.Namespace) -> int:
|
||||||
cfg.tx_allowed_freq_ranges = [[float(lo), float(hi)] for lo, hi in freq_ranges]
|
cfg.tx_allowed_freq_ranges = [[float(lo), float(hi)] for lo, hi in freq_ranges]
|
||||||
path = _config.save(cfg)
|
path = _config.save(cfg)
|
||||||
|
|
||||||
print(f"Registered agent: {agent_id} ({name})")
|
print(f"Registered agent: {agent_id}")
|
||||||
if cfg.tx_enabled:
|
if cfg.tx_enabled:
|
||||||
caps: list[str] = []
|
caps: list[str] = []
|
||||||
if cfg.tx_max_gain_db is not None:
|
if cfg.tx_max_gain_db is not None:
|
||||||
|
|
@ -219,17 +140,8 @@ def main() -> None:
|
||||||
sub.add_parser("detect", help="List available SDR drivers")
|
sub.add_parser("detect", help="List available SDR drivers")
|
||||||
|
|
||||||
p_reg = sub.add_parser("register", help="Register agent with RIA Hub and save credentials")
|
p_reg = sub.add_parser("register", help="Register agent with RIA Hub and save credentials")
|
||||||
p_reg.add_argument("--hub", default=DEFAULT_HUB_URL, help=f"RIA Hub URL (default: {DEFAULT_HUB_URL})")
|
p_reg.add_argument("--hub", required=True, help="RIA Hub URL (e.g. http://whitehorse:3005)")
|
||||||
p_reg.add_argument(
|
p_reg.add_argument("--api-key", dest="api_key", required=True, help="Hub API key")
|
||||||
"--api-key",
|
|
||||||
dest="api_key",
|
|
||||||
required=True,
|
|
||||||
help=(
|
|
||||||
"Personal registration key from the RIA Agents page on the hub "
|
|
||||||
"(format: ria_reg_...). Shown once when generated; save it then. "
|
|
||||||
"The legacy shared API key is also accepted but deprecated."
|
|
||||||
),
|
|
||||||
)
|
|
||||||
p_reg.add_argument("--name", default=None, help="Human-friendly agent name")
|
p_reg.add_argument("--name", default=None, help="Human-friendly agent name")
|
||||||
p_reg.add_argument("--insecure", action="store_true", help="Skip TLS verification")
|
p_reg.add_argument("--insecure", action="store_true", help="Skip TLS verification")
|
||||||
p_reg.add_argument(
|
p_reg.add_argument(
|
||||||
|
|
|
||||||
|
|
@ -45,14 +45,7 @@ def is_annotation_contained(inner: Annotation, outer: Annotation) -> bool:
|
||||||
outer_sample_stop = outer.sample_start + outer.sample_count
|
outer_sample_stop = outer.sample_start + outer.sample_count
|
||||||
|
|
||||||
if inner.sample_start > outer.sample_start and inner_sample_stop < outer_sample_stop:
|
if inner.sample_start > outer.sample_start and inner_sample_stop < outer_sample_stop:
|
||||||
if (
|
if inner.freq_lower_edge > outer.freq_lower_edge and inner.freq_upper_edge < outer.freq_upper_edge:
|
||||||
inner.freq_lower_edge is not None
|
|
||||||
and inner.freq_upper_edge is not None
|
|
||||||
and outer.freq_lower_edge is not None
|
|
||||||
and outer.freq_upper_edge is not None
|
|
||||||
and inner.freq_lower_edge > outer.freq_lower_edge
|
|
||||||
and inner.freq_upper_edge < outer.freq_upper_edge
|
|
||||||
):
|
|
||||||
return True
|
return True
|
||||||
|
|
||||||
return False
|
return False
|
||||||
|
|
|
||||||
|
|
@ -38,13 +38,7 @@ def annotate_with_cusum(
|
||||||
:type annotation_type: str
|
:type annotation_type: str
|
||||||
"""
|
"""
|
||||||
|
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Supply it with --sample-rate when using the CLI, or set "
|
|
||||||
"recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||||
|
|
||||||
# Create an object of the time segmenter
|
# Create an object of the time segmenter
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,6 @@ and occupied bandwidth calculation following ITU-R SM.328 standard.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import warnings
|
|
||||||
from typing import Tuple
|
from typing import Tuple
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
@ -120,17 +119,6 @@ def detect_signals_energy(
|
||||||
if active:
|
if active:
|
||||||
boundaries.append((start, len(smoothed_power) - start))
|
boundaries.append((start, len(smoothed_power) - start))
|
||||||
|
|
||||||
if not boundaries and noise_floor > 0:
|
|
||||||
peak = float(np.max(smoothed_power))
|
|
||||||
dynamic_range = peak / noise_floor
|
|
||||||
if dynamic_range < threshold_factor:
|
|
||||||
warnings.warn(
|
|
||||||
f"detect_signals_energy: no signal boundaries found — dynamic range {dynamic_range:.2f}x is below "
|
|
||||||
f"the threshold factor {threshold_factor}x. The signal may be constant-envelope (e.g. CW or chirp). "
|
|
||||||
"If the entire recording is signal, use 'ria annotate cusum' to segment it as a single region.",
|
|
||||||
stacklevel=2,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Merge boundaries that are closer than min_distance
|
# Merge boundaries that are closer than min_distance
|
||||||
merged_boundaries = []
|
merged_boundaries = []
|
||||||
if boundaries:
|
if boundaries:
|
||||||
|
|
@ -147,13 +135,7 @@ def detect_signals_energy(
|
||||||
merged_boundaries.append((start, length))
|
merged_boundaries.append((start, length))
|
||||||
|
|
||||||
# Create annotations from detected boundaries
|
# Create annotations from detected boundaries
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Supply it with --sample-rate when using the CLI, or set "
|
|
||||||
"recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||||
|
|
||||||
# Validate frequency method
|
# Validate frequency method
|
||||||
|
|
@ -369,12 +351,7 @@ def annotate_with_obw(
|
||||||
>>> annotated = annotate_with_obw(recording, label="signal_obw")
|
>>> annotated = annotate_with_obw(recording, label="signal_obw")
|
||||||
"""
|
"""
|
||||||
signal = recording.data[0]
|
signal = recording.data[0]
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Set recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_freq = recording.metadata.get("center_frequency", 0)
|
center_freq = recording.metadata.get("center_frequency", 0)
|
||||||
|
|
||||||
# Calculate OBW
|
# Calculate OBW
|
||||||
|
|
|
||||||
|
|
@ -49,7 +49,6 @@ allowing splitting of overlapping signals into separate training samples.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import warnings
|
|
||||||
from typing import List, Optional, Tuple
|
from typing import List, Optional, Tuple
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
@ -402,13 +401,7 @@ def split_recording_annotations(
|
||||||
return recording
|
return recording
|
||||||
|
|
||||||
signal = recording.data[0]
|
signal = recording.data[0]
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Supply it with --sample-rate when using the CLI, or set "
|
|
||||||
"recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_frequency = recording.metadata.get("center_frequency", 0.0)
|
center_frequency = recording.metadata.get("center_frequency", 0.0)
|
||||||
|
|
||||||
# Build new annotation list
|
# Build new annotation list
|
||||||
|
|
@ -432,11 +425,8 @@ def split_recording_annotations(
|
||||||
else:
|
else:
|
||||||
# No components found, keep original
|
# No components found, keep original
|
||||||
new_annotations.append(anno)
|
new_annotations.append(anno)
|
||||||
except Exception as e:
|
except Exception:
|
||||||
warnings.warn(
|
# Split failed for any reason, keep original
|
||||||
f"split_recording_annotations: failed to split annotation at index {i} ({e}); keeping original.",
|
|
||||||
stacklevel=2,
|
|
||||||
)
|
|
||||||
new_annotations.append(anno)
|
new_annotations.append(anno)
|
||||||
else:
|
else:
|
||||||
# Not in split list, keep as-is
|
# Not in split list, keep as-is
|
||||||
|
|
|
||||||
|
|
@ -24,7 +24,7 @@ def qualify_slice_from_annotations(recording: Recording, slice_length: int):
|
||||||
|
|
||||||
output_recordings = []
|
output_recordings = []
|
||||||
|
|
||||||
for i in range(len(recording.data[0]) // slice_length):
|
for i in range((len(recording.data[0]) // slice_length) - 1):
|
||||||
start_index = slice_length * i
|
start_index = slice_length * i
|
||||||
end_index = slice_length * (i + 1)
|
end_index = slice_length * (i + 1)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -35,24 +35,17 @@ def isolate_signal(recording: Recording, annotation: Annotation) -> Recording:
|
||||||
|
|
||||||
isolation_bw = anno_bw
|
isolation_bw = anno_bw
|
||||||
|
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Set recording.sample_rate before calling isolate_signal."
|
|
||||||
)
|
|
||||||
|
|
||||||
# frequency shift the center of the box about zero
|
# frequency shift the center of the box about zero
|
||||||
shifted_signal_slice = frequency_shift_iq_samples(
|
shifted_signal_slice = frequency_shift_iq_samples(
|
||||||
iq_samples=signal_slice,
|
iq_samples=signal_slice,
|
||||||
sample_rate=sample_rate,
|
sample_rate=recording.metadata["sample_rate"],
|
||||||
shift_frequency=-1 * anno_base_center_freq,
|
shift_frequency=-1 * anno_base_center_freq,
|
||||||
)
|
)
|
||||||
|
|
||||||
# filter
|
# filter
|
||||||
if isolation_bw < sample_rate - 1:
|
if isolation_bw < recording.metadata["sample_rate"] - 1:
|
||||||
filtered_signal = apply_complex_lowpass_filter(
|
filtered_signal = apply_complex_lowpass_filter(
|
||||||
signal=shifted_signal_slice, cutoff_frequency=isolation_bw, sample_rate=sample_rate
|
signal=shifted_signal_slice, cutoff_frequency=isolation_bw, sample_rate=recording.metadata["sample_rate"]
|
||||||
)
|
)
|
||||||
|
|
||||||
else:
|
else:
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,6 @@ classification or demodulation stages.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import warnings
|
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
@ -217,22 +216,11 @@ def threshold_qualifier(
|
||||||
"""
|
"""
|
||||||
# Extract signal and metadata
|
# Extract signal and metadata
|
||||||
sample_data = recording.data[channel]
|
sample_data = recording.data[channel]
|
||||||
sample_rate = recording.metadata.get("sample_rate")
|
sample_rate = recording.metadata["sample_rate"]
|
||||||
if sample_rate is None:
|
|
||||||
raise ValueError(
|
|
||||||
"Recording metadata does not contain 'sample_rate'. "
|
|
||||||
"Supply it with --sample-rate when using the CLI, or set "
|
|
||||||
"recording.sample_rate before calling this function."
|
|
||||||
)
|
|
||||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||||
|
|
||||||
n_samples = len(sample_data)
|
|
||||||
|
|
||||||
if window_size is None:
|
if window_size is None:
|
||||||
window_size = max(64, int(sample_rate * 0.001))
|
window_size = max(64, int(sample_rate * 0.001))
|
||||||
# Cap at 1% of signal length so short recordings aren't over-smoothed into
|
|
||||||
# a flat envelope that collapses the dynamic range below the early-exit guard.
|
|
||||||
window_size = min(window_size, max(64, n_samples // 100))
|
|
||||||
|
|
||||||
# --- 1. SIGNAL CONDITIONING ---
|
# --- 1. SIGNAL CONDITIONING ---
|
||||||
# Convert to power (Magnitude squared)
|
# Convert to power (Magnitude squared)
|
||||||
|
|
@ -249,12 +237,6 @@ def threshold_qualifier(
|
||||||
# Soft early exit: keep a guard for low-contrast noise, but compute it from
|
# Soft early exit: keep a guard for low-contrast noise, but compute it from
|
||||||
# the quieter tail of the envelope so burst-heavy captures are not rejected.
|
# the quieter tail of the envelope so burst-heavy captures are not rejected.
|
||||||
if dynamic_range_ratio < 1.5:
|
if dynamic_range_ratio < 1.5:
|
||||||
warnings.warn(
|
|
||||||
f"threshold_qualifier: dynamic range ratio {dynamic_range_ratio:.2f} is below 1.5 — "
|
|
||||||
"the signal appears to be constant-envelope or pure noise, so no burst boundaries can be found. "
|
|
||||||
"If the entire recording is signal, use 'ria annotate cusum' to segment it as a single region.",
|
|
||||||
stacklevel=2,
|
|
||||||
)
|
|
||||||
return Recording(data=recording.data, metadata=recording.metadata, annotations=recording.annotations)
|
return Recording(data=recording.data, metadata=recording.metadata, annotations=recording.annotations)
|
||||||
|
|
||||||
trigger_val = noise_floor + threshold * (max_power - noise_floor)
|
trigger_val = noise_floor + threshold * (max_power - noise_floor)
|
||||||
|
|
@ -314,7 +296,7 @@ def threshold_qualifier(
|
||||||
# burst energy does not bleed through the long window into adjacent regions,
|
# burst energy does not bleed through the long window into adjacent regions,
|
||||||
# which would inflate macro_residual_max and push the trigger above the
|
# which would inflate macro_residual_max and push the trigger above the
|
||||||
# faint burst's average power.
|
# faint burst's average power.
|
||||||
macro_window_size = min(max(window_size * 16, int(sample_rate * 0.02)), max(window_size * 2, n_samples // 4))
|
macro_window_size = max(window_size * 16, int(sample_rate * 0.02))
|
||||||
macro_kernel = np.ones(macro_window_size, dtype=np.float64) / macro_window_size
|
macro_kernel = np.ones(macro_window_size, dtype=np.float64) / macro_window_size
|
||||||
# Expand each annotated range by half the macro window on both sides so that
|
# Expand each annotated range by half the macro window on both sides so that
|
||||||
# the long convolution cannot "see" the leading/trailing edges of already-
|
# the long convolution cannot "see" the leading/trailing edges of already-
|
||||||
|
|
|
||||||
|
|
@ -17,10 +17,10 @@ class Annotation:
|
||||||
:type sample_start: int
|
:type sample_start: int
|
||||||
:param sample_count: The index of the ending sample of the annotation, inclusive.
|
:param sample_count: The index of the ending sample of the annotation, inclusive.
|
||||||
:type sample_count: int
|
:type sample_count: int
|
||||||
:param freq_lower_edge: The lower frequency of the annotation. Optional; None if not specified in source.
|
:param freq_lower_edge: The lower frequency of the annotation.
|
||||||
:type freq_lower_edge: float, optional
|
:type freq_lower_edge: float
|
||||||
:param freq_upper_edge: The upper frequency of the annotation. Optional; None if not specified in source.
|
:param freq_upper_edge: The upper frequency of the annotation.
|
||||||
:type freq_upper_edge: float, optional
|
:type freq_upper_edge: float
|
||||||
:param label: The label that will be displayed with the bounding box in compatible viewers including IQEngine.
|
:param label: The label that will be displayed with the bounding box in compatible viewers including IQEngine.
|
||||||
Defaults to an emtpy string.
|
Defaults to an emtpy string.
|
||||||
:type label: str, optional
|
:type label: str, optional
|
||||||
|
|
@ -34,8 +34,8 @@ class Annotation:
|
||||||
self,
|
self,
|
||||||
sample_start: int,
|
sample_start: int,
|
||||||
sample_count: int,
|
sample_count: int,
|
||||||
freq_lower_edge: Optional[float] = None,
|
freq_lower_edge: float,
|
||||||
freq_upper_edge: Optional[float] = None,
|
freq_upper_edge: float,
|
||||||
label: Optional[str] = "",
|
label: Optional[str] = "",
|
||||||
comment: Optional[str] = "",
|
comment: Optional[str] = "",
|
||||||
detail: Optional[dict] = None,
|
detail: Optional[dict] = None,
|
||||||
|
|
@ -43,8 +43,8 @@ class Annotation:
|
||||||
"""Initialize a new Annotation instance."""
|
"""Initialize a new Annotation instance."""
|
||||||
self.sample_start = int(sample_start)
|
self.sample_start = int(sample_start)
|
||||||
self.sample_count = int(sample_count)
|
self.sample_count = int(sample_count)
|
||||||
self.freq_lower_edge = float(freq_lower_edge) if freq_lower_edge is not None else None
|
self.freq_lower_edge = float(freq_lower_edge)
|
||||||
self.freq_upper_edge = float(freq_upper_edge) if freq_upper_edge is not None else None
|
self.freq_upper_edge = float(freq_upper_edge)
|
||||||
self.label = str(label)
|
self.label = str(label)
|
||||||
self.comment = str(comment)
|
self.comment = str(comment)
|
||||||
|
|
||||||
|
|
@ -62,8 +62,6 @@ class Annotation:
|
||||||
:returns: True if valid, False if not.
|
:returns: True if valid, False if not.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
if self.freq_lower_edge is None or self.freq_upper_edge is None:
|
|
||||||
return self.sample_count > 0
|
|
||||||
return self.sample_count > 0 and self.freq_lower_edge < self.freq_upper_edge
|
return self.sample_count > 0 and self.freq_lower_edge < self.freq_upper_edge
|
||||||
|
|
||||||
def overlap(self, other):
|
def overlap(self, other):
|
||||||
|
|
@ -75,14 +73,6 @@ class Annotation:
|
||||||
|
|
||||||
:returns: The area of the overlap in samples*frequency, or 0 if they do not overlap."""
|
:returns: The area of the overlap in samples*frequency, or 0 if they do not overlap."""
|
||||||
|
|
||||||
if (
|
|
||||||
self.freq_lower_edge is None
|
|
||||||
or self.freq_upper_edge is None
|
|
||||||
or other.freq_lower_edge is None
|
|
||||||
or other.freq_upper_edge is None
|
|
||||||
):
|
|
||||||
return 0
|
|
||||||
|
|
||||||
sample_overlap_start = max(self.sample_start, other.sample_start)
|
sample_overlap_start = max(self.sample_start, other.sample_start)
|
||||||
sample_overlap_end = min(self.sample_start + self.sample_count, other.sample_start + other.sample_count)
|
sample_overlap_end = min(self.sample_start + self.sample_count, other.sample_start + other.sample_count)
|
||||||
|
|
||||||
|
|
@ -101,8 +91,6 @@ class Annotation:
|
||||||
|
|
||||||
:returns: sample length multiplied by bandwidth."""
|
:returns: sample length multiplied by bandwidth."""
|
||||||
|
|
||||||
if self.freq_lower_edge is None or self.freq_upper_edge is None:
|
|
||||||
return 0
|
|
||||||
return self.sample_count * (self.freq_upper_edge - self.freq_lower_edge)
|
return self.sample_count * (self.freq_upper_edge - self.freq_lower_edge)
|
||||||
|
|
||||||
def __eq__(self, other: Annotation) -> bool:
|
def __eq__(self, other: Annotation) -> bool:
|
||||||
|
|
@ -115,16 +103,13 @@ class Annotation:
|
||||||
|
|
||||||
annotation_dict = {SigMFFile.START_INDEX_KEY: self.sample_start, SigMFFile.LENGTH_INDEX_KEY: self.sample_count}
|
annotation_dict = {SigMFFile.START_INDEX_KEY: self.sample_start, SigMFFile.LENGTH_INDEX_KEY: self.sample_count}
|
||||||
|
|
||||||
metadata = {
|
annotation_dict["metadata"] = {
|
||||||
SigMFFile.LABEL_KEY: self.label,
|
SigMFFile.LABEL_KEY: self.label,
|
||||||
SigMFFile.COMMENT_KEY: self.comment,
|
SigMFFile.COMMENT_KEY: self.comment,
|
||||||
|
SigMFFile.FHI_KEY: self.freq_upper_edge,
|
||||||
|
SigMFFile.FLO_KEY: self.freq_lower_edge,
|
||||||
"ria:detail": self.detail,
|
"ria:detail": self.detail,
|
||||||
}
|
}
|
||||||
if self.freq_upper_edge is not None:
|
|
||||||
metadata[SigMFFile.FHI_KEY] = self.freq_upper_edge
|
|
||||||
if self.freq_lower_edge is not None:
|
|
||||||
metadata[SigMFFile.FLO_KEY] = self.freq_lower_edge
|
|
||||||
annotation_dict["metadata"] = metadata
|
|
||||||
|
|
||||||
if _is_jsonable(annotation_dict):
|
if _is_jsonable(annotation_dict):
|
||||||
return annotation_dict
|
return annotation_dict
|
||||||
|
|
|
||||||
|
|
@ -175,15 +175,6 @@ def from_npy(file: os.PathLike | str, legacy: bool = False) -> Recording:
|
||||||
)
|
)
|
||||||
data = first # already loaded without pickle (numeric array)
|
data = first # already loaded without pickle (numeric array)
|
||||||
metadata = np.load(f, allow_pickle=True).tolist()
|
metadata = np.load(f, allow_pickle=True).tolist()
|
||||||
# Normalize namespaced keys (e.g. "BlockGenerator:Foo:sample_rate") to
|
|
||||||
# their bare equivalents so downstream code can find them reliably.
|
|
||||||
_STANDARD_KEYS = {"sample_rate", "center_frequency", "bandwidth"}
|
|
||||||
if isinstance(metadata, dict):
|
|
||||||
for k in list(metadata):
|
|
||||||
if ":" in k:
|
|
||||||
bare = k.rsplit(":", 1)[-1]
|
|
||||||
if bare in _STANDARD_KEYS and bare not in metadata:
|
|
||||||
metadata[bare] = metadata[k]
|
|
||||||
try:
|
try:
|
||||||
annotations = list(np.load(f, allow_pickle=True))
|
annotations = list(np.load(f, allow_pickle=True))
|
||||||
except EOFError:
|
except EOFError:
|
||||||
|
|
|
||||||
|
|
@ -58,7 +58,7 @@ class HackRF(SDR):
|
||||||
:param channel: The channel the HackRF is set to. (Not actually used)
|
:param channel: The channel the HackRF is set to. (Not actually used)
|
||||||
:type channel: int
|
:type channel: int
|
||||||
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
||||||
'relative' means that gain should be a negative value, and it will be subtracted from the max gain (40).
|
'relative' means that gain should be a negative value, and it will be subtracted from the max gain (40).
|
||||||
:type gain_mode: str
|
:type gain_mode: str
|
||||||
"""
|
"""
|
||||||
print("Initializing RX")
|
print("Initializing RX")
|
||||||
|
|
|
||||||
|
|
@ -54,7 +54,7 @@ class USRP(SDR):
|
||||||
:param channel: The channel the USRP is set to.
|
:param channel: The channel the USRP is set to.
|
||||||
:type channel: int
|
:type channel: int
|
||||||
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
||||||
'relative' means that gain should be a negative value, and it will be subtracted from the max gain.
|
'relative' means that gain should be a negative value, and it will be subtracted from the max gain.
|
||||||
:type gain_mode: str
|
:type gain_mode: str
|
||||||
:param rx_buffer_size: Internal buffer size for receiving samples. Defaults to 960000.
|
:param rx_buffer_size: Internal buffer size for receiving samples. Defaults to 960000.
|
||||||
:type rx_buffer_size: int
|
:type rx_buffer_size: int
|
||||||
|
|
@ -285,7 +285,7 @@ class USRP(SDR):
|
||||||
:param channel: The channel the USRP is set to.
|
:param channel: The channel the USRP is set to.
|
||||||
:type channel: int
|
:type channel: int
|
||||||
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
:param gain_mode: 'absolute' passes gain directly to the sdr,
|
||||||
'relative' means that gain should be a negative value, and it will be subtracted from the max gain.
|
'relative' means that gain should be a negative value, and it will be subtracted from the max gain.
|
||||||
:type gain_mode: str
|
:type gain_mode: str
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -32,15 +32,16 @@ def extract_metadata_fields(metadata):
|
||||||
|
|
||||||
|
|
||||||
def set_path(output_path):
|
def set_path(output_path):
|
||||||
path = pathlib.Path(output_path)
|
split_path = output_path.split("/")
|
||||||
|
|
||||||
# If only filename provided (no directory), use default 'images' folder
|
if len(split_path) == 1:
|
||||||
if len(path.parts) == 1:
|
folder = "images"
|
||||||
folder = pathlib.Path("images")
|
file = split_path[0]
|
||||||
file = path.name
|
elif len(split_path) > 2:
|
||||||
|
file = split_path[-1]
|
||||||
|
folder = "/".join(split_path[:-1])
|
||||||
else:
|
else:
|
||||||
folder = path.parent
|
folder, file = split_path
|
||||||
file = path.name
|
|
||||||
|
|
||||||
split_file = file.split(".")
|
split_file = file.split(".")
|
||||||
if len(split_file) == 2:
|
if len(split_file) == 2:
|
||||||
|
|
@ -52,5 +53,5 @@ def set_path(output_path):
|
||||||
extension = "png"
|
extension = "png"
|
||||||
file = file + ".png"
|
file = file + ".png"
|
||||||
|
|
||||||
folder.mkdir(parents=True, exist_ok=True)
|
pathlib.Path(folder).mkdir(parents=True, exist_ok=True)
|
||||||
return str(folder / file), extension
|
return "/".join([folder, file]), extension
|
||||||
|
|
|
||||||
|
|
@ -3,12 +3,11 @@ import os
|
||||||
import textwrap
|
import textwrap
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
import matplotlib
|
|
||||||
import matplotlib.pyplot as plt
|
import matplotlib.pyplot as plt
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from matplotlib import gridspec, ticker
|
from matplotlib import gridspec
|
||||||
from matplotlib.patches import Patch
|
from matplotlib.patches import Patch
|
||||||
from PIL import Image, UnidentifiedImageError
|
from PIL import Image
|
||||||
from scipy.fft import fft, fftshift
|
from scipy.fft import fft, fftshift
|
||||||
from scipy.signal import spectrogram
|
from scipy.signal import spectrogram
|
||||||
from scipy.signal.windows import hann
|
from scipy.signal.windows import hann
|
||||||
|
|
@ -81,8 +80,6 @@ def view_annotations(
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
for annotation in sorted(annotations, key=_threshold_sort_key, reverse=True):
|
for annotation in sorted(annotations, key=_threshold_sort_key, reverse=True):
|
||||||
if annotation.freq_lower_edge is None or annotation.freq_upper_edge is None:
|
|
||||||
continue
|
|
||||||
t_start = annotation.sample_start / sample_rate
|
t_start = annotation.sample_start / sample_rate
|
||||||
t_width = annotation.sample_count / sample_rate
|
t_width = annotation.sample_count / sample_rate
|
||||||
f_start = annotation.freq_lower_edge
|
f_start = annotation.freq_lower_edge
|
||||||
|
|
@ -188,7 +185,7 @@ def view_sig(
|
||||||
logo: Optional[bool] = True,
|
logo: Optional[bool] = True,
|
||||||
dark: Optional[bool] = True,
|
dark: Optional[bool] = True,
|
||||||
spines: Optional[bool] = False,
|
spines: Optional[bool] = False,
|
||||||
title_fontsize: Optional[int] = 25,
|
title_fontsize: Optional[int] = 35,
|
||||||
subtitle_fontsize: Optional[int] = 15,
|
subtitle_fontsize: Optional[int] = 15,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""
|
"""
|
||||||
|
|
@ -233,26 +230,11 @@ def view_sig(
|
||||||
complex_signal = recording.data[0]
|
complex_signal = recording.data[0]
|
||||||
sample_rate, center_frequency, _ = extract_metadata_fields(recording.metadata)
|
sample_rate, center_frequency, _ = extract_metadata_fields(recording.metadata)
|
||||||
|
|
||||||
subplot_height = 3 * (plot_spectrogram) + 2 * (iq + frequency) + 3 * (constellation or metadata or logo)
|
subplot_height = 2 * (plot_spectrogram + iq + frequency) + 3 * (constellation or metadata or logo)
|
||||||
subplot_width = max((constellation + metadata or 1), logo * 3)
|
subplot_width = max((constellation + metadata or 1), logo * 3)
|
||||||
|
|
||||||
if dark:
|
if dark:
|
||||||
plt.style.use("dark_background")
|
plt.style.use("dark_background")
|
||||||
matplotlib.rcParams.update(
|
|
||||||
{
|
|
||||||
"figure.facecolor": "#161616",
|
|
||||||
"axes.facecolor": "#161616",
|
|
||||||
"savefig.facecolor": "#161616",
|
|
||||||
"savefig.edgecolor": "#161616",
|
|
||||||
"font.size": 10,
|
|
||||||
"axes.titlesize": 15,
|
|
||||||
"axes.labelsize": 10,
|
|
||||||
"xtick.labelsize": 10,
|
|
||||||
"ytick.labelsize": 10,
|
|
||||||
"legend.frameon": False,
|
|
||||||
"legend.facecolor": "none",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
logo_path = os.path.dirname(__file__) + "/graphics/Qoherent-logo-white-transparent.png"
|
logo_path = os.path.dirname(__file__) + "/graphics/Qoherent-logo-white-transparent.png"
|
||||||
else:
|
else:
|
||||||
plt.style.use("default")
|
plt.style.use("default")
|
||||||
|
|
@ -270,8 +252,8 @@ def view_sig(
|
||||||
plot_x_indx = 0
|
plot_x_indx = 0
|
||||||
|
|
||||||
if plot_spectrogram:
|
if plot_spectrogram:
|
||||||
spec_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 3, :])
|
spec_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
|
||||||
plot_y_indx = plot_y_indx + 3
|
plot_y_indx = plot_y_indx + 2
|
||||||
fft_size = get_fft_size(plot_length=plot_length)
|
fft_size = get_fft_size(plot_length=plot_length)
|
||||||
|
|
||||||
_, t_spec, Sxx = spectrogram(
|
_, t_spec, Sxx = spectrogram(
|
||||||
|
|
@ -298,10 +280,7 @@ def view_sig(
|
||||||
)
|
)
|
||||||
|
|
||||||
set_spines(spec_ax, spines)
|
set_spines(spec_ax, spines)
|
||||||
spec_ax.set_title("Spectrogram", loc="left", fontsize=subtitle_fontsize)
|
spec_ax.set_title("Spectrogram", loc="center", fontsize=subtitle_fontsize)
|
||||||
spec_ax.set_xlabel("Time (s)")
|
|
||||||
spec_ax.set_ylabel("Frequency (MHz)")
|
|
||||||
spec_ax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
|
||||||
|
|
||||||
if iq:
|
if iq:
|
||||||
iq_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
|
iq_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
|
||||||
|
|
@ -312,13 +291,12 @@ def view_sig(
|
||||||
|
|
||||||
iq_ax.plot(t, plot_iq.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
|
iq_ax.plot(t, plot_iq.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
|
||||||
iq_ax.plot(t, plot_iq.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
|
iq_ax.plot(t, plot_iq.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
|
||||||
iq_ax.grid(True, alpha=0.2, linewidth=0.5)
|
iq_ax.grid(False)
|
||||||
|
|
||||||
iq_ax.set_ylabel("Amplitude")
|
iq_ax.set_ylabel("Amplitude")
|
||||||
iq_ax.set_xlim([min(t), max(t)])
|
iq_ax.set_xlim([min(t), max(t)])
|
||||||
iq_ax.set_xlabel("Time (s)")
|
iq_ax.set_xlabel("Time (s)")
|
||||||
iq_ax.set_title("IQ Sample Plot", loc="left", fontsize=subtitle_fontsize)
|
iq_ax.set_title("IQ Sample Plot", fontsize=subtitle_fontsize)
|
||||||
iq_ax.legend(loc="upper right", fontsize=10)
|
|
||||||
set_spines(iq_ax, spines)
|
set_spines(iq_ax, spines)
|
||||||
|
|
||||||
if frequency:
|
if frequency:
|
||||||
|
|
@ -332,14 +310,10 @@ def view_sig(
|
||||||
# Convert to dB
|
# Convert to dB
|
||||||
spectrum_db = 20 * np.log10(spectrum + 1e-12) # 20*log for magnitude
|
spectrum_db = 20 * np.log10(spectrum + 1e-12) # 20*log for magnitude
|
||||||
|
|
||||||
freqs = (
|
freqs = np.linspace(-sample_rate / 2, sample_rate / 2, len(complex_signal[:plot_length])) + center_frequency
|
||||||
np.linspace(-sample_rate / 2, sample_rate / 2, len(complex_signal[:plot_length])) + center_frequency
|
|
||||||
) / 1e6
|
|
||||||
freq_ax.plot(freqs, spectrum_db, color=COLORS["accent"], linewidth=0.8)
|
freq_ax.plot(freqs, spectrum_db, color=COLORS["accent"], linewidth=0.8)
|
||||||
freq_ax.set_xlabel("Frequency (MHz)")
|
|
||||||
freq_ax.set_ylabel("Magnitude (dB)")
|
freq_ax.set_ylabel("Magnitude (dB)")
|
||||||
freq_ax.grid(True, alpha=0.2, linewidth=0.5)
|
freq_ax.set_title("Frequency Spectrum (Windowed FFT)", fontsize=subtitle_fontsize)
|
||||||
freq_ax.set_title("Frequency Spectrum (Windowed FFT)", loc="left", fontsize=subtitle_fontsize)
|
|
||||||
set_spines(freq_ax, spines)
|
set_spines(freq_ax, spines)
|
||||||
|
|
||||||
if constellation:
|
if constellation:
|
||||||
|
|
@ -352,7 +326,7 @@ def view_sig(
|
||||||
const_ax.set_ylim([-1 * dimension, dimension])
|
const_ax.set_ylim([-1 * dimension, dimension])
|
||||||
const_ax.set_xlabel("In-phase (I)")
|
const_ax.set_xlabel("In-phase (I)")
|
||||||
const_ax.set_ylabel("Quadrature (Q)")
|
const_ax.set_ylabel("Quadrature (Q)")
|
||||||
const_ax.set_title("Constellation", loc="left", fontsize=subtitle_fontsize)
|
const_ax.set_title("Constellation", fontsize=subtitle_fontsize)
|
||||||
const_ax.set_aspect("equal")
|
const_ax.set_aspect("equal")
|
||||||
|
|
||||||
if not spines:
|
if not spines:
|
||||||
|
|
@ -401,8 +375,8 @@ def view_sig(
|
||||||
image = Image.open(logo_path) # Open the PNG image using PIL
|
image = Image.open(logo_path) # Open the PNG image using PIL
|
||||||
logo_ax.imshow(image)
|
logo_ax.imshow(image)
|
||||||
|
|
||||||
except (FileNotFoundError, UnidentifiedImageError, OSError) as exc:
|
except FileNotFoundError:
|
||||||
print(f"Warning, could not load logo image: {logo_path}. Reason: {exc}")
|
print(f"Warning, {logo_path} not found.")
|
||||||
|
|
||||||
fig.subplots_adjust(
|
fig.subplots_adjust(
|
||||||
left=0.1, # Left margin
|
left=0.1, # Left margin
|
||||||
|
|
|
||||||
|
|
@ -119,19 +119,24 @@ def setup_style(*, labels_mode: bool = False, compact_mode: bool = False) -> Non
|
||||||
label_font = 14
|
label_font = 14
|
||||||
else:
|
else:
|
||||||
base_font = 10
|
base_font = 10
|
||||||
title_font = 15
|
title_font = 12
|
||||||
label_font = 10
|
label_font = 10
|
||||||
|
|
||||||
matplotlib.rcParams.update(
|
matplotlib.rcParams.update(
|
||||||
{
|
{
|
||||||
"figure.facecolor": "#161616",
|
"figure.facecolor": "#0f172a",
|
||||||
"axes.facecolor": "#161616",
|
"axes.facecolor": "#1e293b",
|
||||||
"savefig.facecolor": "#161616",
|
"axes.edgecolor": COLORS["muted"],
|
||||||
"savefig.edgecolor": "#161616",
|
"axes.labelcolor": COLORS["light"],
|
||||||
|
"text.color": COLORS["light"],
|
||||||
|
"xtick.color": COLORS["muted"],
|
||||||
|
"ytick.color": COLORS["muted"],
|
||||||
|
"grid.color": COLORS["muted"],
|
||||||
|
"grid.alpha": 0.3,
|
||||||
"font.size": base_font,
|
"font.size": base_font,
|
||||||
"axes.titlesize": title_font,
|
"axes.titlesize": title_font,
|
||||||
"axes.labelsize": label_font,
|
"axes.labelsize": label_font,
|
||||||
"figure.titlesize": title_font + 4,
|
"figure.titlesize": title_font + 2,
|
||||||
"legend.frameon": False,
|
"legend.frameon": False,
|
||||||
"legend.facecolor": "none",
|
"legend.facecolor": "none",
|
||||||
"xtick.labelsize": base_font,
|
"xtick.labelsize": base_font,
|
||||||
|
|
@ -189,7 +194,7 @@ def view_simple_sig(
|
||||||
constellation_mode: Optional[bool] = False,
|
constellation_mode: Optional[bool] = False,
|
||||||
labels_mode: Optional[bool] = False,
|
labels_mode: Optional[bool] = False,
|
||||||
slice: Optional[tuple] = None,
|
slice: Optional[tuple] = None,
|
||||||
title: Optional[str] = "Signal Plot",
|
title: Optional[str] = "Signal",
|
||||||
):
|
):
|
||||||
"""
|
"""
|
||||||
Create a simple plot of various signal visualizations as a png or svg image.
|
Create a simple plot of various signal visualizations as a png or svg image.
|
||||||
|
|
@ -232,7 +237,7 @@ def view_simple_sig(
|
||||||
spec_signal = signal
|
spec_signal = signal
|
||||||
|
|
||||||
if compact_mode:
|
if compact_mode:
|
||||||
fig, (ax2, ax1) = plt.subplots(2, 1, figsize=(12, 6), gridspec_kw={"height_ratios": [5, 1]})
|
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 6), gridspec_kw={"height_ratios": [1, 5]})
|
||||||
show_title = False
|
show_title = False
|
||||||
show_labels = False
|
show_labels = False
|
||||||
ax_constellation = ax_psd = None
|
ax_constellation = ax_psd = None
|
||||||
|
|
@ -248,24 +253,25 @@ def view_simple_sig(
|
||||||
ax_psd = None
|
ax_psd = None
|
||||||
else:
|
else:
|
||||||
if constellation_mode:
|
if constellation_mode:
|
||||||
fig, ((ax2, ax1), (ax3, ax4)) = plt.subplots(2, 2, figsize=(16, 12))
|
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(16, 12))
|
||||||
ax_constellation, ax_psd = ax3, ax4
|
ax_constellation, ax_psd = ax3, ax4
|
||||||
else:
|
else:
|
||||||
fig, (ax2, ax1) = plt.subplots(2, 1, figsize=(14, 10))
|
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 10))
|
||||||
ax_constellation = ax_psd = None
|
ax_constellation = ax_psd = None
|
||||||
show_title = True
|
show_title = True
|
||||||
show_labels = labels_mode
|
show_labels = labels_mode
|
||||||
|
|
||||||
if show_title:
|
if show_title:
|
||||||
fig.suptitle(title, fontsize=25)
|
fig.suptitle(title, fontsize=16, color=COLORS["light"], y=0.96)
|
||||||
fig.patch.set_facecolor(matplotlib.rcParams["figure.facecolor"])
|
fig.patch.set_facecolor("#0f172a")
|
||||||
|
|
||||||
total_duration_s = len(signal) / sample_rate_hz if sample_rate_hz else 0.0
|
total_duration_s = len(signal) / sample_rate_hz if sample_rate_hz else 0.0
|
||||||
t_s = np.linspace(0, total_duration_s, len(display_signal)) if len(display_signal) else np.array([])
|
t_s = np.linspace(0, total_duration_s, len(display_signal)) if len(display_signal) else np.array([])
|
||||||
|
|
||||||
ax1.plot(t_s, display_signal.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
|
ax1.plot(t_s, display_signal.real, color=COLORS["purple"], linewidth=0.8, alpha=0.8, label="I")
|
||||||
ax1.plot(t_s, display_signal.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
|
ax1.plot(t_s, display_signal.imag, color=COLORS["magenta"], linewidth=0.8, alpha=0.8, label="Q")
|
||||||
ax1.grid(True, alpha=0.2, linewidth=0.5)
|
ax1.set_xlim(0, total_duration_s)
|
||||||
|
ax1.grid(True, alpha=0.3)
|
||||||
|
|
||||||
nfft, overlap = _get_nfft_size(signal=signal, fast_mode=fast_mode)
|
nfft, overlap = _get_nfft_size(signal=signal, fast_mode=fast_mode)
|
||||||
|
|
||||||
|
|
@ -279,7 +285,7 @@ def view_simple_sig(
|
||||||
)
|
)
|
||||||
|
|
||||||
ax2.set_ylim(center_freq_hz - sample_rate_hz / 2, center_freq_hz + sample_rate_hz / 2)
|
ax2.set_ylim(center_freq_hz - sample_rate_hz / 2, center_freq_hz + sample_rate_hz / 2)
|
||||||
ax1.set_xlim(ax2.get_xlim())
|
ax2.set_xlim(0, total_duration_s)
|
||||||
|
|
||||||
if show_labels:
|
if show_labels:
|
||||||
if horizontal_mode:
|
if horizontal_mode:
|
||||||
|
|
@ -288,25 +294,20 @@ def view_simple_sig(
|
||||||
ax2.set_xlabel("Time (s)")
|
ax2.set_xlabel("Time (s)")
|
||||||
|
|
||||||
ax1.set_ylabel("Amplitude")
|
ax1.set_ylabel("Amplitude")
|
||||||
ax1.set_title(f"IQ Sample Plot - {sdr} SDR", loc="left", pad=10, fontsize=15)
|
ax1.set_title(f"Time Series - {sdr} SDR", loc="left", pad=10)
|
||||||
ax1.legend(loc="upper right", fontsize=10)
|
ax1.legend(loc="upper right")
|
||||||
|
|
||||||
ax2.set_ylabel("Frequency (MHz)")
|
ax2.set_ylabel("Frequency (Hz)")
|
||||||
ax2.set_title(
|
ax2.set_title(
|
||||||
f"Spectrogram - {center_freq_hz / 1e6:.1f} MHz ± {sample_rate_hz / 2e6:.1f} MHz",
|
f"Spectrogram - {center_freq_hz / 1e6:.1f} MHz ± {sample_rate_hz / 2e6:.1f} MHz", loc="left", pad=10
|
||||||
loc="left",
|
|
||||||
pad=10,
|
|
||||||
fontsize=15,
|
|
||||||
)
|
)
|
||||||
ax2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
yticks = ax2.get_yticks()
|
||||||
|
ax2.set_yticklabels([f"{y / 1e6:.1f}" for y in yticks])
|
||||||
elif not compact_mode:
|
elif not compact_mode:
|
||||||
ax1.set_title("IQ Sample Plot", loc="left", pad=10, fontsize=15)
|
ax1.set_title("Time Series", loc="left", pad=10)
|
||||||
ax1.legend(loc="upper right", fontsize=10)
|
ax1.legend(loc="upper right", fontsize=8)
|
||||||
|
|
||||||
ax2.set_xlabel("Time (s)")
|
ax2.set_title("Spectrogram", loc="left", pad=10)
|
||||||
ax2.set_ylabel("Frequency (MHz)")
|
|
||||||
ax2.set_title("Spectrogram", loc="left", pad=10, fontsize=15)
|
|
||||||
ax2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
|
||||||
|
|
||||||
_add_annotations(
|
_add_annotations(
|
||||||
annotations=annotations,
|
annotations=annotations,
|
||||||
|
|
@ -338,8 +339,8 @@ def view_simple_sig(
|
||||||
)
|
)
|
||||||
ax_constellation.set_xlabel("In-phase (I)")
|
ax_constellation.set_xlabel("In-phase (I)")
|
||||||
ax_constellation.set_ylabel("Quadrature (Q)")
|
ax_constellation.set_ylabel("Quadrature (Q)")
|
||||||
ax_constellation.set_title("Constellation", loc="left", fontsize=15)
|
ax_constellation.set_title("Constellation")
|
||||||
ax_constellation.grid(True, alpha=0.2, linewidth=0.5)
|
ax_constellation.grid(True, alpha=0.3)
|
||||||
ax_constellation.set_aspect("equal")
|
ax_constellation.set_aspect("equal")
|
||||||
|
|
||||||
if ax_psd is not None:
|
if ax_psd is not None:
|
||||||
|
|
@ -350,11 +351,11 @@ def view_simple_sig(
|
||||||
freqs = freqs + center_freq_hz
|
freqs = freqs + center_freq_hz
|
||||||
spectrum_db = 10 * np.log10(spectrum + 1e-12)
|
spectrum_db = 10 * np.log10(spectrum + 1e-12)
|
||||||
|
|
||||||
ax_psd.plot(freqs / 1e6, spectrum_db, color=COLORS["accent"], linewidth=0.8)
|
ax_psd.plot(freqs / 1e6, spectrum_db, color=COLORS["accent"], linewidth=1.0)
|
||||||
ax_psd.set_xlabel("Frequency (MHz)")
|
ax_psd.set_xlabel("Frequency (MHz)")
|
||||||
ax_psd.set_ylabel("Power (dB)")
|
ax_psd.set_ylabel("Power (dB)")
|
||||||
ax_psd.set_title("Power Spectral Density", loc="left", fontsize=15)
|
ax_psd.set_title("Power Spectral Density")
|
||||||
ax_psd.grid(True, alpha=0.2, linewidth=0.5)
|
ax_psd.grid(True, alpha=0.3)
|
||||||
|
|
||||||
if compact_mode:
|
if compact_mode:
|
||||||
ax1.set_xticks([])
|
ax1.set_xticks([])
|
||||||
|
|
@ -366,20 +367,13 @@ def view_simple_sig(
|
||||||
else:
|
else:
|
||||||
plt.tight_layout()
|
plt.tight_layout()
|
||||||
if show_title:
|
if show_title:
|
||||||
plt.subplots_adjust(top=0.9)
|
plt.subplots_adjust(top=0.92)
|
||||||
|
|
||||||
if saveplot:
|
if saveplot:
|
||||||
output_path, extension = set_path(output_path=output_path)
|
output_path, extension = set_path(output_path=output_path)
|
||||||
dpi_value = _set_dpi(fast_mode=fast_mode, labels_mode=labels_mode, extension=extension)
|
dpi_value = _set_dpi(fast_mode=fast_mode, labels_mode=labels_mode, extension=extension)
|
||||||
|
|
||||||
plt.savefig(
|
plt.savefig(output_path, dpi=dpi_value, bbox_inches="tight", facecolor="#0f172a", edgecolor="none")
|
||||||
output_path,
|
|
||||||
dpi=dpi_value,
|
|
||||||
bbox_inches="tight",
|
|
||||||
pad_inches=0.3,
|
|
||||||
facecolor=matplotlib.rcParams["savefig.facecolor"],
|
|
||||||
edgecolor=matplotlib.rcParams["savefig.edgecolor"],
|
|
||||||
)
|
|
||||||
print(f"Saved signal plot to {output_path}")
|
print(f"Saved signal plot to {output_path}")
|
||||||
return output_path
|
return output_path
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -2,57 +2,15 @@
|
||||||
This module contains the main group for the ria toolkit oss CLI.
|
This module contains the main group for the ria toolkit oss CLI.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import subprocess
|
import click
|
||||||
import sys
|
|
||||||
import warnings
|
|
||||||
|
|
||||||
warnings.filterwarnings(
|
from ria_toolkit_oss_cli.ria_toolkit_oss import commands
|
||||||
"ignore",
|
|
||||||
message="Unable to import Axes3D",
|
|
||||||
category=UserWarning,
|
|
||||||
module="matplotlib",
|
|
||||||
)
|
|
||||||
|
|
||||||
import click # noqa: E402
|
|
||||||
|
|
||||||
from ria_toolkit_oss_cli.ria_toolkit_oss import commands # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def _git_lfs_installed() -> bool:
|
@click.group()
|
||||||
"""Return True if git-lfs is available on PATH."""
|
|
||||||
try:
|
|
||||||
return (
|
|
||||||
subprocess.run(
|
|
||||||
["git", "lfs", "version"],
|
|
||||||
capture_output=True,
|
|
||||||
).returncode
|
|
||||||
== 0
|
|
||||||
)
|
|
||||||
except FileNotFoundError:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
@click.group(invoke_without_command=True)
|
|
||||||
@click.option("-v", "--verbose", is_flag=True, type=bool, help="Increase verbosity, especially useful for debugging.")
|
@click.option("-v", "--verbose", is_flag=True, type=bool, help="Increase verbosity, especially useful for debugging.")
|
||||||
@click.pass_context
|
def cli(verbose):
|
||||||
def cli(ctx, verbose):
|
pass
|
||||||
lfs_missing = not _git_lfs_installed()
|
|
||||||
if lfs_missing:
|
|
||||||
click.echo(
|
|
||||||
"Warning: git-lfs is not installed. RIA Hub projects require git-lfs to\n"
|
|
||||||
"track large binary files (models, recordings, datasets).\n"
|
|
||||||
"\n"
|
|
||||||
" Linux: sudo apt-get install git-lfs\n"
|
|
||||||
" macOS: brew install git-lfs\n"
|
|
||||||
" Other platforms: https://git-lfs.com\n"
|
|
||||||
"\n"
|
|
||||||
"After installing, run: git lfs install",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
if ctx.invoked_subcommand is None:
|
|
||||||
if lfs_missing and sys.stdin.isatty():
|
|
||||||
click.pause(info="\nPress Enter to continue...", err=True)
|
|
||||||
click.echo(ctx.get_help())
|
|
||||||
|
|
||||||
|
|
||||||
# Loop through project commands, binding them all to the CLI.
|
# Loop through project commands, binding them all to the CLI.
|
||||||
|
|
|
||||||
|
|
@ -1,97 +0,0 @@
|
||||||
"""Shared authentication and security helpers for RIA Hub API calls."""
|
|
||||||
|
|
||||||
import base64
|
|
||||||
import subprocess
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
DEFAULT_HUB = "https://riahub.ai"
|
|
||||||
|
|
||||||
|
|
||||||
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
|
|
||||||
"""Block redirects on authenticated requests to prevent credential exfiltration.
|
|
||||||
|
|
||||||
urllib re-sends the Authorization header on same-host redirects by default.
|
|
||||||
A malicious server could redirect a POST to a different host to harvest
|
|
||||||
credentials. We refuse all redirects — API clients should not encounter them
|
|
||||||
in normal operation.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
|
||||||
raise urllib.error.URLError(f"Unexpected redirect ({code}) to {newurl} — aborting to protect credentials")
|
|
||||||
|
|
||||||
|
|
||||||
def hub_opener() -> urllib.request.OpenerDirector:
|
|
||||||
"""Return a urllib opener that blocks redirects."""
|
|
||||||
return urllib.request.build_opener(_NoRedirectHandler)
|
|
||||||
|
|
||||||
|
|
||||||
def warn_if_insecure(hub: str) -> None:
|
|
||||||
"""Warn when credentials would be sent over plain HTTP to a non-localhost host."""
|
|
||||||
parsed = urllib.parse.urlparse(hub)
|
|
||||||
if parsed.scheme == "http":
|
|
||||||
host = parsed.hostname or ""
|
|
||||||
if host not in ("localhost", "127.0.0.1", "::1"):
|
|
||||||
click.echo(
|
|
||||||
f"Warning: sending credentials over plain HTTP to {host}. " "Use HTTPS in production.",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def basic_auth(username: str, password: str) -> str:
|
|
||||||
return "Basic " + base64.b64encode(f"{username}:{password}".encode()).decode()
|
|
||||||
|
|
||||||
|
|
||||||
def get_stored_credentials(hub_url: str) -> tuple[str | None, str | None]:
|
|
||||||
"""Ask git credential fill for stored creds. Returns (username, password) or (None, None)."""
|
|
||||||
parsed = urllib.parse.urlparse(hub_url)
|
|
||||||
payload = f"protocol={parsed.scheme}\nhost={parsed.netloc}\n\n"
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
["git", "credential", "fill"],
|
|
||||||
input=payload,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=5,
|
|
||||||
)
|
|
||||||
creds = {}
|
|
||||||
for line in result.stdout.splitlines():
|
|
||||||
# Partition on the FIRST '=' only so passwords containing '=' are preserved.
|
|
||||||
k, sep, v = line.partition("=")
|
|
||||||
if sep:
|
|
||||||
creds[k.strip()] = v # keep value verbatim
|
|
||||||
return creds.get("username"), creds.get("password")
|
|
||||||
except Exception:
|
|
||||||
return None, None
|
|
||||||
|
|
||||||
|
|
||||||
def store_credentials(hub_url: str, username: str, password: str) -> None:
|
|
||||||
"""Cache credentials via git credential approve (uses the system keychain/store)."""
|
|
||||||
parsed = urllib.parse.urlparse(hub_url)
|
|
||||||
payload = (
|
|
||||||
f"protocol={parsed.scheme}\n" f"host={parsed.netloc}\n" f"username={username}\n" f"password={password}\n\n"
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
subprocess.run(
|
|
||||||
["git", "credential", "approve"],
|
|
||||||
input=payload,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=5,
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
pass # non-fatal — next push just prompts again
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_credentials(hub: str) -> tuple[str, str]:
|
|
||||||
"""Return (username, password), prompting interactively if not cached."""
|
|
||||||
username, password = get_stored_credentials(hub)
|
|
||||||
if username and password:
|
|
||||||
return username, password
|
|
||||||
click.echo(f"No stored credentials found for {hub}.")
|
|
||||||
username = click.prompt("RIA Hub username")
|
|
||||||
password = click.prompt("Password / personal access token", hide_input=True)
|
|
||||||
return username, password
|
|
||||||
|
|
@ -51,7 +51,7 @@ def detect_input_format(filepath):
|
||||||
raise click.ClickException(f"Unknown format for '{filepath}'. Supported: .sigmf, .npy, .wav, .blue")
|
raise click.ClickException(f"Unknown format for '{filepath}'. Supported: .sigmf, .npy, .wav, .blue")
|
||||||
|
|
||||||
|
|
||||||
def determine_output_path(input_path, output_path, fmt, overwrite):
|
def determine_output_path(input_path, output_path, fmt, quiet, overwrite):
|
||||||
input_path = Path(input_path)
|
input_path = Path(input_path)
|
||||||
input_is_annotated = input_path.stem.endswith("_annotated")
|
input_is_annotated = input_path.stem.endswith("_annotated")
|
||||||
|
|
||||||
|
|
@ -63,20 +63,24 @@ def determine_output_path(input_path, output_path, fmt, overwrite):
|
||||||
else:
|
else:
|
||||||
target = input_path.with_name(f"{input_path.stem}_annotated{input_path.suffix}")
|
target = input_path.with_name(f"{input_path.stem}_annotated{input_path.suffix}")
|
||||||
|
|
||||||
final_path = normalize_sigmf_path(target) if fmt == "sigmf" else target
|
if fmt == "sigmf":
|
||||||
|
final_path = normalize_sigmf_path(target)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(f"Saving SigMF metadata to: {final_path}")
|
||||||
|
else:
|
||||||
|
final_path = target
|
||||||
|
if not quiet:
|
||||||
|
click.echo(f"Saving to: {final_path}")
|
||||||
|
|
||||||
if final_path.exists() and not overwrite:
|
# Always allow writing to _annotated files; guard against overwriting originals
|
||||||
raise click.ClickException(f"{final_path} already exists. Use --overwrite to replace it.")
|
target_is_annotated = final_path.stem.endswith("_annotated")
|
||||||
|
if final_path.exists() and not target_is_annotated and final_path != input_path:
|
||||||
|
click.echo(f"Error: {final_path} is not an annotated file and cannot be overwritten.", err=True)
|
||||||
|
return None
|
||||||
|
|
||||||
return final_path
|
return final_path
|
||||||
|
|
||||||
|
|
||||||
def check_output_available(input_path, output_path, overwrite):
|
|
||||||
"""Raise ClickException before any work begins if the output file already exists."""
|
|
||||||
fmt = detect_input_format(Path(input_path))
|
|
||||||
determine_output_path(input_path=input_path, output_path=output_path, fmt=fmt, overwrite=overwrite)
|
|
||||||
|
|
||||||
|
|
||||||
def save_recording_auto(recording, output_path, input_path, quiet=False, overwrite=False):
|
def save_recording_auto(recording, output_path, input_path, quiet=False, overwrite=False):
|
||||||
"""Save recording, auto-detecting format from extension.
|
"""Save recording, auto-detecting format from extension.
|
||||||
|
|
||||||
|
|
@ -86,13 +90,10 @@ def save_recording_auto(recording, output_path, input_path, quiet=False, overwri
|
||||||
input_path = Path(input_path)
|
input_path = Path(input_path)
|
||||||
fmt = detect_input_format(input_path)
|
fmt = detect_input_format(input_path)
|
||||||
|
|
||||||
output_path = determine_output_path(input_path=input_path, output_path=output_path, fmt=fmt, overwrite=overwrite)
|
# Determine output path
|
||||||
|
output_path = determine_output_path(
|
||||||
if not quiet:
|
input_path=input_path, output_path=output_path, fmt=fmt, quiet=quiet, overwrite=overwrite
|
||||||
if fmt == "sigmf":
|
)
|
||||||
click.echo(f"Saving SigMF metadata to: {output_path}")
|
|
||||||
else:
|
|
||||||
click.echo(f"Saving to: {output_path}")
|
|
||||||
|
|
||||||
if fmt == "sigmf":
|
if fmt == "sigmf":
|
||||||
# Normalize path for SigMF
|
# Normalize path for SigMF
|
||||||
|
|
@ -256,11 +257,7 @@ def list(input, verbose):
|
||||||
user_comment = ann.comment or ""
|
user_comment = ann.comment or ""
|
||||||
|
|
||||||
# Basic info
|
# Basic info
|
||||||
freq_range = (
|
freq_range = f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
||||||
f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
|
||||||
if ann.freq_lower_edge is not None and ann.freq_upper_edge is not None
|
|
||||||
else "N/A"
|
|
||||||
)
|
|
||||||
click.echo(
|
click.echo(
|
||||||
f" [{i}] Samples {format_sample_count(ann.sample_start)}-"
|
f" [{i}] Samples {format_sample_count(ann.sample_start)}-"
|
||||||
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {ann.label}"
|
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {ann.label}"
|
||||||
|
|
@ -315,8 +312,6 @@ def add(input, start, count, label, freq_lower, freq_upper, comment, annotation_
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
# Validate sample range
|
# Validate sample range
|
||||||
n_samples = len(recording.data[0])
|
n_samples = len(recording.data[0])
|
||||||
if start < 0:
|
if start < 0:
|
||||||
|
|
@ -368,9 +363,12 @@ def add(input, start, count, label, freq_lower, freq_upper, comment, annotation_
|
||||||
if comment:
|
if comment:
|
||||||
click.echo(f" Comment: {comment}")
|
click.echo(f" Comment: {comment}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
try:
|
||||||
if not quiet:
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
click.echo(" ✓ Saved")
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
|
except Exception as e:
|
||||||
|
raise click.ClickException(f"Failed to save: {e}")
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
@ -468,6 +466,8 @@ def clear(input, output, overwrite, force, quiet):
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo(f"\nCleared {count_before} annotation(s)")
|
click.echo(f"\nCleared {count_before} annotation(s)")
|
||||||
|
|
||||||
|
recording._annotations = []
|
||||||
|
|
||||||
try:
|
try:
|
||||||
save_recording_auto(recording, output_path=input, input_path=input, quiet=quiet, overwrite=True)
|
save_recording_auto(recording, output_path=input, input_path=input, quiet=quiet, overwrite=True)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -503,9 +503,6 @@ def clear(input, output, overwrite, force, quiet):
|
||||||
default="standalone",
|
default="standalone",
|
||||||
help="Annotation type",
|
help="Annotation type",
|
||||||
)
|
)
|
||||||
@click.option(
|
|
||||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
|
||||||
)
|
|
||||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||||
|
|
@ -520,7 +517,6 @@ def energy(
|
||||||
nfft,
|
nfft,
|
||||||
obw_power,
|
obw_power,
|
||||||
annotation_type,
|
annotation_type,
|
||||||
sample_rate,
|
|
||||||
output,
|
output,
|
||||||
overwrite,
|
overwrite,
|
||||||
quiet,
|
quiet,
|
||||||
|
|
@ -543,11 +539,8 @@ def energy(
|
||||||
ria annotate energy signal.npy --threshold 1.5 --min-distance 10000
|
ria annotate energy signal.npy --threshold 1.5 --min-distance 10000
|
||||||
ria annotate energy signal.sigmf-data --freq-method obw
|
ria annotate energy signal.sigmf-data --freq-method obw
|
||||||
ria annotate energy signal.sigmf-data --freq-method full-detected
|
ria annotate energy signal.sigmf-data --freq-method full-detected
|
||||||
ria annotate energy signal.npy --sample-rate 1e6
|
|
||||||
|
|
||||||
"""
|
"""
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
recording = load_recording(input)
|
recording = load_recording(input)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -555,15 +548,6 @@ def energy(
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
if sample_rate is not None:
|
|
||||||
recording.sample_rate = sample_rate
|
|
||||||
|
|
||||||
if recording.sample_rate is None:
|
|
||||||
raise click.ClickException(
|
|
||||||
"Recording metadata does not contain a sample rate. "
|
|
||||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
|
||||||
)
|
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo("\nDetecting signals using energy-based method...")
|
click.echo("\nDetecting signals using energy-based method...")
|
||||||
click.echo(" Time detection:")
|
click.echo(" Time detection:")
|
||||||
|
|
@ -591,13 +575,13 @@ def energy(
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo(f" ✓ Added {added} annotation(s)")
|
click.echo(f" ✓ Added {added} annotation(s)")
|
||||||
|
|
||||||
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Energy detection failed: {e}")
|
raise click.ClickException(f"Energy detection failed: {e}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
|
||||||
if not quiet:
|
|
||||||
click.echo(" ✓ Saved")
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# CUSUM detection subcommand
|
# CUSUM detection subcommand
|
||||||
|
|
@ -617,13 +601,10 @@ def energy(
|
||||||
default="standalone",
|
default="standalone",
|
||||||
help="Annotation type",
|
help="Annotation type",
|
||||||
)
|
)
|
||||||
@click.option(
|
|
||||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
|
||||||
)
|
|
||||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||||
def cusum(input, label, min_duration, window_size, tolerance, annotation_type, sample_rate, output, overwrite, quiet):
|
def cusum(input, label, min_duration, window_size, tolerance, annotation_type, output, overwrite, quiet):
|
||||||
"""Auto-detect segments using CUSUM method.
|
"""Auto-detect segments using CUSUM method.
|
||||||
|
|
||||||
Detects signal state changes (on/off, amplitude transitions). Best for
|
Detects signal state changes (on/off, amplitude transitions). Best for
|
||||||
|
|
@ -635,10 +616,7 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
||||||
Examples:
|
Examples:
|
||||||
ria annotate cusum signal.sigmf-data --min-duration 5.0
|
ria annotate cusum signal.sigmf-data --min-duration 5.0
|
||||||
ria annotate cusum data.npy --min-duration 10.0 --label state
|
ria annotate cusum data.npy --min-duration 10.0 --label state
|
||||||
ria annotate cusum data.npy --sample-rate 1e6
|
|
||||||
"""
|
"""
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
recording = load_recording(input)
|
recording = load_recording(input)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -646,15 +624,6 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
if sample_rate is not None:
|
|
||||||
recording.sample_rate = sample_rate
|
|
||||||
|
|
||||||
if recording.sample_rate is None:
|
|
||||||
raise click.ClickException(
|
|
||||||
"Recording metadata does not contain a sample rate. "
|
|
||||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
|
||||||
)
|
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo("\nDetecting segments using CUSUM...")
|
click.echo("\nDetecting segments using CUSUM...")
|
||||||
click.echo(f" Min duration: {min_duration} ms")
|
click.echo(f" Min duration: {min_duration} ms")
|
||||||
|
|
@ -675,13 +644,13 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo(f" ✓ Added {added} annotation(s)")
|
click.echo(f" ✓ Added {added} annotation(s)")
|
||||||
|
|
||||||
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"CUSUM detection failed: {e}")
|
raise click.ClickException(f"CUSUM detection failed: {e}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
|
||||||
if not quiet:
|
|
||||||
click.echo(" ✓ Saved")
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Threshold detection subcommand
|
# Threshold detection subcommand
|
||||||
|
|
@ -706,13 +675,10 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
||||||
help="Annotation type",
|
help="Annotation type",
|
||||||
)
|
)
|
||||||
@click.option("--channel", type=int, default=0, help="Channel index to annotate (default: 0)")
|
@click.option("--channel", type=int, default=0, help="Channel index to annotate (default: 0)")
|
||||||
@click.option(
|
|
||||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
|
||||||
)
|
|
||||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||||
def threshold(input, threshold, label, window_size, annotation_type, channel, sample_rate, output, overwrite, quiet):
|
def threshold(input, threshold, label, window_size, annotation_type, channel, output, overwrite, quiet):
|
||||||
"""Auto-detect signals using threshold method.
|
"""Auto-detect signals using threshold method.
|
||||||
|
|
||||||
Detects samples above a percentage of maximum magnitude. Best for simple
|
Detects samples above a percentage of maximum magnitude. Best for simple
|
||||||
|
|
@ -722,13 +688,10 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
||||||
Examples:
|
Examples:
|
||||||
ria annotate threshold signal.sigmf-data --threshold 0.7 --label wifi
|
ria annotate threshold signal.sigmf-data --threshold 0.7 --label wifi
|
||||||
ria annotate threshold data.npy --threshold 0.5 --window-size 2048
|
ria annotate threshold data.npy --threshold 0.5 --window-size 2048
|
||||||
ria annotate threshold data.npy --threshold 0.4 --sample-rate 1e6
|
|
||||||
"""
|
"""
|
||||||
if not (0.0 <= threshold <= 1.0):
|
if not (0.0 <= threshold <= 1.0):
|
||||||
raise click.ClickException(f"--threshold must be between 0.0 and 1.0, got {threshold}")
|
raise click.ClickException(f"--threshold must be between 0.0 and 1.0, got {threshold}")
|
||||||
|
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
recording = load_recording(input)
|
recording = load_recording(input)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -736,21 +699,11 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
if sample_rate is not None:
|
|
||||||
recording.sample_rate = sample_rate
|
|
||||||
|
|
||||||
if recording.sample_rate is None:
|
|
||||||
raise click.ClickException(
|
|
||||||
"Recording metadata does not contain a sample rate. "
|
|
||||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
|
||||||
)
|
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo("\nDetecting signals using threshold qualifier...")
|
click.echo("\nDetecting signals using threshold qualifier...")
|
||||||
click.echo(f" Threshold: {threshold * 100:.1f}% of max magnitude")
|
click.echo(f" Threshold: {threshold * 100:.1f}% of max magnitude")
|
||||||
click.echo(f" Window size: {'auto (1ms)' if window_size is None else f'{window_size} samples'}")
|
click.echo(f" Window size: {'auto (1ms)' if window_size is None else f'{window_size} samples'}")
|
||||||
click.echo(f" Channel: {channel}")
|
click.echo(f" Channel: {channel}")
|
||||||
click.echo(f" Sample rate: {recording.sample_rate:.0f} Hz")
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
initial_count = len(recording.annotations)
|
initial_count = len(recording.annotations)
|
||||||
|
|
@ -766,13 +719,13 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
click.echo(f" ✓ Added {added} annotation(s)")
|
click.echo(f" ✓ Added {added} annotation(s)")
|
||||||
|
|
||||||
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Threshold detection failed: {e}")
|
raise click.ClickException(f"Threshold detection failed: {e}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
|
||||||
if not quiet:
|
|
||||||
click.echo(" ✓ Saved")
|
|
||||||
|
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Separate subcommand (Phase 2: Parallel signal separation)
|
# Separate subcommand (Phase 2: Parallel signal separation)
|
||||||
|
|
@ -785,30 +738,11 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
||||||
@click.option("--nfft", type=int, default=65536, help="FFT size for spectral analysis")
|
@click.option("--nfft", type=int, default=65536, help="FFT size for spectral analysis")
|
||||||
@click.option("--noise-threshold-db", type=float, help="Noise floor threshold in dB (auto-estimated if not specified)")
|
@click.option("--noise-threshold-db", type=float, help="Noise floor threshold in dB (auto-estimated if not specified)")
|
||||||
@click.option("--min-component-bw", type=float, default=50e3, help="Min component bandwidth in Hz")
|
@click.option("--min-component-bw", type=float, default=50e3, help="Min component bandwidth in Hz")
|
||||||
@click.option(
|
|
||||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
|
||||||
)
|
|
||||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||||
@click.option("--verbose", is_flag=True, help="Verbose output (show detected components)")
|
@click.option("--verbose", is_flag=True, help="Verbose output (show detected components)")
|
||||||
def _log_separate_start(quiet, recording, indices_list, nfft, noise_threshold_db, min_component_bw):
|
def separate(input, indices, nfft, noise_threshold_db, min_component_bw, output, overwrite, quiet, verbose):
|
||||||
if not quiet:
|
|
||||||
click.echo("\nSplitting annotations by frequency components...")
|
|
||||||
click.echo(f" Input annotations: {len(recording.annotations)}")
|
|
||||||
if indices_list:
|
|
||||||
click.echo(f" Splitting indices: {indices_list}")
|
|
||||||
click.echo(f" FFT size: {nfft}")
|
|
||||||
if noise_threshold_db is not None:
|
|
||||||
click.echo(f" Noise threshold: {noise_threshold_db} dB")
|
|
||||||
else:
|
|
||||||
click.echo(" Noise threshold: auto-estimated")
|
|
||||||
click.echo(f" Min component BW: {format_frequency(min_component_bw)}")
|
|
||||||
|
|
||||||
|
|
||||||
def separate(
|
|
||||||
input, indices, nfft, noise_threshold_db, min_component_bw, sample_rate, output, overwrite, quiet, verbose
|
|
||||||
):
|
|
||||||
"""
|
"""
|
||||||
Auto-detect parallel frequency-offset signals and split into sub-bands.
|
Auto-detect parallel frequency-offset signals and split into sub-bands.
|
||||||
|
|
||||||
|
|
@ -834,8 +768,6 @@ def separate(
|
||||||
ria annotate separate signal.npy --min-component-bw 100000
|
ria annotate separate signal.npy --min-component-bw 100000
|
||||||
|
|
||||||
"""
|
"""
|
||||||
check_output_available(input, output, overwrite)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
recording = load_recording(input)
|
recording = load_recording(input)
|
||||||
if not quiet:
|
if not quiet:
|
||||||
|
|
@ -843,15 +775,6 @@ def separate(
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Failed to load recording: {e}")
|
raise click.ClickException(f"Failed to load recording: {e}")
|
||||||
|
|
||||||
if sample_rate is not None:
|
|
||||||
recording.sample_rate = sample_rate
|
|
||||||
|
|
||||||
if recording.sample_rate is None:
|
|
||||||
raise click.ClickException(
|
|
||||||
"Recording metadata does not contain a sample rate. "
|
|
||||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Parse indices if specified
|
# Parse indices if specified
|
||||||
indices_list = get_indices_list(indices=indices, recording=recording)
|
indices_list = get_indices_list(indices=indices, recording=recording)
|
||||||
|
|
||||||
|
|
@ -860,7 +783,17 @@ def separate(
|
||||||
click.echo("No annotations to split")
|
click.echo("No annotations to split")
|
||||||
return
|
return
|
||||||
|
|
||||||
_log_separate_start(quiet, recording, indices_list, nfft, noise_threshold_db, min_component_bw)
|
if not quiet:
|
||||||
|
click.echo("\nSplitting annotations by frequency components...")
|
||||||
|
click.echo(f" Input annotations: {len(recording.annotations)}")
|
||||||
|
if indices_list:
|
||||||
|
click.echo(f" Splitting indices: {indices_list}")
|
||||||
|
click.echo(f" FFT size: {nfft}")
|
||||||
|
if noise_threshold_db is not None:
|
||||||
|
click.echo(f" Noise threshold: {noise_threshold_db} dB")
|
||||||
|
else:
|
||||||
|
click.echo(" Noise threshold: auto-estimated")
|
||||||
|
click.echo(f" Min component BW: {format_frequency(min_component_bw)}")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
initial_count = len(recording.annotations)
|
initial_count = len(recording.annotations)
|
||||||
|
|
@ -882,19 +815,14 @@ def separate(
|
||||||
click.echo("\n Details:")
|
click.echo("\n Details:")
|
||||||
for i in range(initial_count, final_count):
|
for i in range(initial_count, final_count):
|
||||||
ann = recording.annotations[i]
|
ann = recording.annotations[i]
|
||||||
freq_range = (
|
freq_range = f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
||||||
f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
|
||||||
if ann.freq_lower_edge is not None and ann.freq_upper_edge is not None
|
|
||||||
else "N/A"
|
|
||||||
)
|
|
||||||
click.echo(
|
click.echo(
|
||||||
f" [{i}] samples {format_sample_count(ann.sample_start)}-"
|
f" [{i}] samples {format_sample_count(ann.sample_start)}-"
|
||||||
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {freq_range}"
|
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {freq_range}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||||
|
if not quiet:
|
||||||
|
click.echo(" ✓ Saved")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise click.ClickException(f"Spectral separation failed: {e}")
|
raise click.ClickException(f"Spectral separation failed: {e}")
|
||||||
|
|
||||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
|
||||||
if not quiet:
|
|
||||||
click.echo(" ✓ Saved")
|
|
||||||
|
|
|
||||||
|
|
@ -16,11 +16,9 @@ from .generate import generate
|
||||||
# from .generate import generate
|
# from .generate import generate
|
||||||
from .init import init
|
from .init import init
|
||||||
from .serve import serve
|
from .serve import serve
|
||||||
from .setup_repo import setup_repo
|
|
||||||
from .split import split
|
from .split import split
|
||||||
from .transform import transform
|
from .transform import transform
|
||||||
from .transmit import transmit
|
from .transmit import transmit
|
||||||
from .upload import upload
|
|
||||||
from .view import view
|
from .view import view
|
||||||
|
|
||||||
# Aliases
|
# Aliases
|
||||||
|
|
|
||||||
|
|
@ -1,401 +0,0 @@
|
||||||
"""ria setup_repo — create and configure a RIA Hub Project repo."""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
from ._hub_auth import (
|
|
||||||
DEFAULT_HUB,
|
|
||||||
_NoRedirectHandler,
|
|
||||||
basic_auth,
|
|
||||||
resolve_credentials,
|
|
||||||
store_credentials,
|
|
||||||
warn_if_insecure,
|
|
||||||
)
|
|
||||||
|
|
||||||
RIA_LFS_RULES = [
|
|
||||||
("*.pt", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.pth", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.onnx", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.sigmf", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.sigmf-data", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.sigmf-meta", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.npy", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.npz", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.h5", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.hdf5", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.bin", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
("*.pkl", "filter=lfs diff=lfs merge=lfs -text"),
|
|
||||||
]
|
|
||||||
|
|
||||||
# Repo names must be safe directory names and valid git remote path components.
|
|
||||||
_SAFE_NAME_RE = re.compile(r"^[A-Za-z0-9._-]{1,100}$")
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# API helpers
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _api_request(
|
|
||||||
hub: str,
|
|
||||||
path: str,
|
|
||||||
method: str,
|
|
||||||
username: str,
|
|
||||||
password: str,
|
|
||||||
body: dict | None = None,
|
|
||||||
) -> tuple[dict, int]:
|
|
||||||
"""
|
|
||||||
Make an authenticated request to the RIA Hub API.
|
|
||||||
Returns (parsed_response_body, http_status_code).
|
|
||||||
Status 0 means a network/connection error.
|
|
||||||
Credentials are sent as HTTP Basic auth — safe over HTTPS and localhost HTTP.
|
|
||||||
Redirects are blocked to prevent credential exfiltration.
|
|
||||||
"""
|
|
||||||
url = f"{hub.rstrip('/')}/api/v1{path}"
|
|
||||||
data = json.dumps(body).encode() if body is not None else None
|
|
||||||
req = urllib.request.Request(url, data=data, method=method)
|
|
||||||
req.add_header("Content-Type", "application/json")
|
|
||||||
req.add_header("Authorization", basic_auth(username, password))
|
|
||||||
|
|
||||||
opener = urllib.request.build_opener(_NoRedirectHandler)
|
|
||||||
try:
|
|
||||||
with opener.open(req, timeout=15) as resp:
|
|
||||||
return json.loads(resp.read() or b"{}"), resp.status
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
try:
|
|
||||||
resp_body = json.loads(e.read() or b"{}")
|
|
||||||
except Exception:
|
|
||||||
resp_body = {}
|
|
||||||
return resp_body, e.code
|
|
||||||
except urllib.error.URLError as e:
|
|
||||||
return {"message": str(e.reason)}, 0
|
|
||||||
|
|
||||||
|
|
||||||
def _get_authenticated_username(hub: str, username: str, password: str) -> str | None:
|
|
||||||
"""Return the login name of the authenticated user from GET /api/v1/user.
|
|
||||||
|
|
||||||
This is the canonical username for URL construction — it may differ from
|
|
||||||
git config user.name which is a display name, not a login.
|
|
||||||
"""
|
|
||||||
body, status = _api_request(hub, "/user", "GET", username, password)
|
|
||||||
if status == 200:
|
|
||||||
return body.get("login")
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _repo_exists(hub: str, owner: str, name: str, username: str, password: str) -> bool:
|
|
||||||
body, status = _api_request(
|
|
||||||
hub,
|
|
||||||
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(name, safe='')}",
|
|
||||||
"GET",
|
|
||||||
username,
|
|
||||||
password,
|
|
||||||
)
|
|
||||||
return status == 200
|
|
||||||
|
|
||||||
|
|
||||||
def _create_repo_on_hub(hub: str, name: str, username: str, password: str, private: bool) -> bool:
|
|
||||||
"""Create an RIA Hub Project repo via API.
|
|
||||||
|
|
||||||
Returns True if the repo was freshly created (server seeded README.md and
|
|
||||||
.gitattributes via auto_init + is_ria), False if the hub was unreachable
|
|
||||||
(local fallback needed). Exits on fatal errors (auth, quota, name taken).
|
|
||||||
"""
|
|
||||||
body, status = _api_request(
|
|
||||||
hub,
|
|
||||||
"/user/repos",
|
|
||||||
"POST",
|
|
||||||
username,
|
|
||||||
password,
|
|
||||||
{
|
|
||||||
"name": name,
|
|
||||||
"auto_init": True,
|
|
||||||
"is_ria": True,
|
|
||||||
"private": private,
|
|
||||||
"default_branch": "main",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
if status == 201:
|
|
||||||
click.echo(f"Repository '{name}' created on RIA Hub.")
|
|
||||||
return True
|
|
||||||
|
|
||||||
if status == 0:
|
|
||||||
click.echo(
|
|
||||||
f"Warning: could not reach RIA Hub at {hub}: {body.get('message', 'connection failed')}",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
click.echo("Continuing with local setup only — create the repo manually on RIA Hub.", err=True)
|
|
||||||
return False
|
|
||||||
|
|
||||||
msg = body.get("message", "")
|
|
||||||
|
|
||||||
if status == 401:
|
|
||||||
click.echo("Error: authentication failed — check your username/password.", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
if status in (403, 413) or "quota" in msg.lower() or "limit" in msg.lower():
|
|
||||||
click.echo("Error: cannot create repository — storage quota or account limit reached.", err=True)
|
|
||||||
if msg:
|
|
||||||
click.echo(f" Server message: {msg}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
if status == 422 or "already exist" in msg.lower():
|
|
||||||
click.echo(f"Repository '{name}' already exists on RIA Hub.")
|
|
||||||
return False
|
|
||||||
|
|
||||||
click.echo(f"Error creating repository (HTTP {status}): {msg}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Local git helpers
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _tracked_patterns(ga_path: str) -> set:
|
|
||||||
if not os.path.exists(ga_path):
|
|
||||||
return set()
|
|
||||||
patterns = set()
|
|
||||||
with open(ga_path, encoding="utf-8") as f:
|
|
||||||
for line in f:
|
|
||||||
line = line.strip()
|
|
||||||
if not line or line.startswith("#"):
|
|
||||||
continue
|
|
||||||
m = re.match(r"^(\S+)\s+", line)
|
|
||||||
if m:
|
|
||||||
patterns.add(m.group(1))
|
|
||||||
return patterns
|
|
||||||
|
|
||||||
|
|
||||||
def _write_local_ria_files(repo_path: str, repo_name: str) -> None:
|
|
||||||
"""Seed README.md and .gitattributes locally (used when hub is unreachable or --no-remote)."""
|
|
||||||
# README
|
|
||||||
for candidate in ("README.md", "README.rst", "README.txt", "README"):
|
|
||||||
if os.path.exists(os.path.join(repo_path, candidate)):
|
|
||||||
click.echo(f"README: {candidate} already exists, skipping")
|
|
||||||
break
|
|
||||||
else:
|
|
||||||
with open(os.path.join(repo_path, "README.md"), "w", encoding="utf-8") as f:
|
|
||||||
f.write(
|
|
||||||
f"# {repo_name}\n"
|
|
||||||
"\n"
|
|
||||||
"A RIA Hub project.\n"
|
|
||||||
"\n"
|
|
||||||
"## Description\n"
|
|
||||||
"\n"
|
|
||||||
"<!-- Add your project description here -->\n"
|
|
||||||
"\n"
|
|
||||||
"## Contents\n"
|
|
||||||
"\n"
|
|
||||||
"<!-- Describe the signals, models, or datasets in this repository -->\n"
|
|
||||||
)
|
|
||||||
click.echo("README.md: created")
|
|
||||||
|
|
||||||
# .gitattributes
|
|
||||||
ga_path = os.path.join(repo_path, ".gitattributes")
|
|
||||||
existing = _tracked_patterns(ga_path)
|
|
||||||
new_rules = [(p, a) for p, a in RIA_LFS_RULES if p not in existing]
|
|
||||||
|
|
||||||
if new_rules:
|
|
||||||
existing_content = ""
|
|
||||||
if os.path.exists(ga_path):
|
|
||||||
with open(ga_path, encoding="utf-8") as f:
|
|
||||||
existing_content = f.read()
|
|
||||||
|
|
||||||
separator = "" if (not existing_content or existing_content.endswith("\n")) else "\n"
|
|
||||||
addition = separator + "".join(f"{pattern} {attrs}\n" for pattern, attrs in new_rules)
|
|
||||||
|
|
||||||
with open(ga_path, "a", encoding="utf-8") as f:
|
|
||||||
f.write(addition)
|
|
||||||
click.echo(f".gitattributes: {len(new_rules)} rule(s) added")
|
|
||||||
else:
|
|
||||||
click.echo(".gitattributes: all RIA Hub rules are already present")
|
|
||||||
|
|
||||||
|
|
||||||
def _git(repo_path: str, *args: str, check: bool = True) -> subprocess.CompletedProcess:
|
|
||||||
return subprocess.run(
|
|
||||||
["git", "-C", repo_path, *args],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
check=check,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve_path_and_name(name: str | None, local_path: str | None) -> tuple[str, str]:
|
|
||||||
if local_path:
|
|
||||||
repo_path = os.path.abspath(local_path)
|
|
||||||
repo_name = name or os.path.basename(repo_path)
|
|
||||||
elif name:
|
|
||||||
repo_path = os.path.abspath(name)
|
|
||||||
repo_name = name
|
|
||||||
else:
|
|
||||||
repo_path = os.path.abspath(".")
|
|
||||||
repo_name = os.path.basename(repo_path)
|
|
||||||
return repo_path, repo_name
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve_owner(hub: str, username: str | None, password: str | None, owner: str | None) -> str:
|
|
||||||
if not owner and username and password:
|
|
||||||
api_login = _get_authenticated_username(hub, username, password)
|
|
||||||
owner = api_login or username
|
|
||||||
return owner or "unknown"
|
|
||||||
|
|
||||||
|
|
||||||
def _git_init(repo_path: str) -> None:
|
|
||||||
if os.path.isdir(os.path.join(repo_path, ".git")):
|
|
||||||
return
|
|
||||||
result = _git(repo_path, "init", "-b", "main", check=False)
|
|
||||||
if result.returncode != 0:
|
|
||||||
# Older git (< 2.28) doesn't support -b; fall back and rename.
|
|
||||||
_git(repo_path, "init")
|
|
||||||
_git(repo_path, "symbolic-ref", "HEAD", "refs/heads/main")
|
|
||||||
click.echo("git init: done (branch: main)")
|
|
||||||
|
|
||||||
|
|
||||||
def _configure_remote(
|
|
||||||
repo_path: str, hub: str, resolved_owner: str, repo_name: str, username: str | None, no_remote: bool
|
|
||||||
) -> None:
|
|
||||||
if no_remote or not username:
|
|
||||||
click.echo(
|
|
||||||
f"Skipped remote setup. Add it manually:\n"
|
|
||||||
f" git -C {repo_path} remote add origin "
|
|
||||||
f"{hub.rstrip('/')}/{resolved_owner}/{repo_name}.git"
|
|
||||||
)
|
|
||||||
return
|
|
||||||
remote_url = f"{hub.rstrip('/')}/{resolved_owner}/{repo_name}.git"
|
|
||||||
existing = _git(repo_path, "remote", "get-url", "origin", check=False)
|
|
||||||
if existing.returncode == 0:
|
|
||||||
existing_url = existing.stdout.strip()
|
|
||||||
if existing_url == remote_url:
|
|
||||||
click.echo(f"remote origin: {remote_url} (already set)")
|
|
||||||
else:
|
|
||||||
click.echo(
|
|
||||||
f"remote 'origin' already points to {existing_url}.\n"
|
|
||||||
f" To update: git remote set-url origin {remote_url}"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
_git(repo_path, "remote", "add", "origin", remote_url)
|
|
||||||
click.echo(f"remote origin: {remote_url}")
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Command
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@click.command("setup_repo")
|
|
||||||
@click.argument("name", required=False)
|
|
||||||
@click.option(
|
|
||||||
"--path", "local_path", default=None, help="Local directory (default: current dir, or created from NAME)."
|
|
||||||
)
|
|
||||||
@click.option("--hub", default=DEFAULT_HUB, show_default=True, metavar="URL", help="RIA Hub base URL.")
|
|
||||||
@click.option(
|
|
||||||
"--owner",
|
|
||||||
default=None,
|
|
||||||
metavar="USER",
|
|
||||||
help="RIA Hub login username (default: looked up from the API using your credentials).",
|
|
||||||
)
|
|
||||||
@click.option("--private", is_flag=True, default=False, help="Create the repository as private.")
|
|
||||||
@click.option(
|
|
||||||
"--no-remote", is_flag=True, default=False, help="Skip creating the repository on RIA Hub (local setup only)."
|
|
||||||
)
|
|
||||||
def setup_repo(
|
|
||||||
name: str | None,
|
|
||||||
local_path: str | None,
|
|
||||||
hub: str,
|
|
||||||
owner: str | None,
|
|
||||||
private: bool,
|
|
||||||
no_remote: bool,
|
|
||||||
) -> None:
|
|
||||||
"""Create and configure a RIA Hub Project repo.
|
|
||||||
|
|
||||||
NAME is the repository name. If the local directory does not exist or is
|
|
||||||
not a git repo, it will be initialised automatically. Credentials are
|
|
||||||
retrieved from git's credential store — no token setup required if you
|
|
||||||
have used RIA Hub with git before.
|
|
||||||
|
|
||||||
\b
|
|
||||||
Examples:
|
|
||||||
ria setup_repo my-dataset
|
|
||||||
ria setup_repo my-dataset --hub https://riahub.example.com
|
|
||||||
ria setup_repo --path ./existing-dir
|
|
||||||
ria setup_repo my-dataset --private
|
|
||||||
"""
|
|
||||||
repo_path, repo_name = _resolve_path_and_name(name, local_path)
|
|
||||||
|
|
||||||
if not _SAFE_NAME_RE.match(repo_name):
|
|
||||||
click.echo(
|
|
||||||
f"Error: '{repo_name}' is not a valid repository name.\n"
|
|
||||||
"Use only letters, numbers, hyphens, underscores, and dots (max 100 chars).",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
if not no_remote:
|
|
||||||
warn_if_insecure(hub)
|
|
||||||
|
|
||||||
username, password = (None, None) if no_remote else resolve_credentials(hub)
|
|
||||||
resolved_owner = _resolve_owner(hub, username, password, owner)
|
|
||||||
|
|
||||||
# newly_created=True means the server ran auto_init+is_ria and seeded
|
|
||||||
# README.md + .gitattributes in the initial commit; local setup pulls
|
|
||||||
# those files via fetch rather than writing them from scratch.
|
|
||||||
newly_created = False
|
|
||||||
if not no_remote and username and password:
|
|
||||||
if _repo_exists(hub, resolved_owner, repo_name, username, password):
|
|
||||||
click.echo(f"Repository '{resolved_owner}/{repo_name}' already exists on RIA Hub.")
|
|
||||||
else:
|
|
||||||
newly_created = _create_repo_on_hub(hub, repo_name, username, password, private)
|
|
||||||
store_credentials(hub, username, password)
|
|
||||||
|
|
||||||
if not os.path.exists(repo_path):
|
|
||||||
os.makedirs(repo_path)
|
|
||||||
click.echo(f"Created directory: {repo_path}")
|
|
||||||
|
|
||||||
_git_init(repo_path)
|
|
||||||
|
|
||||||
if subprocess.run(["git", "lfs", "version"], capture_output=True).returncode != 0:
|
|
||||||
click.echo(
|
|
||||||
"Error: git-lfs is not installed.\n"
|
|
||||||
" Linux: sudo apt-get install git-lfs\n"
|
|
||||||
" macOS: brew install git-lfs\n"
|
|
||||||
" Other platforms: https://git-lfs.com",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
_git(repo_path, "lfs", "install", "--local")
|
|
||||||
click.echo("git lfs install --local: done")
|
|
||||||
|
|
||||||
_configure_remote(repo_path, hub, resolved_owner, repo_name, username, no_remote)
|
|
||||||
|
|
||||||
if newly_created:
|
|
||||||
fetch = _git(repo_path, "fetch", "origin", check=False)
|
|
||||||
if fetch.returncode == 0:
|
|
||||||
_git(repo_path, "reset", "--hard", "origin/main")
|
|
||||||
click.echo("Pulled initial commit from RIA Hub (README.md + .gitattributes)")
|
|
||||||
else:
|
|
||||||
click.echo("Warning: fetch failed — falling back to local file setup.", err=True)
|
|
||||||
_write_local_ria_files(repo_path, repo_name)
|
|
||||||
else:
|
|
||||||
_write_local_ria_files(repo_path, repo_name)
|
|
||||||
|
|
||||||
if newly_created:
|
|
||||||
click.echo(f"\nRepo is ready. Push your work:\n cd {repo_path}\n git push -u origin main")
|
|
||||||
else:
|
|
||||||
click.echo(
|
|
||||||
f"\nRepo is ready. Commit and push:\n"
|
|
||||||
f" cd {repo_path}\n"
|
|
||||||
f" git add README.md .gitattributes\n"
|
|
||||||
f" git commit -m 'chore: initialise RIA Hub project'\n"
|
|
||||||
f" git push -u origin main"
|
|
||||||
)
|
|
||||||
|
|
@ -1,392 +0,0 @@
|
||||||
"""ria upload — stream large files to a RIA Hub Project via the LFS API.
|
|
||||||
|
|
||||||
How it works
|
|
||||||
------------
|
|
||||||
1. The file is hashed locally (SHA-256 + size) — this is the LFS object ID.
|
|
||||||
2. A single POST to the repo's LFS batch endpoint returns an upload URL
|
|
||||||
(and headers) for any object the server does not already have.
|
|
||||||
3. The file is streamed to that URL in fixed-size chunks — nothing is ever
|
|
||||||
fully loaded into memory, so files of any size work.
|
|
||||||
4. A commit is created via the Gitea contents API that records the LFS
|
|
||||||
pointer (a small text file) so the file appears in the repo tree.
|
|
||||||
|
|
||||||
No server-side changes are required — this uses the same authenticated LFS
|
|
||||||
protocol that `git lfs push` uses internally.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import base64
|
|
||||||
import hashlib
|
|
||||||
import http.client
|
|
||||||
import json
|
|
||||||
import math
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
from ._hub_auth import (
|
|
||||||
DEFAULT_HUB,
|
|
||||||
basic_auth,
|
|
||||||
hub_opener,
|
|
||||||
resolve_credentials,
|
|
||||||
warn_if_insecure,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Read buffer for hashing and streaming — 8 MB keeps memory use flat
|
|
||||||
# for arbitrarily large files.
|
|
||||||
_CHUNK = 8 * 1024 * 1024
|
|
||||||
|
|
||||||
LFS_MEDIA_TYPE = "application/vnd.git-lfs+json"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# File helpers
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _hash_file(path: str) -> tuple[str, int]:
|
|
||||||
"""Return (sha256_hex, byte_size) by streaming the file."""
|
|
||||||
h = hashlib.sha256()
|
|
||||||
size = 0
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
while True:
|
|
||||||
chunk = f.read(_CHUNK)
|
|
||||||
if not chunk:
|
|
||||||
break
|
|
||||||
h.update(chunk)
|
|
||||||
size += len(chunk)
|
|
||||||
return h.hexdigest(), size
|
|
||||||
|
|
||||||
|
|
||||||
def _lfs_pointer_text(oid: str, size: int) -> str:
|
|
||||||
return f"version https://git-lfs.github.com/spec/v1\noid sha256:{oid}\nsize {size}\n"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# LFS batch API
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _lfs_batch(
|
|
||||||
hub: str,
|
|
||||||
owner: str,
|
|
||||||
repo: str,
|
|
||||||
objects: list[dict],
|
|
||||||
username: str,
|
|
||||||
password: str,
|
|
||||||
) -> dict:
|
|
||||||
"""
|
|
||||||
POST to /{owner}/{repo}.git/info/lfs/objects/batch.
|
|
||||||
Returns the parsed JSON response.
|
|
||||||
Raises on HTTP error or JSON decode failure.
|
|
||||||
"""
|
|
||||||
url = (
|
|
||||||
f"{hub.rstrip('/')}"
|
|
||||||
f"/{urllib.parse.quote(owner, safe='')}"
|
|
||||||
f"/{urllib.parse.quote(repo, safe='')}"
|
|
||||||
f".git/info/lfs/objects/batch"
|
|
||||||
)
|
|
||||||
body = json.dumps(
|
|
||||||
{
|
|
||||||
"operation": "upload",
|
|
||||||
"transfers": ["basic"],
|
|
||||||
"objects": objects,
|
|
||||||
}
|
|
||||||
).encode()
|
|
||||||
|
|
||||||
req = urllib.request.Request(url, data=body, method="POST")
|
|
||||||
req.add_header("Content-Type", LFS_MEDIA_TYPE)
|
|
||||||
req.add_header("Accept", LFS_MEDIA_TYPE)
|
|
||||||
req.add_header("Authorization", basic_auth(username, password))
|
|
||||||
|
|
||||||
opener = hub_opener()
|
|
||||||
try:
|
|
||||||
with opener.open(req, timeout=30) as resp:
|
|
||||||
return json.loads(resp.read())
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
body_text = e.read().decode(errors="replace")
|
|
||||||
raise RuntimeError(f"LFS batch request failed (HTTP {e.code}): {body_text}") from e
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Gitea contents API — create / update a file to record the LFS pointer
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _get_file_sha(
|
|
||||||
hub: str,
|
|
||||||
owner: str,
|
|
||||||
repo: str,
|
|
||||||
path: str,
|
|
||||||
branch: str,
|
|
||||||
username: str,
|
|
||||||
password: str,
|
|
||||||
) -> str | None:
|
|
||||||
"""Return the blob SHA of an existing file, or None if it doesn't exist."""
|
|
||||||
url = (
|
|
||||||
f"{hub.rstrip('/')}/api/v1"
|
|
||||||
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(repo, safe='')}"
|
|
||||||
f"/contents/{urllib.parse.quote(path)}"
|
|
||||||
f"?ref={urllib.parse.quote(branch)}"
|
|
||||||
)
|
|
||||||
req = urllib.request.Request(url)
|
|
||||||
req.add_header("Authorization", basic_auth(username, password))
|
|
||||||
try:
|
|
||||||
with hub_opener().open(req, timeout=15) as resp:
|
|
||||||
return json.loads(resp.read()).get("sha")
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
if e.code == 404:
|
|
||||||
return None
|
|
||||||
raise
|
|
||||||
|
|
||||||
|
|
||||||
def _commit_lfs_pointer(
|
|
||||||
hub: str,
|
|
||||||
owner: str,
|
|
||||||
repo: str,
|
|
||||||
remote_path: str,
|
|
||||||
pointer_text: str,
|
|
||||||
branch: str,
|
|
||||||
message: str,
|
|
||||||
username: str,
|
|
||||||
password: str,
|
|
||||||
) -> None:
|
|
||||||
"""Create or update a file in the repo containing the LFS pointer."""
|
|
||||||
url = (
|
|
||||||
f"{hub.rstrip('/')}/api/v1"
|
|
||||||
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(repo, safe='')}"
|
|
||||||
f"/contents/{urllib.parse.quote(remote_path)}"
|
|
||||||
)
|
|
||||||
|
|
||||||
existing_sha = _get_file_sha(hub, owner, repo, remote_path, branch, username, password)
|
|
||||||
|
|
||||||
body: dict = {
|
|
||||||
"message": message,
|
|
||||||
"content": base64.b64encode(pointer_text.encode()).decode(),
|
|
||||||
"branch": branch,
|
|
||||||
}
|
|
||||||
if existing_sha:
|
|
||||||
body["sha"] = existing_sha
|
|
||||||
|
|
||||||
method = "PUT" if existing_sha else "POST"
|
|
||||||
req = urllib.request.Request(url, data=json.dumps(body).encode(), method=method)
|
|
||||||
req.add_header("Content-Type", "application/json")
|
|
||||||
req.add_header("Authorization", basic_auth(username, password))
|
|
||||||
|
|
||||||
try:
|
|
||||||
with hub_opener().open(req, timeout=30) as resp:
|
|
||||||
resp.read()
|
|
||||||
except urllib.error.HTTPError as e:
|
|
||||||
body_text = e.read().decode(errors="replace")
|
|
||||||
raise RuntimeError(f"Failed to commit LFS pointer for '{remote_path}' (HTTP {e.code}): {body_text}") from e
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Per-file upload logic
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _upload_single_file(
|
|
||||||
hub: str,
|
|
||||||
owner: str,
|
|
||||||
repo_name: str,
|
|
||||||
username: str,
|
|
||||||
password: str,
|
|
||||||
file_path: str,
|
|
||||||
remote_dir: str,
|
|
||||||
message: str | None,
|
|
||||||
branch: str,
|
|
||||||
) -> None:
|
|
||||||
"""Hash, upload (if needed), and commit the LFS pointer for one file."""
|
|
||||||
filename = os.path.basename(file_path)
|
|
||||||
file_size = os.path.getsize(file_path)
|
|
||||||
size_mb = file_size / (1024 * 1024)
|
|
||||||
|
|
||||||
click.echo(f"\n {filename} ({size_mb:.1f} MB)")
|
|
||||||
|
|
||||||
click.echo(" Hashing...", nl=False)
|
|
||||||
oid, size = _hash_file(file_path)
|
|
||||||
click.echo(f" sha256:{oid[:12]}...")
|
|
||||||
|
|
||||||
try:
|
|
||||||
batch = _lfs_batch(hub, owner, repo_name, [{"oid": oid, "size": size}], username, password)
|
|
||||||
except RuntimeError as e:
|
|
||||||
click.echo(f"\n Error: {e}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
objects = batch.get("objects", [])
|
|
||||||
if not objects:
|
|
||||||
click.echo(" Already in LFS — skipping upload.")
|
|
||||||
else:
|
|
||||||
obj = objects[0]
|
|
||||||
if "error" in obj:
|
|
||||||
err_msg = obj["error"].get("message", "unknown error")
|
|
||||||
err_code = obj["error"].get("code", 0)
|
|
||||||
if err_code == 413 or "quota" in err_msg.lower() or "limit" in err_msg.lower():
|
|
||||||
click.echo(
|
|
||||||
f"\n Error: storage quota exceeded for this repo.\n Server: {err_msg}",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
click.echo(f"\n Error from server: {err_msg}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
upload_action = obj.get("actions", {}).get("upload")
|
|
||||||
if not upload_action:
|
|
||||||
click.echo(" Already in LFS — skipping upload.")
|
|
||||||
else:
|
|
||||||
href = upload_action["href"]
|
|
||||||
up_headers = upload_action.get("header", {})
|
|
||||||
chunks = math.ceil(size / _CHUNK)
|
|
||||||
click.echo(f" Uploading ({size_mb:.1f} MB, {chunks} chunk{'s' if chunks != 1 else ''})...")
|
|
||||||
try:
|
|
||||||
_stream_upload_progress(href, up_headers, file_path, size)
|
|
||||||
except RuntimeError as e:
|
|
||||||
click.echo(f"\n Upload failed: {e}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
click.echo(" Upload complete.")
|
|
||||||
|
|
||||||
verify_action = obj.get("actions", {}).get("verify")
|
|
||||||
if verify_action:
|
|
||||||
try:
|
|
||||||
vreq = urllib.request.Request(
|
|
||||||
verify_action["href"],
|
|
||||||
data=json.dumps({"oid": oid, "size": size}).encode(),
|
|
||||||
method="POST",
|
|
||||||
)
|
|
||||||
vreq.add_header("Content-Type", LFS_MEDIA_TYPE)
|
|
||||||
vreq.add_header("Accept", LFS_MEDIA_TYPE)
|
|
||||||
for k, v in verify_action.get("header", {}).items():
|
|
||||||
vreq.add_header(k, v)
|
|
||||||
with urllib.request.urlopen(vreq, timeout=15):
|
|
||||||
pass
|
|
||||||
except Exception:
|
|
||||||
pass # verify is optional; non-fatal on failure
|
|
||||||
|
|
||||||
pointer = _lfs_pointer_text(oid, size)
|
|
||||||
remote_path = (f"{remote_dir.rstrip('/')}/{filename}").lstrip("/") if remote_dir else filename
|
|
||||||
commit_msg = message or f"chore: upload {filename} via ria"
|
|
||||||
|
|
||||||
click.echo(f" Committing pointer → {remote_path}...", nl=False)
|
|
||||||
try:
|
|
||||||
_commit_lfs_pointer(hub, owner, repo_name, remote_path, pointer, branch, commit_msg, username, password)
|
|
||||||
click.echo(" done.")
|
|
||||||
except RuntimeError as e:
|
|
||||||
click.echo(f"\n Error: {e}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
def _stream_upload_progress(href: str, headers: dict, file_path: str, size: int) -> None:
|
|
||||||
"""Stream file_path to href with a click progress bar."""
|
|
||||||
parsed = urllib.parse.urlparse(href)
|
|
||||||
host = parsed.netloc
|
|
||||||
path_q = parsed.path + (f"?{parsed.query}" if parsed.query else "")
|
|
||||||
|
|
||||||
if parsed.scheme == "https":
|
|
||||||
conn = http.client.HTTPSConnection(host, timeout=300)
|
|
||||||
else:
|
|
||||||
conn = http.client.HTTPConnection(host, timeout=300)
|
|
||||||
|
|
||||||
all_headers = dict(headers)
|
|
||||||
all_headers.setdefault("Content-Type", "application/octet-stream")
|
|
||||||
all_headers["Content-Length"] = str(size)
|
|
||||||
|
|
||||||
try:
|
|
||||||
conn.connect()
|
|
||||||
conn.putrequest("PUT", path_q)
|
|
||||||
for k, v in all_headers.items():
|
|
||||||
conn.putheader(k, v)
|
|
||||||
conn.endheaders()
|
|
||||||
|
|
||||||
with click.progressbar(
|
|
||||||
length=size,
|
|
||||||
label=" ",
|
|
||||||
width=40,
|
|
||||||
show_eta=True,
|
|
||||||
show_percent=True,
|
|
||||||
fill_char="█",
|
|
||||||
empty_char="░",
|
|
||||||
) as bar:
|
|
||||||
with open(file_path, "rb") as f:
|
|
||||||
while True:
|
|
||||||
chunk = f.read(_CHUNK)
|
|
||||||
if not chunk:
|
|
||||||
break
|
|
||||||
conn.send(chunk)
|
|
||||||
bar.update(len(chunk))
|
|
||||||
|
|
||||||
resp = conn.getresponse()
|
|
||||||
resp.read()
|
|
||||||
if resp.status not in (200, 201):
|
|
||||||
raise RuntimeError(f"HTTP {resp.status}")
|
|
||||||
finally:
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Command
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@click.command("upload")
|
|
||||||
@click.argument("files", nargs=-1, required=True)
|
|
||||||
@click.option(
|
|
||||||
"--repo", required=True, metavar="OWNER/NAME", help="Target repository on RIA Hub (e.g. benchinnery/my-dataset)."
|
|
||||||
)
|
|
||||||
@click.option("--hub", default=DEFAULT_HUB, show_default=True, metavar="URL", help="RIA Hub base URL.")
|
|
||||||
@click.option("--branch", default="main", show_default=True, help="Branch to commit the files to.")
|
|
||||||
@click.option(
|
|
||||||
"--path",
|
|
||||||
"remote_dir",
|
|
||||||
default="",
|
|
||||||
metavar="DIR",
|
|
||||||
help="Remote directory path inside the repo (default: repo root).",
|
|
||||||
)
|
|
||||||
@click.option("--message", "-m", default=None, help="Commit message (default: 'chore: upload <filename> via ria').")
|
|
||||||
def upload(
|
|
||||||
files: tuple[str],
|
|
||||||
repo: str,
|
|
||||||
hub: str,
|
|
||||||
branch: str,
|
|
||||||
remote_dir: str,
|
|
||||||
message: str | None,
|
|
||||||
) -> None:
|
|
||||||
"""Upload large files to a RIA Hub Project via Git LFS.
|
|
||||||
|
|
||||||
Files are streamed directly to the repo's LFS object store — nothing is
|
|
||||||
buffered into memory, so files of any size work. Each file creates one
|
|
||||||
commit recording the LFS pointer.
|
|
||||||
|
|
||||||
\b
|
|
||||||
Examples:
|
|
||||||
ria upload recording.sigmf-data --repo benchinnery/my-recordings
|
|
||||||
ria upload *.npy --repo benchinnery/my-recordings --branch main
|
|
||||||
ria upload big.pt --repo benchinnery/models --path weights/
|
|
||||||
"""
|
|
||||||
# Validate repo argument
|
|
||||||
if "/" not in repo:
|
|
||||||
click.echo("Error: --repo must be in the form OWNER/NAME.", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
owner, repo_name = repo.split("/", 1)
|
|
||||||
|
|
||||||
# Expand and validate files
|
|
||||||
resolved = []
|
|
||||||
for pattern in files:
|
|
||||||
if not os.path.isfile(pattern):
|
|
||||||
click.echo(f"Error: '{pattern}' is not a file or does not exist.", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
resolved.append(os.path.abspath(pattern))
|
|
||||||
|
|
||||||
hub = hub.rstrip("/")
|
|
||||||
warn_if_insecure(hub)
|
|
||||||
username, password = resolve_credentials(hub)
|
|
||||||
|
|
||||||
click.echo(f"Uploading {len(resolved)} file(s) to {owner}/{repo_name} on {hub}...")
|
|
||||||
|
|
||||||
for file_path in resolved:
|
|
||||||
_upload_single_file(hub, owner, repo_name, username, password, file_path, remote_dir, message, branch)
|
|
||||||
|
|
||||||
click.echo(f"\nAll done. {len(resolved)} file(s) uploaded to {owner}/{repo_name}.")
|
|
||||||
|
|
@ -1,145 +0,0 @@
|
||||||
"""Structured error reporting for `ria-agent register` (T2)."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
import urllib.error
|
|
||||||
from io import BytesIO
|
|
||||||
from unittest.mock import patch
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from ria_toolkit_oss.agent import cli as agent_cli
|
|
||||||
|
|
||||||
|
|
||||||
def _structured(reason: str) -> bytes:
|
|
||||||
return json.dumps({"detail": {"reason": reason}}).encode()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"reason",
|
|
||||||
["invalid_key", "expired", "revoked", "already_consumed"],
|
|
||||||
)
|
|
||||||
def test_explain_maps_known_reasons(reason):
|
|
||||||
msg = agent_cli._explain_registration_failure(403, _structured(reason))
|
|
||||||
assert msg == agent_cli.REGISTRATION_REASON_MESSAGES[reason]
|
|
||||||
|
|
||||||
|
|
||||||
def test_explain_unknown_reason_falls_through_with_code():
|
|
||||||
msg = agent_cli._explain_registration_failure(403, _structured("brand_new_thing"))
|
|
||||||
assert "brand_new_thing" in msg
|
|
||||||
assert "rejected" in msg.lower()
|
|
||||||
|
|
||||||
|
|
||||||
def test_explain_string_detail():
|
|
||||||
body = json.dumps({"detail": "Forbidden"}).encode()
|
|
||||||
msg = agent_cli._explain_registration_failure(403, body)
|
|
||||||
assert msg == "Registration rejected: Forbidden"
|
|
||||||
|
|
||||||
|
|
||||||
def test_explain_429_with_string_detail():
|
|
||||||
body = json.dumps({"detail": "Too many attempts; try again shortly"}).encode()
|
|
||||||
msg = agent_cli._explain_registration_failure(429, body)
|
|
||||||
assert "rate-limited" in msg
|
|
||||||
assert "Too many attempts" in msg
|
|
||||||
|
|
||||||
|
|
||||||
def test_explain_429_with_no_body():
|
|
||||||
msg = agent_cli._explain_registration_failure(429, b"")
|
|
||||||
assert "rate-limited" in msg
|
|
||||||
|
|
||||||
|
|
||||||
def test_explain_malformed_json():
|
|
||||||
msg = agent_cli._explain_registration_failure(500, b"<html>boom</html>")
|
|
||||||
assert msg.startswith("HTTP 500")
|
|
||||||
assert "boom" in msg
|
|
||||||
|
|
||||||
|
|
||||||
def test_explain_empty_body():
|
|
||||||
msg = agent_cli._explain_registration_failure(502, b"")
|
|
||||||
assert msg == "HTTP 502: no body"
|
|
||||||
|
|
||||||
|
|
||||||
def _http_error(status: int, body: bytes) -> urllib.error.HTTPError:
|
|
||||||
return urllib.error.HTTPError(
|
|
||||||
url="http://hub/screens/agents/register",
|
|
||||||
code=status,
|
|
||||||
msg="",
|
|
||||||
hdrs=None, # type: ignore[arg-type]
|
|
||||||
fp=BytesIO(body),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_user_agent_is_set_and_not_python_default():
|
|
||||||
"""Cloudflare on `riahub.ai` returns 403 code 1010 to `Python-urllib/*`.
|
|
||||||
|
|
||||||
Guarding the UA explicitly is the entire point of the register-flow fix;
|
|
||||||
if this test ever breaks, the production bug is back.
|
|
||||||
"""
|
|
||||||
ua = agent_cli._user_agent()
|
|
||||||
assert ua, "User-Agent must not be empty"
|
|
||||||
assert not ua.lower().startswith(
|
|
||||||
"python-urllib"
|
|
||||||
), f"User-Agent must not be Python's default (got {ua!r}) — Cloudflare blocks it"
|
|
||||||
assert ua.startswith("ria-agent/")
|
|
||||||
|
|
||||||
|
|
||||||
def test_register_request_carries_explicit_user_agent(tmp_path):
|
|
||||||
"""Capture the outbound urllib Request and verify the UA header is set."""
|
|
||||||
cfg_path = tmp_path / "agent.json"
|
|
||||||
captured: dict = {}
|
|
||||||
|
|
||||||
def _fake_urlopen(req, *args, **kwargs):
|
|
||||||
# urllib normalizes header names; get_header takes the title-cased form.
|
|
||||||
captured["ua"] = req.get_header("User-agent")
|
|
||||||
captured["api_key"] = req.get_header("X-api-key")
|
|
||||||
captured["timeout"] = kwargs.get("timeout")
|
|
||||||
raise urllib.error.HTTPError(
|
|
||||||
url=req.full_url,
|
|
||||||
code=403,
|
|
||||||
msg="",
|
|
||||||
hdrs=None, # type: ignore[arg-type]
|
|
||||||
fp=BytesIO(_structured("invalid_key")),
|
|
||||||
)
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch.dict("os.environ", {"RIA_AGENT_CONFIG": str(cfg_path)}, clear=False),
|
|
||||||
patch("urllib.request.urlopen", side_effect=_fake_urlopen),
|
|
||||||
patch.object(
|
|
||||||
sys,
|
|
||||||
"argv",
|
|
||||||
["ria-agent", "register", "--hub", "http://hub:3005", "--api-key", "ria_reg_x"],
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(SystemExit):
|
|
||||||
agent_cli.main()
|
|
||||||
|
|
||||||
assert captured["ua"], "User-Agent header was not sent"
|
|
||||||
assert not captured["ua"].lower().startswith("python-urllib")
|
|
||||||
assert captured["api_key"] == "ria_reg_x"
|
|
||||||
assert captured["timeout"] is not None, "register must pass a timeout to urlopen"
|
|
||||||
|
|
||||||
|
|
||||||
def test_register_surfaces_reason_on_http_error(tmp_path, capsys):
|
|
||||||
cfg_path = tmp_path / "agent.json"
|
|
||||||
err = _http_error(403, _structured("revoked"))
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch.dict("os.environ", {"RIA_AGENT_CONFIG": str(cfg_path)}, clear=False),
|
|
||||||
patch("urllib.request.urlopen", side_effect=err),
|
|
||||||
patch.object(
|
|
||||||
sys,
|
|
||||||
"argv",
|
|
||||||
["ria-agent", "register", "--hub", "http://hub:3005", "--api-key", "ria_reg_x"],
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with pytest.raises(SystemExit) as exc:
|
|
||||||
agent_cli.main()
|
|
||||||
|
|
||||||
assert exc.value.code == 1
|
|
||||||
captured = capsys.readouterr()
|
|
||||||
assert "revoked" in captured.err.lower()
|
|
||||||
assert "Settings → RIA Agents" in captured.err
|
|
||||||
# Config must NOT be written on failure.
|
|
||||||
assert not cfg_path.exists()
|
|
||||||
|
|
@ -199,44 +199,3 @@ def test_annotation_to_sigmf_format_values():
|
||||||
values = list(result.values())
|
values = list(result.values())
|
||||||
assert 50 in values or ann.sample_start in values
|
assert 50 in values or ann.sample_start in values
|
||||||
assert 100 in values or ann.sample_count in values
|
assert 100 in values or ann.sample_count in values
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# None freq-edge regression tests (SigMF optional fields)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_annotation_no_freq_edges():
|
|
||||||
ann = Annotation(sample_start=0, sample_count=10, label="burst")
|
|
||||||
assert ann.freq_lower_edge is None
|
|
||||||
assert ann.freq_upper_edge is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_annotation_is_valid_no_freq_edges():
|
|
||||||
ann = Annotation(sample_start=0, sample_count=10, label="burst")
|
|
||||||
assert ann.is_valid() is True
|
|
||||||
|
|
||||||
ann_zero = Annotation(sample_start=0, sample_count=0, label="burst")
|
|
||||||
assert ann_zero.is_valid() is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_annotation_overlap_none_edges_returns_zero():
|
|
||||||
ann1 = Annotation(sample_start=0, sample_count=10)
|
|
||||||
ann2 = Annotation(sample_start=0, sample_count=10, freq_lower_edge=0, freq_upper_edge=100)
|
|
||||||
assert ann1.overlap(ann2) == 0
|
|
||||||
assert ann2.overlap(ann1) == 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_annotation_area_none_edges_returns_zero():
|
|
||||||
ann = Annotation(sample_start=0, sample_count=10, label="burst")
|
|
||||||
assert ann.area() == 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_annotation_to_sigmf_omits_freq_keys_when_none():
|
|
||||||
from sigmf import SigMFFile
|
|
||||||
|
|
||||||
ann = Annotation(sample_start=0, sample_count=10, label="burst")
|
|
||||||
result = ann.to_sigmf_format()
|
|
||||||
metadata = result["metadata"]
|
|
||||||
assert SigMFFile.FLO_KEY not in metadata
|
|
||||||
assert SigMFFile.FHI_KEY not in metadata
|
|
||||||
|
|
|
||||||
|
|
@ -189,21 +189,3 @@ def test_sigmf_3(tmp_path):
|
||||||
to_sigmf(recording=recording1, path=tmp_path, filename=filename.name)
|
to_sigmf(recording=recording1, path=tmp_path, filename=filename.name)
|
||||||
except IOError as e:
|
except IOError as e:
|
||||||
assert str(e) == "File already exists"
|
assert str(e) == "File already exists"
|
||||||
|
|
||||||
|
|
||||||
def test_sigmf_annotation_without_freq_edges(tmp_path):
|
|
||||||
# Regression: annotations that omit the optional SigMF freq edge fields must
|
|
||||||
# load without error; edges should be None and the annotation still valid.
|
|
||||||
ann = Annotation(sample_start=0, sample_count=5, label="burst")
|
|
||||||
recording1 = Recording(data=complex_data_1, metadata=sample_metadata, annotations=[ann])
|
|
||||||
|
|
||||||
filename = tmp_path / "test"
|
|
||||||
to_sigmf(recording=recording1, path=tmp_path, filename=filename.name, overwrite=True)
|
|
||||||
recording2 = from_sigmf(filename)
|
|
||||||
|
|
||||||
assert len(recording2.annotations) == 1
|
|
||||||
loaded = recording2.annotations[0]
|
|
||||||
assert loaded.freq_lower_edge is None
|
|
||||||
assert loaded.freq_upper_edge is None
|
|
||||||
assert loaded.is_valid() is True
|
|
||||||
assert loaded.label == "burst"
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue
Block a user