Compare commits
No commits in common. "main" and "zfp-oss" have entirely different histories.
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -99,5 +99,4 @@ cython_debug/
|
|||
*.sigmf-meta
|
||||
*.blue
|
||||
*.wav
|
||||
/images/
|
||||
!docs/source/images/**
|
||||
images/
|
||||
|
|
|
|||
19
CHANGELOG.md
19
CHANGELOG.md
|
|
@ -1,24 +1,5 @@
|
|||
# Changelog
|
||||
|
||||
## [0.1.7] - 2026-05-26
|
||||
|
||||
### Added
|
||||
|
||||
- **Human-readable agent names** — `ria-agent register` now generates a default `adjective-colour-animal` name (e.g. `swift-teal-falcon`) via the new `namegen` module when `--name` is omitted, instead of registering with an empty string.
|
||||
- **Structured registration error messages** — `ria-agent register` translates hub responses into actionable English for the known failure reasons (`invalid_key`, `expired`, `revoked`, `already_consumed`) and rate-limit (`HTTP 429`) responses, instead of surfacing raw `HTTP 4xx` text.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`ria-agent register` `--api-key` help** — now describes the personal `ria_reg_*` registration key flow (minted from **Settings → RIA Agents** on the hub, shown once at mint time). The legacy shared `[wac] API_KEY` is still accepted by the hub for back-compat, but the CLI documents the per-user flow as preferred.
|
||||
- **`ria-agent register` success output** — now prints both the hub-assigned agent ID and the chosen name: `Registered agent: <id> (<name>)`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`ria-agent register` blocked by Cloudflare on hubs behind it** — set an explicit `User-Agent` (`ria-agent/<package-version> (+https://riahub.ai/qoherent/ria-toolkit-oss)`) so the request isn't rejected as `Python-urllib/<ver>` (Cloudflare Browser Integrity Check returns HTTP 403, edge error code 1010). Version is read from package metadata so it tracks releases automatically.
|
||||
- **`ria-agent register` could hang indefinitely** — added a 15-second timeout to the hub request; previously `urllib`'s default of no timeout meant a stuck hub would block the CLI forever.
|
||||
|
||||
---
|
||||
|
||||
## [0.1.0] - 2026-02-20
|
||||
|
||||
### Added
|
||||
|
|
|
|||
|
|
@ -185,7 +185,7 @@ RIA Toolkit OSS is developed and maintained by [Qoherent](https://qoherent.ai/),
|
|||
If you are doing research with RIA Toolkit OSS, please cite the project:
|
||||
|
||||
```
|
||||
[1] Qoherent Inc., "Radio Intelligence Apps Toolkit OSS," 2026. [Online]. Available: https://riahub.ai/qoherent/ria-toolkit-oss
|
||||
[1] Qoherent Inc., "Radio Intelligence Apps Toolkit OSS," 2025. [Online]. Available: https://riahub.ai/qoherent/ria-toolkit-oss
|
||||
```
|
||||
|
||||
If you like what we're doing, don't forget to give the project a star! ⭐
|
||||
|
|
|
|||
|
|
@ -6,10 +6,10 @@ This is a practical reference for the ``ria`` CLI from ``ria-toolkit-oss``.
|
|||
|
||||
**Scope of this guide:**
|
||||
|
||||
* **Installation and SDR driver prerequisites** — how to install RIA Toolkit OSS and configure the system drivers your hardware requires
|
||||
* **End-to-end CLI workflow** — a step-by-step walkthrough from hardware discovery through capture, annotation, and processing
|
||||
* **Full command reference** — options, flags, and examples for every ``ria`` command
|
||||
* **Python scripting preview** — using the toolkit API directly without the CLI
|
||||
* Installation and setup
|
||||
* End-to-end CLI workflows
|
||||
* Full command reference for CLI features
|
||||
* Brief scripting section
|
||||
|
||||
**Official resources:**
|
||||
|
||||
|
|
@ -18,15 +18,76 @@ This is a practical reference for the ``ria`` CLI from ``ria-toolkit-oss``.
|
|||
* `PyPI package <https://pypi.org/project/ria-toolkit-oss/>`_
|
||||
* `RIA Hub Conda package <https://riahub.ai/qoherent/-/packages/conda/ria-toolkit-oss>`_
|
||||
|
||||
.. contents:: Contents
|
||||
:local:
|
||||
:depth: 2
|
||||
:backlinks: none
|
||||
|
||||
|
||||
1) Installation and Setup
|
||||
==========================
|
||||
|
||||
Before using the ``ria`` CLI, follow the :doc:`Installation <installation>` guide to
|
||||
install RIA Toolkit OSS and any SDR drivers required for your hardware.
|
||||
1.1 Installation with Conda
|
||||
----------------------------
|
||||
|
||||
RIA Toolkit OSS is available as a Conda package on RIA Hub. This is typically the easiest
|
||||
path when using SDR tooling that depends on native/system libraries.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
conda update --force conda
|
||||
conda config --add channels https://riahub.ai/api/packages/qoherent/conda
|
||||
conda activate base
|
||||
conda install ria-toolkit-oss
|
||||
|
||||
Verify:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
conda list | grep ria-toolkit-oss
|
||||
|
||||
|
||||
1.1 SDR driver prerequisites
|
||||
1.2 Installation with pip
|
||||
--------------------------
|
||||
|
||||
Use pip unless you specifically need to edit toolkit source.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
pip install ria-toolkit-oss
|
||||
|
||||
Verify CLI entrypoint:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ria --help
|
||||
|
||||
``pyproject.toml`` defines two script entry points:
|
||||
|
||||
* ``ria``
|
||||
* ``ria-tools``
|
||||
|
||||
Both point to the same CLI module (``ria_toolkit_oss_cli.cli:cli``).
|
||||
|
||||
|
||||
1.3 Optional install from source
|
||||
----------------------------------
|
||||
|
||||
Use this for local development or testing unreleased changes.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://riahub.ai/qoherent/ria-toolkit-oss.git
|
||||
cd ria-toolkit-oss
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e .
|
||||
|
||||
|
||||
1.4 SDR driver prerequisites
|
||||
-----------------------------
|
||||
|
||||
Toolkit package install does not install all system SDR drivers. Install vendor/runtime
|
||||
|
|
@ -34,22 +95,11 @@ dependencies for the hardware you use.
|
|||
|
||||
Examples (depends on device and OS):
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:header-rows: 1
|
||||
|
||||
* - Device
|
||||
- Driver Package
|
||||
* - USRP
|
||||
- UHD drivers
|
||||
* - Pluto
|
||||
- libiio / IIO utilities
|
||||
* - BladeRF
|
||||
- libbladeRF
|
||||
* - HackRF
|
||||
- libhackrf
|
||||
* - RTL-SDR
|
||||
- librtlsdr
|
||||
* USRP: UHD drivers
|
||||
* Pluto: libiio / IIO utilities
|
||||
* BladeRF: libbladeRF
|
||||
* HackRF: libhackrf
|
||||
* RTL-SDR: librtlsdr
|
||||
|
||||
See repo docs under ``docs/source/sdr_guides/*`` and your OS package instructions.
|
||||
|
||||
|
|
@ -69,34 +119,18 @@ Top-level CLI follows this model:
|
|||
|
||||
**Top-level commands:**
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:header-rows: 1
|
||||
|
||||
* - Command
|
||||
- Purpose
|
||||
* - :ref:`discover <cmd-discover>`
|
||||
- Probe SDR drivers and enumerate attached hardware
|
||||
* - :ref:`init <cmd-init>`
|
||||
- Create and manage user metadata defaults
|
||||
* - :ref:`capture <cmd-capture>`
|
||||
- Record IQ samples from a connected SDR
|
||||
* - :ref:`view <cmd-view>`
|
||||
- Generate visualizations from IQ files
|
||||
* - :ref:`annotate <cmd-annotate>`
|
||||
- Label signal regions manually or with auto-detection (group)
|
||||
* - :ref:`convert <cmd-convert>`
|
||||
- Convert between IQ file formats
|
||||
* - :ref:`split <cmd-split>`
|
||||
- Split, trim, or extract recordings
|
||||
* - :ref:`combine <cmd-combine>`
|
||||
- Merge multiple recordings by concatenation or addition
|
||||
* - :ref:`generate / synth <cmd-generate>`
|
||||
- Generate synthetic IQ signals (group; ``synth`` is an alias)
|
||||
* - :ref:`transform <cmd-transform>`
|
||||
- Apply augmentations or impairments to recordings (group)
|
||||
* - :ref:`transmit <cmd-transmit>`
|
||||
- Transmit IQ through a TX-capable SDR
|
||||
* ``discover``
|
||||
* ``init``
|
||||
* ``capture``
|
||||
* ``view``
|
||||
* ``annotate`` (group)
|
||||
* ``convert``
|
||||
* ``split``
|
||||
* ``combine``
|
||||
* ``generate`` (group)
|
||||
* ``transform`` (group)
|
||||
* ``transmit``
|
||||
* ``synth`` (alias of ``generate`` in command bindings)
|
||||
|
||||
|
||||
3) Quick End-to-End Workflow
|
||||
|
|
@ -124,8 +158,10 @@ provenance fields.
|
|||
.. code-block:: bash
|
||||
|
||||
ria init
|
||||
ria init --author "Jane Doe" --project "rf-campaign-1" --location "Lab-A" # non-interactive
|
||||
ria init --show # show config
|
||||
# or non-interactive
|
||||
ria init --author "Jane Doe" --project "rf-campaign-1" --location "Lab-A"
|
||||
# show config
|
||||
ria init --show
|
||||
|
||||
|
||||
3.3 Capture IQ
|
||||
|
|
@ -191,14 +227,13 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
|||
.. code-block:: bash
|
||||
|
||||
ria transmit -d hackrf -f 2.44G -s 2e6 --input capture.sigmf-data
|
||||
ria transmit -d hackrf --generate lfm --continuous # generated waveform
|
||||
# or generated waveform
|
||||
ria transmit -d hackrf --generate lfm --continuous
|
||||
|
||||
|
||||
4) Command Reference
|
||||
=====================
|
||||
|
||||
.. _cmd-discover:
|
||||
|
||||
4.1 ``discover``
|
||||
-----------------
|
||||
|
||||
|
|
@ -228,8 +263,6 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
|||
hidden in default output.
|
||||
|
||||
|
||||
.. _cmd-init:
|
||||
|
||||
4.2 ``init``
|
||||
-------------
|
||||
|
||||
|
|
@ -276,8 +309,6 @@ Replay recorded or synthesized IQ through a transmit-capable SDR.
|
|||
generate metadata, and YAML config loading paths).
|
||||
|
||||
|
||||
.. _cmd-capture:
|
||||
|
||||
4.3 ``capture``
|
||||
----------------
|
||||
|
||||
|
|
@ -351,8 +382,6 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
|||
ria capture -c capture_config.yaml
|
||||
|
||||
|
||||
.. _cmd-view:
|
||||
|
||||
4.4 ``view``
|
||||
-------------
|
||||
|
||||
|
|
@ -413,21 +442,7 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
|||
ria view capture.npy --type full --title "Test Capture" --format pdf
|
||||
ria view capture.npy --show --no-save
|
||||
ria view old.npy --legacy --type simple
|
||||
ria view recordings\qam64_35.npy --type simple
|
||||
ria view recordings\qam64_35.npy --type full
|
||||
|
||||
.. figure:: ../images/recordings/qam64_35.png
|
||||
:alt: Example output of ria view recordings\qam64_35.npy --type simple
|
||||
|
||||
Output of ``ria view recordings\qam64_35.npy --type simple``
|
||||
|
||||
.. figure:: ../images/recordings/qam64_35-full.png
|
||||
:alt: Example output of ria view recordings\qam64_35.npy --type full
|
||||
|
||||
Output of ``ria view recordings\qam64_35.npy --type full``
|
||||
|
||||
|
||||
.. _cmd-annotate:
|
||||
|
||||
4.5 ``annotate`` group
|
||||
-----------------------
|
||||
|
|
@ -444,30 +459,8 @@ Device selection (``--device``) is optional if only one device is detected. Exac
|
|||
|
||||
ria annotate <subcommand> ...
|
||||
|
||||
**Subcommands:**
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:header-rows: 1
|
||||
|
||||
* - Subcommand
|
||||
- Purpose
|
||||
* - ``list``
|
||||
- Inspect all annotations on a recording
|
||||
* - ``add``
|
||||
- Add one annotation with explicit sample-domain bounds
|
||||
* - ``remove``
|
||||
- Remove one annotation by index
|
||||
* - ``clear``
|
||||
- Remove all annotations from a recording
|
||||
* - ``energy``
|
||||
- Auto-detect regions above the estimated noise floor
|
||||
* - ``cusum``
|
||||
- Auto-detect regime changes using change-point detection
|
||||
* - ``threshold``
|
||||
- Auto-detect regions using normalized magnitude thresholding
|
||||
* - ``separate``
|
||||
- Decompose annotations into narrower spectral components
|
||||
**Subcommands:** ``list``, ``add``, ``remove``, ``clear``, ``energy``, ``cusum``,
|
||||
``threshold``, ``separate``
|
||||
|
||||
**General behavior:**
|
||||
|
||||
|
|
@ -594,16 +587,8 @@ annotations.
|
|||
ria annotate add capture.sigmf-data --start 10000 --count 5000 --label burst
|
||||
ria annotate energy capture.sigmf-data --label signal --threshold 1.3
|
||||
ria annotate cusum capture.sigmf-data --min-duration 5
|
||||
ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%
|
||||
ria annotate separate capture.sigmf-data --indices 0,1 --verbose
|
||||
|
||||
.. figure:: ../images/recordings/sample_recording3_annotated.png
|
||||
:alt: Example output of ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%
|
||||
|
||||
Output of ``ria annotate threshold recordings/sample_recording3.npy --threshold 0.7 --label 70%``
|
||||
|
||||
|
||||
.. _cmd-convert:
|
||||
|
||||
4.6 ``convert``
|
||||
----------------
|
||||
|
|
@ -644,8 +629,6 @@ inferred from the output file extension.
|
|||
ria convert old.npy --format sigmf --legacy --overwrite
|
||||
|
||||
|
||||
.. _cmd-split:
|
||||
|
||||
4.7 ``split``
|
||||
--------------
|
||||
|
||||
|
|
@ -687,8 +670,6 @@ Choose exactly one operation per invocation:
|
|||
ria split annotated.sigmf-data --extract-annotations --annotation-label payload
|
||||
|
||||
|
||||
.. _cmd-combine:
|
||||
|
||||
4.8 ``combine``
|
||||
----------------
|
||||
|
||||
|
|
@ -736,8 +717,6 @@ Choose exactly one operation per invocation:
|
|||
ria combine signal.npy pattern.npy out.npy --mode add --align-mode repeat-spaced --repeat-spacing 10000
|
||||
|
||||
|
||||
.. _cmd-generate:
|
||||
|
||||
4.9 ``generate`` group (and ``synth`` alias)
|
||||
---------------------------------------------
|
||||
|
||||
|
|
@ -749,34 +728,15 @@ Choose exactly one operation per invocation:
|
|||
|
||||
``ria synth ...`` is an alias for ``ria generate ...``.
|
||||
|
||||
**Usage:**
|
||||
**Shape:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ria generate <subcommand> [subcommand options] [common options]
|
||||
|
||||
**Available subcommands:**
|
||||
|
||||
.. list-table::
|
||||
:widths: 30 70
|
||||
:header-rows: 1
|
||||
|
||||
* - Subcommand(s)
|
||||
- Description
|
||||
* - ``tone``
|
||||
- Clean sinusoidal calibration/reference source
|
||||
* - ``noise``
|
||||
- Baseline noise floor data or controlled additive-noise synthesis
|
||||
* - ``chirp``
|
||||
- Sweep-based radar/sonar-style signals and bandwidth occupancy tests
|
||||
* - ``square``, ``sawtooth``
|
||||
- Periodic waveform primitives
|
||||
* - ``qam``, ``apsk``, ``pam``, ``psk``
|
||||
- Digital modulation families with pulse-shaping filter support
|
||||
* - ``fsk``
|
||||
- Frequency-shift keying with configurable tone spacing
|
||||
* - ``ook``, ``oqpsk``, ``gmsk``
|
||||
- On-off keying and continuous-phase modulation schemes
|
||||
``tone``, ``noise``, ``chirp``, ``square``, ``sawtooth``, ``qam``, ``apsk``, ``pam``,
|
||||
``fsk``, ``ook``, ``oqpsk``, ``gmsk``, ``psk``
|
||||
|
||||
**Common options shared across all generators:**
|
||||
|
||||
|
|
@ -800,16 +760,22 @@ Multipath and IQ imbalance flags apply impairment-style post-processing during g
|
|||
|
||||
Options: ``--frequency``, ``--amplitude``, ``--phase``
|
||||
|
||||
Clean sinusoidal calibration/reference source.
|
||||
|
||||
``noise``
|
||||
~~~~~~~~~~
|
||||
|
||||
Options: ``--noise-type {gaussian,uniform}``, ``--power``
|
||||
|
||||
Baseline noise floor data or controlled additive-noise synthesis.
|
||||
|
||||
``chirp``
|
||||
~~~~~~~~~~
|
||||
|
||||
Options: ``--bandwidth`` (required), ``--period`` (required), ``--type {up,down,up_down}``
|
||||
|
||||
Sweep-based radar/sonar-style signals and bandwidth occupancy tests.
|
||||
|
||||
``square``
|
||||
~~~~~~~~~~~
|
||||
|
||||
|
|
@ -860,8 +826,6 @@ symbol transition sharpness).
|
|||
ria synth psk -s 2e6 -r 100e3 -M 8 -N 8000 -o psk8.npy
|
||||
|
||||
|
||||
.. _cmd-transform:
|
||||
|
||||
4.10 ``transform`` group
|
||||
-------------------------
|
||||
|
||||
|
|
@ -870,7 +834,7 @@ symbol transition sharpness).
|
|||
* Apply algorithmic transforms to existing recordings.
|
||||
* Run reusable augmentations/impairments for dataset diversity and robustness testing.
|
||||
|
||||
**Usage:**
|
||||
**Shape:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -931,8 +895,6 @@ inspect parameter hints. ``--view`` writes a PNG preview alongside transform out
|
|||
ria transform custom my_filter in.npy out.npy --transform-dir ./my_transforms --params cutoff=0.2
|
||||
|
||||
|
||||
.. _cmd-transmit:
|
||||
|
||||
4.11 ``transmit``
|
||||
------------------
|
||||
|
||||
|
|
@ -1031,7 +993,17 @@ experiment-specific fields on the CLI.
|
|||
ria generate noise --config generate.yaml
|
||||
|
||||
|
||||
6) Version Notes
|
||||
6) Practical Tips and Safety
|
||||
=============================
|
||||
|
||||
* Use ``ria discover`` before capture/transmit sessions.
|
||||
* Keep TX gain conservative first; validate with attenuators/dummy loads when needed.
|
||||
* Prefer SigMF for interoperable metadata and annotations.
|
||||
* For long workflows, keep outputs organized by campaign directories and consistent prefixes.
|
||||
* Use ``--verbose`` when debugging device init or driver issues.
|
||||
|
||||
|
||||
7) Version Notes
|
||||
=================
|
||||
|
||||
These notes are based on the current implementation and should be re-validated against future
|
||||
|
|
@ -1044,19 +1016,18 @@ releases.
|
|||
3. Multiple non-CLI modules still import ``utils.*``, which can create runtime dependency
|
||||
coupling when using only ``ria-toolkit-oss`` in isolation.
|
||||
|
||||
.. tip::
|
||||
If you observe unexpected import errors after install, check the package version and
|
||||
changelog, then test ``ria --help`` in a clean virtual environment.
|
||||
If you observe unexpected import errors after install, check the package version and
|
||||
changelog, then test ``ria --help`` in a clean virtual environment.
|
||||
|
||||
|
||||
7) Brief Scripting (Python) Preview
|
||||
8) Brief Scripting (Python) Preview
|
||||
=====================================
|
||||
|
||||
For quick non-CLI use:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ria_toolkit_oss.data import Recording
|
||||
from ria_toolkit_oss.datatypes import Recording
|
||||
from ria_toolkit_oss.io import load_recording, to_sigmf
|
||||
from ria_toolkit_oss.transforms import iq_augmentations, iq_impairments
|
||||
|
||||
|
|
@ -1066,3 +1037,47 @@ For quick non-CLI use:
|
|||
to_sigmf(imp, filename="capture_awgn", path=".")
|
||||
|
||||
You can also call annotation algorithms and block-generator primitives from Python directly.
|
||||
|
||||
|
||||
9) Cheat Sheet
|
||||
===============
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Install
|
||||
pip install ria-toolkit-oss
|
||||
|
||||
# Discover
|
||||
ria discover -v
|
||||
|
||||
# Init defaults
|
||||
ria init --author "Jane" --project "rf1" --location "Lab-A"
|
||||
|
||||
# Capture
|
||||
ria capture -d pluto -f 2.44G -s 2e6 -n 1000000 -o cap.sigmf-data
|
||||
|
||||
# View
|
||||
ria view cap.sigmf-data --type simple
|
||||
|
||||
# Annotate
|
||||
ria annotate energy cap.sigmf-data --threshold 1.2
|
||||
ria annotate list cap.sigmf-data --verbose
|
||||
|
||||
# Convert
|
||||
ria convert cap.sigmf-data cap.npy
|
||||
|
||||
# Split
|
||||
ria split cap.sigmf-data --split-every 100000 --output-dir chunks
|
||||
|
||||
# Combine
|
||||
ria combine chunks/a.npy chunks/b.npy merged.npy
|
||||
|
||||
# Generate
|
||||
ria generate qam -s 2e6 -r 100e3 -M 16 -N 5000 -o qam16.npy
|
||||
|
||||
# Transform
|
||||
ria transform augment channel_swap cap.npy
|
||||
ria transform impair add_awgn_to_signal cap.npy --params snr=10
|
||||
|
||||
# Transmit
|
||||
ria transmit -d hackrf --input cap.sigmf-data -f 2.44G -s 2e6
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
/* Change the hex values below to customize heading colours */
|
||||
|
||||
.rst-content { color: #e0e0e0; }
|
||||
.rst-content h1 { color: #ffffff; }
|
||||
.rst-content h1 { color: #2c3e50; }
|
||||
.rst-content h2,
|
||||
.rst-content h2 a { color: #ffffff !important; font-size: 22px !important; }
|
||||
|
||||
|
|
@ -23,20 +22,8 @@
|
|||
.rst-content .admonition.warning p {
|
||||
color: #ffffff !important;
|
||||
}
|
||||
.rst-content h4 { color: #cccccc; }
|
||||
.rst-content h4 { color: #404040; }
|
||||
|
||||
.highlight * { color: #ffffff !important; }
|
||||
|
||||
.ria-cmd { color: #2980b9 !important; }
|
||||
|
||||
|
||||
/* Table header text */
|
||||
.rst-content table.docutils th {
|
||||
color: #ffffff !important;
|
||||
}
|
||||
|
||||
/* Remove alternating row background colors from tables */
|
||||
.rst-content table.docutils td,
|
||||
.rst-content table.docutils tr:nth-child(2n-1) td {
|
||||
background-color: transparent !important;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12,9 +12,9 @@ sys.path.insert(0, os.path.abspath(os.path.join('..', '..')))
|
|||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
|
||||
|
||||
project = 'ria-toolkit-oss'
|
||||
copyright = '2026, Qoherent Inc'
|
||||
copyright = '2025, Qoherent Inc'
|
||||
author = 'Qoherent Inc.'
|
||||
release = '0.1.7'
|
||||
release = '0.1.5'
|
||||
|
||||
# -- General configuration ---------------------------------------------------
|
||||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
.. _sdr_examples:
|
||||
.. _examples:
|
||||
|
||||
############
|
||||
SDR Examples
|
||||
|
|
|
|||
BIN
docs/source/images/recordings/qam64_35-full.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/qam64_35-full.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/qam64_35.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/qam64_35.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3-full.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3-full.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_annotated.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_annotated.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_awgn.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_awgn.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_cusum.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_cusum.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording3_threshold.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording3_threshold.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording5_after_separate.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording5_after_separate.png
(Stored with Git LFS)
Binary file not shown.
BIN
docs/source/images/recordings/sample_recording5_before_separate.png
(Stored with Git LFS)
BIN
docs/source/images/recordings/sample_recording5_before_separate.png
(Stored with Git LFS)
Binary file not shown.
File diff suppressed because it is too large
Load Diff
|
|
@ -4,26 +4,7 @@ Installation
|
|||
RIA Hub Toolkit OSS can be installed either as a Conda package or as a standard Python package.
|
||||
|
||||
Please note that SDR drivers must be installed separately. Refer to the relevant guide in the
|
||||
:ref:`SDR Guides <sdr_guides>` section of the documentation for additional setup instructions.
|
||||
|
||||
Common driver packages by device (exact package names depend on your OS):
|
||||
|
||||
.. list-table::
|
||||
:widths: 25 75
|
||||
:header-rows: 1
|
||||
|
||||
* - Device
|
||||
- Driver Package
|
||||
* - USRP
|
||||
- UHD drivers
|
||||
* - Pluto
|
||||
- libiio / IIO utilities
|
||||
* - BladeRF
|
||||
- libbladeRF
|
||||
* - HackRF
|
||||
- libhackrf
|
||||
* - RTL-SDR
|
||||
- librtlsdr
|
||||
:ref:`SDR Guides <sdr_guides>` section of the documentation for addition setup instructions.
|
||||
|
||||
We want your experience with RIA Toolkit OSS to be as smooth and frictionless as possible. If you run into any
|
||||
issues during installation, please reach out to our support team: ``support@qoherent.ai``.
|
||||
|
|
@ -103,22 +84,12 @@ Please follow the steps below to install RIA Toolkit OSS using pip:
|
|||
python -m venv venv
|
||||
venv\Scripts\activate
|
||||
|
||||
2. Upgrade pip and install RIA Toolkit OSS:
|
||||
2. Install RIA Toolkit OSS from PyPI with pip:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install --upgrade pip
|
||||
pip install ria-toolkit-oss
|
||||
|
||||
3. Verify the CLI is available:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ria --help
|
||||
|
||||
A successful install prints the top-level help text. ``pyproject.toml`` registers two
|
||||
entrypoints — ``ria`` and ``ria-tools`` — that both point to the same CLI module.
|
||||
|
||||
RIA Toolkit OSS can also be installed from RIA Hub. However, RIA Hub does not yet support a proxy or cache for public packages.
|
||||
We intend to add this missing functionality soon. In the meantime, please use the ``--no-deps`` option with pip to skip automatic
|
||||
dependency installation, and then manually install each dependency afterward.
|
||||
|
|
@ -148,6 +119,3 @@ Follow the steps below to install RIA Toolkit OSS from source:
|
|||
.. code-block:: bash
|
||||
|
||||
pip install .
|
||||
|
||||
For local development, use ``pip install -e .`` instead to install in editable mode
|
||||
so local changes take effect immediately without reinstalling.
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
Data Package (ria_toolkit_oss.data)
|
||||
=======================================
|
||||
Datatypes Package (ria_toolkit_oss.data)
|
||||
=============================================
|
||||
|
||||
.. |br| raw:: html
|
||||
|
||||
|
|
@ -40,36 +40,26 @@ Limitations
|
|||
- USB 3.0 connectivity is required for optimal performance; using USB 2.0 will significantly limit data
|
||||
transfer rates.
|
||||
|
||||
Set up instructions (Linux)
|
||||
---------------------------
|
||||
Set up instructions (Linux, Radioconda)
|
||||
---------------------------------------
|
||||
|
||||
No additional Python packages are required for BladeRF beyond the base RIA Toolkit OSS installation.
|
||||
|
||||
1. Install the system library:
|
||||
1. Activate your Radioconda environment.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo apt install libbladerf-dev
|
||||
conda activate <your-env-name>
|
||||
|
||||
For a more complete installation including CLI tools and FPGA images, use the Nuand PPA:
|
||||
2. Install the base dependencies and drivers (*Easy method*):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo add-apt-repository ppa:nuandllc/bladerf
|
||||
sudo apt-get update
|
||||
sudo apt-get install bladerf libbladerf-dev
|
||||
sudo apt-get install bladerf-fpga-hostedxa4 # Necessary for BladeRF 2.0 Micro xA4
|
||||
sudo apt-get install bladerf
|
||||
sudo apt-get install libbladerf-dev
|
||||
sudo apt-get install bladerf-fpga-hostedxa4 # Necessary for installation of bladeRF 2.0 Micro A4.
|
||||
|
||||
2. Install udev rules:
|
||||
|
||||
For most users:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
|
||||
For **Radioconda** users, create symlinks from your conda environment instead:
|
||||
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
|
|||
|
|
@ -39,28 +39,23 @@ Limitations
|
|||
- Bandwidth is limited to 20 MHz.
|
||||
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
||||
|
||||
Set up instructions (Linux)
|
||||
---------------------------
|
||||
Set up instructions (Linux, Radioconda)
|
||||
---------------------------------------
|
||||
|
||||
HackRF is supported out of the box after installing RIA Toolkit OSS.
|
||||
|
||||
1. Ensure ``libhackrf`` is installed at the system level. On most Ubuntu installations this is already
|
||||
present. If not:
|
||||
1. Activate your Radioconda environment:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo apt install libhackrf-dev
|
||||
conda activate <your-env-name>
|
||||
|
||||
2. Install udev rules to allow non-root device access:
|
||||
|
||||
For most users:
|
||||
2. Install the System Package (Ubuntu / Debian):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
sudo apt-get update
|
||||
sudo apt-get install hackrf
|
||||
|
||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
||||
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -68,7 +63,7 @@ HackRF is supported out of the box after installing RIA Toolkit OSS.
|
|||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
|
||||
Make sure your user account belongs to the ``plugdev`` group in order to access your device:
|
||||
Make sure your user account belongs to the plugdev group in order to access your device:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -76,7 +71,7 @@ HackRF is supported out of the box after installing RIA Toolkit OSS.
|
|||
|
||||
.. note::
|
||||
|
||||
You may have to restart your system for group membership changes to take effect.
|
||||
You may have to restart your system for changes to take effect.
|
||||
|
||||
Further information
|
||||
-------------------
|
||||
|
|
|
|||
|
|
@ -43,34 +43,34 @@ Limitations
|
|||
affect stability.
|
||||
- USB 2.0 connectivity might limit data transfer rates compared to USB 3.0 or Ethernet-based SDRs.
|
||||
|
||||
Set up instructions (Linux)
|
||||
---------------------------
|
||||
Set up instructions (Linux, Radioconda)
|
||||
---------------------------------------
|
||||
|
||||
The PlutoSDR is supported out of the box after installing RIA Toolkit OSS. The required Python package
|
||||
(``pyadi-iio``) is included in the toolkit's dependencies.
|
||||
|
||||
1. Ensure ``libiio`` is installed at the system level. On most Ubuntu installations this is already present.
|
||||
If not:
|
||||
1. Activate your Radioconda environment:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo apt install libiio-dev libiio-utils libiio0
|
||||
conda activate <your-env-name>
|
||||
|
||||
.. note::
|
||||
|
||||
PlutoSDR devices are discoverable over both USB and network (mDNS). Network discovery uses Avahi — if
|
||||
``avahi-daemon`` is not running, network discovery will be skipped but USB discovery still works.
|
||||
|
||||
2. Install a ``udev`` rule to allow non-root device access:
|
||||
|
||||
For most users:
|
||||
2. Install system dependencies:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y \
|
||||
build-essential \
|
||||
git \
|
||||
libxml2-dev \
|
||||
bison \
|
||||
flex \
|
||||
libcdk5-dev \
|
||||
cmake \
|
||||
libusb-1.0-0-dev \
|
||||
libavahi-client-dev \
|
||||
libavahi-common-dev \
|
||||
libaio-dev
|
||||
|
||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
||||
3. Install a ``udev`` rule by creating a link into your Radioconda installation:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -78,18 +78,11 @@ The PlutoSDR is supported out of the box after installing RIA Toolkit OSS. The r
|
|||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
|
||||
Once you can communicate with the hardware, you may want to perform the post-install steps detailed on
|
||||
the `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_.
|
||||
Once you can talk to the hardware, you may want to perform the post-install steps detailed on the `PlutoSDR Documentation <https://wiki.analog.com/university/tools/pluto>`_.
|
||||
|
||||
3. (Optional) Building ``libiio`` or ``libad9361-iio`` from source:
|
||||
4. (Optional) Building ``libiio`` or ``libad9361-iio`` from source:
|
||||
|
||||
This step is only required if you need a version not available via ``apt``. First install build
|
||||
dependencies:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo apt-get install -y build-essential git libxml2-dev bison flex libcdk5-dev cmake \
|
||||
libusb-1.0-0-dev libavahi-client-dev libavahi-common-dev libaio-dev
|
||||
This step is only required if you want the latest version of these libraries not provided in Radioconda.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
|
|||
|
|
@ -30,10 +30,18 @@ Limitations
|
|||
- Sensitivity and performance can vary depending on the specific model and components.
|
||||
- Requires external software for signal processing and analysis.
|
||||
|
||||
Set up instructions (Linux)
|
||||
---------------------------
|
||||
Set up instructions (Linux, Radioconda)
|
||||
---------------------------------------
|
||||
|
||||
1. If you previously had RTL-SDR drivers installed, purge them first:
|
||||
1. Activate your Radioconda environment:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
conda activate <your-env-name>
|
||||
|
||||
2. Purge drivers:
|
||||
|
||||
If you already have other drivers installed, purge them from your system.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -45,95 +53,47 @@ Set up instructions (Linux)
|
|||
sudo rm -rvf /usr/local/include/rtl_*
|
||||
sudo rm -rvf /usr/local/bin/rtl_*
|
||||
|
||||
2. Install build dependencies:
|
||||
3. Install RTL-SDR Blog drivers:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo apt install libusb-1.0-0-dev git cmake pkg-config build-essential
|
||||
|
||||
3. Build ``librtlsdr`` from source:
|
||||
|
||||
The standard ``librtlsdr`` package available via ``apt`` is missing symbols required by the Python
|
||||
bindings. Build from the **rtl-sdr-blog fork**:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/rtlsdrblog/rtl-sdr-blog.git
|
||||
cd rtl-sdr-blog
|
||||
mkdir build && cd build
|
||||
cmake .. -DINSTALL_UDEV_RULES=ON
|
||||
sudo apt-get install libusb-1.0-0-dev git cmake pkg-config build-essential
|
||||
git clone https://github.com/osmocom/rtl-sdr
|
||||
cd rtl-sdr
|
||||
mkdir build
|
||||
cd build
|
||||
cmake ../ -DINSTALL_UDEV_RULES=ON
|
||||
make
|
||||
sudo make install
|
||||
sudo cp ../rtl-sdr.rules /etc/udev/rules.d/
|
||||
sudo ldconfig
|
||||
|
||||
.. important::
|
||||
|
||||
Do not use the osmocom ``rtl-sdr`` repository or the Ubuntu ``librtlsdr-dev`` apt package. Neither
|
||||
provides the ``rtlsdr_set_dithering`` symbol that the Python bindings require.
|
||||
|
||||
4. Blacklist the kernel DVB driver:
|
||||
|
||||
The kernel DVB-T driver (``dvb_usb_rtl28xxu``) claims the RTL-SDR device and prevents ``librtlsdr``
|
||||
from accessing it.
|
||||
|
||||
For most users:
|
||||
4. Blacklist the DVB-T modules that would otherwise claim the device:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
echo 'blacklist dvb_usb_rtl28xxu' | sudo tee /etc/modprobe.d/blacklist-rtlsdr.conf
|
||||
sudo modprobe -r dvb_usb_rtl28xxu
|
||||
|
||||
For **Radioconda** users, a blacklist configuration is already provided in your conda environment:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo ln -s $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf /etc/modprobe.d/radioconda-rtl-sdr-blacklist.conf
|
||||
sudo modprobe -r $(cat $CONDA_PREFIX/etc/modprobe.d/rtl-sdr-blacklist.conf | sed -n -e 's/^blacklist //p')
|
||||
|
||||
If ``modprobe -r`` fails with "Module is in use", unplug the RTL-SDR dongle, run the command again,
|
||||
then plug it back in. Alternatively, reboot — the blacklist takes effect on next boot.
|
||||
.. note::
|
||||
|
||||
.. note::
|
||||
|
||||
Some systems also require blacklisting additional DVB-T modules. Add these entries to your
|
||||
blacklist configuration if needed:
|
||||
In addition to the Radioconda blacklist file, some systems also require
|
||||
manually blacklisting the following DVB-T modules to prevent them from
|
||||
claiming the device:
|
||||
|
||||
- ``dvb_usb_rtl28xxu``
|
||||
- ``rtl2832``
|
||||
- ``rtl2830``
|
||||
|
||||
5. Reload udev rules:
|
||||
Add these entries to ``rtlsdr.conf`` (or create the file at
|
||||
``/etc/modprobe.d/rtlsdr.conf``) if they are not already present.
|
||||
|
||||
For most users (rules are installed by the build step above):
|
||||
5. Install a udev rule by creating a link into your radioconda installation:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
|
||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo ln -s $CONDA_PREFIX/lib/udev/rules.d/rtl-sdr.rules /etc/udev/rules.d/radioconda-rtl-sdr.rules
|
||||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
|
||||
6. Install Python packages:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install pyrtlsdr==0.3.0
|
||||
pip install setuptools==69.5.1
|
||||
|
||||
.. note::
|
||||
|
||||
``pyrtlsdr`` 0.4.0 references a ``rtlsdr_set_dithering`` symbol not present in standard
|
||||
``librtlsdr`` builds. Version 0.3.0 works correctly.
|
||||
|
||||
``pyrtlsdr`` 0.3.0 depends on ``pkg_resources``, which was removed in ``setuptools`` >= 82.
|
||||
Pinning to 69.5.1 ensures ``pkg_resources`` is available.
|
||||
|
||||
Further Information
|
||||
-------------------
|
||||
- `RTL-SDR Official Website <https://www.rtl-sdr.com/>`_
|
||||
|
|
|
|||
|
|
@ -39,48 +39,18 @@ Limitations
|
|||
Set up instructions (Linux)
|
||||
---------------------------------
|
||||
|
||||
ThinkRF devices require the ``pyrf`` package, which is written in Python 2 syntax and must be patched
|
||||
after installation to work with Python 3.
|
||||
|
||||
.. note::
|
||||
|
||||
``lib2to3`` was fully removed in Python 3.13. ThinkRF support is currently limited to
|
||||
**Python 3.12 and below**.
|
||||
|
||||
1. Install ``lib2to3``:
|
||||
|
||||
On some distributions (including Ubuntu 24.04+), ``lib2to3`` is not included by default:
|
||||
Install PyRF
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo apt install python3-lib2to3
|
||||
pip install 'pyrf>=2.8.0'
|
||||
|
||||
2. Install ``pyrf``:
|
||||
Convert PyRF scripts to Python 3
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install pyrf
|
||||
|
||||
3. Patch ``pyrf`` for Python 3:
|
||||
|
||||
The ``pyrf`` package contains Python 2 syntax throughout (e.g., ``dict.iteritems()``, ``print``
|
||||
statements). Run the following to automatically convert the entire package to Python 3:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -c "
|
||||
from lib2to3.refactor import RefactoringTool, get_fixers_from_package
|
||||
import pyrf, os
|
||||
pyrf_path = os.path.dirname(pyrf.__file__)
|
||||
fixers = get_fixers_from_package('lib2to3.fixes')
|
||||
tool = RefactoringTool(fixers)
|
||||
tool.refactor_dir(pyrf_path, write=True)
|
||||
print('Done')
|
||||
"
|
||||
|
||||
.. note::
|
||||
|
||||
This patches the entire ``pyrf`` package in place, which is required for the driver to fully load.
|
||||
cd ../scripts
|
||||
./convert_pyrf_to_python3.sh
|
||||
|
||||
Further Information
|
||||
-------------------
|
||||
|
|
|
|||
|
|
@ -41,97 +41,34 @@ Limitations
|
|||
- Compatibility with certain software tools may vary depending on the version of the UHD.
|
||||
- Price range can be a consideration, especially for high-end models.
|
||||
|
||||
Set up instructions (Linux)
|
||||
---------------------------
|
||||
Set up instructions (Linux, Radioconda)
|
||||
---------------------------------------
|
||||
|
||||
USRP devices require the UHD (USRP Hardware Driver) library with Python bindings. There is no pip-installable
|
||||
UHD package — it must either be installed via conda or built from source.
|
||||
1. Activate your Radioconda environment:
|
||||
|
||||
**Option A: Install via conda (recommended for conda environments)**
|
||||
.. code-block:: bash
|
||||
|
||||
conda activate <your-env-name>
|
||||
|
||||
2. Install UHD and Python bindings:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
conda install conda-forge::uhd
|
||||
|
||||
**Option B: Build from source (required for pip/venv environments)**
|
||||
|
||||
The Python bindings must target the same Python version used in your virtual environment.
|
||||
|
||||
1. Install build dependencies:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo apt install cmake build-essential libboost-all-dev libusb-1.0-0-dev \
|
||||
python3-dev python3-numpy libncurses-dev
|
||||
|
||||
2. Install the Mako template library into your virtual environment (used by UHD's build system):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install mako
|
||||
|
||||
3. Clone and build UHD with your virtual environment activated:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/EttusResearch/uhd.git
|
||||
cd uhd
|
||||
git checkout v4.7.0.0
|
||||
cd host
|
||||
mkdir build && cd build
|
||||
cmake -DENABLE_PYTHON_API=ON -DPYTHON_EXECUTABLE=$(which python3) ..
|
||||
make -j$(nproc)
|
||||
sudo make install
|
||||
sudo ldconfig
|
||||
|
||||
.. important::
|
||||
|
||||
Run the ``cmake`` command with your virtual environment activated so ``$(which python3)`` points
|
||||
to the correct interpreter. Before running ``make``, verify the cmake output includes::
|
||||
|
||||
-- * LibUHD - Python API → must say "Enabling"
|
||||
-- Python interpreter: .../your-venv/bin/python3
|
||||
|
||||
If "LibUHD - Python API" is not listed under enabled components, the Python bindings will not be
|
||||
built. The build typically takes 10–30 minutes.
|
||||
|
||||
4. Copy the Python bindings into your virtual environment if ``import uhd`` fails after installation:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cp -r ~/uhd/host/build/python/uhd ~/.venv/lib/python3.XX/site-packages/
|
||||
|
||||
Replace ``python3.XX`` with your Python version (e.g., ``python3.12``).
|
||||
|
||||
.. note::
|
||||
|
||||
If you have a pre-existing UHD installation built against a different Python version, you will see
|
||||
a circular import error. The bindings must match the Python version in your virtual environment exactly.
|
||||
|
||||
**After either installation method:**
|
||||
|
||||
1. Download UHD FPGA/firmware images:
|
||||
3. Download UHD images:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uhd_images_downloader
|
||||
|
||||
2. Verify device access:
|
||||
4. Verify access to your device:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uhd_find_devices
|
||||
|
||||
For USB devices (e.g. B-series), install a ``udev`` rule.
|
||||
|
||||
For most users:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
|
||||
For **Radioconda** users, create a symlink from your conda environment instead:
|
||||
For USB devices only (e.g. B series), install a ``udev`` rule by creating a link into your Radioconda installation.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -139,7 +76,7 @@ UHD package — it must either be installed via conda or built from source.
|
|||
sudo udevadm control --reload
|
||||
sudo udevadm trigger
|
||||
|
||||
3. (Optional) Update firmware/FPGA images:
|
||||
5. (Optional) Update firmware/FPGA images:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
|
|||
1407
poetry.lock
generated
1407
poetry.lock
generated
File diff suppressed because it is too large
Load Diff
|
|
@ -1,6 +1,6 @@
|
|||
[project]
|
||||
name = "ria-toolkit-oss"
|
||||
version = "0.1.7"
|
||||
version = "0.1.5"
|
||||
description = "An open-source version of the RIA Toolkit, including the fundamental tools to get started developing, testing, and deploying radio intelligence applications"
|
||||
license = { text = "AGPL-3.0-only" }
|
||||
readme = "README.md"
|
||||
|
|
|
|||
|
|
@ -5,11 +5,8 @@ Subcommands:
|
|||
- ``ria-agent run [legacy args]`` — legacy long-poll NodeAgent (unchanged).
|
||||
- ``ria-agent stream`` — new WebSocket-based IQ streamer.
|
||||
- ``ria-agent detect`` — print SDR drivers whose modules import cleanly.
|
||||
- ``ria-agent register --hub URL --api-key KEY`` — register with the hub
|
||||
using a personal registration key (minted from **Settings → RIA Agents**
|
||||
on the hub, shown once at mint time) and save credentials (and optional
|
||||
TX interlocks) to ``~/.ria/agent.json``. The hub also accepts the legacy
|
||||
shared ``[wac] API_KEY`` for back-compat, but that path is deprecated.
|
||||
- ``ria-agent register --hub URL --api-key KEY`` — register with the hub and
|
||||
save credentials (and optional TX interlocks) to ``~/.ria/agent.json``.
|
||||
|
||||
Invoking ``ria-agent`` with no subcommand falls through to the legacy
|
||||
long-poll behavior for back-compatibility with existing deployments.
|
||||
|
|
@ -28,75 +25,9 @@ from .hardware import available_devices
|
|||
from .legacy_executor import main as _legacy_main
|
||||
from .namegen import generate_agent_name
|
||||
|
||||
DEFAULT_HUB_URL = "https://riahub.ai"
|
||||
|
||||
_LEGACY_ALIASES = {"--hub", "--key", "--name", "--device", "--insecure", "--log-level", "--config"}
|
||||
|
||||
|
||||
def _user_agent() -> str:
|
||||
"""Build the User-Agent header for hub requests.
|
||||
|
||||
Set explicitly so we don't fall back to Python's default `Python-urllib/<ver>`,
|
||||
which is blocked by Cloudflare's Browser Integrity Check on `riahub.ai`
|
||||
(HTTP 403 edge code 1010). Version is read from package metadata so it
|
||||
tracks releases instead of going stale.
|
||||
"""
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
|
||||
try:
|
||||
pkg_version = version("ria-toolkit-oss")
|
||||
except PackageNotFoundError:
|
||||
pkg_version = "unknown"
|
||||
return f"ria-agent/{pkg_version} (+https://riahub.ai/qoherent/ria-toolkit-oss)"
|
||||
|
||||
|
||||
# How long to wait on the hub before giving up. The register endpoint is a
|
||||
# small DB lookup + insert; anything past this is a stuck hub, not a slow one.
|
||||
_REGISTER_TIMEOUT_S = 15
|
||||
|
||||
|
||||
REGISTRATION_REASON_MESSAGES = {
|
||||
"invalid_key": ("Registration key not recognized. Generate a fresh key from " "Settings → RIA Agents on the hub."),
|
||||
"expired": ("This registration key has expired. Generate a new one from " "Settings → RIA Agents on the hub."),
|
||||
"revoked": ("This registration key was revoked. Generate a new one from " "Settings → RIA Agents on the hub."),
|
||||
"already_consumed": (
|
||||
"This single-use registration key has already been used. "
|
||||
"Generate a new one, or mint a reusable key instead."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _explain_registration_failure(status: int, body: bytes) -> str:
|
||||
"""Return a human-readable explanation for a failed register call."""
|
||||
try:
|
||||
parsed = json.loads(body) if body else None
|
||||
except ValueError:
|
||||
parsed = None
|
||||
|
||||
if status == 429:
|
||||
# 429 carries a plain string detail, never a reason code.
|
||||
if isinstance(parsed, dict) and parsed.get("detail"):
|
||||
detail = parsed["detail"]
|
||||
else:
|
||||
detail = body.decode("utf-8", "replace") or "rate limited"
|
||||
return f"Registration rate-limited by the hub: {detail}"
|
||||
|
||||
if not isinstance(parsed, dict):
|
||||
text = body.decode("utf-8", "replace")
|
||||
return f"HTTP {status}: {text or 'no body'}"
|
||||
|
||||
detail = parsed.get("detail")
|
||||
if isinstance(detail, dict):
|
||||
reason = detail.get("reason")
|
||||
if reason in REGISTRATION_REASON_MESSAGES:
|
||||
return REGISTRATION_REASON_MESSAGES[reason]
|
||||
if reason:
|
||||
return f"Registration rejected ({reason})"
|
||||
if isinstance(detail, str) and detail:
|
||||
return f"Registration rejected: {detail}"
|
||||
return f"HTTP {status}: {parsed}"
|
||||
|
||||
|
||||
def _cmd_detect(_args: argparse.Namespace) -> int:
|
||||
devices = available_devices()
|
||||
if not devices:
|
||||
|
|
@ -108,7 +39,6 @@ def _cmd_detect(_args: argparse.Namespace) -> int:
|
|||
|
||||
|
||||
def _cmd_register(args: argparse.Namespace) -> int:
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
hub_url = args.hub.rstrip("/")
|
||||
|
|
@ -121,20 +51,11 @@ def _cmd_register(args: argparse.Namespace) -> int:
|
|||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"X-API-Key": args.api_key,
|
||||
"User-Agent": _user_agent(),
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=_REGISTER_TIMEOUT_S) as resp:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
data = json.loads(resp.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
try:
|
||||
err_body = e.read()
|
||||
except Exception:
|
||||
err_body = b""
|
||||
msg = _explain_registration_failure(e.code, err_body)
|
||||
print(f"error: registration failed: {msg}", file=sys.stderr)
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"error: registration failed: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
|
@ -159,7 +80,7 @@ def _cmd_register(args: argparse.Namespace) -> int:
|
|||
cfg.tx_allowed_freq_ranges = [[float(lo), float(hi)] for lo, hi in freq_ranges]
|
||||
path = _config.save(cfg)
|
||||
|
||||
print(f"Registered agent: {agent_id} ({name})")
|
||||
print(f"Registered agent: {agent_id}")
|
||||
if cfg.tx_enabled:
|
||||
caps: list[str] = []
|
||||
if cfg.tx_max_gain_db is not None:
|
||||
|
|
@ -219,17 +140,8 @@ def main() -> None:
|
|||
sub.add_parser("detect", help="List available SDR drivers")
|
||||
|
||||
p_reg = sub.add_parser("register", help="Register agent with RIA Hub and save credentials")
|
||||
p_reg.add_argument("--hub", default=DEFAULT_HUB_URL, help=f"RIA Hub URL (default: {DEFAULT_HUB_URL})")
|
||||
p_reg.add_argument(
|
||||
"--api-key",
|
||||
dest="api_key",
|
||||
required=True,
|
||||
help=(
|
||||
"Personal registration key from the RIA Agents page on the hub "
|
||||
"(format: ria_reg_...). Shown once when generated; save it then. "
|
||||
"The legacy shared API key is also accepted but deprecated."
|
||||
),
|
||||
)
|
||||
p_reg.add_argument("--hub", required=True, help="RIA Hub URL (e.g. http://whitehorse:3005)")
|
||||
p_reg.add_argument("--api-key", dest="api_key", required=True, help="Hub API key")
|
||||
p_reg.add_argument("--name", default=None, help="Human-friendly agent name")
|
||||
p_reg.add_argument("--insecure", action="store_true", help="Skip TLS verification")
|
||||
p_reg.add_argument(
|
||||
|
|
|
|||
|
|
@ -45,14 +45,7 @@ def is_annotation_contained(inner: Annotation, outer: Annotation) -> bool:
|
|||
outer_sample_stop = outer.sample_start + outer.sample_count
|
||||
|
||||
if inner.sample_start > outer.sample_start and inner_sample_stop < outer_sample_stop:
|
||||
if (
|
||||
inner.freq_lower_edge is not None
|
||||
and inner.freq_upper_edge is not None
|
||||
and outer.freq_lower_edge is not None
|
||||
and outer.freq_upper_edge is not None
|
||||
and inner.freq_lower_edge > outer.freq_lower_edge
|
||||
and inner.freq_upper_edge < outer.freq_upper_edge
|
||||
):
|
||||
if inner.freq_lower_edge > outer.freq_lower_edge and inner.freq_upper_edge < outer.freq_upper_edge:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
|
|
|||
|
|
@ -38,13 +38,7 @@ def annotate_with_cusum(
|
|||
:type annotation_type: str
|
||||
"""
|
||||
|
||||
sample_rate = recording.metadata.get("sample_rate")
|
||||
if sample_rate is None:
|
||||
raise ValueError(
|
||||
"Recording metadata does not contain 'sample_rate'. "
|
||||
"Supply it with --sample-rate when using the CLI, or set "
|
||||
"recording.sample_rate before calling this function."
|
||||
)
|
||||
sample_rate = recording.metadata["sample_rate"]
|
||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||
|
||||
# Create an object of the time segmenter
|
||||
|
|
|
|||
|
|
@ -6,7 +6,6 @@ and occupied bandwidth calculation following ITU-R SM.328 standard.
|
|||
"""
|
||||
|
||||
import json
|
||||
import warnings
|
||||
from typing import Tuple
|
||||
|
||||
import numpy as np
|
||||
|
|
@ -120,17 +119,6 @@ def detect_signals_energy(
|
|||
if active:
|
||||
boundaries.append((start, len(smoothed_power) - start))
|
||||
|
||||
if not boundaries and noise_floor > 0:
|
||||
peak = float(np.max(smoothed_power))
|
||||
dynamic_range = peak / noise_floor
|
||||
if dynamic_range < threshold_factor:
|
||||
warnings.warn(
|
||||
f"detect_signals_energy: no signal boundaries found — dynamic range {dynamic_range:.2f}x is below "
|
||||
f"the threshold factor {threshold_factor}x. The signal may be constant-envelope (e.g. CW or chirp). "
|
||||
"If the entire recording is signal, use 'ria annotate cusum' to segment it as a single region.",
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
# Merge boundaries that are closer than min_distance
|
||||
merged_boundaries = []
|
||||
if boundaries:
|
||||
|
|
@ -147,13 +135,7 @@ def detect_signals_energy(
|
|||
merged_boundaries.append((start, length))
|
||||
|
||||
# Create annotations from detected boundaries
|
||||
sample_rate = recording.metadata.get("sample_rate")
|
||||
if sample_rate is None:
|
||||
raise ValueError(
|
||||
"Recording metadata does not contain 'sample_rate'. "
|
||||
"Supply it with --sample-rate when using the CLI, or set "
|
||||
"recording.sample_rate before calling this function."
|
||||
)
|
||||
sample_rate = recording.metadata["sample_rate"]
|
||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||
|
||||
# Validate frequency method
|
||||
|
|
@ -369,12 +351,7 @@ def annotate_with_obw(
|
|||
>>> annotated = annotate_with_obw(recording, label="signal_obw")
|
||||
"""
|
||||
signal = recording.data[0]
|
||||
sample_rate = recording.metadata.get("sample_rate")
|
||||
if sample_rate is None:
|
||||
raise ValueError(
|
||||
"Recording metadata does not contain 'sample_rate'. "
|
||||
"Set recording.sample_rate before calling this function."
|
||||
)
|
||||
sample_rate = recording.metadata["sample_rate"]
|
||||
center_freq = recording.metadata.get("center_frequency", 0)
|
||||
|
||||
# Calculate OBW
|
||||
|
|
|
|||
|
|
@ -49,7 +49,6 @@ allowing splitting of overlapping signals into separate training samples.
|
|||
"""
|
||||
|
||||
import json
|
||||
import warnings
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
import numpy as np
|
||||
|
|
@ -402,13 +401,7 @@ def split_recording_annotations(
|
|||
return recording
|
||||
|
||||
signal = recording.data[0]
|
||||
sample_rate = recording.metadata.get("sample_rate")
|
||||
if sample_rate is None:
|
||||
raise ValueError(
|
||||
"Recording metadata does not contain 'sample_rate'. "
|
||||
"Supply it with --sample-rate when using the CLI, or set "
|
||||
"recording.sample_rate before calling this function."
|
||||
)
|
||||
sample_rate = recording.metadata["sample_rate"]
|
||||
center_frequency = recording.metadata.get("center_frequency", 0.0)
|
||||
|
||||
# Build new annotation list
|
||||
|
|
@ -432,11 +425,8 @@ def split_recording_annotations(
|
|||
else:
|
||||
# No components found, keep original
|
||||
new_annotations.append(anno)
|
||||
except Exception as e:
|
||||
warnings.warn(
|
||||
f"split_recording_annotations: failed to split annotation at index {i} ({e}); keeping original.",
|
||||
stacklevel=2,
|
||||
)
|
||||
except Exception:
|
||||
# Split failed for any reason, keep original
|
||||
new_annotations.append(anno)
|
||||
else:
|
||||
# Not in split list, keep as-is
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ def qualify_slice_from_annotations(recording: Recording, slice_length: int):
|
|||
|
||||
output_recordings = []
|
||||
|
||||
for i in range(len(recording.data[0]) // slice_length):
|
||||
for i in range((len(recording.data[0]) // slice_length) - 1):
|
||||
start_index = slice_length * i
|
||||
end_index = slice_length * (i + 1)
|
||||
|
||||
|
|
|
|||
|
|
@ -35,24 +35,17 @@ def isolate_signal(recording: Recording, annotation: Annotation) -> Recording:
|
|||
|
||||
isolation_bw = anno_bw
|
||||
|
||||
sample_rate = recording.metadata.get("sample_rate")
|
||||
if sample_rate is None:
|
||||
raise ValueError(
|
||||
"Recording metadata does not contain 'sample_rate'. "
|
||||
"Set recording.sample_rate before calling isolate_signal."
|
||||
)
|
||||
|
||||
# frequency shift the center of the box about zero
|
||||
shifted_signal_slice = frequency_shift_iq_samples(
|
||||
iq_samples=signal_slice,
|
||||
sample_rate=sample_rate,
|
||||
sample_rate=recording.metadata["sample_rate"],
|
||||
shift_frequency=-1 * anno_base_center_freq,
|
||||
)
|
||||
|
||||
# filter
|
||||
if isolation_bw < sample_rate - 1:
|
||||
if isolation_bw < recording.metadata["sample_rate"] - 1:
|
||||
filtered_signal = apply_complex_lowpass_filter(
|
||||
signal=shifted_signal_slice, cutoff_frequency=isolation_bw, sample_rate=sample_rate
|
||||
signal=shifted_signal_slice, cutoff_frequency=isolation_bw, sample_rate=recording.metadata["sample_rate"]
|
||||
)
|
||||
|
||||
else:
|
||||
|
|
|
|||
|
|
@ -42,7 +42,6 @@ classification or demodulation stages.
|
|||
"""
|
||||
|
||||
import json
|
||||
import warnings
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
|
|
@ -217,22 +216,11 @@ def threshold_qualifier(
|
|||
"""
|
||||
# Extract signal and metadata
|
||||
sample_data = recording.data[channel]
|
||||
sample_rate = recording.metadata.get("sample_rate")
|
||||
if sample_rate is None:
|
||||
raise ValueError(
|
||||
"Recording metadata does not contain 'sample_rate'. "
|
||||
"Supply it with --sample-rate when using the CLI, or set "
|
||||
"recording.sample_rate before calling this function."
|
||||
)
|
||||
sample_rate = recording.metadata["sample_rate"]
|
||||
center_frequency = recording.metadata.get("center_frequency", 0)
|
||||
|
||||
n_samples = len(sample_data)
|
||||
|
||||
if window_size is None:
|
||||
window_size = max(64, int(sample_rate * 0.001))
|
||||
# Cap at 1% of signal length so short recordings aren't over-smoothed into
|
||||
# a flat envelope that collapses the dynamic range below the early-exit guard.
|
||||
window_size = min(window_size, max(64, n_samples // 100))
|
||||
|
||||
# --- 1. SIGNAL CONDITIONING ---
|
||||
# Convert to power (Magnitude squared)
|
||||
|
|
@ -249,12 +237,6 @@ def threshold_qualifier(
|
|||
# Soft early exit: keep a guard for low-contrast noise, but compute it from
|
||||
# the quieter tail of the envelope so burst-heavy captures are not rejected.
|
||||
if dynamic_range_ratio < 1.5:
|
||||
warnings.warn(
|
||||
f"threshold_qualifier: dynamic range ratio {dynamic_range_ratio:.2f} is below 1.5 — "
|
||||
"the signal appears to be constant-envelope or pure noise, so no burst boundaries can be found. "
|
||||
"If the entire recording is signal, use 'ria annotate cusum' to segment it as a single region.",
|
||||
stacklevel=2,
|
||||
)
|
||||
return Recording(data=recording.data, metadata=recording.metadata, annotations=recording.annotations)
|
||||
|
||||
trigger_val = noise_floor + threshold * (max_power - noise_floor)
|
||||
|
|
@ -314,7 +296,7 @@ def threshold_qualifier(
|
|||
# burst energy does not bleed through the long window into adjacent regions,
|
||||
# which would inflate macro_residual_max and push the trigger above the
|
||||
# faint burst's average power.
|
||||
macro_window_size = min(max(window_size * 16, int(sample_rate * 0.02)), max(window_size * 2, n_samples // 4))
|
||||
macro_window_size = max(window_size * 16, int(sample_rate * 0.02))
|
||||
macro_kernel = np.ones(macro_window_size, dtype=np.float64) / macro_window_size
|
||||
# Expand each annotated range by half the macro window on both sides so that
|
||||
# the long convolution cannot "see" the leading/trailing edges of already-
|
||||
|
|
|
|||
|
|
@ -17,10 +17,10 @@ class Annotation:
|
|||
:type sample_start: int
|
||||
:param sample_count: The index of the ending sample of the annotation, inclusive.
|
||||
:type sample_count: int
|
||||
:param freq_lower_edge: The lower frequency of the annotation. Optional; None if not specified in source.
|
||||
:type freq_lower_edge: float, optional
|
||||
:param freq_upper_edge: The upper frequency of the annotation. Optional; None if not specified in source.
|
||||
:type freq_upper_edge: float, optional
|
||||
:param freq_lower_edge: The lower frequency of the annotation.
|
||||
:type freq_lower_edge: float
|
||||
:param freq_upper_edge: The upper frequency of the annotation.
|
||||
:type freq_upper_edge: float
|
||||
:param label: The label that will be displayed with the bounding box in compatible viewers including IQEngine.
|
||||
Defaults to an emtpy string.
|
||||
:type label: str, optional
|
||||
|
|
@ -34,8 +34,8 @@ class Annotation:
|
|||
self,
|
||||
sample_start: int,
|
||||
sample_count: int,
|
||||
freq_lower_edge: Optional[float] = None,
|
||||
freq_upper_edge: Optional[float] = None,
|
||||
freq_lower_edge: float,
|
||||
freq_upper_edge: float,
|
||||
label: Optional[str] = "",
|
||||
comment: Optional[str] = "",
|
||||
detail: Optional[dict] = None,
|
||||
|
|
@ -43,8 +43,8 @@ class Annotation:
|
|||
"""Initialize a new Annotation instance."""
|
||||
self.sample_start = int(sample_start)
|
||||
self.sample_count = int(sample_count)
|
||||
self.freq_lower_edge = float(freq_lower_edge) if freq_lower_edge is not None else None
|
||||
self.freq_upper_edge = float(freq_upper_edge) if freq_upper_edge is not None else None
|
||||
self.freq_lower_edge = float(freq_lower_edge)
|
||||
self.freq_upper_edge = float(freq_upper_edge)
|
||||
self.label = str(label)
|
||||
self.comment = str(comment)
|
||||
|
||||
|
|
@ -62,8 +62,6 @@ class Annotation:
|
|||
:returns: True if valid, False if not.
|
||||
"""
|
||||
|
||||
if self.freq_lower_edge is None or self.freq_upper_edge is None:
|
||||
return self.sample_count > 0
|
||||
return self.sample_count > 0 and self.freq_lower_edge < self.freq_upper_edge
|
||||
|
||||
def overlap(self, other):
|
||||
|
|
@ -75,14 +73,6 @@ class Annotation:
|
|||
|
||||
:returns: The area of the overlap in samples*frequency, or 0 if they do not overlap."""
|
||||
|
||||
if (
|
||||
self.freq_lower_edge is None
|
||||
or self.freq_upper_edge is None
|
||||
or other.freq_lower_edge is None
|
||||
or other.freq_upper_edge is None
|
||||
):
|
||||
return 0
|
||||
|
||||
sample_overlap_start = max(self.sample_start, other.sample_start)
|
||||
sample_overlap_end = min(self.sample_start + self.sample_count, other.sample_start + other.sample_count)
|
||||
|
||||
|
|
@ -101,8 +91,6 @@ class Annotation:
|
|||
|
||||
:returns: sample length multiplied by bandwidth."""
|
||||
|
||||
if self.freq_lower_edge is None or self.freq_upper_edge is None:
|
||||
return 0
|
||||
return self.sample_count * (self.freq_upper_edge - self.freq_lower_edge)
|
||||
|
||||
def __eq__(self, other: Annotation) -> bool:
|
||||
|
|
@ -115,16 +103,13 @@ class Annotation:
|
|||
|
||||
annotation_dict = {SigMFFile.START_INDEX_KEY: self.sample_start, SigMFFile.LENGTH_INDEX_KEY: self.sample_count}
|
||||
|
||||
metadata = {
|
||||
annotation_dict["metadata"] = {
|
||||
SigMFFile.LABEL_KEY: self.label,
|
||||
SigMFFile.COMMENT_KEY: self.comment,
|
||||
SigMFFile.FHI_KEY: self.freq_upper_edge,
|
||||
SigMFFile.FLO_KEY: self.freq_lower_edge,
|
||||
"ria:detail": self.detail,
|
||||
}
|
||||
if self.freq_upper_edge is not None:
|
||||
metadata[SigMFFile.FHI_KEY] = self.freq_upper_edge
|
||||
if self.freq_lower_edge is not None:
|
||||
metadata[SigMFFile.FLO_KEY] = self.freq_lower_edge
|
||||
annotation_dict["metadata"] = metadata
|
||||
|
||||
if _is_jsonable(annotation_dict):
|
||||
return annotation_dict
|
||||
|
|
|
|||
|
|
@ -175,15 +175,6 @@ def from_npy(file: os.PathLike | str, legacy: bool = False) -> Recording:
|
|||
)
|
||||
data = first # already loaded without pickle (numeric array)
|
||||
metadata = np.load(f, allow_pickle=True).tolist()
|
||||
# Normalize namespaced keys (e.g. "BlockGenerator:Foo:sample_rate") to
|
||||
# their bare equivalents so downstream code can find them reliably.
|
||||
_STANDARD_KEYS = {"sample_rate", "center_frequency", "bandwidth"}
|
||||
if isinstance(metadata, dict):
|
||||
for k in list(metadata):
|
||||
if ":" in k:
|
||||
bare = k.rsplit(":", 1)[-1]
|
||||
if bare in _STANDARD_KEYS and bare not in metadata:
|
||||
metadata[bare] = metadata[k]
|
||||
try:
|
||||
annotations = list(np.load(f, allow_pickle=True))
|
||||
except EOFError:
|
||||
|
|
|
|||
|
|
@ -32,15 +32,16 @@ def extract_metadata_fields(metadata):
|
|||
|
||||
|
||||
def set_path(output_path):
|
||||
path = pathlib.Path(output_path)
|
||||
split_path = output_path.split("/")
|
||||
|
||||
# If only filename provided (no directory), use default 'images' folder
|
||||
if len(path.parts) == 1:
|
||||
folder = pathlib.Path("images")
|
||||
file = path.name
|
||||
if len(split_path) == 1:
|
||||
folder = "images"
|
||||
file = split_path[0]
|
||||
elif len(split_path) > 2:
|
||||
file = split_path[-1]
|
||||
folder = "/".join(split_path[:-1])
|
||||
else:
|
||||
folder = path.parent
|
||||
file = path.name
|
||||
folder, file = split_path
|
||||
|
||||
split_file = file.split(".")
|
||||
if len(split_file) == 2:
|
||||
|
|
@ -52,5 +53,5 @@ def set_path(output_path):
|
|||
extension = "png"
|
||||
file = file + ".png"
|
||||
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
return str(folder / file), extension
|
||||
pathlib.Path(folder).mkdir(parents=True, exist_ok=True)
|
||||
return "/".join([folder, file]), extension
|
||||
|
|
|
|||
|
|
@ -3,12 +3,11 @@ import os
|
|||
import textwrap
|
||||
from typing import Optional
|
||||
|
||||
import matplotlib
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
from matplotlib import gridspec, ticker
|
||||
from matplotlib import gridspec
|
||||
from matplotlib.patches import Patch
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
from PIL import Image
|
||||
from scipy.fft import fft, fftshift
|
||||
from scipy.signal import spectrogram
|
||||
from scipy.signal.windows import hann
|
||||
|
|
@ -81,8 +80,6 @@ def view_annotations(
|
|||
return 0
|
||||
|
||||
for annotation in sorted(annotations, key=_threshold_sort_key, reverse=True):
|
||||
if annotation.freq_lower_edge is None or annotation.freq_upper_edge is None:
|
||||
continue
|
||||
t_start = annotation.sample_start / sample_rate
|
||||
t_width = annotation.sample_count / sample_rate
|
||||
f_start = annotation.freq_lower_edge
|
||||
|
|
@ -188,7 +185,7 @@ def view_sig(
|
|||
logo: Optional[bool] = True,
|
||||
dark: Optional[bool] = True,
|
||||
spines: Optional[bool] = False,
|
||||
title_fontsize: Optional[int] = 25,
|
||||
title_fontsize: Optional[int] = 35,
|
||||
subtitle_fontsize: Optional[int] = 15,
|
||||
) -> None:
|
||||
"""
|
||||
|
|
@ -233,26 +230,11 @@ def view_sig(
|
|||
complex_signal = recording.data[0]
|
||||
sample_rate, center_frequency, _ = extract_metadata_fields(recording.metadata)
|
||||
|
||||
subplot_height = 3 * (plot_spectrogram) + 2 * (iq + frequency) + 3 * (constellation or metadata or logo)
|
||||
subplot_height = 2 * (plot_spectrogram + iq + frequency) + 3 * (constellation or metadata or logo)
|
||||
subplot_width = max((constellation + metadata or 1), logo * 3)
|
||||
|
||||
if dark:
|
||||
plt.style.use("dark_background")
|
||||
matplotlib.rcParams.update(
|
||||
{
|
||||
"figure.facecolor": "#161616",
|
||||
"axes.facecolor": "#161616",
|
||||
"savefig.facecolor": "#161616",
|
||||
"savefig.edgecolor": "#161616",
|
||||
"font.size": 10,
|
||||
"axes.titlesize": 15,
|
||||
"axes.labelsize": 10,
|
||||
"xtick.labelsize": 10,
|
||||
"ytick.labelsize": 10,
|
||||
"legend.frameon": False,
|
||||
"legend.facecolor": "none",
|
||||
}
|
||||
)
|
||||
logo_path = os.path.dirname(__file__) + "/graphics/Qoherent-logo-white-transparent.png"
|
||||
else:
|
||||
plt.style.use("default")
|
||||
|
|
@ -270,8 +252,8 @@ def view_sig(
|
|||
plot_x_indx = 0
|
||||
|
||||
if plot_spectrogram:
|
||||
spec_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 3, :])
|
||||
plot_y_indx = plot_y_indx + 3
|
||||
spec_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
|
||||
plot_y_indx = plot_y_indx + 2
|
||||
fft_size = get_fft_size(plot_length=plot_length)
|
||||
|
||||
_, t_spec, Sxx = spectrogram(
|
||||
|
|
@ -298,10 +280,7 @@ def view_sig(
|
|||
)
|
||||
|
||||
set_spines(spec_ax, spines)
|
||||
spec_ax.set_title("Spectrogram", loc="left", fontsize=subtitle_fontsize)
|
||||
spec_ax.set_xlabel("Time (s)")
|
||||
spec_ax.set_ylabel("Frequency (MHz)")
|
||||
spec_ax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
||||
spec_ax.set_title("Spectrogram", loc="center", fontsize=subtitle_fontsize)
|
||||
|
||||
if iq:
|
||||
iq_ax = plt.subplot(gs[plot_y_indx : plot_y_indx + 2, :])
|
||||
|
|
@ -312,13 +291,12 @@ def view_sig(
|
|||
|
||||
iq_ax.plot(t, plot_iq.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
|
||||
iq_ax.plot(t, plot_iq.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
|
||||
iq_ax.grid(True, alpha=0.2, linewidth=0.5)
|
||||
iq_ax.grid(False)
|
||||
|
||||
iq_ax.set_ylabel("Amplitude")
|
||||
iq_ax.set_xlim([min(t), max(t)])
|
||||
iq_ax.set_xlabel("Time (s)")
|
||||
iq_ax.set_title("IQ Sample Plot", loc="left", fontsize=subtitle_fontsize)
|
||||
iq_ax.legend(loc="upper right", fontsize=10)
|
||||
iq_ax.set_title("IQ Sample Plot", fontsize=subtitle_fontsize)
|
||||
set_spines(iq_ax, spines)
|
||||
|
||||
if frequency:
|
||||
|
|
@ -332,14 +310,10 @@ def view_sig(
|
|||
# Convert to dB
|
||||
spectrum_db = 20 * np.log10(spectrum + 1e-12) # 20*log for magnitude
|
||||
|
||||
freqs = (
|
||||
np.linspace(-sample_rate / 2, sample_rate / 2, len(complex_signal[:plot_length])) + center_frequency
|
||||
) / 1e6
|
||||
freqs = np.linspace(-sample_rate / 2, sample_rate / 2, len(complex_signal[:plot_length])) + center_frequency
|
||||
freq_ax.plot(freqs, spectrum_db, color=COLORS["accent"], linewidth=0.8)
|
||||
freq_ax.set_xlabel("Frequency (MHz)")
|
||||
freq_ax.set_ylabel("Magnitude (dB)")
|
||||
freq_ax.grid(True, alpha=0.2, linewidth=0.5)
|
||||
freq_ax.set_title("Frequency Spectrum (Windowed FFT)", loc="left", fontsize=subtitle_fontsize)
|
||||
freq_ax.set_title("Frequency Spectrum (Windowed FFT)", fontsize=subtitle_fontsize)
|
||||
set_spines(freq_ax, spines)
|
||||
|
||||
if constellation:
|
||||
|
|
@ -352,7 +326,7 @@ def view_sig(
|
|||
const_ax.set_ylim([-1 * dimension, dimension])
|
||||
const_ax.set_xlabel("In-phase (I)")
|
||||
const_ax.set_ylabel("Quadrature (Q)")
|
||||
const_ax.set_title("Constellation", loc="left", fontsize=subtitle_fontsize)
|
||||
const_ax.set_title("Constellation", fontsize=subtitle_fontsize)
|
||||
const_ax.set_aspect("equal")
|
||||
|
||||
if not spines:
|
||||
|
|
@ -401,8 +375,8 @@ def view_sig(
|
|||
image = Image.open(logo_path) # Open the PNG image using PIL
|
||||
logo_ax.imshow(image)
|
||||
|
||||
except (FileNotFoundError, UnidentifiedImageError, OSError) as exc:
|
||||
print(f"Warning, could not load logo image: {logo_path}. Reason: {exc}")
|
||||
except FileNotFoundError:
|
||||
print(f"Warning, {logo_path} not found.")
|
||||
|
||||
fig.subplots_adjust(
|
||||
left=0.1, # Left margin
|
||||
|
|
|
|||
|
|
@ -119,19 +119,24 @@ def setup_style(*, labels_mode: bool = False, compact_mode: bool = False) -> Non
|
|||
label_font = 14
|
||||
else:
|
||||
base_font = 10
|
||||
title_font = 15
|
||||
title_font = 12
|
||||
label_font = 10
|
||||
|
||||
matplotlib.rcParams.update(
|
||||
{
|
||||
"figure.facecolor": "#161616",
|
||||
"axes.facecolor": "#161616",
|
||||
"savefig.facecolor": "#161616",
|
||||
"savefig.edgecolor": "#161616",
|
||||
"figure.facecolor": "#0f172a",
|
||||
"axes.facecolor": "#1e293b",
|
||||
"axes.edgecolor": COLORS["muted"],
|
||||
"axes.labelcolor": COLORS["light"],
|
||||
"text.color": COLORS["light"],
|
||||
"xtick.color": COLORS["muted"],
|
||||
"ytick.color": COLORS["muted"],
|
||||
"grid.color": COLORS["muted"],
|
||||
"grid.alpha": 0.3,
|
||||
"font.size": base_font,
|
||||
"axes.titlesize": title_font,
|
||||
"axes.labelsize": label_font,
|
||||
"figure.titlesize": title_font + 4,
|
||||
"figure.titlesize": title_font + 2,
|
||||
"legend.frameon": False,
|
||||
"legend.facecolor": "none",
|
||||
"xtick.labelsize": base_font,
|
||||
|
|
@ -189,7 +194,7 @@ def view_simple_sig(
|
|||
constellation_mode: Optional[bool] = False,
|
||||
labels_mode: Optional[bool] = False,
|
||||
slice: Optional[tuple] = None,
|
||||
title: Optional[str] = "Signal Plot",
|
||||
title: Optional[str] = "Signal",
|
||||
):
|
||||
"""
|
||||
Create a simple plot of various signal visualizations as a png or svg image.
|
||||
|
|
@ -232,7 +237,7 @@ def view_simple_sig(
|
|||
spec_signal = signal
|
||||
|
||||
if compact_mode:
|
||||
fig, (ax2, ax1) = plt.subplots(2, 1, figsize=(12, 6), gridspec_kw={"height_ratios": [5, 1]})
|
||||
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 6), gridspec_kw={"height_ratios": [1, 5]})
|
||||
show_title = False
|
||||
show_labels = False
|
||||
ax_constellation = ax_psd = None
|
||||
|
|
@ -248,24 +253,25 @@ def view_simple_sig(
|
|||
ax_psd = None
|
||||
else:
|
||||
if constellation_mode:
|
||||
fig, ((ax2, ax1), (ax3, ax4)) = plt.subplots(2, 2, figsize=(16, 12))
|
||||
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(16, 12))
|
||||
ax_constellation, ax_psd = ax3, ax4
|
||||
else:
|
||||
fig, (ax2, ax1) = plt.subplots(2, 1, figsize=(14, 10))
|
||||
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 10))
|
||||
ax_constellation = ax_psd = None
|
||||
show_title = True
|
||||
show_labels = labels_mode
|
||||
|
||||
if show_title:
|
||||
fig.suptitle(title, fontsize=25)
|
||||
fig.patch.set_facecolor(matplotlib.rcParams["figure.facecolor"])
|
||||
fig.suptitle(title, fontsize=16, color=COLORS["light"], y=0.96)
|
||||
fig.patch.set_facecolor("#0f172a")
|
||||
|
||||
total_duration_s = len(signal) / sample_rate_hz if sample_rate_hz else 0.0
|
||||
t_s = np.linspace(0, total_duration_s, len(display_signal)) if len(display_signal) else np.array([])
|
||||
|
||||
ax1.plot(t_s, display_signal.real, color=COLORS["purple"], linewidth=0.6, alpha=0.8, label="I")
|
||||
ax1.plot(t_s, display_signal.imag, color=COLORS["magenta"], linewidth=0.6, alpha=0.8, label="Q")
|
||||
ax1.grid(True, alpha=0.2, linewidth=0.5)
|
||||
ax1.plot(t_s, display_signal.real, color=COLORS["purple"], linewidth=0.8, alpha=0.8, label="I")
|
||||
ax1.plot(t_s, display_signal.imag, color=COLORS["magenta"], linewidth=0.8, alpha=0.8, label="Q")
|
||||
ax1.set_xlim(0, total_duration_s)
|
||||
ax1.grid(True, alpha=0.3)
|
||||
|
||||
nfft, overlap = _get_nfft_size(signal=signal, fast_mode=fast_mode)
|
||||
|
||||
|
|
@ -279,7 +285,7 @@ def view_simple_sig(
|
|||
)
|
||||
|
||||
ax2.set_ylim(center_freq_hz - sample_rate_hz / 2, center_freq_hz + sample_rate_hz / 2)
|
||||
ax1.set_xlim(ax2.get_xlim())
|
||||
ax2.set_xlim(0, total_duration_s)
|
||||
|
||||
if show_labels:
|
||||
if horizontal_mode:
|
||||
|
|
@ -288,25 +294,20 @@ def view_simple_sig(
|
|||
ax2.set_xlabel("Time (s)")
|
||||
|
||||
ax1.set_ylabel("Amplitude")
|
||||
ax1.set_title(f"IQ Sample Plot - {sdr} SDR", loc="left", pad=10, fontsize=15)
|
||||
ax1.legend(loc="upper right", fontsize=10)
|
||||
ax1.set_title(f"Time Series - {sdr} SDR", loc="left", pad=10)
|
||||
ax1.legend(loc="upper right")
|
||||
|
||||
ax2.set_ylabel("Frequency (MHz)")
|
||||
ax2.set_ylabel("Frequency (Hz)")
|
||||
ax2.set_title(
|
||||
f"Spectrogram - {center_freq_hz / 1e6:.1f} MHz ± {sample_rate_hz / 2e6:.1f} MHz",
|
||||
loc="left",
|
||||
pad=10,
|
||||
fontsize=15,
|
||||
f"Spectrogram - {center_freq_hz / 1e6:.1f} MHz ± {sample_rate_hz / 2e6:.1f} MHz", loc="left", pad=10
|
||||
)
|
||||
ax2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
||||
yticks = ax2.get_yticks()
|
||||
ax2.set_yticklabels([f"{y / 1e6:.1f}" for y in yticks])
|
||||
elif not compact_mode:
|
||||
ax1.set_title("IQ Sample Plot", loc="left", pad=10, fontsize=15)
|
||||
ax1.legend(loc="upper right", fontsize=10)
|
||||
ax1.set_title("Time Series", loc="left", pad=10)
|
||||
ax1.legend(loc="upper right", fontsize=8)
|
||||
|
||||
ax2.set_xlabel("Time (s)")
|
||||
ax2.set_ylabel("Frequency (MHz)")
|
||||
ax2.set_title("Spectrogram", loc="left", pad=10, fontsize=15)
|
||||
ax2.yaxis.set_major_formatter(matplotlib.ticker.FuncFormatter(lambda x, _: f"{x / 1e6:.1f}"))
|
||||
ax2.set_title("Spectrogram", loc="left", pad=10)
|
||||
|
||||
_add_annotations(
|
||||
annotations=annotations,
|
||||
|
|
@ -338,8 +339,8 @@ def view_simple_sig(
|
|||
)
|
||||
ax_constellation.set_xlabel("In-phase (I)")
|
||||
ax_constellation.set_ylabel("Quadrature (Q)")
|
||||
ax_constellation.set_title("Constellation", loc="left", fontsize=15)
|
||||
ax_constellation.grid(True, alpha=0.2, linewidth=0.5)
|
||||
ax_constellation.set_title("Constellation")
|
||||
ax_constellation.grid(True, alpha=0.3)
|
||||
ax_constellation.set_aspect("equal")
|
||||
|
||||
if ax_psd is not None:
|
||||
|
|
@ -350,11 +351,11 @@ def view_simple_sig(
|
|||
freqs = freqs + center_freq_hz
|
||||
spectrum_db = 10 * np.log10(spectrum + 1e-12)
|
||||
|
||||
ax_psd.plot(freqs / 1e6, spectrum_db, color=COLORS["accent"], linewidth=0.8)
|
||||
ax_psd.plot(freqs / 1e6, spectrum_db, color=COLORS["accent"], linewidth=1.0)
|
||||
ax_psd.set_xlabel("Frequency (MHz)")
|
||||
ax_psd.set_ylabel("Power (dB)")
|
||||
ax_psd.set_title("Power Spectral Density", loc="left", fontsize=15)
|
||||
ax_psd.grid(True, alpha=0.2, linewidth=0.5)
|
||||
ax_psd.set_title("Power Spectral Density")
|
||||
ax_psd.grid(True, alpha=0.3)
|
||||
|
||||
if compact_mode:
|
||||
ax1.set_xticks([])
|
||||
|
|
@ -366,20 +367,13 @@ def view_simple_sig(
|
|||
else:
|
||||
plt.tight_layout()
|
||||
if show_title:
|
||||
plt.subplots_adjust(top=0.9)
|
||||
plt.subplots_adjust(top=0.92)
|
||||
|
||||
if saveplot:
|
||||
output_path, extension = set_path(output_path=output_path)
|
||||
dpi_value = _set_dpi(fast_mode=fast_mode, labels_mode=labels_mode, extension=extension)
|
||||
|
||||
plt.savefig(
|
||||
output_path,
|
||||
dpi=dpi_value,
|
||||
bbox_inches="tight",
|
||||
pad_inches=0.3,
|
||||
facecolor=matplotlib.rcParams["savefig.facecolor"],
|
||||
edgecolor=matplotlib.rcParams["savefig.edgecolor"],
|
||||
)
|
||||
plt.savefig(output_path, dpi=dpi_value, bbox_inches="tight", facecolor="#0f172a", edgecolor="none")
|
||||
print(f"Saved signal plot to {output_path}")
|
||||
return output_path
|
||||
|
||||
|
|
|
|||
|
|
@ -2,57 +2,15 @@
|
|||
This module contains the main group for the ria toolkit oss CLI.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import warnings
|
||||
import click
|
||||
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message="Unable to import Axes3D",
|
||||
category=UserWarning,
|
||||
module="matplotlib",
|
||||
)
|
||||
|
||||
import click # noqa: E402
|
||||
|
||||
from ria_toolkit_oss_cli.ria_toolkit_oss import commands # noqa: E402
|
||||
from ria_toolkit_oss_cli.ria_toolkit_oss import commands
|
||||
|
||||
|
||||
def _git_lfs_installed() -> bool:
|
||||
"""Return True if git-lfs is available on PATH."""
|
||||
try:
|
||||
return (
|
||||
subprocess.run(
|
||||
["git", "lfs", "version"],
|
||||
capture_output=True,
|
||||
).returncode
|
||||
== 0
|
||||
)
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
|
||||
|
||||
@click.group(invoke_without_command=True)
|
||||
@click.group()
|
||||
@click.option("-v", "--verbose", is_flag=True, type=bool, help="Increase verbosity, especially useful for debugging.")
|
||||
@click.pass_context
|
||||
def cli(ctx, verbose):
|
||||
lfs_missing = not _git_lfs_installed()
|
||||
if lfs_missing:
|
||||
click.echo(
|
||||
"Warning: git-lfs is not installed. RIA Hub projects require git-lfs to\n"
|
||||
"track large binary files (models, recordings, datasets).\n"
|
||||
"\n"
|
||||
" Linux: sudo apt-get install git-lfs\n"
|
||||
" macOS: brew install git-lfs\n"
|
||||
" Other platforms: https://git-lfs.com\n"
|
||||
"\n"
|
||||
"After installing, run: git lfs install",
|
||||
err=True,
|
||||
)
|
||||
if ctx.invoked_subcommand is None:
|
||||
if lfs_missing and sys.stdin.isatty():
|
||||
click.pause(info="\nPress Enter to continue...", err=True)
|
||||
click.echo(ctx.get_help())
|
||||
def cli(verbose):
|
||||
pass
|
||||
|
||||
|
||||
# Loop through project commands, binding them all to the CLI.
|
||||
|
|
|
|||
|
|
@ -1,97 +0,0 @@
|
|||
"""Shared authentication and security helpers for RIA Hub API calls."""
|
||||
|
||||
import base64
|
||||
import subprocess
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
import click
|
||||
|
||||
DEFAULT_HUB = "https://riahub.ai"
|
||||
|
||||
|
||||
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
|
||||
"""Block redirects on authenticated requests to prevent credential exfiltration.
|
||||
|
||||
urllib re-sends the Authorization header on same-host redirects by default.
|
||||
A malicious server could redirect a POST to a different host to harvest
|
||||
credentials. We refuse all redirects — API clients should not encounter them
|
||||
in normal operation.
|
||||
"""
|
||||
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
raise urllib.error.URLError(f"Unexpected redirect ({code}) to {newurl} — aborting to protect credentials")
|
||||
|
||||
|
||||
def hub_opener() -> urllib.request.OpenerDirector:
|
||||
"""Return a urllib opener that blocks redirects."""
|
||||
return urllib.request.build_opener(_NoRedirectHandler)
|
||||
|
||||
|
||||
def warn_if_insecure(hub: str) -> None:
|
||||
"""Warn when credentials would be sent over plain HTTP to a non-localhost host."""
|
||||
parsed = urllib.parse.urlparse(hub)
|
||||
if parsed.scheme == "http":
|
||||
host = parsed.hostname or ""
|
||||
if host not in ("localhost", "127.0.0.1", "::1"):
|
||||
click.echo(
|
||||
f"Warning: sending credentials over plain HTTP to {host}. " "Use HTTPS in production.",
|
||||
err=True,
|
||||
)
|
||||
|
||||
|
||||
def basic_auth(username: str, password: str) -> str:
|
||||
return "Basic " + base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
|
||||
|
||||
def get_stored_credentials(hub_url: str) -> tuple[str | None, str | None]:
|
||||
"""Ask git credential fill for stored creds. Returns (username, password) or (None, None)."""
|
||||
parsed = urllib.parse.urlparse(hub_url)
|
||||
payload = f"protocol={parsed.scheme}\nhost={parsed.netloc}\n\n"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "credential", "fill"],
|
||||
input=payload,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
creds = {}
|
||||
for line in result.stdout.splitlines():
|
||||
# Partition on the FIRST '=' only so passwords containing '=' are preserved.
|
||||
k, sep, v = line.partition("=")
|
||||
if sep:
|
||||
creds[k.strip()] = v # keep value verbatim
|
||||
return creds.get("username"), creds.get("password")
|
||||
except Exception:
|
||||
return None, None
|
||||
|
||||
|
||||
def store_credentials(hub_url: str, username: str, password: str) -> None:
|
||||
"""Cache credentials via git credential approve (uses the system keychain/store)."""
|
||||
parsed = urllib.parse.urlparse(hub_url)
|
||||
payload = (
|
||||
f"protocol={parsed.scheme}\n" f"host={parsed.netloc}\n" f"username={username}\n" f"password={password}\n\n"
|
||||
)
|
||||
try:
|
||||
subprocess.run(
|
||||
["git", "credential", "approve"],
|
||||
input=payload,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
except Exception:
|
||||
pass # non-fatal — next push just prompts again
|
||||
|
||||
|
||||
def resolve_credentials(hub: str) -> tuple[str, str]:
|
||||
"""Return (username, password), prompting interactively if not cached."""
|
||||
username, password = get_stored_credentials(hub)
|
||||
if username and password:
|
||||
return username, password
|
||||
click.echo(f"No stored credentials found for {hub}.")
|
||||
username = click.prompt("RIA Hub username")
|
||||
password = click.prompt("Password / personal access token", hide_input=True)
|
||||
return username, password
|
||||
|
|
@ -51,7 +51,7 @@ def detect_input_format(filepath):
|
|||
raise click.ClickException(f"Unknown format for '{filepath}'. Supported: .sigmf, .npy, .wav, .blue")
|
||||
|
||||
|
||||
def determine_output_path(input_path, output_path, fmt, overwrite):
|
||||
def determine_output_path(input_path, output_path, fmt, quiet, overwrite):
|
||||
input_path = Path(input_path)
|
||||
input_is_annotated = input_path.stem.endswith("_annotated")
|
||||
|
||||
|
|
@ -63,20 +63,24 @@ def determine_output_path(input_path, output_path, fmt, overwrite):
|
|||
else:
|
||||
target = input_path.with_name(f"{input_path.stem}_annotated{input_path.suffix}")
|
||||
|
||||
final_path = normalize_sigmf_path(target) if fmt == "sigmf" else target
|
||||
if fmt == "sigmf":
|
||||
final_path = normalize_sigmf_path(target)
|
||||
if not quiet:
|
||||
click.echo(f"Saving SigMF metadata to: {final_path}")
|
||||
else:
|
||||
final_path = target
|
||||
if not quiet:
|
||||
click.echo(f"Saving to: {final_path}")
|
||||
|
||||
if final_path.exists() and not overwrite:
|
||||
raise click.ClickException(f"{final_path} already exists. Use --overwrite to replace it.")
|
||||
# Always allow writing to _annotated files; guard against overwriting originals
|
||||
target_is_annotated = final_path.stem.endswith("_annotated")
|
||||
if final_path.exists() and not target_is_annotated and final_path != input_path:
|
||||
click.echo(f"Error: {final_path} is not an annotated file and cannot be overwritten.", err=True)
|
||||
return None
|
||||
|
||||
return final_path
|
||||
|
||||
|
||||
def check_output_available(input_path, output_path, overwrite):
|
||||
"""Raise ClickException before any work begins if the output file already exists."""
|
||||
fmt = detect_input_format(Path(input_path))
|
||||
determine_output_path(input_path=input_path, output_path=output_path, fmt=fmt, overwrite=overwrite)
|
||||
|
||||
|
||||
def save_recording_auto(recording, output_path, input_path, quiet=False, overwrite=False):
|
||||
"""Save recording, auto-detecting format from extension.
|
||||
|
||||
|
|
@ -86,13 +90,10 @@ def save_recording_auto(recording, output_path, input_path, quiet=False, overwri
|
|||
input_path = Path(input_path)
|
||||
fmt = detect_input_format(input_path)
|
||||
|
||||
output_path = determine_output_path(input_path=input_path, output_path=output_path, fmt=fmt, overwrite=overwrite)
|
||||
|
||||
if not quiet:
|
||||
if fmt == "sigmf":
|
||||
click.echo(f"Saving SigMF metadata to: {output_path}")
|
||||
else:
|
||||
click.echo(f"Saving to: {output_path}")
|
||||
# Determine output path
|
||||
output_path = determine_output_path(
|
||||
input_path=input_path, output_path=output_path, fmt=fmt, quiet=quiet, overwrite=overwrite
|
||||
)
|
||||
|
||||
if fmt == "sigmf":
|
||||
# Normalize path for SigMF
|
||||
|
|
@ -256,11 +257,7 @@ def list(input, verbose):
|
|||
user_comment = ann.comment or ""
|
||||
|
||||
# Basic info
|
||||
freq_range = (
|
||||
f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
||||
if ann.freq_lower_edge is not None and ann.freq_upper_edge is not None
|
||||
else "N/A"
|
||||
)
|
||||
freq_range = f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
||||
click.echo(
|
||||
f" [{i}] Samples {format_sample_count(ann.sample_start)}-"
|
||||
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {ann.label}"
|
||||
|
|
@ -315,8 +312,6 @@ def add(input, start, count, label, freq_lower, freq_upper, comment, annotation_
|
|||
except Exception as e:
|
||||
raise click.ClickException(f"Failed to load recording: {e}")
|
||||
|
||||
check_output_available(input, output, overwrite)
|
||||
|
||||
# Validate sample range
|
||||
n_samples = len(recording.data[0])
|
||||
if start < 0:
|
||||
|
|
@ -368,9 +363,12 @@ def add(input, start, count, label, freq_lower, freq_upper, comment, annotation_
|
|||
if comment:
|
||||
click.echo(f" Comment: {comment}")
|
||||
|
||||
try:
|
||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||
if not quiet:
|
||||
click.echo(" ✓ Saved")
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"Failed to save: {e}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
|
|
@ -468,6 +466,8 @@ def clear(input, output, overwrite, force, quiet):
|
|||
if not quiet:
|
||||
click.echo(f"\nCleared {count_before} annotation(s)")
|
||||
|
||||
recording._annotations = []
|
||||
|
||||
try:
|
||||
save_recording_auto(recording, output_path=input, input_path=input, quiet=quiet, overwrite=True)
|
||||
if not quiet:
|
||||
|
|
@ -503,9 +503,6 @@ def clear(input, output, overwrite, force, quiet):
|
|||
default="standalone",
|
||||
help="Annotation type",
|
||||
)
|
||||
@click.option(
|
||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
||||
)
|
||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||
|
|
@ -520,7 +517,6 @@ def energy(
|
|||
nfft,
|
||||
obw_power,
|
||||
annotation_type,
|
||||
sample_rate,
|
||||
output,
|
||||
overwrite,
|
||||
quiet,
|
||||
|
|
@ -543,11 +539,8 @@ def energy(
|
|||
ria annotate energy signal.npy --threshold 1.5 --min-distance 10000
|
||||
ria annotate energy signal.sigmf-data --freq-method obw
|
||||
ria annotate energy signal.sigmf-data --freq-method full-detected
|
||||
ria annotate energy signal.npy --sample-rate 1e6
|
||||
|
||||
"""
|
||||
check_output_available(input, output, overwrite)
|
||||
|
||||
try:
|
||||
recording = load_recording(input)
|
||||
if not quiet:
|
||||
|
|
@ -555,15 +548,6 @@ def energy(
|
|||
except Exception as e:
|
||||
raise click.ClickException(f"Failed to load recording: {e}")
|
||||
|
||||
if sample_rate is not None:
|
||||
recording.sample_rate = sample_rate
|
||||
|
||||
if recording.sample_rate is None:
|
||||
raise click.ClickException(
|
||||
"Recording metadata does not contain a sample rate. "
|
||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
||||
)
|
||||
|
||||
if not quiet:
|
||||
click.echo("\nDetecting signals using energy-based method...")
|
||||
click.echo(" Time detection:")
|
||||
|
|
@ -591,12 +575,12 @@ def energy(
|
|||
|
||||
if not quiet:
|
||||
click.echo(f" ✓ Added {added} annotation(s)")
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"Energy detection failed: {e}")
|
||||
|
||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||
if not quiet:
|
||||
click.echo(" ✓ Saved")
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"Energy detection failed: {e}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
|
|
@ -617,13 +601,10 @@ def energy(
|
|||
default="standalone",
|
||||
help="Annotation type",
|
||||
)
|
||||
@click.option(
|
||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
||||
)
|
||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||
def cusum(input, label, min_duration, window_size, tolerance, annotation_type, sample_rate, output, overwrite, quiet):
|
||||
def cusum(input, label, min_duration, window_size, tolerance, annotation_type, output, overwrite, quiet):
|
||||
"""Auto-detect segments using CUSUM method.
|
||||
|
||||
Detects signal state changes (on/off, amplitude transitions). Best for
|
||||
|
|
@ -635,10 +616,7 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
|||
Examples:
|
||||
ria annotate cusum signal.sigmf-data --min-duration 5.0
|
||||
ria annotate cusum data.npy --min-duration 10.0 --label state
|
||||
ria annotate cusum data.npy --sample-rate 1e6
|
||||
"""
|
||||
check_output_available(input, output, overwrite)
|
||||
|
||||
try:
|
||||
recording = load_recording(input)
|
||||
if not quiet:
|
||||
|
|
@ -646,15 +624,6 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
|||
except Exception as e:
|
||||
raise click.ClickException(f"Failed to load recording: {e}")
|
||||
|
||||
if sample_rate is not None:
|
||||
recording.sample_rate = sample_rate
|
||||
|
||||
if recording.sample_rate is None:
|
||||
raise click.ClickException(
|
||||
"Recording metadata does not contain a sample rate. "
|
||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
||||
)
|
||||
|
||||
if not quiet:
|
||||
click.echo("\nDetecting segments using CUSUM...")
|
||||
click.echo(f" Min duration: {min_duration} ms")
|
||||
|
|
@ -675,12 +644,12 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
|||
|
||||
if not quiet:
|
||||
click.echo(f" ✓ Added {added} annotation(s)")
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"CUSUM detection failed: {e}")
|
||||
|
||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||
if not quiet:
|
||||
click.echo(" ✓ Saved")
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"CUSUM detection failed: {e}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
|
|
@ -706,13 +675,10 @@ def cusum(input, label, min_duration, window_size, tolerance, annotation_type, s
|
|||
help="Annotation type",
|
||||
)
|
||||
@click.option("--channel", type=int, default=0, help="Channel index to annotate (default: 0)")
|
||||
@click.option(
|
||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
||||
)
|
||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||
def threshold(input, threshold, label, window_size, annotation_type, channel, sample_rate, output, overwrite, quiet):
|
||||
def threshold(input, threshold, label, window_size, annotation_type, channel, output, overwrite, quiet):
|
||||
"""Auto-detect signals using threshold method.
|
||||
|
||||
Detects samples above a percentage of maximum magnitude. Best for simple
|
||||
|
|
@ -722,13 +688,10 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
|||
Examples:
|
||||
ria annotate threshold signal.sigmf-data --threshold 0.7 --label wifi
|
||||
ria annotate threshold data.npy --threshold 0.5 --window-size 2048
|
||||
ria annotate threshold data.npy --threshold 0.4 --sample-rate 1e6
|
||||
"""
|
||||
if not (0.0 <= threshold <= 1.0):
|
||||
raise click.ClickException(f"--threshold must be between 0.0 and 1.0, got {threshold}")
|
||||
|
||||
check_output_available(input, output, overwrite)
|
||||
|
||||
try:
|
||||
recording = load_recording(input)
|
||||
if not quiet:
|
||||
|
|
@ -736,21 +699,11 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
|||
except Exception as e:
|
||||
raise click.ClickException(f"Failed to load recording: {e}")
|
||||
|
||||
if sample_rate is not None:
|
||||
recording.sample_rate = sample_rate
|
||||
|
||||
if recording.sample_rate is None:
|
||||
raise click.ClickException(
|
||||
"Recording metadata does not contain a sample rate. "
|
||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
||||
)
|
||||
|
||||
if not quiet:
|
||||
click.echo("\nDetecting signals using threshold qualifier...")
|
||||
click.echo(f" Threshold: {threshold * 100:.1f}% of max magnitude")
|
||||
click.echo(f" Window size: {'auto (1ms)' if window_size is None else f'{window_size} samples'}")
|
||||
click.echo(f" Channel: {channel}")
|
||||
click.echo(f" Sample rate: {recording.sample_rate:.0f} Hz")
|
||||
|
||||
try:
|
||||
initial_count = len(recording.annotations)
|
||||
|
|
@ -766,12 +719,12 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
|||
|
||||
if not quiet:
|
||||
click.echo(f" ✓ Added {added} annotation(s)")
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"Threshold detection failed: {e}")
|
||||
|
||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||
if not quiet:
|
||||
click.echo(" ✓ Saved")
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"Threshold detection failed: {e}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
|
|
@ -785,30 +738,11 @@ def threshold(input, threshold, label, window_size, annotation_type, channel, sa
|
|||
@click.option("--nfft", type=int, default=65536, help="FFT size for spectral analysis")
|
||||
@click.option("--noise-threshold-db", type=float, help="Noise floor threshold in dB (auto-estimated if not specified)")
|
||||
@click.option("--min-component-bw", type=float, default=50e3, help="Min component bandwidth in Hz")
|
||||
@click.option(
|
||||
"--sample-rate", type=float, default=None, help="Sample rate in Hz (overrides metadata; required if not in file)"
|
||||
)
|
||||
@click.option("--output", "-o", type=click.Path(), help="Output file path")
|
||||
@click.option("--overwrite", is_flag=True, help="Overwrite input file (non-SigMF only)")
|
||||
@click.option("--quiet", is_flag=True, help="Quiet mode")
|
||||
@click.option("--verbose", is_flag=True, help="Verbose output (show detected components)")
|
||||
def _log_separate_start(quiet, recording, indices_list, nfft, noise_threshold_db, min_component_bw):
|
||||
if not quiet:
|
||||
click.echo("\nSplitting annotations by frequency components...")
|
||||
click.echo(f" Input annotations: {len(recording.annotations)}")
|
||||
if indices_list:
|
||||
click.echo(f" Splitting indices: {indices_list}")
|
||||
click.echo(f" FFT size: {nfft}")
|
||||
if noise_threshold_db is not None:
|
||||
click.echo(f" Noise threshold: {noise_threshold_db} dB")
|
||||
else:
|
||||
click.echo(" Noise threshold: auto-estimated")
|
||||
click.echo(f" Min component BW: {format_frequency(min_component_bw)}")
|
||||
|
||||
|
||||
def separate(
|
||||
input, indices, nfft, noise_threshold_db, min_component_bw, sample_rate, output, overwrite, quiet, verbose
|
||||
):
|
||||
def separate(input, indices, nfft, noise_threshold_db, min_component_bw, output, overwrite, quiet, verbose):
|
||||
"""
|
||||
Auto-detect parallel frequency-offset signals and split into sub-bands.
|
||||
|
||||
|
|
@ -834,8 +768,6 @@ def separate(
|
|||
ria annotate separate signal.npy --min-component-bw 100000
|
||||
|
||||
"""
|
||||
check_output_available(input, output, overwrite)
|
||||
|
||||
try:
|
||||
recording = load_recording(input)
|
||||
if not quiet:
|
||||
|
|
@ -843,15 +775,6 @@ def separate(
|
|||
except Exception as e:
|
||||
raise click.ClickException(f"Failed to load recording: {e}")
|
||||
|
||||
if sample_rate is not None:
|
||||
recording.sample_rate = sample_rate
|
||||
|
||||
if recording.sample_rate is None:
|
||||
raise click.ClickException(
|
||||
"Recording metadata does not contain a sample rate. "
|
||||
"Provide it with --sample-rate (e.g. --sample-rate 1e6)."
|
||||
)
|
||||
|
||||
# Parse indices if specified
|
||||
indices_list = get_indices_list(indices=indices, recording=recording)
|
||||
|
||||
|
|
@ -860,7 +783,17 @@ def separate(
|
|||
click.echo("No annotations to split")
|
||||
return
|
||||
|
||||
_log_separate_start(quiet, recording, indices_list, nfft, noise_threshold_db, min_component_bw)
|
||||
if not quiet:
|
||||
click.echo("\nSplitting annotations by frequency components...")
|
||||
click.echo(f" Input annotations: {len(recording.annotations)}")
|
||||
if indices_list:
|
||||
click.echo(f" Splitting indices: {indices_list}")
|
||||
click.echo(f" FFT size: {nfft}")
|
||||
if noise_threshold_db is not None:
|
||||
click.echo(f" Noise threshold: {noise_threshold_db} dB")
|
||||
else:
|
||||
click.echo(" Noise threshold: auto-estimated")
|
||||
click.echo(f" Min component BW: {format_frequency(min_component_bw)}")
|
||||
|
||||
try:
|
||||
initial_count = len(recording.annotations)
|
||||
|
|
@ -882,19 +815,14 @@ def separate(
|
|||
click.echo("\n Details:")
|
||||
for i in range(initial_count, final_count):
|
||||
ann = recording.annotations[i]
|
||||
freq_range = (
|
||||
f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
||||
if ann.freq_lower_edge is not None and ann.freq_upper_edge is not None
|
||||
else "N/A"
|
||||
)
|
||||
freq_range = f"{format_frequency(ann.freq_lower_edge)} - {format_frequency(ann.freq_upper_edge)}"
|
||||
click.echo(
|
||||
f" [{i}] samples {format_sample_count(ann.sample_start)}-"
|
||||
f"{format_sample_count(ann.sample_start + ann.sample_count)}: {freq_range}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"Spectral separation failed: {e}")
|
||||
|
||||
save_recording_auto(recording, output, input, quiet, overwrite)
|
||||
if not quiet:
|
||||
click.echo(" ✓ Saved")
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"Spectral separation failed: {e}")
|
||||
|
|
|
|||
|
|
@ -16,11 +16,9 @@ from .generate import generate
|
|||
# from .generate import generate
|
||||
from .init import init
|
||||
from .serve import serve
|
||||
from .setup_repo import setup_repo
|
||||
from .split import split
|
||||
from .transform import transform
|
||||
from .transmit import transmit
|
||||
from .upload import upload
|
||||
from .view import view
|
||||
|
||||
# Aliases
|
||||
|
|
|
|||
|
|
@ -1,401 +0,0 @@
|
|||
"""ria setup_repo — create and configure a RIA Hub Project repo."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
import click
|
||||
|
||||
from ._hub_auth import (
|
||||
DEFAULT_HUB,
|
||||
_NoRedirectHandler,
|
||||
basic_auth,
|
||||
resolve_credentials,
|
||||
store_credentials,
|
||||
warn_if_insecure,
|
||||
)
|
||||
|
||||
RIA_LFS_RULES = [
|
||||
("*.pt", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.pth", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.onnx", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.sigmf", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.sigmf-data", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.sigmf-meta", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.npy", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.npz", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.h5", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.hdf5", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.bin", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
("*.pkl", "filter=lfs diff=lfs merge=lfs -text"),
|
||||
]
|
||||
|
||||
# Repo names must be safe directory names and valid git remote path components.
|
||||
_SAFE_NAME_RE = re.compile(r"^[A-Za-z0-9._-]{1,100}$")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# API helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _api_request(
|
||||
hub: str,
|
||||
path: str,
|
||||
method: str,
|
||||
username: str,
|
||||
password: str,
|
||||
body: dict | None = None,
|
||||
) -> tuple[dict, int]:
|
||||
"""
|
||||
Make an authenticated request to the RIA Hub API.
|
||||
Returns (parsed_response_body, http_status_code).
|
||||
Status 0 means a network/connection error.
|
||||
Credentials are sent as HTTP Basic auth — safe over HTTPS and localhost HTTP.
|
||||
Redirects are blocked to prevent credential exfiltration.
|
||||
"""
|
||||
url = f"{hub.rstrip('/')}/api/v1{path}"
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
req = urllib.request.Request(url, data=data, method=method)
|
||||
req.add_header("Content-Type", "application/json")
|
||||
req.add_header("Authorization", basic_auth(username, password))
|
||||
|
||||
opener = urllib.request.build_opener(_NoRedirectHandler)
|
||||
try:
|
||||
with opener.open(req, timeout=15) as resp:
|
||||
return json.loads(resp.read() or b"{}"), resp.status
|
||||
except urllib.error.HTTPError as e:
|
||||
try:
|
||||
resp_body = json.loads(e.read() or b"{}")
|
||||
except Exception:
|
||||
resp_body = {}
|
||||
return resp_body, e.code
|
||||
except urllib.error.URLError as e:
|
||||
return {"message": str(e.reason)}, 0
|
||||
|
||||
|
||||
def _get_authenticated_username(hub: str, username: str, password: str) -> str | None:
|
||||
"""Return the login name of the authenticated user from GET /api/v1/user.
|
||||
|
||||
This is the canonical username for URL construction — it may differ from
|
||||
git config user.name which is a display name, not a login.
|
||||
"""
|
||||
body, status = _api_request(hub, "/user", "GET", username, password)
|
||||
if status == 200:
|
||||
return body.get("login")
|
||||
return None
|
||||
|
||||
|
||||
def _repo_exists(hub: str, owner: str, name: str, username: str, password: str) -> bool:
|
||||
body, status = _api_request(
|
||||
hub,
|
||||
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(name, safe='')}",
|
||||
"GET",
|
||||
username,
|
||||
password,
|
||||
)
|
||||
return status == 200
|
||||
|
||||
|
||||
def _create_repo_on_hub(hub: str, name: str, username: str, password: str, private: bool) -> bool:
|
||||
"""Create an RIA Hub Project repo via API.
|
||||
|
||||
Returns True if the repo was freshly created (server seeded README.md and
|
||||
.gitattributes via auto_init + is_ria), False if the hub was unreachable
|
||||
(local fallback needed). Exits on fatal errors (auth, quota, name taken).
|
||||
"""
|
||||
body, status = _api_request(
|
||||
hub,
|
||||
"/user/repos",
|
||||
"POST",
|
||||
username,
|
||||
password,
|
||||
{
|
||||
"name": name,
|
||||
"auto_init": True,
|
||||
"is_ria": True,
|
||||
"private": private,
|
||||
"default_branch": "main",
|
||||
},
|
||||
)
|
||||
|
||||
if status == 201:
|
||||
click.echo(f"Repository '{name}' created on RIA Hub.")
|
||||
return True
|
||||
|
||||
if status == 0:
|
||||
click.echo(
|
||||
f"Warning: could not reach RIA Hub at {hub}: {body.get('message', 'connection failed')}",
|
||||
err=True,
|
||||
)
|
||||
click.echo("Continuing with local setup only — create the repo manually on RIA Hub.", err=True)
|
||||
return False
|
||||
|
||||
msg = body.get("message", "")
|
||||
|
||||
if status == 401:
|
||||
click.echo("Error: authentication failed — check your username/password.", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
if status in (403, 413) or "quota" in msg.lower() or "limit" in msg.lower():
|
||||
click.echo("Error: cannot create repository — storage quota or account limit reached.", err=True)
|
||||
if msg:
|
||||
click.echo(f" Server message: {msg}", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
if status == 422 or "already exist" in msg.lower():
|
||||
click.echo(f"Repository '{name}' already exists on RIA Hub.")
|
||||
return False
|
||||
|
||||
click.echo(f"Error creating repository (HTTP {status}): {msg}", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Local git helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _tracked_patterns(ga_path: str) -> set:
|
||||
if not os.path.exists(ga_path):
|
||||
return set()
|
||||
patterns = set()
|
||||
with open(ga_path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^(\S+)\s+", line)
|
||||
if m:
|
||||
patterns.add(m.group(1))
|
||||
return patterns
|
||||
|
||||
|
||||
def _write_local_ria_files(repo_path: str, repo_name: str) -> None:
|
||||
"""Seed README.md and .gitattributes locally (used when hub is unreachable or --no-remote)."""
|
||||
# README
|
||||
for candidate in ("README.md", "README.rst", "README.txt", "README"):
|
||||
if os.path.exists(os.path.join(repo_path, candidate)):
|
||||
click.echo(f"README: {candidate} already exists, skipping")
|
||||
break
|
||||
else:
|
||||
with open(os.path.join(repo_path, "README.md"), "w", encoding="utf-8") as f:
|
||||
f.write(
|
||||
f"# {repo_name}\n"
|
||||
"\n"
|
||||
"A RIA Hub project.\n"
|
||||
"\n"
|
||||
"## Description\n"
|
||||
"\n"
|
||||
"<!-- Add your project description here -->\n"
|
||||
"\n"
|
||||
"## Contents\n"
|
||||
"\n"
|
||||
"<!-- Describe the signals, models, or datasets in this repository -->\n"
|
||||
)
|
||||
click.echo("README.md: created")
|
||||
|
||||
# .gitattributes
|
||||
ga_path = os.path.join(repo_path, ".gitattributes")
|
||||
existing = _tracked_patterns(ga_path)
|
||||
new_rules = [(p, a) for p, a in RIA_LFS_RULES if p not in existing]
|
||||
|
||||
if new_rules:
|
||||
existing_content = ""
|
||||
if os.path.exists(ga_path):
|
||||
with open(ga_path, encoding="utf-8") as f:
|
||||
existing_content = f.read()
|
||||
|
||||
separator = "" if (not existing_content or existing_content.endswith("\n")) else "\n"
|
||||
addition = separator + "".join(f"{pattern} {attrs}\n" for pattern, attrs in new_rules)
|
||||
|
||||
with open(ga_path, "a", encoding="utf-8") as f:
|
||||
f.write(addition)
|
||||
click.echo(f".gitattributes: {len(new_rules)} rule(s) added")
|
||||
else:
|
||||
click.echo(".gitattributes: all RIA Hub rules are already present")
|
||||
|
||||
|
||||
def _git(repo_path: str, *args: str, check: bool = True) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
["git", "-C", repo_path, *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=check,
|
||||
)
|
||||
|
||||
|
||||
def _resolve_path_and_name(name: str | None, local_path: str | None) -> tuple[str, str]:
|
||||
if local_path:
|
||||
repo_path = os.path.abspath(local_path)
|
||||
repo_name = name or os.path.basename(repo_path)
|
||||
elif name:
|
||||
repo_path = os.path.abspath(name)
|
||||
repo_name = name
|
||||
else:
|
||||
repo_path = os.path.abspath(".")
|
||||
repo_name = os.path.basename(repo_path)
|
||||
return repo_path, repo_name
|
||||
|
||||
|
||||
def _resolve_owner(hub: str, username: str | None, password: str | None, owner: str | None) -> str:
|
||||
if not owner and username and password:
|
||||
api_login = _get_authenticated_username(hub, username, password)
|
||||
owner = api_login or username
|
||||
return owner or "unknown"
|
||||
|
||||
|
||||
def _git_init(repo_path: str) -> None:
|
||||
if os.path.isdir(os.path.join(repo_path, ".git")):
|
||||
return
|
||||
result = _git(repo_path, "init", "-b", "main", check=False)
|
||||
if result.returncode != 0:
|
||||
# Older git (< 2.28) doesn't support -b; fall back and rename.
|
||||
_git(repo_path, "init")
|
||||
_git(repo_path, "symbolic-ref", "HEAD", "refs/heads/main")
|
||||
click.echo("git init: done (branch: main)")
|
||||
|
||||
|
||||
def _configure_remote(
|
||||
repo_path: str, hub: str, resolved_owner: str, repo_name: str, username: str | None, no_remote: bool
|
||||
) -> None:
|
||||
if no_remote or not username:
|
||||
click.echo(
|
||||
f"Skipped remote setup. Add it manually:\n"
|
||||
f" git -C {repo_path} remote add origin "
|
||||
f"{hub.rstrip('/')}/{resolved_owner}/{repo_name}.git"
|
||||
)
|
||||
return
|
||||
remote_url = f"{hub.rstrip('/')}/{resolved_owner}/{repo_name}.git"
|
||||
existing = _git(repo_path, "remote", "get-url", "origin", check=False)
|
||||
if existing.returncode == 0:
|
||||
existing_url = existing.stdout.strip()
|
||||
if existing_url == remote_url:
|
||||
click.echo(f"remote origin: {remote_url} (already set)")
|
||||
else:
|
||||
click.echo(
|
||||
f"remote 'origin' already points to {existing_url}.\n"
|
||||
f" To update: git remote set-url origin {remote_url}"
|
||||
)
|
||||
else:
|
||||
_git(repo_path, "remote", "add", "origin", remote_url)
|
||||
click.echo(f"remote origin: {remote_url}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Command
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@click.command("setup_repo")
|
||||
@click.argument("name", required=False)
|
||||
@click.option(
|
||||
"--path", "local_path", default=None, help="Local directory (default: current dir, or created from NAME)."
|
||||
)
|
||||
@click.option("--hub", default=DEFAULT_HUB, show_default=True, metavar="URL", help="RIA Hub base URL.")
|
||||
@click.option(
|
||||
"--owner",
|
||||
default=None,
|
||||
metavar="USER",
|
||||
help="RIA Hub login username (default: looked up from the API using your credentials).",
|
||||
)
|
||||
@click.option("--private", is_flag=True, default=False, help="Create the repository as private.")
|
||||
@click.option(
|
||||
"--no-remote", is_flag=True, default=False, help="Skip creating the repository on RIA Hub (local setup only)."
|
||||
)
|
||||
def setup_repo(
|
||||
name: str | None,
|
||||
local_path: str | None,
|
||||
hub: str,
|
||||
owner: str | None,
|
||||
private: bool,
|
||||
no_remote: bool,
|
||||
) -> None:
|
||||
"""Create and configure a RIA Hub Project repo.
|
||||
|
||||
NAME is the repository name. If the local directory does not exist or is
|
||||
not a git repo, it will be initialised automatically. Credentials are
|
||||
retrieved from git's credential store — no token setup required if you
|
||||
have used RIA Hub with git before.
|
||||
|
||||
\b
|
||||
Examples:
|
||||
ria setup_repo my-dataset
|
||||
ria setup_repo my-dataset --hub https://riahub.example.com
|
||||
ria setup_repo --path ./existing-dir
|
||||
ria setup_repo my-dataset --private
|
||||
"""
|
||||
repo_path, repo_name = _resolve_path_and_name(name, local_path)
|
||||
|
||||
if not _SAFE_NAME_RE.match(repo_name):
|
||||
click.echo(
|
||||
f"Error: '{repo_name}' is not a valid repository name.\n"
|
||||
"Use only letters, numbers, hyphens, underscores, and dots (max 100 chars).",
|
||||
err=True,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
if not no_remote:
|
||||
warn_if_insecure(hub)
|
||||
|
||||
username, password = (None, None) if no_remote else resolve_credentials(hub)
|
||||
resolved_owner = _resolve_owner(hub, username, password, owner)
|
||||
|
||||
# newly_created=True means the server ran auto_init+is_ria and seeded
|
||||
# README.md + .gitattributes in the initial commit; local setup pulls
|
||||
# those files via fetch rather than writing them from scratch.
|
||||
newly_created = False
|
||||
if not no_remote and username and password:
|
||||
if _repo_exists(hub, resolved_owner, repo_name, username, password):
|
||||
click.echo(f"Repository '{resolved_owner}/{repo_name}' already exists on RIA Hub.")
|
||||
else:
|
||||
newly_created = _create_repo_on_hub(hub, repo_name, username, password, private)
|
||||
store_credentials(hub, username, password)
|
||||
|
||||
if not os.path.exists(repo_path):
|
||||
os.makedirs(repo_path)
|
||||
click.echo(f"Created directory: {repo_path}")
|
||||
|
||||
_git_init(repo_path)
|
||||
|
||||
if subprocess.run(["git", "lfs", "version"], capture_output=True).returncode != 0:
|
||||
click.echo(
|
||||
"Error: git-lfs is not installed.\n"
|
||||
" Linux: sudo apt-get install git-lfs\n"
|
||||
" macOS: brew install git-lfs\n"
|
||||
" Other platforms: https://git-lfs.com",
|
||||
err=True,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
_git(repo_path, "lfs", "install", "--local")
|
||||
click.echo("git lfs install --local: done")
|
||||
|
||||
_configure_remote(repo_path, hub, resolved_owner, repo_name, username, no_remote)
|
||||
|
||||
if newly_created:
|
||||
fetch = _git(repo_path, "fetch", "origin", check=False)
|
||||
if fetch.returncode == 0:
|
||||
_git(repo_path, "reset", "--hard", "origin/main")
|
||||
click.echo("Pulled initial commit from RIA Hub (README.md + .gitattributes)")
|
||||
else:
|
||||
click.echo("Warning: fetch failed — falling back to local file setup.", err=True)
|
||||
_write_local_ria_files(repo_path, repo_name)
|
||||
else:
|
||||
_write_local_ria_files(repo_path, repo_name)
|
||||
|
||||
if newly_created:
|
||||
click.echo(f"\nRepo is ready. Push your work:\n cd {repo_path}\n git push -u origin main")
|
||||
else:
|
||||
click.echo(
|
||||
f"\nRepo is ready. Commit and push:\n"
|
||||
f" cd {repo_path}\n"
|
||||
f" git add README.md .gitattributes\n"
|
||||
f" git commit -m 'chore: initialise RIA Hub project'\n"
|
||||
f" git push -u origin main"
|
||||
)
|
||||
|
|
@ -1,392 +0,0 @@
|
|||
"""ria upload — stream large files to a RIA Hub Project via the LFS API.
|
||||
|
||||
How it works
|
||||
------------
|
||||
1. The file is hashed locally (SHA-256 + size) — this is the LFS object ID.
|
||||
2. A single POST to the repo's LFS batch endpoint returns an upload URL
|
||||
(and headers) for any object the server does not already have.
|
||||
3. The file is streamed to that URL in fixed-size chunks — nothing is ever
|
||||
fully loaded into memory, so files of any size work.
|
||||
4. A commit is created via the Gitea contents API that records the LFS
|
||||
pointer (a small text file) so the file appears in the repo tree.
|
||||
|
||||
No server-side changes are required — this uses the same authenticated LFS
|
||||
protocol that `git lfs push` uses internally.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import http.client
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
import click
|
||||
|
||||
from ._hub_auth import (
|
||||
DEFAULT_HUB,
|
||||
basic_auth,
|
||||
hub_opener,
|
||||
resolve_credentials,
|
||||
warn_if_insecure,
|
||||
)
|
||||
|
||||
# Read buffer for hashing and streaming — 8 MB keeps memory use flat
|
||||
# for arbitrarily large files.
|
||||
_CHUNK = 8 * 1024 * 1024
|
||||
|
||||
LFS_MEDIA_TYPE = "application/vnd.git-lfs+json"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# File helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _hash_file(path: str) -> tuple[str, int]:
|
||||
"""Return (sha256_hex, byte_size) by streaming the file."""
|
||||
h = hashlib.sha256()
|
||||
size = 0
|
||||
with open(path, "rb") as f:
|
||||
while True:
|
||||
chunk = f.read(_CHUNK)
|
||||
if not chunk:
|
||||
break
|
||||
h.update(chunk)
|
||||
size += len(chunk)
|
||||
return h.hexdigest(), size
|
||||
|
||||
|
||||
def _lfs_pointer_text(oid: str, size: int) -> str:
|
||||
return f"version https://git-lfs.github.com/spec/v1\noid sha256:{oid}\nsize {size}\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LFS batch API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _lfs_batch(
|
||||
hub: str,
|
||||
owner: str,
|
||||
repo: str,
|
||||
objects: list[dict],
|
||||
username: str,
|
||||
password: str,
|
||||
) -> dict:
|
||||
"""
|
||||
POST to /{owner}/{repo}.git/info/lfs/objects/batch.
|
||||
Returns the parsed JSON response.
|
||||
Raises on HTTP error or JSON decode failure.
|
||||
"""
|
||||
url = (
|
||||
f"{hub.rstrip('/')}"
|
||||
f"/{urllib.parse.quote(owner, safe='')}"
|
||||
f"/{urllib.parse.quote(repo, safe='')}"
|
||||
f".git/info/lfs/objects/batch"
|
||||
)
|
||||
body = json.dumps(
|
||||
{
|
||||
"operation": "upload",
|
||||
"transfers": ["basic"],
|
||||
"objects": objects,
|
||||
}
|
||||
).encode()
|
||||
|
||||
req = urllib.request.Request(url, data=body, method="POST")
|
||||
req.add_header("Content-Type", LFS_MEDIA_TYPE)
|
||||
req.add_header("Accept", LFS_MEDIA_TYPE)
|
||||
req.add_header("Authorization", basic_auth(username, password))
|
||||
|
||||
opener = hub_opener()
|
||||
try:
|
||||
with opener.open(req, timeout=30) as resp:
|
||||
return json.loads(resp.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
body_text = e.read().decode(errors="replace")
|
||||
raise RuntimeError(f"LFS batch request failed (HTTP {e.code}): {body_text}") from e
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gitea contents API — create / update a file to record the LFS pointer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _get_file_sha(
|
||||
hub: str,
|
||||
owner: str,
|
||||
repo: str,
|
||||
path: str,
|
||||
branch: str,
|
||||
username: str,
|
||||
password: str,
|
||||
) -> str | None:
|
||||
"""Return the blob SHA of an existing file, or None if it doesn't exist."""
|
||||
url = (
|
||||
f"{hub.rstrip('/')}/api/v1"
|
||||
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(repo, safe='')}"
|
||||
f"/contents/{urllib.parse.quote(path)}"
|
||||
f"?ref={urllib.parse.quote(branch)}"
|
||||
)
|
||||
req = urllib.request.Request(url)
|
||||
req.add_header("Authorization", basic_auth(username, password))
|
||||
try:
|
||||
with hub_opener().open(req, timeout=15) as resp:
|
||||
return json.loads(resp.read()).get("sha")
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
return None
|
||||
raise
|
||||
|
||||
|
||||
def _commit_lfs_pointer(
|
||||
hub: str,
|
||||
owner: str,
|
||||
repo: str,
|
||||
remote_path: str,
|
||||
pointer_text: str,
|
||||
branch: str,
|
||||
message: str,
|
||||
username: str,
|
||||
password: str,
|
||||
) -> None:
|
||||
"""Create or update a file in the repo containing the LFS pointer."""
|
||||
url = (
|
||||
f"{hub.rstrip('/')}/api/v1"
|
||||
f"/repos/{urllib.parse.quote(owner, safe='')}/{urllib.parse.quote(repo, safe='')}"
|
||||
f"/contents/{urllib.parse.quote(remote_path)}"
|
||||
)
|
||||
|
||||
existing_sha = _get_file_sha(hub, owner, repo, remote_path, branch, username, password)
|
||||
|
||||
body: dict = {
|
||||
"message": message,
|
||||
"content": base64.b64encode(pointer_text.encode()).decode(),
|
||||
"branch": branch,
|
||||
}
|
||||
if existing_sha:
|
||||
body["sha"] = existing_sha
|
||||
|
||||
method = "PUT" if existing_sha else "POST"
|
||||
req = urllib.request.Request(url, data=json.dumps(body).encode(), method=method)
|
||||
req.add_header("Content-Type", "application/json")
|
||||
req.add_header("Authorization", basic_auth(username, password))
|
||||
|
||||
try:
|
||||
with hub_opener().open(req, timeout=30) as resp:
|
||||
resp.read()
|
||||
except urllib.error.HTTPError as e:
|
||||
body_text = e.read().decode(errors="replace")
|
||||
raise RuntimeError(f"Failed to commit LFS pointer for '{remote_path}' (HTTP {e.code}): {body_text}") from e
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-file upload logic
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _upload_single_file(
|
||||
hub: str,
|
||||
owner: str,
|
||||
repo_name: str,
|
||||
username: str,
|
||||
password: str,
|
||||
file_path: str,
|
||||
remote_dir: str,
|
||||
message: str | None,
|
||||
branch: str,
|
||||
) -> None:
|
||||
"""Hash, upload (if needed), and commit the LFS pointer for one file."""
|
||||
filename = os.path.basename(file_path)
|
||||
file_size = os.path.getsize(file_path)
|
||||
size_mb = file_size / (1024 * 1024)
|
||||
|
||||
click.echo(f"\n {filename} ({size_mb:.1f} MB)")
|
||||
|
||||
click.echo(" Hashing...", nl=False)
|
||||
oid, size = _hash_file(file_path)
|
||||
click.echo(f" sha256:{oid[:12]}...")
|
||||
|
||||
try:
|
||||
batch = _lfs_batch(hub, owner, repo_name, [{"oid": oid, "size": size}], username, password)
|
||||
except RuntimeError as e:
|
||||
click.echo(f"\n Error: {e}", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
objects = batch.get("objects", [])
|
||||
if not objects:
|
||||
click.echo(" Already in LFS — skipping upload.")
|
||||
else:
|
||||
obj = objects[0]
|
||||
if "error" in obj:
|
||||
err_msg = obj["error"].get("message", "unknown error")
|
||||
err_code = obj["error"].get("code", 0)
|
||||
if err_code == 413 or "quota" in err_msg.lower() or "limit" in err_msg.lower():
|
||||
click.echo(
|
||||
f"\n Error: storage quota exceeded for this repo.\n Server: {err_msg}",
|
||||
err=True,
|
||||
)
|
||||
else:
|
||||
click.echo(f"\n Error from server: {err_msg}", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
upload_action = obj.get("actions", {}).get("upload")
|
||||
if not upload_action:
|
||||
click.echo(" Already in LFS — skipping upload.")
|
||||
else:
|
||||
href = upload_action["href"]
|
||||
up_headers = upload_action.get("header", {})
|
||||
chunks = math.ceil(size / _CHUNK)
|
||||
click.echo(f" Uploading ({size_mb:.1f} MB, {chunks} chunk{'s' if chunks != 1 else ''})...")
|
||||
try:
|
||||
_stream_upload_progress(href, up_headers, file_path, size)
|
||||
except RuntimeError as e:
|
||||
click.echo(f"\n Upload failed: {e}", err=True)
|
||||
sys.exit(1)
|
||||
click.echo(" Upload complete.")
|
||||
|
||||
verify_action = obj.get("actions", {}).get("verify")
|
||||
if verify_action:
|
||||
try:
|
||||
vreq = urllib.request.Request(
|
||||
verify_action["href"],
|
||||
data=json.dumps({"oid": oid, "size": size}).encode(),
|
||||
method="POST",
|
||||
)
|
||||
vreq.add_header("Content-Type", LFS_MEDIA_TYPE)
|
||||
vreq.add_header("Accept", LFS_MEDIA_TYPE)
|
||||
for k, v in verify_action.get("header", {}).items():
|
||||
vreq.add_header(k, v)
|
||||
with urllib.request.urlopen(vreq, timeout=15):
|
||||
pass
|
||||
except Exception:
|
||||
pass # verify is optional; non-fatal on failure
|
||||
|
||||
pointer = _lfs_pointer_text(oid, size)
|
||||
remote_path = (f"{remote_dir.rstrip('/')}/{filename}").lstrip("/") if remote_dir else filename
|
||||
commit_msg = message or f"chore: upload {filename} via ria"
|
||||
|
||||
click.echo(f" Committing pointer → {remote_path}...", nl=False)
|
||||
try:
|
||||
_commit_lfs_pointer(hub, owner, repo_name, remote_path, pointer, branch, commit_msg, username, password)
|
||||
click.echo(" done.")
|
||||
except RuntimeError as e:
|
||||
click.echo(f"\n Error: {e}", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def _stream_upload_progress(href: str, headers: dict, file_path: str, size: int) -> None:
|
||||
"""Stream file_path to href with a click progress bar."""
|
||||
parsed = urllib.parse.urlparse(href)
|
||||
host = parsed.netloc
|
||||
path_q = parsed.path + (f"?{parsed.query}" if parsed.query else "")
|
||||
|
||||
if parsed.scheme == "https":
|
||||
conn = http.client.HTTPSConnection(host, timeout=300)
|
||||
else:
|
||||
conn = http.client.HTTPConnection(host, timeout=300)
|
||||
|
||||
all_headers = dict(headers)
|
||||
all_headers.setdefault("Content-Type", "application/octet-stream")
|
||||
all_headers["Content-Length"] = str(size)
|
||||
|
||||
try:
|
||||
conn.connect()
|
||||
conn.putrequest("PUT", path_q)
|
||||
for k, v in all_headers.items():
|
||||
conn.putheader(k, v)
|
||||
conn.endheaders()
|
||||
|
||||
with click.progressbar(
|
||||
length=size,
|
||||
label=" ",
|
||||
width=40,
|
||||
show_eta=True,
|
||||
show_percent=True,
|
||||
fill_char="█",
|
||||
empty_char="░",
|
||||
) as bar:
|
||||
with open(file_path, "rb") as f:
|
||||
while True:
|
||||
chunk = f.read(_CHUNK)
|
||||
if not chunk:
|
||||
break
|
||||
conn.send(chunk)
|
||||
bar.update(len(chunk))
|
||||
|
||||
resp = conn.getresponse()
|
||||
resp.read()
|
||||
if resp.status not in (200, 201):
|
||||
raise RuntimeError(f"HTTP {resp.status}")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Command
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@click.command("upload")
|
||||
@click.argument("files", nargs=-1, required=True)
|
||||
@click.option(
|
||||
"--repo", required=True, metavar="OWNER/NAME", help="Target repository on RIA Hub (e.g. benchinnery/my-dataset)."
|
||||
)
|
||||
@click.option("--hub", default=DEFAULT_HUB, show_default=True, metavar="URL", help="RIA Hub base URL.")
|
||||
@click.option("--branch", default="main", show_default=True, help="Branch to commit the files to.")
|
||||
@click.option(
|
||||
"--path",
|
||||
"remote_dir",
|
||||
default="",
|
||||
metavar="DIR",
|
||||
help="Remote directory path inside the repo (default: repo root).",
|
||||
)
|
||||
@click.option("--message", "-m", default=None, help="Commit message (default: 'chore: upload <filename> via ria').")
|
||||
def upload(
|
||||
files: tuple[str],
|
||||
repo: str,
|
||||
hub: str,
|
||||
branch: str,
|
||||
remote_dir: str,
|
||||
message: str | None,
|
||||
) -> None:
|
||||
"""Upload large files to a RIA Hub Project via Git LFS.
|
||||
|
||||
Files are streamed directly to the repo's LFS object store — nothing is
|
||||
buffered into memory, so files of any size work. Each file creates one
|
||||
commit recording the LFS pointer.
|
||||
|
||||
\b
|
||||
Examples:
|
||||
ria upload recording.sigmf-data --repo benchinnery/my-recordings
|
||||
ria upload *.npy --repo benchinnery/my-recordings --branch main
|
||||
ria upload big.pt --repo benchinnery/models --path weights/
|
||||
"""
|
||||
# Validate repo argument
|
||||
if "/" not in repo:
|
||||
click.echo("Error: --repo must be in the form OWNER/NAME.", err=True)
|
||||
sys.exit(1)
|
||||
owner, repo_name = repo.split("/", 1)
|
||||
|
||||
# Expand and validate files
|
||||
resolved = []
|
||||
for pattern in files:
|
||||
if not os.path.isfile(pattern):
|
||||
click.echo(f"Error: '{pattern}' is not a file or does not exist.", err=True)
|
||||
sys.exit(1)
|
||||
resolved.append(os.path.abspath(pattern))
|
||||
|
||||
hub = hub.rstrip("/")
|
||||
warn_if_insecure(hub)
|
||||
username, password = resolve_credentials(hub)
|
||||
|
||||
click.echo(f"Uploading {len(resolved)} file(s) to {owner}/{repo_name} on {hub}...")
|
||||
|
||||
for file_path in resolved:
|
||||
_upload_single_file(hub, owner, repo_name, username, password, file_path, remote_dir, message, branch)
|
||||
|
||||
click.echo(f"\nAll done. {len(resolved)} file(s) uploaded to {owner}/{repo_name}.")
|
||||
|
|
@ -1,145 +0,0 @@
|
|||
"""Structured error reporting for `ria-agent register` (T2)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import urllib.error
|
||||
from io import BytesIO
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from ria_toolkit_oss.agent import cli as agent_cli
|
||||
|
||||
|
||||
def _structured(reason: str) -> bytes:
|
||||
return json.dumps({"detail": {"reason": reason}}).encode()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"reason",
|
||||
["invalid_key", "expired", "revoked", "already_consumed"],
|
||||
)
|
||||
def test_explain_maps_known_reasons(reason):
|
||||
msg = agent_cli._explain_registration_failure(403, _structured(reason))
|
||||
assert msg == agent_cli.REGISTRATION_REASON_MESSAGES[reason]
|
||||
|
||||
|
||||
def test_explain_unknown_reason_falls_through_with_code():
|
||||
msg = agent_cli._explain_registration_failure(403, _structured("brand_new_thing"))
|
||||
assert "brand_new_thing" in msg
|
||||
assert "rejected" in msg.lower()
|
||||
|
||||
|
||||
def test_explain_string_detail():
|
||||
body = json.dumps({"detail": "Forbidden"}).encode()
|
||||
msg = agent_cli._explain_registration_failure(403, body)
|
||||
assert msg == "Registration rejected: Forbidden"
|
||||
|
||||
|
||||
def test_explain_429_with_string_detail():
|
||||
body = json.dumps({"detail": "Too many attempts; try again shortly"}).encode()
|
||||
msg = agent_cli._explain_registration_failure(429, body)
|
||||
assert "rate-limited" in msg
|
||||
assert "Too many attempts" in msg
|
||||
|
||||
|
||||
def test_explain_429_with_no_body():
|
||||
msg = agent_cli._explain_registration_failure(429, b"")
|
||||
assert "rate-limited" in msg
|
||||
|
||||
|
||||
def test_explain_malformed_json():
|
||||
msg = agent_cli._explain_registration_failure(500, b"<html>boom</html>")
|
||||
assert msg.startswith("HTTP 500")
|
||||
assert "boom" in msg
|
||||
|
||||
|
||||
def test_explain_empty_body():
|
||||
msg = agent_cli._explain_registration_failure(502, b"")
|
||||
assert msg == "HTTP 502: no body"
|
||||
|
||||
|
||||
def _http_error(status: int, body: bytes) -> urllib.error.HTTPError:
|
||||
return urllib.error.HTTPError(
|
||||
url="http://hub/screens/agents/register",
|
||||
code=status,
|
||||
msg="",
|
||||
hdrs=None, # type: ignore[arg-type]
|
||||
fp=BytesIO(body),
|
||||
)
|
||||
|
||||
|
||||
def test_user_agent_is_set_and_not_python_default():
|
||||
"""Cloudflare on `riahub.ai` returns 403 code 1010 to `Python-urllib/*`.
|
||||
|
||||
Guarding the UA explicitly is the entire point of the register-flow fix;
|
||||
if this test ever breaks, the production bug is back.
|
||||
"""
|
||||
ua = agent_cli._user_agent()
|
||||
assert ua, "User-Agent must not be empty"
|
||||
assert not ua.lower().startswith(
|
||||
"python-urllib"
|
||||
), f"User-Agent must not be Python's default (got {ua!r}) — Cloudflare blocks it"
|
||||
assert ua.startswith("ria-agent/")
|
||||
|
||||
|
||||
def test_register_request_carries_explicit_user_agent(tmp_path):
|
||||
"""Capture the outbound urllib Request and verify the UA header is set."""
|
||||
cfg_path = tmp_path / "agent.json"
|
||||
captured: dict = {}
|
||||
|
||||
def _fake_urlopen(req, *args, **kwargs):
|
||||
# urllib normalizes header names; get_header takes the title-cased form.
|
||||
captured["ua"] = req.get_header("User-agent")
|
||||
captured["api_key"] = req.get_header("X-api-key")
|
||||
captured["timeout"] = kwargs.get("timeout")
|
||||
raise urllib.error.HTTPError(
|
||||
url=req.full_url,
|
||||
code=403,
|
||||
msg="",
|
||||
hdrs=None, # type: ignore[arg-type]
|
||||
fp=BytesIO(_structured("invalid_key")),
|
||||
)
|
||||
|
||||
with (
|
||||
patch.dict("os.environ", {"RIA_AGENT_CONFIG": str(cfg_path)}, clear=False),
|
||||
patch("urllib.request.urlopen", side_effect=_fake_urlopen),
|
||||
patch.object(
|
||||
sys,
|
||||
"argv",
|
||||
["ria-agent", "register", "--hub", "http://hub:3005", "--api-key", "ria_reg_x"],
|
||||
),
|
||||
):
|
||||
with pytest.raises(SystemExit):
|
||||
agent_cli.main()
|
||||
|
||||
assert captured["ua"], "User-Agent header was not sent"
|
||||
assert not captured["ua"].lower().startswith("python-urllib")
|
||||
assert captured["api_key"] == "ria_reg_x"
|
||||
assert captured["timeout"] is not None, "register must pass a timeout to urlopen"
|
||||
|
||||
|
||||
def test_register_surfaces_reason_on_http_error(tmp_path, capsys):
|
||||
cfg_path = tmp_path / "agent.json"
|
||||
err = _http_error(403, _structured("revoked"))
|
||||
|
||||
with (
|
||||
patch.dict("os.environ", {"RIA_AGENT_CONFIG": str(cfg_path)}, clear=False),
|
||||
patch("urllib.request.urlopen", side_effect=err),
|
||||
patch.object(
|
||||
sys,
|
||||
"argv",
|
||||
["ria-agent", "register", "--hub", "http://hub:3005", "--api-key", "ria_reg_x"],
|
||||
),
|
||||
):
|
||||
with pytest.raises(SystemExit) as exc:
|
||||
agent_cli.main()
|
||||
|
||||
assert exc.value.code == 1
|
||||
captured = capsys.readouterr()
|
||||
assert "revoked" in captured.err.lower()
|
||||
assert "Settings → RIA Agents" in captured.err
|
||||
# Config must NOT be written on failure.
|
||||
assert not cfg_path.exists()
|
||||
|
|
@ -199,44 +199,3 @@ def test_annotation_to_sigmf_format_values():
|
|||
values = list(result.values())
|
||||
assert 50 in values or ann.sample_start in values
|
||||
assert 100 in values or ann.sample_count in values
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# None freq-edge regression tests (SigMF optional fields)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_annotation_no_freq_edges():
|
||||
ann = Annotation(sample_start=0, sample_count=10, label="burst")
|
||||
assert ann.freq_lower_edge is None
|
||||
assert ann.freq_upper_edge is None
|
||||
|
||||
|
||||
def test_annotation_is_valid_no_freq_edges():
|
||||
ann = Annotation(sample_start=0, sample_count=10, label="burst")
|
||||
assert ann.is_valid() is True
|
||||
|
||||
ann_zero = Annotation(sample_start=0, sample_count=0, label="burst")
|
||||
assert ann_zero.is_valid() is False
|
||||
|
||||
|
||||
def test_annotation_overlap_none_edges_returns_zero():
|
||||
ann1 = Annotation(sample_start=0, sample_count=10)
|
||||
ann2 = Annotation(sample_start=0, sample_count=10, freq_lower_edge=0, freq_upper_edge=100)
|
||||
assert ann1.overlap(ann2) == 0
|
||||
assert ann2.overlap(ann1) == 0
|
||||
|
||||
|
||||
def test_annotation_area_none_edges_returns_zero():
|
||||
ann = Annotation(sample_start=0, sample_count=10, label="burst")
|
||||
assert ann.area() == 0
|
||||
|
||||
|
||||
def test_annotation_to_sigmf_omits_freq_keys_when_none():
|
||||
from sigmf import SigMFFile
|
||||
|
||||
ann = Annotation(sample_start=0, sample_count=10, label="burst")
|
||||
result = ann.to_sigmf_format()
|
||||
metadata = result["metadata"]
|
||||
assert SigMFFile.FLO_KEY not in metadata
|
||||
assert SigMFFile.FHI_KEY not in metadata
|
||||
|
|
|
|||
|
|
@ -189,21 +189,3 @@ def test_sigmf_3(tmp_path):
|
|||
to_sigmf(recording=recording1, path=tmp_path, filename=filename.name)
|
||||
except IOError as e:
|
||||
assert str(e) == "File already exists"
|
||||
|
||||
|
||||
def test_sigmf_annotation_without_freq_edges(tmp_path):
|
||||
# Regression: annotations that omit the optional SigMF freq edge fields must
|
||||
# load without error; edges should be None and the annotation still valid.
|
||||
ann = Annotation(sample_start=0, sample_count=5, label="burst")
|
||||
recording1 = Recording(data=complex_data_1, metadata=sample_metadata, annotations=[ann])
|
||||
|
||||
filename = tmp_path / "test"
|
||||
to_sigmf(recording=recording1, path=tmp_path, filename=filename.name, overwrite=True)
|
||||
recording2 = from_sigmf(filename)
|
||||
|
||||
assert len(recording2.annotations) == 1
|
||||
loaded = recording2.annotations[0]
|
||||
assert loaded.freq_lower_edge is None
|
||||
assert loaded.freq_upper_edge is None
|
||||
assert loaded.is_valid() is True
|
||||
assert loaded.label == "burst"
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user