Building Reliable RF Anomaly Detection

During my internship as a Machine Learning Engineer at Palitronica, I worked across Anvil’s RF-based electronics assurance pipeline. Here, I’ll focus on two projects: calibrating a one-class anomaly-detection model in the inference framework and building historical tracking around an existing RF data-drift calculation. Both dealt with how new measurements are interpreted relative to a reference. Calibration refined the detector’s operating point using labeled examples, while drift tracking made changes in incoming measurement groups visible over time.

Electronics assurance through RF measurements

Palitronica, a YC W22 company, develops technology for verifying the physical integrity of electronics. Its Anvil Checkpoint platform uses radio-frequency reflectometry and machine learning to compare a device’s response with a nominal (known-good) profile. The public product description covers components, assemblies, and complete modules, with measurements taken through existing connectors or a test fixture. This black-box approach supports testing when the purchaser has limited access to the device’s schematics, bill of materials, or firmware.

That capability matters when electronics arrive through a third-party supply chain. A component substitution, an undocumented board revision, or tampering can change the physical device while its part number and apparent function remain familiar. Palitronica describes supply-chain assurance applications in incoming inspection, supplier validation, supply-chain security, and aerospace and defense electronics. Government procurement contexts present similar assurance needs for purchased hardware and mission-critical systems; the company’s public CanadaBuys profile describes screening incoming commercial off-the-shelf electronics and assemblies. These are public use cases for the technology, rather than descriptions of customers or deployments I worked on.

An RF measurement captures how a device responds across a range of frequencies. Reflected and transmitted signals provide different views of its electrical structure, and a physical change can alter that response. Nominal devices also vary, so a useful verification system has to distinguish ordinary variation from departures that deserve investigation.

From RF traces to NND within Anvil’s inference framework

Anvil’s feature pipeline represents captures across RF paths, signal representations, and frequency bins. After feature extraction, the values associated with a particular path form a feature vector. A path can correspond to a reflection port or a transmission port combination, providing a distinct view of the device and its nominal variation.

The inference framework supported several anomaly-detection models for interpreting those features. One of the models I worked on was the Normal-distribution-based Novelty Detector (NND), which represented nominal measurements with a multivariate distribution and scored new measurements using Mahalanobis distance. The calibration project described here focused on this detector within the broader Anvil ML stack.

detector: nnd
split_by: rf_path

NND learned its reference distribution from nominal measurements in a one-class setup. For each feature split, corresponding here to an RF path, it fitted a multivariate nominal model with a mean and covariance structure. Samples labeled Nominal and Anomaly were available for validation, while the underlying nominal model was fitted using the nominal group.

For path $i$, the squared Mahalanobis distance of a feature vector $x$ is

where $\mu_i$ is the fitted nominal mean, $\Sigma_i$ is its covariance matrix, and $\Sigma_i^{+}$ denotes the covariance pseudoinverse. I use $q_i$ throughout to keep the squared distance distinct from its square root.

NND’s covariance accounts for the direction of a departure as well as its size. A feature can move substantially along a direction that varies naturally among nominal captures, while a smaller movement across a tightly concentrated direction can be more unusual. Correlations between features also matter: several frequency bins can move together as part of ordinary variation.

[ 1.63  0.95 ]
[ 0.95  0.79 ]

Why calibrate a statistically derived boundary?

The existing NND implementation used a statistically derived threshold based on the nominal model. That produced one distance threshold shared across the path submodels. Each path had its own fitted mean and covariance, while the decision boundary did not come from labeled examples.

This gave the one-class detector a principled decision boundary, but it did not use labeled Nominal and Anomaly examples to choose its operating point. Those validation examples still described classification behavior: which nominal samples fell beyond the boundary and which anomaly samples remained inside it.

My NND tuning work added a calibration step that evaluated the labeled distances separately for each split and selected a threshold for each one. The fitted nominal model continued to supply the distance; calibration supplied the decision boundary. This made it possible to express the desired false-positive/false-negative tradeoff using observed validation behavior.

At inference time, the Mahalanobis distance is compared with the calibrated threshold to determine whether the sample remains within the nominal boundary.

Choosing a threshold when the classes separate

For one path, let $N$ contain the squared distances of labeled nominal examples and $A$ contain those of labeled anomaly examples. When

the observations leave a gap between the two classes. The tuner places the threshold within that gap using a configurable interpolation position $g$:

An interior choice, $0<g<1$, separates all observed examples. The interpolation controls the boundary’s position between the most distant nominal example and the nearest anomaly example; it does not establish how unseen samples will behave.

The exact NND boundary rule matters here. NND classifies a sample as Anomaly only when

Equality remains nominal. Consequently, the observed zero-error interval is $[\max(N),\min(A))$. The nominal endpoint is valid under this rule, while the Anomaly-class endpoint would accept the nearest anomaly observation.

Choosing a tradeoff when the classes overlap

Overlapping distances require a different decision. Lowering the threshold flags more anomalies, but it can also reject more nominal measurements. Raising it accepts more nominal variation while allowing more anomaly examples through. For the strict comparison used by NND, the relevant counts are

The tuning routine evaluated candidate boundaries and compared their observed FP/FN ratios with a configurable target. The extreme settings directly prioritized minimizing false positives or false negatives; finite targets selected the candidate closest to the requested FP/FN ratio. These are counts on the supplied labeled data, so the balance of nominal and anomaly examples affects the ratio.

After selecting a candidate operating point, the routine placed the boundary in an adjacent interval between observations. An interior threshold preserves a classification partition while avoiding unnecessary coincidence with a sample value. When two differently labeled examples have exactly the same distance, threshold movement alone cannot separate them, so the strict comparison must apply consistently to both.

This calibration changes which mistakes NND prefers on the available validation data. Its performance on future measurements still depends on how representative those examples are. The useful outcome of the project was an explicit, deterministic calibration mechanism, rather than an assumed universal improvement in production error rates.

Within NND, the architectural change connected calibration to the existing path submodels. Before calibration, NND used a shared threshold across those submodels. Calibration allowed each RF path to use a threshold informed by its own validation behavior. A threshold still belongs to the path whose fitted mean and covariance produced its distances.

Following the measurement distribution over time

NND calibration refined how individual measurements were classified, while the second project looked at how groups of incoming RF measurements changed relative to a reference over time. A detector can retain a well-defined threshold while the data reaching it gradually shifts, making that history useful context for interpreting its behavior.

The existing drift calculation already compared two groups of RF-derived features and summarized their separation for each RF path. Unlike NND, which evaluates an individual measurement against a fitted nominal distribution, drift compares groups using their average responses and observed variation. My contribution was the tracking system around that calculation: identifying new measurement groups, resolving suitable references, reconstructing comparable inputs, and retaining the results over time.

The drift metric increases as the average RF responses separate relative to their spread, with larger departures contributing more strongly. It is not an anomaly probability or universal acceptance threshold, and its interpretation depends on comparing compatible RF paths, representations, and frequency grids.

Building a reproducible history

Turning a one-shot drift calculation into a monitoring system raised a different set of engineering questions:

• Which new datasets should be evaluated?
• Which historical data should be considered the reference?
• How do you ensure both groups use compatible feature extraction?
• What happens if the tracker misses several days?
• Can somebody reproduce a suspicious drift point later?

I built a tracker around these concerns. The prototype recorded how far evaluation had progressed, evaluated new data incrementally, and could replay missed daily windows to preserve the temporal meaning of the history.

Each comparison also needed a suitable reference dataset and compatible feature extraction. Because a drift value depends on both, the tracker preserved enough provenance to reconstruct how each historical point was produced.

The early implementation kept tracker state and outputs in a form that made replay and reference reconstruction easy to test. The resulting drift metrics were later integrated into Prometheus and Grafana, giving the team a monitoring view of RF drift over time.

Validating the tracking workflow

I built helper tooling to inspect historical collections and models, find device/setup groups with usable data, and replay the tracker with a controlled notion of the present. Replay used separate state so historical runs could advance without overwriting the active tracker state.

Another helper reconstructed baseline and current feature arrays for a selected comparison and ran the same drift calculation directly. This provided a way to inspect the inputs independently of the window-discovery workflow. I also verified that historical comparisons could be reconstructed from their stored provenance.

These tools made the surrounding system easier to inspect. A suspicious history point could be traced back to its inputs, while a missing point could be investigated through collection availability, reference selection, and retrieval. The historical value became something the team could examine, rather than a scalar detached from the measurements that produced it.

What I took away from the work

These two projects improved different parts of the same measurement pipeline. Threshold tuning used labeled examples to make NND’s operating point explicit for each RF path, while drift tracking connected recurring group comparisons to a reproducible history and monitoring view. Their value depended on details that could easily disappear behind a single score: the exact inequality at the boundary, which threshold belonged to which RF path, the chosen reference, and the configuration used to reconstruct the data.

Working on both reinforced how much ML engineering depends on making those details inspectable. A statistical model provides a foundation, labeled calibration defines an operating choice, and historical monitoring gives the team context as measurements evolve. For electronics assurance, that connection between physical measurements, decisions, and their provenance was one of the most interesting engineering problems I worked on during the internship.