spike_detection_vickers97.spike_detection_vickers97

spike_detection_vickers97.spike_detection_vickers97(data: ndarray, spike_mode: int = 1, max_pass: int = 10, avrg_len: int = 30, ac_freq: int = 10, spike_limit: float = 3.5, max_consec_spikes: int = 3, ctrplot: bool = False) → Tuple[ndarray, ndarray][source]

Detect and remove spikes in high-frequency eddy covariance data.

This function implements the Vickers and Mahrt (1997) despiking algorithm for eddy covariance data. It uses an iterative moving window approach to identify outliers based on local statistics. The algorithm can either flag spikes or both flag and remove them through linear interpolation.

Parameters:

  • data (numpy.ndarray) – 1D input data array containing high-frequency measurements (e.g., wind components, scalar concentrations)

  • spike_mode ({1, 2}, optional) – Operation mode: - 1: Only detect spikes - 2: Detect and remove spikes via linear interpolation Default is 1

  • max_pass (int, optional) – Maximum number of iterations for spike detection. Each pass may use progressively larger thresholds. Default is 10

  • avrg_len (int, optional) – Averaging period length in minutes. Used to determine the window size for local statistics. Default is 30

  • ac_freq (int, optional) – Data acquisition frequency in Hz. Used to calculate the number of samples in each window. Default is 10

  • spike_limit (float, optional) – Initial threshold for spike detection in standard deviations. Points exceeding mean ± (spike_limit × std) are flagged. Default is 3.5

  • max_consec_spikes (int, optional) – Maximum number of consecutive points that can be flagged as spikes. Longer sequences are not considered spikes. Default is 3

  • ctrplot (bool, optional) – If True, generates diagnostic plots showing: - Original data with detected spikes - Cleaned data with interpolated values Default is False

Returns:

  • data_out (numpy.ndarray) – If spike_mode=1: Copy of input with spikes still present If spike_mode=2: Data with spikes replaced by linear interpolation

  • is_spike (numpy.ndarray) – Boolean array same length as input, True where spikes were detected

Notes

The algorithm follows these steps: 1. Divides data into overlapping windows 2. Calculates local mean and standard deviation 3. Flags points exceeding threshold as potential spikes 4. Checks for consecutive outliers 5. Optionally interpolates across spike locations 6. Repeats with adjusted threshold if spikes found

The window advancement step is currently set to 100 samples, which differs from both the original VM97 paper (1 sample) and the EddyPro manual recommendation (half window size).

See also

linear_interpolate_spikes

Function used to replace detected spikes

References

Examples

>>> # Generate sample data with artificial spikes
>>> import numpy as np
>>> data = np.random.normal(0, 1, 18000)  # 30 min at 10 Hz
>>> data[1000:1002] = 10  # Add artificial spikes
>>> cleaned, spikes = spike_detection_vickers97(
...     data, spike_mode=2, ctrplot=True
... )
>>> print(f'Found {np.sum(spikes)} spikes')

Author

Written by Bernard Heinesch University of Liege, Gembloux Agro-Bio Tech