FREQCOR_functions

Utility functions used across the FREQCOR processing pipeline.

Functions

ReadLUT	Read a LUT class index from a look-up table.
Simpson	Apply Simpson weighting along a 1D array.
TFsonic	Compute the sonic transfer function correction.
av_Kaimal	Function to compute the average Kaimal, Massman and ideal cospectra
check_var_tf	Function that computes the coefficient of variation of the transfer function normalisation factor and compares it to the user-defined limit.
fit_Aslan21	@author: Aslan et al. 2021 Translated by A.
fit_func	@author: Aslan et al. 2021 Translated by A.
fun_Gauss	Non-linear regression according to Gauss
fun_Lorentz	Non-linear regression according to Lorentz
fun_Lorentz_peltola	Non-linear regression according to Lorentz, applied with the square-root according to Peltola et al. 2021.
remove_outliers	Detect and remove outliers from a data series using the IQR method.
spectral_outlier_mask_iqr	Outlier detection using pointwise IQR in a frequency band.
test_and_apply_denoising	Conditionally applies the denoising procedure described by Aslan et al. 2021 to spectral data.
write_cflut_to_csv	Write correction factor LUT to an open CSV file handle.
write_coflut_to_csv	Write cut-off frequency LUT to CSV.

FREQCOR_functions.check_var_tf(trafun, jtmin, jtmax, varclim)

Function that computes the coefficient of variation of the transfer function normalisation factor and compares it to the user-defined limit. If the coefficient of variation exceeds its limit value, the transfer function is discarded and the cut-off frequency won’t be computed for that specific half-hour or wind class.

Parameters:

• trafun (TYPE) – Experimental transfer function computed as the ratio of real to ideal (co)spectra.

• jtmin (int) – Index for the lower threshold of the user-defined normalisation range.

• jtmax (int) – Index for the upper threshold of the user-defined normalisation range.

• varclim (float) – User-defined limit for the coefficient of variation of the normalisation factor. If exceeded, the transfer function is discarded.

Returns:

trafun – Experimental transfer function after potential filtering performed according to the results of the check of the coefficient of variation against the set limit. Is “False” if it didn’t pass the check.

Return type:

TYPE

FREQCOR_functions.fun_Lorentz(x, a, c)

Non-linear regression according to Lorentz

FREQCOR_functions.fun_Gauss(x, a, c)

Non-linear regression according to Gauss

FREQCOR_functions.fun_Lorentz_peltola(x, a, c)

Non-linear regression according to Lorentz, applied with the square-root according to Peltola et al. 2021

FREQCOR_functions.av_Kaimal(Icos, freq, freqn, zL, A0, kf0, mu)

Function to compute the average Kaimal, Massman and ideal cospectra

Parameters:

• Icos (pd.DataFrame) – [nfreq,nspec_f]: contains 1/2 hour (co)spectral data of sonic temperature, already filtered for best data quality.

• freq (pd.Series) – [nfreq,]: natural frequency data, in log-spaced bins.

• freqn (pd.DataFrame) – [nfreq,nspec_f]: normalised frequency data, in log-spaced bins, for each available 1/2 hour also present in the “Icos” DataFrame.

Returns:

grouped_data – [nfreq,3]: DataFrame with newly log-binned frequencies and averaged values for the ideal, Kaimal and Massman cospectra

Return type:

pd.DataFrame

FREQCOR_functions.ReadLUT(Var, LUT, mxcl, gss, var0=None)

Read a LUT class index from a look-up table.

This helper supports two cases: - 1D LUT access when var0 is None. - 2-level LUT access (first by WD/RH, then by wind speed) when var0 is given.

Parameters:

• Var (float) – Query value for the second-level variable (typically wind speed).

• LUT (pandas.DataFrame or dict) – Look-up table.

• mxcl (int) – Number of classes for 1D LUT access (ignored for 2-level access).

• gss (int) – Gas species selector.

• var0 (float, optional) – First-level query value (wind direction for CO2 or relative humidity for H2O).

Returns:

• VarI (int or float) – Index of the selected wind-speed class (0-based). Returns NaN if inputs are not finite.

• clrh (int or float) – Selected first-level class key (WD/RH class). Returns NaN when var0 is None or not finite.

FREQCOR_functions.Simpson(nonweighted)

Apply Simpson weighting along a 1D array.

Parameters:

nonweighted (numpy.ndarray or pandas.Series) – Input array.

Returns:

weighted – Weighted array computed as (f[i-1] + f[i+1] + 4*f[i]) / 6.

Return type:

numpy.ndarray

FREQCOR_functions.TFsonic(freq, ws)

Compute the sonic transfer function correction.

Parameters:

• freq (numpy.ndarray or pandas.Series) – Natural frequencies (Hz).

• ws (float) – Mean wind speed (m s-1).

Returns:

TFs – Transfer function values.

Return type:

numpy.ndarray

FREQCOR_functions.spectral_outlier_mask_iqr(df, freq, fmin=0.005, fmax=0.1, k=2.5, min_fraction=0.15)

Outlier detection using pointwise IQR in a frequency band.

FREQCOR_functions.remove_outliers(data, class_id, ws_values, outlier_info=None)

Detect and remove outliers from a data series using the IQR method. Optionally stores information about removed outliers in outlier_info dict. Returns filtered data (same type as input), and optionally updates outlier_info.

FREQCOR_functions.write_coflut_to_csv(data_dict, gss, outputpath, run_tag=None)

Write cut-off frequency LUT to CSV.

Parameters:

• data_dict (dict) – LUT structure as returned by FREQCOR_LUT_cof.

• gss (int) – Gas species selector.

• outputpath (str) – Output directory.

• run_tag (str, optional) – Run identifier used to generate output filenames.

Raises:

ValueError – If run_tag is missing.

Returns:

Writes 4_LUT_cof__{run_tag}.csv.

Return type:

None

FREQCOR_functions.write_cflut_to_csv(outfile, data_dict, gss)

Write correction factor LUT to an open CSV file handle.

Parameters:

• outfile (io.TextIOBase) – Open file handle in text write mode.

• data_dict (dict) – LUT structure as returned by FREQCOR_LUT_CF.

• gss (int) – Gas species selector.

Returns:

Writes CSV blocks to outfile.

Return type:

None

FREQCOR_functions.fit_func(freq_binned, a, b, d, PS_original)

@author: Aslan et al. 2021 Translated by A. Faurès from Matlab to Python on 21/10/2024

Function to fit power spectra. The function is defined as the sum of an attenuated turbulent signal component and an independent white noise compontent.

Parameters:

• freq_binned (numpy array) – Natural frequency, binned.

• a (float64) – Low-pass filtering time constant (s).

• b (float64) – Transfer function normalisation factor.

• d (float64) – Intercept of white noise line with the y-axis.

• PS_original (numpy array) – Binned power spectra of sonic temperature (unattenuated, noise-free). Is in the form: f*sp(T)/var(T)

Returns:

Attenuated and noise-contaminated power spectra.

Return type:

Numpy array

FREQCOR_functions.fit_Aslan21(f, ideal_sp, real_sp, par_bounds, ini_par)

@author: Aslan et al. 2021 Translated by A. Faurès from Matlab to Python on 21/10/2024

Function to fit the equation developed by Aslan et al. 2021 to any spectral case and dataset.

Parameters:

• f (numpy array) – Natural frequency, binned.

• ideal_sp (numpy array) – Binned power spectra of sonic temperature (unattenuated, noise-free). Is in the form: f*sp(T)/var(T)

• real_sp (numpy array) – Binned power spectra of tracer of interest (attenuated, noisy). Is in the form: f*sp(c)/var(c)

• lim (list of float64) – Thresholds for fitting [Hz]. In the form [Low_lim, Up_lim]

• ini_par (list of float64) – Starting points for a, b, d parameters. In the form [a0, b0, d0] Reminder: a: filter time constant (s); b: normalisation factor; d: intercept of white noise line with y-axis.

Returns:

• popt (numpy array) – a, b, d parameters optimised (least-squares).

• PSA_A21 (numpy array) – Binned power spectra modelled with optimised parameters.

• r_squared (float64) – Goodness of fit (calculated from residuals).

• noise (numpy array) – Calculation of the noise term as f*d.

FREQCOR_functions.test_and_apply_denoising(Rcos, Icos, freq, trafun, tfmin, tfmax, fit_Aslan21, bounds)

Conditionally applies the denoising procedure described by Aslan et al. 2021 to spectral data.

The function:

1. Determines a frequency threshold (3 Hz if max(freq) <= 5 Hz, else 5 Hz).

2. Performs a linear regression on log-log(f * spectrum/variance) above the threshold.

3. If the slope of the regression is > 0.2, applies the denoising fit using fit_Aslan21.

4. Returns the denoising results and the (possibly denoised) transfer function.

Parameters:

• Rcos (array-like or pd.Series) – Real (measured) (co)spectrum.

• Icos (array-like or pd.Series) – Ideal (reference) (co)spectrum.

• freq (array-like or pd.Series) – Frequency vector (Hz).

• trafun (array-like or pd.Series) – Transfer function (real/ideal cospectra ratio).

• tfmin (int) – Index for the lower bound of the fit frequency range.

• tfmax (int) – Index for the upper bound of the fit frequency range.

• fit_Aslan21 (callable) – Function to perform the Aslan et al. 2021 denoising fit.

• bounds (tuple of lists) – Bounds for the fit parameters.

Returns:

• denoising_out (dict) –

  Dictionary with denoising status and results. Contains keys:

  • ’active’: bool, True if denoising was performed.

  • ’sp_re_denoised’: array, denoised spectrum (if denoising performed).

  • ’x’: array, frequency vector used in denoising (if performed).

• y_result (array or None) – The denoised transfer function if denoising was performed, otherwise None.

Notes

• Denoising is only performed if the slope of the log-log regression is > 0.2.

• If denoising fails, denoising_out[‘active’] is False and y_result is None.