How to detect a partial discharge in a noisy signal?
UPDATE  Imagine this is a  $1k competition to a $trillion global power infrastructure problem with ageing grid transformers and switchgear that can explode from dissolved combustible gasses from H2 to Acetylene that requires your data analysis skills.

Here you can see a unique impulse (PD) response of the system due to PD events in high voltage

After all the denoising and event identification when you count repetition rate of events and map a histogram of phase coherence you can readily see it is somewhat stochastic (quasi-random with histogram peaks near 50and 230 deg and in this more on the negative cycle. (230deg)



Design Goal


Optimal Receiver design theory.
Take the spectrum of each data stream with an FFT 


use the one without PD as the spectral shape for your Bandstop filter.
use the one with PD to compare the difference and use the difference as your Bandpass Filter.

choose a 10th order Bessel filter for bandstop and band pass separately.


compute what your spectral SNR becomes and see if it is > 30 dB for good error rates.



PD can be over 1GHz and even in the optical, UV and higher spectrum depending on the bandwidth of your sensors and how close to the PD activity it is. After all PD is the result of detonation of a tiny void of homogeneous dielectric that may have a slightly lower dielectric constant and possible a lower breakdown voltage BDV.







simulate this circuit &ndash; Schematic created using CircuitLab

There are hundreds of ways to detect PD and to do de-noising and thousands of research articles on the web to demonstrate this but the channel spectrum of "Noise" must be characterized 1st so that the PD signal can be extracted by some wavelet definitions with line phase-PD amplitude histograms.

But as I understand this is a statistical analysis competition, not a measurement competition.  The data is poor here, which makes it a challenge.

Below is a PD event I captured on 33kV 5MVA transformer in a factory without power ON doing an offline Hipot PDIV test to the primary bushing using a wire loop around the Ground strap to a long coax going into a 50 Ohm load on the LeCroy scope.

The PDIV was below normal operating voltage so the unit was taken out of service in spite of perfect operational power study parameters but DGA indicated near 4% LEL after 6 mos service. 

Danger! 

The peak voltage I captured was -10.8V @ 50 Ohms using 300MHz BW with a rise time &lt; 1ns and PW50= 9 ns from a low power signal generator putting out 20kVac 60 Hz.  THis was tested on a 5MVA transformer ramping up a HiPot test to PDIV. It failed below the normal operating volge of 33kVac 60Hz.

 

Normally PD clusters around the rising edge of each half sine where dV/dt is high before peak voltage but may have other patterns which indicate different failure modes. It may start triggering 1 pulse every few minutes which is the Inception Voltage (PDIV) and can rapidly increase to several pulses per half cycle like any relaxation unijunction oscillator due to several stress factors like heat, oil pump pressure and common mode delta voltage but the Root Cause is particle size contamination in the insulation that may be invisible to the naked eye.

This is Fourier spectrum of a pulse, and the primary spectrum (1st hump) goes up to Tr=0.35/BW with a null energy at f=/t(PW50) then harmonics replicate this pattern from rise time.
 i.e. the sine wave that fits in the pulse width will have minimal energy but will have harmonics above and fundamental below.

You may only need to denoise the low frequency noise (&lt; 10MHz?)and capture the spectrum above this.



Background

What is a PD? It is a very tiny capacitance of contaminant that detonates in a vast and much larger dielectric medium. The event may be large in voltage but small in &lt;&lt;1 pF such that Q=CV is the charge being measured. The event is a very small duration (ps) of ionization in a dielectric medium but due to low pass filtering effects of the medium with inductive core and windings, the impulse imparts a lower frequency ringing unless nearby UHF antennae are used to capture the events. In Oil, the PD microburst breaks down some molecules of a large hydrocarbon chain \$H_xC_y\$ to create the gas with the lowest activation energy 1st Hydrogen, \$H_2\$. Then if the energy is larger it generates Methane and so on up into the pyrotechnic energy levels where it can generate Acetylene gas which has the highest activation energy and a precursor to grid transformer explosions. 

This is why early PD detection is so vital

In dry insulation, it tends to convert gas void microbubbles or contaminants surrounded by epoxy, mica or other plastic insulation materials and creates a carbon chain which increasing spreads the energy until it fuses out or grows with age and activation levels.

But the initial event is like a abrupt pulse discharge but the measured signal passes thru the LC network of large core and grid inductance and insulating capacitance, so what is recorded is the LC network response of ringing, that depends on the geometry core inductance and dielectric capacitance where the underdamped resonance occurs.  You could easily hear this on a SW radio nearby like lightning static with hum.

The equivalent charge is measured with calibrated PD equipment in nanoCoulombs based on the CV relative to standard test cap and standard impulse voltage.

The result is a group of varying resonant pulses that may vary widely in spectrum depending on the devive under test (DUT) and measurement sensor BW and DAQ data acquisition limitations. The best PD equipment has a spectrum > 3GHz.  The waveform at the bottom of this answer used had an old 300MHz LeCroy scope that I had but I am sure that my sensor had a bandwidth 10x higher than this, and gave a nice 1st order response, unlike so many Rogowski coils that have a resonance from poor design. So in my case, the voltage was reduced by 10 and stretched out to 10x the rise time. The lack of resonance with 50 Ohm coax and termination, indicates a clean recording.

So the de-noising of PD using auto-correlation RMS averaging or real-time FFT is not effective but by characterizing the network with FFT you can design an Optimal Receiver as below where the maximum SNR is possible.

Here is a simulation of 50 Hz with a few handmade glitches up to the 256 harmonic of 50Hz up to 15kHz. Consider this as the line noise and look for PD Signal 2~10MHz. Just use a Bandstop filter and decide where to place it so that there is no group delay in the passband.

There is NO pattern !! except stochastic groupings. It would sound like a geiger counter because a large PD event in 1 location canbe a low voltage on the next cycle as the charge builds up and when the PDIV threshold exceeds results in a large impulse but not necessarily in the same phase from the same source. 

Many PD event locations can exist like random lightning strokes so there may not be any recurring phase pulse but there may be a range of pulse repetition rates from various sources. Where lightning might be an atmospheric discharge that occurs every few seconds, Line induced PD is most stressful just after max dV/dt before peak voltage, but can still can occur later as the event is a particle being accelerated with enough kinetic energy to hit a conductor so there can be inertial effects in a fluid.  

Events can occur from rain on the outside of bushings from dust and occur with dozens of spikes per cycle but since the particle mass and voltage creates a dynamic force since water has a very high dielectric constant the contamination tends to be very low and thus charges up more rapidly with an alternating electrostatic force.

===



Let's call the line frequency &amp; load spectral noise from brushed motors and aggregate Triac powered devices as NOISE 

Let's call your desired PD spectrum = SIGNAL. 

You need to "de-noise" the data to maximize your SNR and reduce your probability of error detection. Let's choose a 30:1 SNR minimum for a target.