Choosing thresholds for SCADA data quality checks: gaps, frozen values, out-of-range

Finding errors in SCADA data is easy; deciding what counts as an error is the hard part. Set the threshold too tight and normal behaviour is flagged as an error; too loose and bad data enters the model. The note Gaps and bad records in SCADA data describes the error types; this note focuses on how to choose the thresholds.

1. Know the logging interval first

The only input to a coverage calculation is the expected record count, and that comes from the logging interval. With 10-minute averages you expect 144 records a day, 4,320 over 30 days. If the file has 4,104 rows, coverage is 4,104 / 4,320 = 95.0%, so 216 records (about 36 hours) are missing. Assuming the wrong interval ruins coverage from the start: read hourly data as 10-minute data and it looks 100% "missing".

2. Gaps: how long becomes a problem?

A single missing record is unimportant in most analyses; a long outage matters. Think of the threshold as a multiple of the logging interval: for example, in 10-minute data a spacing larger than 1.5 times the interval between two consecutive records can count as a gap, but a longer threshold (say, several hours) can be chosen for a "problem gap". Set the threshold by use: a short one for instant alarms, a longer gap is acceptable for daily energy calculations. Never fill a gap with zero or with the last value.

3. Frozen values: window length

A frozen value is the value staying exactly the same over N consecutive records. Choose N by the nature of the signal. On a continuously varying channel such as wind speed or vibration, 6 identical values in 10-minute data (1 hour) is suspicious. On the other hand, the power of a stopped machine or a fixed setpoint may legitimately be constant: there a frozen value is a state, not an error. So set the threshold per channel and assess stopped periods separately.

4. Out of range: physical limits

Derive the limit from physics, not from the data itself. Examples: direction 0-360 degrees, relative humidity 0-100, speed cannot be negative. For fields like temperature, keep the limit wide and catch only impossible values (a bearing temperature below ambient, for instance, is outside this and is a machine-specific rule). Values that are physically possible but unusual are not errors but statistical outliers; they are examined in a separate step.

5. How to read the result

• Look at coverage per channel and the share of problem records, not at the error count.
• If gaps, freezing and out-of-range values appear together on one channel, a sensor or communication problem is likely; compare with the maintenance records.
• Flag, do not delete: a flag stating why a record is questionable lets you pick training windows and report losses honestly.

Try it with the tool

Give your own CSV file to the SCADA data quality check: the file is not uploaded to a server, it is processed in your browser. For timestamp problems the timestamp converter helps. For a first assessment with your own data, get in touch.