Computer vision for maintenance

Two kinds of images

Computer vision enters maintenance from two directions. The first converts sensor signals into images so that image models can classify them. The second analyses real photographs and thermal images of the equipment. Both rely on the same core tool: the convolutional neural network (CNN).

Signals as images

Spectrograms (short-time Fourier transform)
Scalograms (continuous wavelet transform)
Envelope spectrograms, order maps
Recurrence plots, Gramian angular fields

Images of the asset

Thermal (infrared) images: hot bearings, loose electrical joints
Visual inspection: corrosion, cracks, leaks, belt wear
Drone and robot inspection of lines, tanks, blades
Gauge and indicator reading

Why turn a signal into a picture?

A spectrogram shows frequency (vertical) over time (horizontal), with colour for amplitude. Faults leave visual signatures: unbalance is a bright steady line at running speed, misalignment adds a line at twice that, and a bearing defect produces evenly spaced vertical streaks of high-frequency energy, one per impact. An image classifier learns to recognise those shapes in the same way it learns edges and textures in photographs.

Classical baseline · analyst reads the spectrum

A trained analyst looks at the spectrum and waterfall plots and recognises patterns by eye. CNNs automate that pattern recognition. They do not replace the physics: the image still has to contain the information, which depends on sample rate, window length and frequency range.

How a convolution sees a fault

A convolution slides a small filter (here 3 × 3) across the image and records how well each patch matches it. The result is a feature map that lights up wherever the pattern appears. A trained CNN learns hundreds of such filters; the two below are hand-set so you can see the principle.

Convolution on a spectrogramInteractive · animated

Try it: with a bearing defect, the vertical-streak filter lights up at every impact. Switch to unbalance and the horizontal-line filter responds to the strong running-speed line at the bottom. The class probabilities come from pooling the strongest responses, which is what the final layers of a CNN do with its learned filters.

A typical CNN pipeline

Signal to class probabilitiesIllustration

Early layers detect simple shapes such as lines and streaks. Deeper layers combine them into fault patterns. The dense layer turns the pooled responses into probabilities for each class.

Window the signal into fixed lengths, for example 0.5 s.
Transform to a spectrogram or scalogram, often log-scaled, and resize to a fixed image size such as 64 × 64 or 224 × 224.
Normalise per machine so that sensor sensitivity does not dominate.
Train a small CNN from scratch, or fine-tune a pre-trained one (transfer learning).
Explain predictions with saliency or Grad-CAM maps that highlight which region of the image drove the decision.

Transfer learning when labels are scarce

Networks such as ResNet or EfficientNet, pre-trained on millions of photographs, already know how to detect edges, lines and textures. Fine-tuning only the last layers on a few hundred labelled spectrograms often beats training from scratch. The same idea transfers knowledge between machines: pre-train on a lab rig with many seeded faults, then fine-tune on the plant asset with a handful of real examples.

Thermal and visual inspection

For images of the equipment itself, three task types cover most needs:

Task	Question	Model type	Maintenance example
Image classification	Is there a defect in this image?	CNN, vision transformer	Corroded vs clean flange
Object detection	Where are the defects?	YOLO-style detectors	Locate leaking valves on a drone image
Segmentation	Which pixels are defective, and how much area?	U-Net and similar	Measure corrosion area on a tank wall

Thermal images add a useful twist: the temperature of each pixel is a measurement, so a model can combine the image with a rule such as "terminal 15 K hotter than its neighbours". Always record emissivity, distance and load at the time of the image, because they change apparent temperature.

Key takeaways

Spectrograms and scalograms turn signals into images that CNNs classify by visual pattern.
Convolution filters act as pattern detectors; deep layers combine them into fault signatures.
Transfer learning from pre-trained networks reduces the need for labelled data.
For photos and thermal images, choose classification, detection or segmentation by the question you need answered.

Try it in the apps

The Spectrogram Image Classifier trains a small CNN in your browser on spectrogram images and shows a saliency map of what it looked at.

Open Spectrogram Classifier