U-ActionNet: Dual-Pathway Fourier Networks with Region-of-Interest Module for Efficient Action Recognition in UAV Surveillance

Why drone footage breaks ground-camera models

Action recognition has been built on footage from cameras at eye level, where the person is large, upright and centred. Seen from a drone, the same person is a few dozen pixels in a frame dominated by ground, and a network trained on it learns more about the background than about the actor. The drop is not subtle:

Crop to the people, then look for motion in frequency

Figure 2. The server-side U-ActionNet. Object localisation (green) crops each clip to its people, a modified C3D (red) extracts features, two Fourier blocks (blue) run in parallel on them, and two dense layers (yellow) classify.

1 · Region of interest

Every clip goes through YOLOv8x trained on COCO. Only person detections are kept, and a person becomes the region of interest when they are detected in 32 consecutive frames with a confidence above 0.60. The crops are resized to 112 × 112 and stitched back into a clip, so the network sees the 10–15% of the frame where the action happens. This alone cuts the memory each video takes by up to 80%.

The region-of-interest module Figure 3

Figure 3. YOLOv8 detects objects, a filter drops everything but people, the region of interest is selected, and the frames are resized to 112 × 112 × 3 to form a new clip.

2 · Modified C3D features

The backbone is C3D cut after its sixth layer: those more general features transfer better and overfit less on small datasets. The Fourier module works on them, followed by two redesigned dense layers with dropout 0.5. Both Fourier branches read the same features in parallel.

3 · Fourier substance separation

Take the C3D features \(f \in \mathbb{R}^{C \times T' \times H' \times W'}\) and run a 1D FFT along time at every channel and location. A static region puts its energy at low temporal frequencies; motion shows up at high ones. Weighting the power by the squared frequency gives a dynamic mask \(D_{FS}\), and multiplying by the activations keeps what both moves and matters:

The separated features rank the four kinds of region in the order an action classifier needs, from weakest to strongest:

• 1 · static, not salient
• 2 · moving, not salient
• 3 · static, salient
• 4 · moving, salient

4 · Fourier self-attention

Self-attention over every space–time position costs \(O\big(C\,(HWT)^2\big)\). U-ActionNet instead correlates the features with themselves in the frequency domain, the way an autocorrelation does: a 2D FFT over time and space, a product with the complex conjugate, an inverse FFT, and a small scaled residual with \(\lambda_{FA} = 0.01\):

That costs \(O(C \cdot HWT \log HWT)\) and relates distant parts of the clip, such as an actor and the object they act on, without a single attention matrix. The server-side model trains with Adam at a learning rate of 0.001 for 50 epochs, with batches of 16 clips.

A version small enough for the drone

The server-side model has 63.3 million parameters, far too many for an edge device. U-ActionNet Light keeps the region-of-interest crop and the Fourier module, and replaces the rest with parts built for small hardware.

Figure 4. U-ActionNet Light. Pre-processing (region-of-interest crop and frame sampling), feature extraction (MobileNetV2 and the Fourier module), a BiLSTM that keeps temporal context, and dense layers.

1. KeyframesAfter the person crop, FFmpeg's scene-change detection with a threshold of 0.01 keeps only frames that differ from the last, cutting the frame count by up to 30%.
2. FeaturesNormalised 112 × 112 frames go through MobileNetV2, one frame at a time, with only its last ten layers fine-tuned. The same Fourier module then separates motion and adds context.
3. TimeA BiLSTM reads the frame features forwards and backwards, and dense layers with dropout and L2 regularisation (1e-4) classify. It trains with SGD at a learning rate of 0.01 for 50 epochs.

Two aerial datasets, ten actions each

Both datasets are used as ten-class subsets chosen for surveillance: fights, threats and calls for help from UAV-Human, which has 155 classes in full, and ten of the thirteen actions in Drone Action. Each is split 80/20 by video. The scores below are on these subsets and are not comparable to results on the full benchmarks.

Figure 5. Where the server-side model errs. Confusion matrices on UAV-Human (left) and Drone Action (right), in percent of each true class.

The light model against its baseline

The paper's baseline is a MobileNet and BiLSTM pipeline without keyframe sampling. U-ActionNet Light improves on it on both datasets:

Baseline → U-ActionNet Light; changes in percentage points.

From a conference paper to a journal paper

Both papers grew out of a BSc thesis in Robotics and Mechatronics Engineering at the University of Dhaka, Enhancing UAV Based Human Action Recognition: A Deep Learning Approach, supervised by Md Mehedi Hasan.

From Table I of the conference paper; the baselines use two fully connected layers on unmodified networks.

Scope: both datasets are used as ten-class subsets with small test splits (60 and 20 videos), and the method assumes at most two people acting in a clip. Extending it to many actors doing different things at once, and to better frame sampling for the light model, is left for future work.

Citation

If this work is useful to you, please cite it as:

@article{chowdhury2024u,
  title   = {{U-ActionNet}: Dual-Pathway Fourier Networks With Region-of-Interest
             Module for Efficient Action Recognition in {UAV} Surveillance},
  author  = {Chowdhury, Abdul Monaf and Imran, Ahsan and Hasan, Md Mehedi and
             Ahmed, Riad and Azad, Akm and Alyami, Salem A.},
  journal = {IEEE Access},
  volume  = {12},
  year    = {2024},
  doi     = {10.1109/ACCESS.2024.3516586}
}