Counting Through Occlusion: Framework for Open World Amodal Counting

Counters count what they see

Open-world counting went from one model per category to models told what to count at test time, by visual exemplars (CounTR, LOCA), by text (CLIP-Count, CounTX), or by both (CountGD, the current state of the art). All of them assume the targets are mostly visible. Humans do not: we infer the donuts under the napkin from the rows around it.

Amodal counting asks for the total, visible or not. Given an image \(X_I\), exemplar boxes \(\mathcal{B}\), a text prompt \(t\) and an occlusion mask \(M_o\), the model must return

The benchmark CAPTURe tests this on regular patterns, where the hidden objects can be read off the visible ones. Real scenes are often unstructured, and nothing outside the occluder says what is under it. To cover both, the paper builds occlusion-augmented versions of two standard benchmarks, keeping their splits and annotations: FSC-147-OCC (147 categories, with disjoint training, validation and test classes) and CARPK-OCC (parking lots seen from a drone). Black rectangles, at most 256 px on a side, are centred on annotated objects until 25–35% of the instances in each image are covered.

Examples from the two benchmarks FSC-147-OCC and CARPK-OCC

Reconstruct the features, then check the attention

CountOCC builds on CountGD and adds two forms of supervision. One works in feature space: rebuild what the backbone would have seen without the occluder. The other works in attention space: make the occluded view look where the clean view looks.

Figure 2. The CountOCC architecture. At each pyramid level, the Feature Reconstruction Module (FRM) replaces corrupted occluded tokens with reconstructed ones. Visual Equivalence (VisEQ) aligns gradient-based attention maps of a teacher that sees the clean image and a student that sees the occluded one. The reconstructed features flow through CountGD's feature enhancer and cross-modality decoder, which count the visible and the occluded instances.

Feature Reconstruction Module

The Swin backbone produces features at three pyramid levels (256, 512 and 1024 channels). At each level \(\ell\), tokens outside the mask are kept as visible context \(Z_{\text{vis}}\), and every occluded position starts from a learned mask embedding. The queries first attend to each other, then to the visible context, then to the fused text and exemplar tokens \(Z_{v,t}\), which keeps the reconstruction on the right category:

The reconstructed tokens replace the occluded positions, so the decoder sees a complete feature pyramid. They are supervised by a frozen teacher that sees the clean image: on occluded positions only, a Charbonnier term, a cosine term and an \(\ell_2\) term pull the student's reconstruction \(\hat Z_S\) toward the teacher's features \(\hat Z_T\), with \(\Delta = \hat Z_S - \hat Z_T\) and weights \((\lambda_{\text{charb}}, \lambda_{\cos}, \lambda_{\ell_2}) = (0.3, 0.5, 1.0)\):

Visual Equivalence

Good features are not enough if the model looks in the wrong place. Inspired by SelfEQ, CountOCC computes a language-conditioned Grad-CAM map \(G\) for each view: the gradient of the matching score (the mean of the top 900 query confidences) weights the channels of each pyramid level, and the levels are blended by their gradient energy. The student's map \(G_S\) on the occluded image is pulled toward the teacher's \(G_T\) on the clean one:

The second term looks only at the region of interest where either map is confident (\(G_T + G_S \ge \tau\)): it asks for high and uniform activation there, which rules out the trivial solution of two maps that agree by both being flat.

Figure 3. The two modules. (a) FRM: learnable queries at occluded positions attend to each other, to the visible tokens and to the fused prompt, and an MLP turns them into reconstructed features. (b) VisEQ: the teacher's attention on the clean image and the student's on the occluded image are aligned by \(\mathcal{L}_{\text{sim}}\) and made confident by \(\mathcal{L}_{\text{cst}}\).

1. Stage 1 · reconstructionTrain FRM alone on FSC-147 training images with occluders drawn on the fly: 128–256 px black rectangles anchored on objects until 15–50% of them are covered. The Swin-B image and BERT text encoders stay frozen.
2. Stage 2 · attentionStarting from the best FRM checkpoint, train FRM and VisEQ together with the counting loss.
3. InferenceOnly the student runs. The teacher and VisEQ exist only in training, so CountOCC adds no cost beyond the reconstruction module.

Lower error on four occluded benchmarks

Baselines run from their official checkpoints under their own protocols, with the same occluded inputs as CountOCC. One model, trained only on FSC-147, is evaluated everywhere.

The gain is on the hidden objects

Splitting the error by instance type shows where the improvement comes from. Counts inside the mask and outside it are scored separately.

Figure 4. Density maps on occluded FSC-147 scenes. Columns: CLIP-Count, CounTX, CounTR, LOCA, CountGD and CountOCC. The baselines leave the occluded regions empty and undercount; CountOCC places density inside the masks.

What each piece does

Reconstruction and attention supervision fix different errors. FRM lowers the typical error; VisEQ removes the catastrophic ones.

Reconstructed features land where the clean ones are

Figure 5. t-SNE of features at the three pyramid levels (256, 512, 1024 channels). Red: features of the occluded image. Green: features of the clean image. Blue: reconstructions. The occluded features sit apart from the clean ones at every level. At the finest level the reconstructions overlap them almost completely, and at coarser levels they stay close though more spread out.

Limitations

• Totals, not positions. FRM reconstructs features that give the right count, but the density inside a mask need not sit exactly where the hidden objects are.
• It needs the mask. The occlusion mask is an input. Without one, CountOCC behaves like an ordinary open-world counter and cannot estimate hidden instances. Predicting the mask jointly is future work.
• Synthetic occluders. The black rectangles are simple. The gains concentrate on occluded-region error while visible-region error is unchanged, which suggests the model is not exploiting the occluder's appearance, and the CrowdHuman results use natural occlusion.
• A small cost on clean images. On unoccluded FSC-147, CountOCC stays competitive but trails CountGD (test MAE 7.02 against 5.74): a trade-off between robustness to occlusion and accuracy when everything is visible.

Citation

If this work is useful to you, please cite it as:

@article{arib2025counting,
  title   = {Counting Through Occlusion: Framework for Open World Amodal Counting},
  author  = {Arib, Safaeid Hossain and Akter, Rabeya and Chowdhury, Abdul Monaf and
             Sourov, Md Jubair Ahmed and Hasan, Md Mehedi},
  journal = {arXiv preprint arXiv:2511.12702},
  year    = {2025}
}