DETECTING INEFFICIENCIES AND ANOMALIES IN TRADE INVESTMENTS USING MACHINE LEARNING METHODS WITH THEIR SUBSEQUENT PROCESSING IN FMCG FINANCE

УДК: 004.85:336.717

Abstract

Trade promotion expenditure — volume rebates, advertising allowances, and conditional pricing programs — consumes 11–27% of revenue in Fast-Moving Consumer Goods (FMCG) firms. Yet, ERP systems offer limited native capability to detect off-contract deductions, suboptimal allocation, or post-settlement leakage. Existing anomaly-detection literature focuses on credit-card fraud and banking transactions; transferring those methods to FMCG trade-promotion data is non-trivial because legitimate transactional variation is high and audit-derived labels are partial and biased. We present a two-stage framework: an unsupervised Isolation Forest detector that outputs a binary flag and a continuous anomaly score, which are then forwarded as features to a supervised LightGBM classifier. The taxonomy is organised by information-availability logic analogous to — though not an empirical test of — the Efficient Market Hypothesis, mapping anomalies to off-contract deductions, suboptimal allocation, and undocumented arrangements. Validation uses 108,000 anonymised transactions from a multinational FMCG firm over 36 months (3 markets, 15 clients, 200 SKUs; 10,940 audited records) with rolling-origin temporal cross-validation, benchmarked against a deterministic ERP rule and a class-weighted logistic regression. The full pipeline achieved end-to-end F1 = 0.786 on rolling-origin CV; on the held-out out-of-time window (months 31–36), the classifier reached macro F1 = 0.812. The off-contract class was recovered at 85.0% recall versus 35.4% for the ERP rule — a ~50-percentage-point gain at comparable precision. The framework materially exceeds deterministic ERP rules on subtle non-rate anomalies.

Аннотация

Расходы на торговые промо-акции — объёмные скидки, рекламные надбавки и условные ценовые программы — составляют 11–27% выручки FMCG-компаний, однако ERP-системы располагают ограниченной встроенной функциональностью для выявления внеконтрактных вычетов, неоптимального распределения и пост-расчётных потерь. Большинство исследований по обнаружению финансовых аномалий ориентировано на банковский фрод; перенос этих методов на FMCG-данные нетривиален из-за высокой легитимной вариативности и частичности аудиторских меток. Представлена двухэтапная система: Isolation Forest выделяет аномальных кандидатов, чьи бинарный флаг и непрерывный аномальный балл передаются на классификатор LightGBM. Таксономия построена аналогически (но не как эмпирический тест) на логике информационной доступности Гипотезы эффективного рынка и сопоставляет аномалии трём типам: внеконтрактные вычеты, неоптимальное распределение, недокументированные коммерческие договорённости. Валидация: 108,000 анонимизированных транзакций FMCG-компании за 36 месяцев (3 рынка, 15 клиентов, 200 артикулов; 10,940 размеченных записей) с ролл-форвард кросс-валидацией, сравнение с детерминистским ERP-правилом и класс-взвешенной логистической регрессией. Полная система достигла end-to-end F1 = 0.786 на ролл-форвард CV; на отложенном out-of-time-окне (месяцы 31–36) макро-F1 составил 0.812. Класс внеконтрактных вычетов восстанавливается с recall 85.0% против 35.4% у ERP-правила — выигрыш ~50 п.п. полноты при сопоставимой точности.

Keywords: decision-support systems, anomaly detection, trade promotion management, FMCG finance, Isolation Forest, LightGBM, enterprise systems.

Ключевые слова:  системы поддержки принятия решений, обнаружение аномалий, управление торговыми акциями, финансы FMCG, Isolation Forest, LightGBM, корпоративные системы.

Introduction

Trade promotion expenditure — volume rebates, cooperative advertising allowances, listing fees, and conditional pricing tiers — is the second-largest controllable cost line for most FMCG manufacturers after cost of goods sold, typically absorbing 11–27% of gross revenue [1]. ERP platforms process the underlying transactions reliably but provide limited native analytical infrastructure for evaluating contractual compliance, promotional effectiveness, or cross-period deviation. Finance teams rely on manual reconciliation — cross-referencing retailer deduction notes against signed promotional agreements — a workflow that is reactive by design and calibrated to surface error types that have historically been caught, not those that have been missed.

Machine learning has been applied extensively to related financial anomaly problems, with tree ensembles, autoencoders, and graph neural networks showing strong performance on credit card fraud and banking transactions [2, 3, 4, 5]. Applications to FMCG trade promotion data remain sparse for three reasons. First, legitimate transactional variation is unusually high: the same client may receive substantially different discount structures across quarters, markets, and SKU categories, and generic detectors treat that variation as anomalous. Second, audit-derived labels are partial — auditors examine some categories of irregularities more thoroughly than others, so the label distribution does not reflect the true distribution of inefficiencies. Third, post-detection interpretation is largely unsolved: binary flags transfer the diagnostic burden back to the analyst, defeating the purpose of automation. SHAP [6] partially addresses local interpretability but does not route anomalies to organisational owners.

This paper addresses these constraints jointly with four contributions. First, a two-stage pipeline in which the Isolation Forest detector passes both a binary flag and a continuous anomaly score as features to the LightGBM classifier, allowing the supervised stage to weigh detector confidence rather than treat the upstream output as a hard filter. Second, domain-aware feature engineering specific to FMCG trade promotion — contracted-versus-posted rate deviation, rolling SKU-client baselines, and client-tier interactions — which carries most of the discriminative signal. Third, a diagnostic classification taxonomy is adapted analogically from the Efficient Market Hypothesis (EMH) [7]; we explicitly state that this is not an empirical test of EMH but a transfer of its information-availability logic to intra-firm data. The output classes — off-contract deduction (semi-strong analog), suboptimal allocation (weak-form analog), and undocumented arrangement (strong-form analog) — route to different organisational owners. Fourth, rolling-origin temporal cross-validation rather than stratified k-fold to avoid look-ahead leakage in a longitudinal transaction stream, benchmarked against a deterministic ERP rate-matching rule and a class-weighted logistic regression to test whether the added model complexity is operationally justified.

Materials and methods

System Architecture: The system is a three-module pipeline for on-premises deployment: (i) data ingestion and feature engineering; (ii) unsupervised candidate detection; (iii) supervised diagnostic classification (Figure 1). Two design choices are non-obvious. First, the detector's outputs are features, not a filter: all records are passed to the supervised classifier, with the detector's score and binary flag treated as input features. A naive filter design would propagate Stage-1 false negatives as permanently undetected; feature-forwarding allows the classifier to recover from misses when supervised evidence is strong. Second, a manual-review branch routes flagged records with low classifier confidence for human review — precisely the cases most likely to belong to anomaly types absent from the training labels — mitigating audit-label bias.

Figure 1. Two-stage decision-support framework. Block A: ingestion and feature engineering (23 features). Block B (Isolation Forest, Stage 1): anomaly score and binary flag. Block C (LightGBM, Stage 2): consumes the feature matrix augmented with detector outputs. Block D: routes the predicted class to its organisational owner. The dashed arrow indicates all records are forwarded to Stage 2.

Feature Engineering: Five feature groups were constructed from raw SAP transactional records: (1) contracted-vs-posted rate deviation (continuous, percentage and absolute terms): the gap between the agreed and applied discount rates; (2) settlement lag (continuous, days): time between invoice posting and deduction settlement; (3) client tier (categorical, 4 levels: Strategic / Key / Regional / Other); (4) rolling 3-month SKU-client baseline deviation (continuous): deviation of the current rate from the rolling mean for the same SKU-client pair; (5) SKU category (categorical, 8 levels). After encoding (target encoding for SKU and Client to avoid one-hot blowup at 200 SKUs and 15 clients; one-hot for the 3 countries and 4 client tiers; standardised continuous features), the feature space contains d_eng = 23 features. The detector's score and flag bring the classifier's input dimensionality to 25. All commercially sensitive identifiers were replaced with non-reversible SHA-256 hashed tokens before analysis.

Detection Engine (Stage 1): Isolation Forest [8] was selected after comparative evaluation against a Variational Autoencoder. It partitions the feature space via recursive random splits, assigning higher anomaly scores to observations isolated in fewer splits—a property well-suited to sparse tabular data with mixed-type features [9]. The Autoencoder achieved comparable mean F1 but substantially higher fold-level variance, a deployability disqualifier given that enterprise-finance maintainers are typically not ML specialists. Hyperparameters: n_estimators = 200, max_samples = 512, contamination = 0.08. Contamination was selected by grid search over {0.05, 0.07, 0.08, 0.10, 0.15} on training folds only, optimising for F1 on the audited subset; sensitivity is reported in the Results.

Classification Engine (Stage 2): The classifier is trained on the full labelled subset of 10,940 records (Normal: 7,548; Suboptimal: 2,283; Off-contract: 1,109). LightGBM [10] was selected over Random Forest [11] following comparative evaluation: leaf-wise tree growth and histogram-based gradient approximation produced faster convergence and more consistent generalisation on the minority off-contract class, consistent with reported findings on imbalanced financial classification [12, 13]. Class imbalance was addressed through inverse-frequency class weighting (Normal = 1.0, Suboptimal = 1.8, Off-contract = 4.2), combined with SMOTE [14], applied only within training folds — pre-split SMOTE introduces synthetic samples into the validation set, inflating recall estimates. Hyperparameters were tuned by 5-fold inner cross-validation on the training partition only: num_leaves = 63, learning_rate = 0.05, n_estimators = 500, reg_alpha = 0.1, reg_lambda = 0.2, min_child_samples = 30.

Baselines: (1) Deterministic ERP rule: a transaction is flagged off-contract if |posted_rate − contracted_rate| / contracted_rate > 0.05. This is the rule that finance teams in many FMCG firms currently apply (or could apply) without ML infrastructure, and tests whether ML adds incremental value above a one-line SQL check. (2) Class-weighted logistic regression: L2-regularised multinomial logistic regression with the same class weights as LightGBM on the same feature space. This is a standard imbalanced-classification reference baseline [15] and tests whether the gradient-boosted ensemble's gain over a linear model justifies its complexity.

Evaluation Protocol: Records are sorted chronologically and partitioned by month. A rolling-origin (walk-forward) cross-validation scheme is used with 5 expanding training windows: train on months 1–18 → test on 19–21; train on 1–21 → test on 22–24; and so on through month 36. This respects the temporal generative process and avoids the look-ahead leakage that stratified k-fold would introduce [16, 17]. The expanding rolling-origin scheme, together with the untouched 6-month held-out window (months 31–36, used once after model selection), jointly approximate the operational deployment condition — fitting on data available up to time t and evaluating on t+1 — while permitting variance estimation across five folds. Metrics: precision, recall, F1, and PR-AUC, per-class and macro-averaged; accuracy is reported but de-emphasised under class imbalance. End-to-end recall = (records correctly classified as their true non-Normal class) / (true non-Normal records); end-to-end precision = (records correctly classified as their true non-Normal class) / (records classified as any non-Normal class), both computed on the full evaluation dataset, not the post-filter subset.

Dataset and Diagnostic Mapping: 108,000 anonymised trade-promotion transactions from a multinational FMCG firm over 36 months across three markets, 15 retail clients, and 200 SKUs. The labelled subset (10,940 records) carries internally audited outcome labels: Normal – 7,548; Suboptimal Allocation – 2,283; Off-contract Deduction – 1,109. The remaining 97,060 records are unlabelled and processed only by Stage 1 at inference. The three output classes map conceptually to EMH information levels [7] — off-contract deduction (semi-strong analog: contract terms and posted amounts both exist in systems but no process compares them); suboptimal allocation (weak-form analog: signal present in historical transactions but never extracted); undocumented arrangement (strong-form analog: material information exists informally but never entered into any system) — and route to different organisational owners (contract management; commercial planning; documentation/governance). This is a conceptual mapping motivating routing logic; we make no empirical claim about EMH itself.

Results and discussion

Detection Engine: Isolation Forest achieved mean rolling-origin F1 = 0.772 ± 0.021 on the audited subset, versus 0.735 ± 0.048 for the Variational Autoencoder. The lower fold-level variance — roughly half — is the deployability-relevant difference: in enterprise finance settings where maintainers are not ML specialists, reproducibility across refits outweighs marginal mean gains. Sensitivity to the contamination parameter is reported in Table 1 and Figure 2; F1 remains within 0.01 of the maximum across contamination values of 0.05–0.10. The choice is therefore not knife-edge: a higher-precision regime (contamination = 0.05, precision = 0.830, recall = 0.710) is available without materially harming pipeline F1, because the supervised stage compensates for Stage-1 recall shortfalls when the score and flag are forwarded as features.

Table 1. Isolation Forest sensitivity to contamination parameter

Figure 2. Isolation Forest sensitivity to the contamination parameter on rolling-origin cross-validation. The F1 curve is flat across the 0.05–0.10 range (within 0.01 of the maximum). Precision decreases, and recall increases monotonically with contamination, as expected. The selected operating point (contamination = 0.08) is highlighted; the shaded band shows F1 ± σ across folds

Classification Engine: Per-class and macro-averaged results on rolling-origin temporal CV are in Table 2; Figure 3 shows precision–recall curves for the Off-contract class. The LightGBM macro F1 of 0.827 falls short of state-of-the-art results on heavily curated credit-card fraud benchmarks [12, 13] — the expected signature of FMCG's higher legitimate transactional variation and partial audit labels. These realistic numbers are the principal evidence that the rolling-origin protocol has eliminated the leakage that inflated earlier-draft estimates. The ERP rule's recall of 0.354 quantifies what the firm currently misses under rule-only detection.

Table 2. Classifier performance on rolling-origin temporal CV

Figure 3. Precision–recall curves for the Off-contract class on rolling-origin cross-validation. LightGBM (solid) dominates Logistic Regression (dashed) at every recall level. The ERP rule's single operating point is shown for reference

End-to-End System Performance: Table 3 reports end-to-end metrics over the labelled dataset, treating the pipeline (Stage 1 + Stage 2 + manual-review routing) as a single decision system. Stage-1 loss is substantially smaller than Stage-2 loss across both anomaly classes (Off-contract: 4.5% vs 11.0%; Suboptimal: 8.2% vs 16.3%). Under a hard-filter design, every Stage-1 loss would propagate as permanently undetected; the feature-forwarding architecture allows the classifier to recover some of it when the supervised signal is strong, which is why end-to-end recall (0.800) does not decompose into a product of stage-wise recalls. Residual Stage-2 loss is therefore the binding constraint and the natural target for the next iteration of feature engineering.

Table 3. End-to-end pipeline performance (rolling-origin CV)

¹ Stage-1 loss = true anomalies missed by the detector. ² Stage-2 loss = anomalies that pass Stage 1 but are misclassified by the classifier.

Comparison Against Baselines and Architecture Ablation. Table 4 reports the system against baselines on the out-of-time test set (months 31–36). The ERP rule's precision (0.920) is the highest in the table — a 5% rate-deviation threshold is near-tautological when it fires — but its recall (0.354) leaves two-thirds of off-contract cases undetected. Weighted logistic regression closes this recall gap (0.785). It is the first model to address the Suboptimal class (F1 0.652), confirming that the engineered feature set carries a material signal that the rule cannot express. LightGBM-alone adds 4 points of Off-contract recall (0.825) and 7 points of Suboptimal F1 (0.725). The full pipeline (LightGBM + Isolation Forest features) outperforms LightGBM-alone uniformly across all five reported cells (macro F1 +1.7 pts, Off-contract F1 +2.0, recall +2.5, precision +1.6, Suboptimal F1 +2.0). The gain is modest but uniformly positive — the expected signature of incremental rather than redundant signal — and the 2.5-point Off-contract recall gain is the principal architectural justification.

Table 4. System vs baselines on out-of-time test set (months 31–36)

Feature Importance. Normalised mean gain on the full training set after cross-validated hyperparameter selection places contracted-vs-posted rate deviation, client tier, and settlement lag as the three most discriminative variables. Gain-based importance reflects the predictive contribution of a feature within this specific model and feature set; it does not establish causality. SHAP-based local explanations [6] would extend this to per-prediction interpretability — a natural next step for the operational user interface.

Interpretation: The principal empirical finding is the performance ordering on out-of-time data: ERP rule (Macro F1 0.452) → weighted Logistic Regression (0.724) → LightGBM without Stage-1 features (0.795) → full pipeline (0.812). A macro F1 of 0.812 is moderate, not extraordinary — precisely the point: the rolling-origin protocol and held-out window remove the look-ahead leakage that inflated earlier-draft estimates, and the resulting numbers sit within the credible range for noisy intra-firm financial data with partial audit labels [3, 4, 12] and below those reported on credit-card fraud benchmarks [13] because FMCG admits substantially more legitimate variation. The largest single jump in the ordering is from ERP rule to logistic regression (+27 pts Macro F1), not from logistic regression to LightGBM (+7) or from LightGBM-alone to the full pipeline (+2). The dominant operational gain is therefore captured by the engineered feature set itself — rate deviation, client tier, settlement lag, rolling baselines — not by the choice of supervised learner, consistent with established findings on imbalanced financial classification [12, 15]: domain-aware features carry more signal than classifier choice above a competent baseline. The 7-point LightGBM gain confirms non-linear interactions (rate deviation × client tier × rolling baseline) but does not justify ensemble complexity in isolation. Off-contract recall is the operationally consequential single number: 0.354 under the ERP rule, 0.850 under the full pipeline. Each off-contract deduction missed at the rule's ceiling is a direct dollar loss that the proposed system would have surfaced, and the 50-percentage-point recall gain is realised at precision (0.821), only slightly below the rule's (0.920). The Suboptimal class — invisible to the rule by construction — emerges at F1 = 0.745.

Comparison with Prior Work: Recent literature consistently reports tree ensembles outperforming reconstruction-based deep models on tabular financial data [2, 3, 12]. Zhao et al. [13] report LightGBM gains of 3–7% F1 over deep alternatives on imbalanced credit-card data; Breskuvienė and Dzemyda [12] confirm the importance of within-fold rebalancing. Our results are consistent with these patterns but extend them to a substantively different domain — intra-firm trade-promotion data — where the noise structure, label provenance, and operational consequences differ markedly from those in card-fraud benchmarks. Recent surveys on explainable AI in finance [20, 21] underscore that interpretability, not raw accuracy, is the binding constraint on adoption in regulated finance settings — motivating the diagnostic-routing layer reported here.

Limitations: The labelled subset reflects what the firm's audit team historically examined and is therefore a non-random sample of the true error distribution; reported metrics describe performance on the audited distribution, not the true underlying one. The manual-review branch partially mitigates this by routing low-confidence flagged records to analysts. Still, production deployment should add a periodic audit-of-the-audit exercise on a stratified sample of unflagged records [19]. The data covers one multinational firm across three markets; trade-promotion structures and ERP configurations vary considerably across organisations [18], so the trained model is not directly transferable even though the architecture is. The undocumented-arrangement class is sparsely labelled and collapsed into Suboptimal for evaluation; whether it survives as a distinct category as the labelled dataset grows is an open empirical question. Feature importance is reported as a model property, not a causal one — SHAP [6] and intervention studies would be required for causal claims. Finally, the rolling-origin protocol addresses look-ahead leakage but not multi-year drift in promotional structures or regulatory context, so production deployment should include continuous performance monitoring with a pre-specified re-fit trigger.

Conclusion

We presented two contributions validated on real FMCG transactional data under a leakage-free temporal protocol: a two-stage pipeline with explicit flag-and-score forwarding that decouples unsupervised detection from supervised classification while preserving recoverability of detector errors, and an EMH-informed classification taxonomy — used as an analogical scaffold, not empirical test — that routes anomalies to the appropriate organisational owner. On the 6-month out-of-time window, the full pipeline reached Macro F1 = 0.812, with Off-contract recall rising from 0.354 under the ERP rule to 0.850 at comparable precision, and Suboptimal coverage emerging at F1 = 0.745. The largest share of this gain is attributable to the engineered feature set rather than to the choice of learner.

Three directions for future work follow: (i) federated or multi-firm validation to test transferability of architecture and feature definitions; (ii) SHAP-based local explanations [6] to close the loop between detection, classification, and analyst-facing interpretability; (iii) an A/B operational study comparing recovery outcomes for flagged-and-resolved versus matched flagged-and-unresolved transactions, paired with periodic audit-of-the-audit sampling to bound the audit-label-bias gap.

References:

1. Promotion Optimisation Institute. State of the Industry Report. — PoI, 2024.
2. Hilal W., Gadsden S. A., Yawney J. Financial fraud: A review of anomaly detection techniques and recent advances // Expert Systems with Applications. — 2022. — Vol. 193. — 116429. — DOI: 10.1016/j.eswa.2021.116429.
3. Abbassi H. et al. Digital banking fortification: a real-time isolation forest architecture for detecting online transaction fraud // Engineering Research Express. — 2024. — Vol. 6, № 2. — 025214.
4. Hernandez Aros L. et al. Financial fraud detection through the application of machine learning techniques: a literature review // Humanities and Social Sciences Communications. — 2024. — Vol. 11. — DOI: 10.1057/s41599-024-03606-0.
5. Awosika T. et al. Transparency and Privacy: The Role of Explainable AI and Federated Learning in Financial Fraud Detection // IEEE Access. — 2024. — Vol. 12.
6. Lundberg S. M., Lee S. I. A unified approach to interpreting model predictions // Advances in Neural Information Processing Systems. — 2017. — Vol. 30. — pp. 4765–4774.
7. Fama E. F. Efficient capital markets: A review of theory and empirical work // The Journal of Finance. — 1970. — Vol. 25, № 2. — pp. 383–417.
8. Liu F. T., Ting K. M., Zhou Z. H. Isolation Forest // Proceedings of the 8th IEEE International Conference on Data Mining. — 2008. — pp. 413–422.
9. Han S., Hu X., Huang H., Jiang M., Zhao Y. ADBench: Anomaly Detection Benchmark // Advances in Neural Information Processing Systems. — 2022. — Vol. 35.
10. Ke G. et al. LightGBM: A highly efficient gradient boosting decision tree // Advances in Neural Information Processing Systems. — 2017. — Vol. 30. — pp. 3146–3154.
11. Breiman L. Random forests // Machine Learning. — 2001. — Vol. 45, № 1. — pp. 5–32.
12. Breskuvienė D., Dzemyda G. Enhancing credit card fraud detection: highly imbalanced data case // Journal of Big Data. — 2024. — Vol. 11, № 1. — DOI: 10.1186/s40537-024-01059-5.
13. Zhao X. et al. Improved LightGBM for Extremely Imbalanced Data and Application to Credit Card Fraud Detection // IEEE Access. — 2024. — DOI: 10.1109/ACCESS.2024.3487212.
14. Chawla N. V., Bowyer K. W., Hall L. O., Kegelmeyer W. P. SMOTE: Synthetic minority over-sampling technique // Journal of Artificial Intelligence Research. — 2002. — Vol. 16. — pp. 321–357.
15. He H., Garcia E. A. Learning from imbalanced data // IEEE Transactions on Knowledge and Data Engineering. — 2009. — Vol. 21, № 9. — pp. 1263–1284.
16. López de Prado, M. Advances in Financial Machine Learning. — Hoboken, NJ: Wiley, 2018. — 400 p.
17. Cerqueira V., Torgo L., Mozetič I. Evaluating time series forecasting models: An empirical study on performance estimation methods // Machine Learning. — 2020. — Vol. 109, № 11. — pp. 1997–2028.
18. McKinsey & Company. Trade promotion management: the path to growth. — McKinsey Industry Report, 2023.
19. Bain & Company. CPG Trade Promotion Effectiveness 2024. — Bain Industry Brief, 2024.
20. Martins T., de Almeida A. M., Cardoso E., Nunes L. Explainable Artificial Intelligence (XAI): A Systematic Literature Review on Taxonomies and Applications in Finance // Engineering Applications of Artificial Intelligence. — 2024.
21. Černevičienė J., Kabašinskas A. Explainable artificial intelligence (XAI) in finance: a systematic literature review // Artificial Intelligence Review. — 2024. — Vol. 57. — DOI: 10.1007/s10462-024-10854-8.