FEDND: FEDERATED NEWTON DIRECTION FOR FEDERATED LEARNING ENVIRONMENT

УДК 004.8

Аннотация

Федеративное обучение представляет собой децентрализованную парадигму машинного обучения, при которой обучение глобальной модели осуществляется на клиентских устройствах без передачи исходных данных на центральный сервер. В настоящей работе проводится сравнительный анализ алгоритма Федеративного Ньютоновского направления (FedND) и Федеративного стохастического градиентного спуска (FedSGD) при двух схемах взвешивания клиентов. В рамках FedND каждый клиент вычисляет локальное направление Ньютона посредством метода сопряжённых градиентов и передаёт его на сервер для агрегации; применение итеративного приближения в духе ADMM, компрессии матрицы Гессиана или многоуровневых схем аппроксимации при этом не требуется. Рассматриваются два подхода к взвешиванию клиентов: стандартный, пропорциональный объёму обучающей выборки, и адаптивный, основанный на нечёткой системе вывода (FIS). Экспериментальная оценка на датасете MAGIC Gamma Telescope показала стабильное превосходство FedND над FedSGD по показателям точности, AUC и F1-меры. Взвешивание на основе FIS обеспечивает дополнительный прирост показателей за счёт учёта гетерогенности распределений данных клиентов. Основной вклад работы состоит в эмпирической оценке прямой агрегации направлений Ньютона как прозрачного метода второго порядка, а также в исследовании его сочетания с адаптивным взвешиванием клиентов — комбинации, ранее не изучавшейся в литературе.

Abstract

Federated Learning (FL) is a decentralized machine learning paradigm in which a global model is trained collaboratively across client devices without transmitting raw data to a central server. The present study investigates the Federated Newton Direction (FedND) algorithm and evaluates its performance against Federated Stochastic Gradient Descent (FedSGD) under two distinct client weighting schemes. In FedND, each client computes a local Newton direction via the Conjugate Gradient method and transmits the resulting vector to the server for aggregation, without recourse to ADMM-based sub-procedures, Hessian matrix compression, or multi-level approximation schemes. Two weighting strategies are examined: conventional sample-size-proportional weighting and an adaptive scheme based on a Fuzzy Inference System (FIS). Empirical evaluation on the MAGIC Gamma Telescope Dataset demonstrates that FedND consistently surpasses FedSGD across all three performance metrics — classification accuracy, AUC, and F1 Score. FIS-based weighting yields additional performance gains by accounting for distributional heterogeneity among participating clients. The primary contributions of this work are an empirical assessment of direct Newton direction aggregation as a principled second-order federated optimization approach, and a first investigation of combining such aggregation with adaptive client weighting — a combination not previously examined in the literature.

Keywords: Federated Learning, second-order optimization, Newton direction, Fuzzy Inference System, client aggregation.

Ключевые слова: федеративное обучение, методы второго порядка, направление Ньютона, нечёткая система вывода, агрегация клиентов.

Introduction

The widespread adoption of machine learning across scientific and industrial domains has brought to prominence a fundamental tension between model quality and data privacy. Under the conventional centralized paradigm, data collected from multiple sources is aggregated on a single server, where a unified model is trained and subsequently distributed to client devices. While this approach is computationally convenient, it is predicated on the unrestricted transfer of potentially sensitive information — an assumption that is increasingly incompatible with modern data protection frameworks. In particular, regulations such as the General Data Protection Regulation (GDPR) impose stringent constraints on the cross-border transmission and centralized storage of personal data, rendering centralized architectures legally and ethically problematic in many domains, including healthcare, autonomous systems, and financial services [1][2].

The rapid proliferation of Internet of Things (IoT) devices has further complicated this landscape by generating unprecedented volumes of heterogeneous data at the network edge [1]. These data are often highly sensitive and subject to jurisdictional restrictions that preclude centralized collection. At the same time, the growing computational capacity of edge devices makes local model training increasingly feasible, providing a technical basis for distributed learning approaches that do not require raw data to leave the device.

Federated Learning (FL) has emerged as a principled response to these constraints. Introduced as a distributed machine learning framework, FL enables a global model to be trained collaboratively across a population of client devices, with each client computing model updates on its private local dataset and transmitting only gradient or parameter vectors to a coordinating server [2]. The server aggregates these contributions into a refined global model, which is then redistributed to clients. This paradigm preserves data locality, substantially mitigates privacy risks, and has demonstrated practical utility across diverse application domains [2].

A central challenge in FL is the design of effective aggregation algorithms [3]. The predominant first-order approaches — including Federated Stochastic Gradient Descent (FedSGD) and Federated Averaging (FedAvg) — transmit gradient or parameter updates that are inexpensive to compute and communicate but may exhibit slow convergence, particularly in non-IID (non-Independent and Identically Distributed) data environments [3][4]. Second-order methods offer faster convergence by incorporating curvature information from the Hessian matrix, but direct Hessian transmission is generally infeasible due to its quadratic scaling with the number of model parameters [9]. Several algorithms have been proposed to address this challenge: DONE [10] approximates the Newton direction distributively using Richardson iteration; FedNew [11] estimates the inverse Hessian-gradient product via ADMM without transmitting gradients or Hessians; and FedNL [12] transmits compressed Hessian updates using Top-K or Rank-R operators.

The present work investigates an alternative approach in which each client independently computes a local Newton direction via the Conjugate Gradient (CG) method — an iterative solver that obtains the product of the inverse Hessian and gradient without explicitly constructing or inverting the Hessian matrix — and transmits this compact directional vector to the server for weighted aggregation. This algorithm, termed FedND (Federated Newton Direction), is distinguished from prior second-order federated methods by its directness: it requires neither ADMM sub-procedures, multi-level iterative approximation, nor Hessian compression. The approach enables a transparent empirical assessment of the performance gains attributable to second-order curvature information in federated aggregation. A secondary contribution of this work is the investigation of FIS-based adaptive client weighting [7][8] in conjunction with Newton-type aggregation — a combination not previously examined in the literature.

The remainder of this paper is organized as follows. Section II reviews related work on federated optimization algorithms and client weighting strategies. Section III presents the proposed methodology. Section IV describes the experimental setup and discusses the obtained results. Section V summarizes the findings and outlines directions for future research.

Related Works

Federated Stochastic Gradient Descent (FedSGD) constitutes one of the foundational algorithmic approaches in the federated learning literature [4]. In each communication round, every participating client computes a stochastic gradient over its local dataset and transmits it to the central server, which forms a weighted average and performs a global parameter update. The principal advantages of FedSGD are its conceptual simplicity and low per-round computational overhead. However, its requirement for synchronous communication — whereby all clients must complete gradient computation before a global update can be performed — introduces significant latency in large-scale or heterogeneous deployments [4]. Federated Averaging (FedAvg) alleviates this bottleneck by permitting clients to execute multiple local gradient steps before transmitting updated model parameters, substantially reducing the number of required communication rounds [5]. Nevertheless, both algorithms rely on sample-size-proportional aggregation weights, which may be suboptimal under non-IID conditions where heterogeneous class distributions across clients introduce systematic bias into the aggregated update. FedProx addresses this issue by augmenting the local objective with a proximal regularization term that constrains local updates from deviating excessively from the current global model, thereby improving convergence stability in heterogeneous settings [6].

The question of client weighting has been examined through the lens of soft computing methodologies. Aliyev and Ismayilova [7] proposed a fuzzy logic-based framework for client selection and weighting in FL, in which membership functions over dataset size and class balance are used to derive quality-aware aggregation weights. Aliyev [8] extended this line of work by applying the Analytic Hierarchy Process (AHP) to construct multi-criteria weighting schemes. Both studies demonstrated that sample-size-proportional weighting systematically overestimates the contribution of clients with large but heavily imbalanced local datasets, and that distributional quality measures can improve global model performance in heterogeneous environments.

The application of second-order optimization methods to federated learning has attracted considerable research interest, motivated by the well-established convergence advantages of Newton-type methods over first-order approaches for strongly convex objectives [9]. The primary obstacle is the communication cost of the Hessian matrix, which scales quadratically with the number of model parameters and is generally prohibitive in federated settings. Existing methods differ substantially in their strategies for circumventing this cost.

DONE (Distributed Approximate Newton-type Method) [10] computes an approximation to the Newton direction distributively on each client using the classical Richardson iteration. Each client executes a fixed number of local iterations that, in the limit of infinite iterations, converge to the exact Newton direction; in practice, a finite number of steps yields an effective inexact approximation. Under strong convexity, DONE achieves a local linear-quadratic convergence rate [10]. FedNew [11] adopts a fundamentally different formulation, reformulating the global Newton update as a distributed ADMM problem in which clients iteratively refine a shared auxiliary variable. A key property of FedNew is that clients transmit neither gradients nor Hessian matrices at any iteration, sharing only compact auxiliary vectors; this simultaneously reduces communication overhead and provides resistance to gradient reconstruction attacks [13]. FedNL [12] takes a complementary approach, in which clients transmit compressed local Hessian updates — using contractive operators such as Top-K or Rank-R — to the server, which accumulates Hessian information across rounds and performs global Newton steps.

The present work occupies a distinct position in this design space. FedND employs the Conjugate Gradient method to compute the Newton direction locally on each client without explicitly forming or inverting the Hessian matrix, and transmits only the resulting directional vector for server-side aggregation. Unlike DONE and FedNew, no iterative distributed sub-procedure is required; unlike FedNL, no Hessian information is transmitted. This design prioritizes transparency and interpretability, enabling a direct empirical assessment of the contribution of second-order curvature information to federated aggregation quality. The integration of FIS-based adaptive weighting with a Newton-type aggregation scheme constitutes a secondary contribution that, to the best of the authors' knowledge, has not been investigated in prior work.

Materials and Methods

Federated Stochastic Gradient Descent (FedSGD)

In FedSGD, each client k computes a local stochastic gradient gk over its private dataset and transmits it to the central server. The server aggregates the received gradients via weighted averaging and updates the global model parameters according to the following rule:

where  denotes the global model parameters at iteration i, α is the learning rate, K is the number of participating clients,  is the number of training samples held by client k, n = Σ is the total sample count across all clients, and  is the local gradient. The weighting factor  assigns each client influence proportional to its share of the total training data. Under IID data conditions this formulation is well-motivated; however, in non-IID settings — where local class distributions vary substantially across clients — sample-size-proportional weighting may introduce systematic bias into the aggregated gradient, as clients with large but heavily skewed datasets exert disproportionate influence on the global update.

Federated Newton Direction (FedND)

Newton’s method is a canonical second-order optimization algorithm, extending gradient descent by incorporating curvature information encoded in the Hessian matrix of the objective function. The classical Newton update is given by:

where H−1g denotes the Newton direction. By scaling the gradient according to the local curvature of the loss landscape, Newton’s method achieves substantially faster convergence than gradient descent, particularly in ill-conditioned optimization problems and near local optima. A naive federated application of Newton’s method would require each client to transmit its local Hessian matrix and gradient to the server, which would then form an aggregated Hessian, invert it, and apply the resulting Newton step globally. This approach is generally infeasible, as the Hessian of a model with d parameters is a d×d matrix whose transmission cost scales quadratically with the model dimension.

FedND circumvents this limitation by having each client k compute the Newton direction locally using the Conjugate Gradient (CG) method. The CG method is an iterative linear solver that obtains the product without explicitly constructing or inverting the local Hessian matrix Hk. The resulting Newton direction:

is a d-dimensional vector — the same dimensionality as a gradient — and constitutes the sole quantity transmitted by client k to the server in each communication round. The server aggregates the received directional vectors via weighted averaging and updates the global model parameters:

This formulation differs from DONE [10], which approximates  iteratively via Richardson iteration and requires a distributed sub-procedure spanning multiple local rounds. It also differs from FedNew [11], which estimates the global  through an ADMM-based procedure without clients ever explicitly computing . In FedND, the CG solver is applied entirely locally on each client, and the server receives only the resulting directional vectors. Transmitting  rather than the raw gradient  additionally could potentially impede gradient reconstruction attacks [13], as an adversary would require knowledge of  to recover the underlying gradient; formal privacy guarantees, however, are beyond the scope of the present work.

Fuzzy Inference System (FIS)

The standard sample-size-proportional weighting scheme employed by FedSGD, FedAvg, and FedND in its baseline configuration does not account for the quality of individual clients’ local data distributions. In particular, a client whose local dataset is large but severely class-imbalanced may produce a Newton direction that is a poor representative of the true local objective, and may therefore exert a detrimental influence on the global model if weighted by sample size alone.

To address this limitation, the present work employs a Mamdani-type Fuzzy Inference System with two input variables — local dataset size and degree of class balance as quantified by the Gini impurity index — and a single output variable representing the aggregation weight assigned to each client. The Gini impurity index is defined as:

where pi denotes the proportion of samples belonging to class i in the client’s local dataset. A Gini index of zero indicates complete class purity (maximum imbalance), whereas larger values indicate greater class diversity. In binary classification, the maximum value of 0.5 corresponds to perfect class balance. The FIS maps the two input variables to a client-specific aggregation weight through a set of linguistically formulated If-Then rules (Table I).

The FIS operates through three sequential processing stages. In the fuzzification stage, crisp numerical input values are mapped to membership degrees in the interval [0, 1] over predefined fuzzy linguistic categories (Small, Medium, Large). In the inference stage, the fuzzified inputs are evaluated against the rule base using the minimum operator for conjunction and minimum implication for rule activation, yielding a set of truncated consequent fuzzy sets. In the defuzzification stage, the aggregated output fuzzy set is converted to a crisp numerical weight via the Centroid of Area (COA) method:

where μ(x) denotes the membership function of the aggregated output. The resulting crisp weight replaces the factor determined by number of samples in both FedSGD and FedND, assigning higher influence to clients with larger and more balanced local datasets, and attenuating the contribution of clients with small or heavily skewed data distributions.

Membership functions used in experiments are demonstrated in Figure 1

Results and Discussion

Dataset

The experimental evaluation was conducted on the MAGIC Gamma Telescope Dataset, a publicly available binary classification benchmark comprising 19,020 observations described by 11 continuous attributes. Instances are assigned to one of two classes: gamma (signal), representing 12,332 observations, and hadron (background), representing 6,682 observations. The dataset exhibits a natural class imbalance that renders it a suitable testbed for evaluating the sensitivity of different aggregation and weighting strategies to distributional heterogeneity.

For the purposes of federated simulation, the dataset was partitioned into five non-overlapping client subsets through randomized allocation. The partition was deliberately designed to introduce both sample size heterogeneity and label distribution heterogeneity across clients: certain clients received datasets with approximately balanced class proportions, while others were assigned heavily skewed local distributions. This experimental design reflects real-world federated deployment conditions, in which participating devices may observe different underlying data distributions.

Experimental Setup and Results

A logistic regression model was employed as the local learner on each client device. This choice ensures that Hessian computation within the CG solver remains tractable and that model architecture does not serve as a confounding variable; any observed performance differences between configurations are thus attributable to the aggregation strategy and weighting scheme rather than to model complexity. Four experimental configurations were evaluated: FedSGD with conventional sample-size-based weighting, FedSGD with FIS-based weighting, FedND with conventional weighting, and FedND with FIS-based weighting. Performance was assessed across three complementary metrics: classification accuracy, the Area Under the Receiver Operating Characteristic Curve (AUC), and the F1 Score. The AUC metric was included specifically to account for the class imbalance present in the dataset, as it provides a more informative measure of discriminatory capability than accuracy alone under such conditions. Detailed per-client results are presented in Table II.

The experimental results demonstrate a consistent and substantive performance advantage for FedND over FedSGD across all evaluation metrics and both weighting schemes. With conventional weighting, average classification accuracy increased from 0.730 under FedSGD to 0.778 under FedND, representing a gain of approximately 6.6 percentage points. The F1 Score exhibited a comparable improvement, rising from 0.734 to 0.784, while the AUC increased from 0.702 to 0.763. These improvements confirm that the incorporation of second-order curvature information via the CG-based Newton direction substantially enhances the quality of the aggregated global model, particularly under heterogeneous data conditions where first-order gradients may be poorly representative of the global objective.

The introduction of FIS-based adaptive weighting yielded additional performance gains for both algorithms, with the most pronounced effects observed in classification accuracy and F1 Score. For FedND, average accuracy increased marginally from 0.778 to 0.782 with FIS weights applied, while F1 Score remained approximately stable. A noteworthy exception concerns the AUC metric: under FIS-based weighting, the average AUC for FedND decreased from 0.763 to 0.751, indicating that the FIS rule base — calibrated primarily with respect to sample size and class balance — optimizes for accuracy-oriented objectives and does not uniformly improve ranking-based evaluation criteria. This observation highlights the importance of aligning weighting criteria with the specific performance objectives of the target application.

Examination of per-client results reveals that FedND demonstrated substantially lower inter-client performance variance relative to FedSGD. Under the FedSGD configurations, Client 5 consistently achieved the highest accuracy and F1 Score, while Client 3 exhibited the lowest performance across both weighting schemes — a disparity likely attributable to the detrimental effect of severe local class imbalance on the quality of the computed gradient. FedND markedly attenuated these inter-client disparities, producing more uniform performance across all participating clients. This behavior is consistent with the curvature-correcting properties of Newton-type directions, which account for the local geometry of the loss landscape and thereby implicitly reduce the adverse influence of poorly-conditioned local datasets on the global model update.

Table I. If-Then rules of the Fuzzy Inference System

Table II. Experimental results (bold values indicate the best result per client per metric)

Conclusion

The present study investigated the Federated Newton Direction (FedND) algorithm, in which each client employs the Conjugate Gradient method to compute a local Newton direction  and transmits the resulting vector to the server for weighted aggregation. By avoiding ADMM-based sub-procedures, distributed iterative approximation, and Hessian matrix compression, FedND provides a transparent and interpretable approach to second-order federated optimization, enabling a direct empirical assessment of the performance gains attributable to curvature-aware aggregation.

Experimental evaluation on the MAGIC Gamma Telescope Dataset demonstrated consistent and substantive superiority of FedND over FedSGD across all three evaluation metrics — classification accuracy, AUC, and F1 Score — under both conventional and FIS-based client weighting schemes. The average accuracy gain of approximately 6.6 percentage points and the corresponding improvements in AUC and F1 Score confirm that the incorporation of second-order curvature information into the federated aggregation process yields meaningful performance benefits, particularly under the non-IID data conditions imposed by the experimental design. Furthermore, the substantial reduction in inter-client performance variance under FedND indicates that Newton-type directions provide an implicit form of client regularization, attenuating the adverse influence of poorly-conditioned local datasets on the global model.

The integration of FIS-based adaptive client weighting yielded additional improvements in classification accuracy and F1 Score, confirming that soft computing approaches to weight determination constitute a productive complement to second-order aggregation strategies. The observed decrease in AUC under FIS-based weighting for FedND, however, indicates that the current rule base is better calibrated for accuracy-oriented objectives than for ranking-based criteria. Refining the membership functions and rule structure to account explicitly for AUC-related quality signals represents a natural direction for subsequent work.

Several directions for future research merit consideration. Although the Conjugate Gradient method substantially reduces per-client computational cost relative to explicit Hessian inversion, extending FedND to high-dimensional models remains an open challenge. The adoption of limited-memory quasi-Newton approximations, such as L-BFGS [14], could further reduce computational overhead while preserving the direct aggregation structure. A multi-step variant of FedND — in which clients execute several CG-based Newton steps locally prior to transmitting the accumulated direction — may additionally reduce the number of communication rounds required for convergence. Finally, a rigorous theoretical convergence analysis of FedND under non-IID data conditions and a formal privacy analysis of the directional transmission mechanism remain important open questions that the empirical results presented here motivate pursuing.

References:

1. Saeed, N., Malik, H., Naeem, A., & Bashir, U. (2023). Incorporating big data and IoT in intelligent ecosystems: state-of-the-arts, challenges and opportunities, and future directions. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-023-16328-3
2. Khalil, S. S., Tawfik, N. S., & Spruit, M. (2024). Exploring the potential of federated learning in mental health research: a systematic literature review. Applied Intelligence. https://doi.org/10.1007/s10489-023-05095-1
3. Qi, P., Chiaro, D., Guzzo, A., Ianni, M., Fortino, G., & Piccialli, F. (2024). Model aggregation techniques in federated learning: A comprehensive survey. Future Generation Computer Systems, 150, 272–293. https://doi.org/10.1016/j.future.2023.09.008
4. Balik, M. Y. (2024). Comparing Federated Stochastic Gradient Descent and Federated Averaging for Predicting Hospital Length of Stay. arXiv. https://arxiv.org/abs/2407.12741
5. Chen, M., Shlezinger, N., Poor, H. V., Eldar, Y. C., & Cui, S. (2021). Communication-efficient federated learning. Proceedings of the National Academy of Sciences, 118(17), e2024789118. https://doi.org/10.1073/pnas.2024789118
6. Yuan, X., & Li, P. (2022). On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond. Advances in Neural Information Processing Systems, 35, 10752–10765.
7. Aliyev, S., & Ismayilova, N. (2023). FL2: Fuzzy Logic for Device Selection in Federated Learning. In 2023 IEEE 17th International Conference on Application of Information and Communication Technologies (AICT) (pp. 1–6). https://doi.org/10.1109/aict59525.2023.10313160
8. Aliyev, S. (2023). Application of AHP for Weighting Clients in Federated Learning. Azerbaijan Journal of High Performance Computing, 6(2), 153–162.
9. Bischoff, S., Günnemann, S., Jaggi, M., & Stich, S. U. (2021). On Second-order Optimization Methods for Federated Learning. arXiv. https://arxiv.org/abs/2109.02388
10. Dinh, C. T., Tran, N. H., & Nguyen, T. D. (2020). DONE: Distributed Approximate Newton-type Method for Federated Edge Learning. arXiv. https://arxiv.org/abs/2012.05625
11. Elgabli, A., Ben Issaid, C., Bedi, A. S., Rajawat, K., Bennis, M., & Aggarwal, V. (2022). FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning. PMLR, 5861–5877.
12. Safaryan, M., Islamov, R., Qian, X., & Richtárik, P. (2021). FedNL: Making Newton-Type Methods Applicable to Federated Learning. arXiv. https://arxiv.org/abs/2106.02969
13. Bhowmick, A., Duchi, J., Freudiger, J., Kapoor, G., & Rogers, R. (2019). Protection Against Reconstruction and Its Applications in Private Federated Learning. arXiv. https://arxiv.org/abs/1812.00984
14. Yang, K., Fan, T., Chen, T., Shi, Y., & Yang, Q. (2019). A Quasi-Newton Method Based Vertical Federated Learning Framework for Logistic Regression. arXiv. https://arxiv.org/abs/1912.00513