.. raw:: html

.. .. image:: https://img.shields.io/github/last-commit/lishenghui/blades/master?logo=Github :alt: GitHub last commit (branch) :target: https://github.com/lishenghui/blades .. image:: https://github.com/lishenghui/blades/actions/workflows/unit-tests.yml/badge.svg?branch=master :alt: GitHub Workflow Status (with event)

.. container:: badges

    .. image:: https://img.shields.io/github/last-commit/lishenghui/blades/master?logo=Github
       :alt: GitHub last commit (branch)
       :target: https://github.com/lishenghui/blades

    .. image:: https://github.com/lishenghui/blades/actions/workflows/unit-tests.yml/badge.svg?branch=master
       :alt: GitHub Workflow Status (with event)

    .. image:: https://img.shields.io/badge/Pytorch-2.0-brightgreen?logo=pytorch&logoColor=red
       :alt: Static Badge
       :target: https://pytorch.org/get-started/pytorch-2.0/

    .. image:: https://img.shields.io/badge/Ray-2.8-brightgreen?logo=ray&logoColor=blue
       :alt: Static Badge
       :target: https://docs.ray.io/en/releases-2.8.0/

    .. image:: https://readthedocs.org/projects/blades/badge/?version=latest
        :target: https://blades.readthedocs.io/en/latest/?badge=latest
        :alt: Documentation Status

    .. image:: https://img.shields.io/github/license/lishenghui/blades?logo=apache&logoColor=red
        :alt: GitHub
        :target: https://github.com/lishenghui/blades/blob/master/LICENSE

    .. image:: https://img.shields.io/badge/arXiv-2206.05359-red?logo=arxiv&style=flat-square&link=https%3A%2F%2Farxiv.org%2Fpdf%2F2206.05359.pdf
        :alt: Static Badge
        :target: https://arxiv.org/pdf/2206.05359.pdf

.. raw:: html

Installation

.. code-block:: bash

git clone https://github.com/lishenghui/blades
cd blades
pip install -v -e .
# "-v" means verbose, or more output
# "-e" means installing a project in editable mode,
# thus any local modifications made to the code will take effect without reinstallation.

.. code-block:: bash

cd blades/blades
python train.py file ./tuned_examples/fedsgd_cnn_fashion_mnist.yaml

Blades internally calls ray.tune <https://docs.ray.io/en/latest/tune/tutorials/tune-output.html>_; therefore, the experimental results are output to its default directory: ~/ray_results.

Experiment Results

.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/fashion_mnist.png

.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/cifar10.png

Cluster Deployment

To run blades on a cluster, you only need to deploy Ray cluster according to the official guide <https://docs.ray.io/en/latest/cluster/user-guide.html>_.

Built-in Implementations

In detail, the following strategies are currently implemented:

Attacks

General Attacks ^^^^^^^^^^^^^^^^^ +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+ | Strategy | Description | Sourse | +====================+==========================================================================================================================================================================================================+===========================================================================================================+ | Noise | Put random noise to the updates. | Sourse <https://github.com/lishenghui/blades/blob/master/blades/adversaries/noise_adversary.py>_ | +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+ | Labelflipping | Fang et al. Local Model Poisoning Attacks to Byzantine-Robust Federated Learning <https://www.usenix.org/conference/usenixsecurity20/presentation/fang>, USENIX Security' 20 | Sourse <https://github.com/lishenghui/blades/blob/master/blades/adversaries/labelflip_adversary.py> | +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+ | Signflipping | Li et al. RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets <https://ojs.aaai.org/index.php/AAAI/article/view/3968>, AAAI' 19 | Sourse <https://github.com/lishenghui/blades/blob/master/blades/adversaries/signflip_adversary.py> | +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+ | ALIE | Baruch et al. A little is enough: Circumventing defenses for distributed learning <https://proceedings.neurips.cc/paper/2019/hash/ec1c59141046cd1866bbbcdfb6ae31d4-Abstract.html>_ NeurIPS' 19 | Sourse <https://github.com/lishenghui/blades/blob/master/blades/adversaries/alie_adversary.py>_ | +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+ | IPM | Xie et al. Fall of empires: Breaking byzantine- tolerant sgd by inner product manipulation <https://arxiv.org/abs/1903.03936>, UAI' 20 | Sourse <https://github.com/lishenghui/blades/blob/master/blades/adversaries/ipm_adversary.py> | +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------+

Adaptive Attacks ^^^^^^^^^^^^^^^^^ +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+ | Strategy | Description | Sourse | +==========================+=====================================================================================================================================================================================+=================================================================================================================+ | DistanceMaximization | Shejwalkar et al. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning <https://par.nsf.gov/servlets/purl/10286354>, NDSS' 21 | Sourse <https://github.com/lishenghui/blades/blob/master/blades/adversaries/minmax_adversary.py> | +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+

.. | FangAttack | Fang et al. Local Model Poisoning Attacks to Byzantine-Robust Federated Learning <https://www.usenix.org/conference/usenixsecurity20/presentation/fang>, USENIX Security' 20 | Sourse <https://github.com/bladesteam/blades/blob/master/src/blades/attackers/fangattackclient.py> | .. +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------+

Defenses

Robust Aggregation ^^^^^^^^^^^^^^^^^^^

+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | Strategy | Descriptions | Source | +=======================+=============================================================================================================================================================================================================================================================+==========================================================================================================+ | MultiKrum | Blanchard et al. Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent <https://proceedings.neurips.cc/paper/2017/hash/f4b9ec30ad9f68f89b29639786cb62ef-Abstract.html>, NIPS'17 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/multikrum.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | GeoMed | Chen et al. Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent <https://arxiv.org/abs/1705.05491>, POMACS'18 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/aggregators.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | Median | Yin et al. Byzantine-robust distributed learning: Towards optimal statistical rates <https://proceedings.mlr.press/v80/yin18a>, ICML'18 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/aggregators.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | TrimmedMean | Yin et al. Byzantine-robust distributed learning: Towards optimal statistical rates <https://proceedings.mlr.press/v80/yin18a>, ICML'18 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/aggregators.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | CenteredClipping | Karimireddy et al. Learning from History for Byzantine Robust Optimization <http://proceedings.mlr.press/v139/karimireddy21a.html>, ICML'21 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/centeredclipping.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | Clustering | Sattler et al. On the byzantine robustness of clustered federated learning <https://ieeexplore.ieee.org/abstract/document/9054676>, ICASSP'20 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/clippedclustering.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | ClippedClustering | Li et al. An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning <https://ieeexplore.ieee.org/abstract/document/10018261>, IEEE TBD'23 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/clippedclustering.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | DnC | Shejwalkar et al. Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning <https://par.nsf.gov/servlets/purl/10286354>, NDSS'21 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/aggregators.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+ | SignGuard | Xu et al. SignGuard: Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filtering <https://arxiv.org/abs/2109.05872>, ICDCS'22 | Source <https://github.com/lishenghui/blades/blob/master/blades/aggregators/signguard.py> | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------+

Data Partitioners:

Dirichlet Partitioner

.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/dirichlet_partition.png

Sharding Partitioner

.. image:: https://github.com/lishenghui/blades/blob/master/docs/source/images/shard_partition.png

Citation

Please cite our paper <https://arxiv.org/abs/2206.05359>_ (and the respective papers of the methods used) if you use this code in your own work:

::

@inproceedings{li2024blades,
title={Blades: A Unified Benchmark Suite for Byzantine Attacks and Defenses in Federated Learning},
author={Li, Shenghui and Ngai, Edith and Ye, Fanghua and Ju, Li and Zhang, Tianru and Voigt, Thiemo},
booktitle={2024 IEEE/ACM Ninth International Conference on Internet-of-Things Design and Implementation (IoTDI)},
year={2024}
}