Distributed and Private Machine Learning (DPML)

ICLR Workshop, May 07, 2021


The focus of this workshop is to bring together researchers from industry and academia that focus on both distributed and private machine learning. These topics are of increasingly large commercial and policy interest. It is therefore important to build a community for this research area, which collaborating researchers that share insights, code, data, benchmarks, training pipelines, etc and together aim to advance distributed and private machine learning.The scope of this workshop includes, but is not limited to:

  • Special track: privacy of ML and data in COVID-19 era
  • Distributed Machine Learning
  • Federated Learning and Split Learning
  • Differential Privacy in Deep Learning
  • Privacy and security attacks (Model Inversion, Membership Inference etc.)
  • Practical Considerations for Distributed Learning (Communication Efficiency, Compression)
  • Model and data fingerprinting/watermarking methods
  • Non i.i.d, sequential and online distributed learning
  • Differential privacy and other statistical notions of privacy: theory, applications, and implementations
  • Secure multi-party computation (Secure MPC) methods for ML
  • Homomorphic encryption for ML
  • Hardware-based techniques to privacy-preserving ML
  • Trade-offs between privacy and utility
  • Correspondence between different notions of privacy
  • Policy and Compliance for Data Privacy
  • Privacy applications in the real world ( including but not limited to Autonomous Systems, Social Networks)
  • Privacy, Fairness, Accountability and Transparency (F.A.T) in Machine Learning
  • Accessibility: We are committed to making this workshop accessible to everybody. Please notify the organizers in advance of the workshop if you require any accommodations or if there is anything we can do to make this workshop more accessible to you.

    Contact: Please email fatemeh@ucsd.edu for more information.

    Code of Conduct

    Everyone who participates in this workshop is required to conform to the ICLR Code of Conduct.


    You need to be registered at ICLR 2021, in order to be able to attend the workshop. The workshop is hosted on Zoom, except for the Poster session which will be held on Gather town, which you can join through this link. Offline and online Q&A will be taking place on Rocket chat. You can find the links to all of these platforms on our ICLR workshop page here (You should be logged into your ICLR account to be able see this).

    Gather town: Once you have joined our town, you can visit the posters by going to “Posters 1” and “Posters 2” rooms in the town. A list of all the posters and where they are can be found here.

    Call for Papers

    The workshop will include contributed papers. All accepted papers will be allocated either a virtual poster presentation, or a virtual talk slot. There are no proceedings (and concurrent/dual submissions are allowed), but will optionally link the papers and talk recordings on the workshop website.

    The papers should have up to 4 pages (excluding references, acknowledgements, or appendices), and be formatted using the ICLR submission template. Papers should be anonymized. Submission deadline is extended to end of the day, March 1st AOE, 2021 and you can submit your work using this link.

    We invite submissions in three tracks:

    Call for Reviewers

    If you are interested in becoming a reviewer for our workshop, please apply using this link.

    Invited Speakers

    Speaker: David Evans (University of Virginia), Title: Inference Risks for Machine Learning
    Biography: David Evans is a Professor of Computer Science at the University of Virginia where he leads a research group focusing on security and privacy (https://uvasrg.github.io). He won the Outstanding Faculty Award from the State Council of Higher Education for Virginia, and was Program Co-Chair for the 24th ACM Conference on Computer and Communications Security (CCS 2017) and the 30th (2009) and 31st (2010) IEEE Symposia on Security and Privacy, where he initiated the Systematization of Knowledge (SoK) papers. He is the author of an open computer science textbook (https://computingbook.org) and a children’s book on combinatorics and computability (https://dori-mic.org), and co-author of a book on secure multi-party computation (https://securecomputation.org/). He has SB, SM and PhD degrees from MIT and has been a faculty member at the University of Virginia since 1999.

    Speaker: Lalitha Sankar (Arizona State University), Title: A Better Bound Gives a Hundred Rounds: Enhanced Privacy Guarantees via f-Divergences
    Biography: Lalitha Sankar is an Associate Professor in the School of Electrical, Computer, and Energy Engineering at Arizona State University. She received her doctorate from Rutgers University, her masters from the University of Maryland and her Bachelors degree from the Indian Institute of Technology, Bombay. Her research is at the intersection of information theory and learning theory and its applications to identifying meaningful metrics for information privacy and algorithmic fairness. She received the NSF CAREER award in 2014 and currently leads an NSF-and Google-funded effort on using learning techniques to assess COVID-19 exposure risk in a secure and privacy-preserving manner.

    Speaker: Gauri Joshi (Carnegie Mellon University), Title: Biased Client Selection for Improved Convergence of Federated Learning
    Biography: Gauri Joshi is an assistant professor in the ECE department at Carnegie Mellon University since September 2017. Previously, she worked as a Research Staff Member at IBM T. J. Watson Research Center. Gauri completed her Ph.D. from MIT EECS in June 2016, advised by Prof. Gregory Wornell. She received her B.Tech and M.Tech in Electrical Engineering from the Indian Institute of Technology (IIT) Bombay in 2010. Her awards and honors include the NSF CAREER Award (2021), ACM Sigmetrics Best Paper Award (2020), NSF CRII Award (2018), IBM Faculty Research Award (2017), Best Thesis Prize in Computer science at MIT (2012), and Institute Gold Medal of IIT Bombay (2010).

    Speaker: Graham Cormode (University of Warwick), Title: Frequency Estimation in Local and Multiparty Differential Privacy
    Biography: Graham Cormode works on topics in privacy and data summarization. He is a Fellow of the ACM, and recipient of the 2017 Adams Prize for Mathematics. He is co-author of the book “Small Summaries for Big Data”.

    Schedule (PDT)

    08:30 Introduction and Opening Remarks
    08:40 Invited Talk 1: Biased Client Selection for Improved Convergence of Federated Learning (Dr. Gauri Joshi)
    09:10 Invited Talk 2:Frequency Estimation in Local and Multiparty Differential Privacy (Dr. Graham Cormode)
    09:40 Invited Talk 3: Inference Risks for Machine Learning (Dr. David Evans)
    10:10 Coffee Break
    10:30 Contributed Talk 1: Federated Learning with Taskonomy
    10:42 Contributed Talk 2: Privacy Amplification via Iteration for Shuffled and Online PNSGD
    10:54 Contributed Talk 3: TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption
    11:06 Contributed Talk 4: Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization
    11:18 Contributed Talk 5: Leveraging Public Data for Practical Private Query Release
    11:30 Poster Session and Coffee Break (accepted papers here)
    12:15 Invited Talk 4: A Better Bound Gives a Hundred Rounds: Enhanced Privacy Guarantees via f-Divergences (Dr. Lalitha Sankar)
    12:45 Concluding Remarks and Awards
    13:00 End


    Adam Smith (Boston University)

    Ramesh Raskar (MIT)

    Jayashree Kalpathy-Cramer (Harvard)

    Gautam Kamath (University of Waterloo)

    Reza Shokri (NUS)

    Hamed Haddadi (Imperial College London)

    Vivek Sharma (MIT, Harvard, KIT)

    Fatemehsadat Mireshghallah (PhD Student, UCSD)

    Praneeth Vepakomma (PhD Student, MIT)

    Ayush Chopra (PhD Student, MIT)

    Abhishek Singh (PhD Student, MIT)

    PC Members

    Mi Zhang (Michigan State University)

    Gauri Joshi (CMU)

    Clément Canonne (University of Sydney)

    Peter Kairouz (Google)

    Ling Liu (GaTech)

    Lin Zhong (Yale)

    Arya Mazumdar (UMass Amherst)

    Konstantinos Chatzikokolakis (University of Athens)

    Vishnu Boddeti (Michigan State University)

    Mehdi Bennis (University of Oulu)

    Waheed Bajwa (Rutgers University)

    Fragkiskos Koufogiannis (Raytheon)

    Supriyo Chakraborty (IBM T.J. Watson Research Center)

    Lei Yu (IBM T.J. Watson Research Center

    Keke Chen (Marquette University)

    Yuzhe Tang (Syracuse University )

    Yuan Hong (Illinois Institute of Technology)

    Ananda Theertha Suresh (Google)

    Jalaj Upadhyay (Apple)

    Antti Koskela (University of Helsinki)

    Eugene Bagdasaryan (PhD Student, Cornell)

    Sameer Wagh (Post-doc researcher, UC Berkeley)

    Matthew Jagielski (Northeastern University)

    Congzheng Song (Cornell)

    Abdullatif Mohammed Albaseer (Hamad Bin Khalifa University)

    Krystal A Maughan (University of Vermont)

    Adam Hall (Edinburgh Napier University/ OpenMined)

    Théo Ryffel (ENS - INRIA - CNRS)

    Akanksha Atrey (U Mass)

    Jinyuan Jia (Duke University)

    Ranya Aloufi (Imperial College London)

    Kritika Prakash (IIIT Hyderabad/ OpenMined)

    Gharib Gharibi (TripleBlind/ University of Missouri)

    Abinav Ravi Venkatakrishnan (Deepc gmbh/ OpenMined)

    Mohammadkazem Taram (UCSD)

    Benjamin Zhao (UNSW)

    Jianfeng Chi (University of Virginia)

    Ali Shahin Shamsabadi (The Vector Institute/Inria)

    Fan Mo (Imperial College London)

    Guanhong Tao (Purdue University)

    Mohammad Malekzadeh (Imperial College London)

    Mohammad Naseri (University College London)

    Yongjun Zhao (Nanyang Technological University)

    Stefanos Laskaridis (Samsung AI Center)

    Xuechen Li (Stanford University)

    Roozbeh Yousefzadeh (Yale University)

    Sahib Singh (Ford R&D/OpenMined)

    Hanieh Hashemi (University of Southern California)

    Saeedeh Parsaeefard (University of Toronto)

    Stylianos Venieris (Samsung AI)

    Gautham Krishna Gudur (Ericsson R&D)

    Kleomenis Katevas (Telefonica)

    Thomas Chen (The Academy for Mathematics, Science, and Engineering)

    Wenqi Wie (Georgia Tech)

    Muhammad Habib ur Rehman (KCL)

    Vandy Tombs (Oak Ridge National Laboratory)

    Stacey Truex (Georgia Tech)

    Accepted Papers

    Paper Title Authors
    Federated Learning's Blessing: FedAvg has Linear Speedup(pdf) Zhaonan Qu, Kaixiang Lin, Zhaojian Li, Jiayu Zhou
    Distributed Gaussian Differential Privacy Via Shuffling(pdf, room1-1) Kan Chen, Qi Long
    Federated Learning with Taskonomy(pdf, room1-2) Hadi Jamali-Rad, Mohammad Abdizadeh, Attila Szabo
    AsymmetricML: An Asymmetric Decomposition Framework for Privacy-Preserving DNN Training and Inference(pdf, room1-3) Yue Niu, Salman Avestimehr
    Layer-wise Characterization of Latent Information Leakage in Federated Learning(pdf, room1-4) Fan Mo, Anastasia Borovykh, Mohammad Malekzadeh, Hamed Haddadi, Soteris Demetriou
    Causally Constrained Data Synthesis for Private Data Release(pdf, room1-5) Chandrasekaran, Varun*; Edge, Darren; Jha, Somesh; Sharma, Amit; Zhang, Cheng; Tople, Shruti
    MPCLeague: Robust 4-party Computation for Privacy-Preserving Machine Learning(pdf) Nishat Koti, Arpita Patra, Ajith Suresh
    Privacy Amplification via Iteration for Shuffled and Online PNSGD(pdf, room1-8) Matteo Sordello, Zhiqi Bu, Jinshuo Dong, Weijie Su
    Computing Differential Privacy Guarantees for Heterogeneous Compositions Using FFT(pdf, room1-9) Antti Koskela and Antti Honkela
    Membership Inference Attack on Graph Neural Networks(pdf, room1-10) Iyiola E. Olatunji, Wolfgang Nejdl, Megha Khosla
    Privacy and Integrity Preserving Training Using Trusted Hardware(pdf, room1-11) Hanieh Hashemi, Yongqin Wang, Murali Annavaram
    Practical Defences Against Model Inversion Attacks for Split Neural Networks(pdf, room1-12) Tom Titcombe, Adam James Hall, Pavlos Papadopoulos, Daniele Romanini
    TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption(pdf, room1-13) Ayoub Benaissa, Bilal Retiat, Bogdan Cebere, Alaa Eddine Belfedhal
    Understanding Clipped FEDAVG: Convergence and Client-Level Differential Privacy(pdf) Zhang, Xinwei; Chen, Xiangyi; Yi, Jinfeng; Wu, Steven; Hong, Mingyi*
    Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization(pdf, room1-14) Mher Safaryan, Filip Hanzely, Peter Richtárik
    Meta Federated Learning(pdf, room1-15) Omid Aramoon, Gang Qu, Pin-Yu Chen, Yuan Tian
    Talk Less, Smile More: Reducing Communication with Distributed Auto-Differentiation(pdf) Baker, Bradley T*; Calhoun, Vince; Pearlmutter, Barak; Plis, Sergey
    Does Differential Privacy Defeat Data Poisoning?(pdf, room1-16) Matthew Jagielski, Alina Oprea
    PyVertical: A Vertical Federated Learning Framework for Multi-headed SplitNN(pdf, room2-1) Daniele Romanini, Adam J. Hall, Pavlos Papadopoulos, Tom Titcombe, Abbas Ismail, Tudor Cebere, Robert Sandmann, Robin Roehm, Michael A. Hoeh
    On Privacy and Confidentiality of Communications in Organizational Graphs(pdf, room2-2) Masoumeh Shafieinejad, Huseyin Inan, Marcello Hasegawa, Robert Sim
    Towards Prior-Free Approximately Truthful One-Shot Auction Learning via Differential Privacy(pdf, room2-3) Daniel Reusche, Nicolás Della Penna
    Leveraging Public Data for Practical Private Query Release(pdf, room2-4) Liu, Terrance*; Vietri, Giuseppe; Steinke, Thomas; Ullman, Jonathan; Wu, Steven
    FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks(pdf, room2-5) Chaoyang He*, Keshav Balasubramanian*, Emir Ceyani*, Yu Rong, Peilin Zhao, Junzhou Huang, Murali Annavaram, Salman Avestimehr
    Direct Federated Neural Architecture Search(pdf, room2-6) Garg, Anubhav*; Saha, Amit; Dutta, Debojyoti
    Differentially Private Multi-Task Learning(pdf, room2-7) Shengyuan Hu, Zhiwei Steven Wu, Virginia Smith
    A Graphical Model Perspective on Federated Learning(pdf, room2-8) Christos Louizos, Matthias Reisser, Joseph Soriaga, Max Welling
    Syft: A Platform for Universally Deployable Structured Transparency(pdf, room2-9) Adam James Hall, Madhava Jay, Tudor Cebere, Bogdan Cebere, Koen Lennart van der Veen, George Muraru, Tongye Xu, Patrick Cason, William Abramson, Ayoub Benaissa, Chinmay Shah, Alan Aboudib, Théo Ryffel, Kritika Prakash, Tom Titcombe, Varun Kumar Khare, Maddie Shang, Ionesio Junior, Animesh Gupta, Jason Paulmier , Nahua Kang, Andrew Trask
    FedPandemic: A Cross-Device Federated Learning Approach Towards Elementary Prognosis of Diseases During a Pandemic(pdf, room2-10) Aman Priyanshu, Rakshit Naidu
    Towards Causal Federated Learning - For enhanced robustness and privacy(pdf, room2-11) Sreya Francis, Irene Tenison, Irina Rish
    Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques(pdf, room2-12) Filip Hanzely, Boxin Zhao, Mladen Kolar
    Gradient-Masked Federated Optimization(pdf, room2-13) Irene Tenison, Sreya Francis, Irina Rish
    Prior-Free Auctions for the Demand Side of Federated Learning(pdf, room2-14) Andreas Haupt, Vaikkunth Mugunthan
    Heterogeneous Zero-Shot Federated Learning with New Classes for Audio Classification(pdf, room2-15) Gautham Krishna Gudur and Satheesh Kumar Perepu

    Best Paper Award

    The winners are:

    Federated Learning with Taskonomy by Hadi Jamali-Rad, Mohammad Abdizadeh, Attila Szabo

    Leveraging Public Data for Practical Private Query Release by Terrance Liu, Giuseppe Vietri, Thomas Steinke, Jonathan Ullman, Steven Wu