April 23, 2024 · 2 min

I will give a talk at JLESC 16 @ Kobe

I will present our ongoing work entitled “Efficient Distributed Continual Learning for Steering Experiments in Real-Time” at JLESC 16. This presentation is an update on the progress of the JLESC project entitled “Towards Continual Learning at Scale”, which has been running since 2022. You can find the full program here.

Efficient Distributed Continual Learning for Steering Experiments in Real-Time - Project Update

Deep learning has emerged as a powerful method for extracting valuable information from large volumes of data. However, when new training data arrives continuously (i.e., is not fully available from the beginning), incremental training suffers from catastrophic forgetting (i.e., new patterns are reinforced at the expense of previously acquired knowledge). Training from scratch each time new training data becomes available would result in extremely long training times and massive data accumulation. Rehearsal-based continual learning has shown promise for addressing the catastrophic forgetting challenge, but research to date has not addressed performance and scalability. To fill this gap, we propose an approach based on a distributed rehearsal buffer that efficiently complements data-parallel training on multiple GPUs to achieve high accuracy, short runtime, and scalability. It leverages a set of buffers (local to each GPU) and uses several asynchronous techniques for updating these local buffers in an embarrassingly parallel fashion, all while handling the communication overheads necessary to augment input mini-batches (groups of training samples fed to the model) using unbiased, global sampling. After evaluating our approach on classification problems, we further propose a generalization of rehearsal buffers to support generative learning tasks, as well as more advanced rehearsal strategies (notably dark experience replay, leveraging knowledge distillation). We illustrate these extensions with a real-life HPC streaming application from the domain of ptychographic image reconstruction, in which experiments need to be steered in real-time.