Revisiting the Classics: Online RL in the Programmable Dataplane

Simpson, K. A. and Pezaros, D. P. (2022) Revisiting the Classics: Online RL in the Programmable Dataplane. In: IEEE/IFIP Network Operations and Management Symposium 2022, Budapest, Hungary, 25-29 April 2022, ISBN 9781665406017 (doi: 10.1109/NOMS54207.2022.9789930)

[img] Text
262493.pdf - Accepted Version

2MB

Abstract

Data-driven networking is becoming more capable and widely researched, partly driven by the efficacy of Deep Reinforcement Learning (DRL) algorithms. Yet the complexity of both DRL inference and learning force these tasks to be pushed away from the dataplane to hosts, harming latency-sensitive applications. Online learning of such policies cannot occur in the dataplane, despite being useful techniques when problems evolve or are hard to model.We present OPaL—On Path Learning—the first work to bring online reinforcement learning to the dataplane. OPaL makes online learning possible in constrained SmartNIC hardware by returning to classical RL techniques—avoiding neural networks. Our design allows weak yet highly parallel SmartNIC NPUs to be competitive against commodity x86 hosts, despite having fewer features and slower cores. Compared to hosts, we achieve a 21 × reduction in 99.99th tail inference times to 34 µs, and 9.9 × improvement in online throughput for real-world policy designs. In-NIC execution eliminates PCIe transfers, and our asynchronous compute model ensures minimal impact on traffic carried by a co-hosted P4 dataplane. OPaL’s design scales with additional resources at compile-time to improve upon both decision latency and throughput, and is quickly reconfigurable at runtime compared to reinstalling device firmware.

Item Type:Conference Proceedings
Additional Information:This work was supported in part by the Engineering and Physical Sciences Research Council [grants EP/N509668/1, EP/N033957/1].
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Simpson, Mr Kyle and Pezaros, Professor Dimitrios
Authors: Simpson, K. A., and Pezaros, D. P.
College/School:College of Science and Engineering > School of Computing Science
College of Science and Engineering > School of Engineering
ISSN:2374-9709
ISBN:9781665406017
Published Online:09 June 2022
Copyright Holders:Copyright © 2022 IEEE
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
172865EPSRC DTP 16/17 and 17/18Mary Beth KneafseyEngineering and Physical Sciences Research Council (EPSRC)EP/N509668/1Research and Innovation Services
172888Network Measurement as a Service (MaaS)Dimitrios PezarosEngineering and Physical Sciences Research Council (EPSRC)EP/N033957/1Computing Science