Introducing Amazon SageMaker Kubeflow Reinforcement Learning Pipelines for Robotics © 2021, Amazon Web Services, Inc. or its Affiliates. Contributors Max Kelsen • Nicholas Therkelsen-Terry • Leonard O'Sullivan • Matthew Rose Woodside Energy • Kyle Saltmarsh General Electric Aviation • John Karigiannis • Viktor Holovashchenko Amazon Web Services • Sahika Genc • Alex Chung • Ragha Prasad • Anna Luo Outline • Robotics Reinforcement Learning Workflow • Amazon SageMaker and Kubernetes/Kubeflow • Kubeflow Pipelines with AnyScale’s Ray • Customer use cases 1. General Electric Aviation – Manufacturing 2. Woodside Energy – Plant Operations Robotics Reinforcement Learning Workflow Reinforcement Learning for Robotics Reinforcement learning algorithms have the potential to enable robots to acquire complex behaviors adaptively in their environments. Motion planning Warehouse navigation Object manipulation Role of Simulation in Robotics • Easy to reset and explore various scenarios • Easy to alter the world, introduce obstacles, and tweak mechanics Successful Sim2Real with Thousands of Unique RL Agents AWS DeepRacer trains in simulation only with custom reward function. Thousands of RL agents have been successfully transferred at hundreds of in-person events around the world. SageMaker & Kubeflow The Machine Learning workflow is iterative and complex Collect and prepare training data Choose or bring your own ML algorithm Set up and manage environments for training Train, debug, and tune models Manage training runs Deploy model in production Monitor models Validate predictions Scale and manage the production environment Amazon SageMaker helps you build, train, and deploy models Collect and prepare training data Fully managed data processing jobs/ data labeling workflows Choose or bring your own ML algorithm Collaborative notebooks, built-in algorithms/models Set up and manage environments for training One-click training Train, debug, and tune models Debugging and optimization Manage training runs Visually track and compare experiments Deploy model in production One-click deployment and auto-scaling Monitor models Automatically spot concept drift Validate predictions Add human review of predictions Scale and manage the production environment Fully managed with auto-scaling for 75% less WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS SageMaker with Kubernetes & Kubeflow Amazon SageMaker Operators for Kubernetes Components and Kubeflow Pipelines enable the use of fully managed SageMaker machine learning tools across the ML workflow natively from Kubernetes or Kubeflow. SageMaker Kubeflow Pipelines with Ray Cluster SageMaker RL helps you build, train, simulate, and deploy on robots Collect and prepare training data continuously Fully managed data processing jobs/ data labeling workflows Choose or bring your own RL algorithm Collaborative notebooks, built-in algorithms/models Set up and manage environments for training One-click training Train, debug, and tune models Debugging and optimization Manage training runs Visually track and compare experiments Deploy model on robots One-click deployment and auto-scaling Monitor models Automatically spot concept drift Validate predictions Add human review of predictions Scale and manage the production environment Fully managed with auto-scaling for 75% less WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS Choose or bring your own simulation Cloud simulation environment, built-in algorithms/models Ray/RLLib Kubeflow Pipelines Orchestrates AWS RL Services Ray/RLLib Ray Cluster Utilizes External Cloud Simulation Servers VPC Amazon SageMaker Ray Cluster Trainer Policy Server Input Policy Rollout Data AWS RoboMaker Simulation Job AWS RoboMaker Simulation Job Rollout Fragments Policy Weights Amazon CloudWatch S3 Bucket Training Metrics Model Checkpoints Episode Visuals Console User AWS RoboMaker Simulation Job Policy Client Policy Episode Runner Agent Simulator Get_action() step() step() Faster training • Run a large number of simulations in parallel • Run at faster than real-time speed Policy V0 Episode 1 Episode 2 Episode 3 Episode N Policy V1 Episode 1 Episode 2 Episode 3 Episode N Epoch 1 Epoch 2 Policy Vi Policy V_N Episode 1 Episode 2 Episode 3 Episode N Epoch N Scaling Simulation Environments Benchmarking with Popular Gaming Environment NeurIPS 2020 Competition Track ProcGen BenchMark SageMaker Notebooks to measure sample efficiency & generalization in RL. Link to code repo: https://github.com/aws-samples/sagemaker-rl-procgen-ray Customer Uses Cases Use AWS RoboMaker with Amazon SageMaker to train, tune, and deploy reinforcement learning agents, and help improve operations and customer experience. Reinforcement Learning for Robotics Woodside Energy GE Aviation Thank You