CS Colloquium

The National Energy Research Scientific Computing Center (NERSC) is the mission high-performance computing center for the United States Department of Energy Office of Science (DOE-SC). NERSC operates cutting-edge, large-scale supercomputers for users performing simulations and data analysis in science areas of interest to DOE-SC. Experimental data analysis, in which data from large scientific instruments such as telescopes or light sources is computationally analyzed, is a growing portion of the compute workload at NERSC. Many of these scientific instruments increasingly require real-time or fast computing turnaround. NERSC's challenge is to support this urgent workload within its existing operations. Checkpoint/restart (C/R) can enable this new workload pattern and address other operational challenges, such as resiliency. In this presentation, we describe our efforts to create a robust offering of checkpointing approaches and to simplify and incentivize their use, in order to optimize the user experience, minimize wasted cycles, and maximize compute-system utilization.

The talk covers use cases, special challenges and solutions for Deep Learning for Medical Image Analysis. You will learn about:

• Use cases for Deep Learning in Medical Image Analysis
• Different DNN architectures used for Medical Image Analysis
• Special purpose compute / accelerators for Deep Learning (in the Cloud / On-prem)
• How to parallelize your models for faster training of models and serving for inferenceing.
• Optimization techniques to get the best performance from your cluster (like Kubernetes/ Apache Mesos / Spark)
• How to build an efficient Data Pipeline for Medical Image Analysis using Deep Learning
• Resources to jump start your journey - like public data sets, common models used in Medical Image Analysis
   

Over the past decade, deep learning has been remarkably successful at solving a massive set of problems on data types including images and sequential data.  This success drove the extension of deep learning to other discrete domains such as sets, point clouds, graphs, 3D shapes, and discrete manifolds. While many of the extended schemes have successfully tackled notable challenges in each particular domain, the plethora of fragmented frameworks have created or resurfaced many long-standing problems in deep learning such as explainability, expressiveness and generalizability. Moreover, theoretical development proven over one discrete domain does not naturally apply to the other domains.  Finally, the lack of a cohesive mathematical framework has created many ad hoc and inorganic implementations and ultimately limited the set of practitioners that can potentially benefit from deep learning technologies. In this talk I will talk about a generalized higher-order domain called combinatorial complex (CC) and utilize it to build a new class of attention-based neural networks called higher-order attention networks (HOANs). CCs generalize many discrete domains that are of practical importance such as point clouds, 3D shapes, (hyper)graphs, simplicial complexes, and cell complexes. The topological structure of a CC encodes arbitrary higher-order interactions among elements of the CC. By exploiting the rich combinatorial and topological structure of CCs, HOANs define a new class of higher-order message passing attention-based networks that unify existing higher-order models based on hypergraphs and cell complexes. I will demonstrate the reducibility of any CC to a special graph called the Hasse graph, which allows the characterization of certain aspects of HOANs and other higher order models in terms of graph-based models. Finally, the predictive capacity of HOANs will be demonstrated in shape analysis and in graph learning, competing against state-of-the-art task-specific neural networks.

Managing the data movement is a major bottleneck in many computing systems. Today, there is an explosion in data set sizes. For instance, terabytes of active memory are required for training machine learning models. Unfortunately, at the same time there has been relatively little growth in the capacities of memory devices. These trends are leading to increasingly heterogeneous memory systems with a small amount of high-performance memory and high-capacity memories with lower performance. In this talk, I will explain why hardware-managed data movement techniques perform poorly in these heterogeneous systems, and I will describe a software-based technique, AutoTM, which outperforms a hardware DRAM cache. Further, I will discuss our ongoing work developing a "data management ISA" to enable software-directed and hardware-accelerated heterogeneous memory management.

Bio: Jason Lowe-Power is an Assistant Professor at University of California, Davis where he leads the Davis Computer Architecture Research Lab (DArchR). His research interests include optimizing data movement in heterogeneous systems, hardware support for security, and simulation infrastructure. Professor Lowe-Power is also the Chair of the Project Management Committee for the gem5 open-source simulation infrastructure. He received his PhD in 2017 from the University of Wisconsin, Madison, and received an NSF CAREER Award and a Google Research Scholar Award.