Sandia Mission Computing: 2013 HPC Annual Report

Summary

The 2013 HPC Annual Report highlights accomplishments and high-performance computing capabilities across Sandia National Laboratories’ key mission areas, including Defense Systems & Assessments, Energy & Climate, International, Homeland & Nuclear Security, Nuclear Weapons, and Laboratory Directed Research & Development (LDRD). It showcases advanced modeling and simulation projects on supercomputing platforms such as Red Sky, Chama, and Uno, addressing critical challenges ranging from nuclear forensics and blast-induced traumatic brain injury to climate simulation, fusion research, and big data analytics.

Cover

SANDIA MISSION COMPUTING 2013 HPC ANNUAL REPORT Sandia National Laboratories

On the Cover

ON THE COVER

A computer generated rendering of a simulated jet fuel fire in a crosswind displayed at Sandia’s Joint Computational Engineering Lab. Running the 400-million-variable simulation occupied 5,000 processors for 24 hours on the Red Storm supercomputer. The simulation helped engineers prepare for a recent experiment at Sandia’s Thermal Test Complex.

With unmatched experimental facilities, some of the fastest computers on Earth, codes for mindboggling physics, and a world-class research staff, Sandia is uniquely qualified to solve the nation’s most pressing problems.

Credits: Editor – Barbara Jennings Contributing Writers – Robert Ballance, Laura Sowko, John Noe Design – Michael Vittitow

Table of Contents

SANDIA MISSION COMPUTING TABLE OF CONTENTS

Mission Excellence

• Mike Vahle, Vice President, Chief Information Office & IT Services … 4
• John Zepper, Director, Computing & Network Services… 5

Defense Systems & Assessments

• Anthony Thornton, Deputy for Technology & Programs … 6
• Modeling Optical Signals from Urban Nuclear Explosions for Yield Determination…7
• Investigation of Blast-induced Traumatic Brain Injury… 8
• Sandia Teams with Army in Launching Advanced Hypersonic Weapons… 9
• Absorption of Noble Gases into Porous Materials … 10

Energy & Climate

• John Mitchiner, Senior Manager, Energy & Climate … 11
• Atmosphere Component in Community Earth System Model… 12
• Computational Fluid Dynamics and Large-scale Uncertainty Quantification for Wind Energy … 13
• Atomistic Molecular Dynamics of Ion-Containing Polymers… 14
• Geomechanical Modeling to Investigate the Cause of Oil Leaks in Wellbores at Big Hill Strategic Petroleum Reserve … 15
• Modeling the Degradation of Used Nuclear Fuel from First Principles… 16

International, Homeland, & Nuclear Security

• Holly Dockery , Deputy for International, Homeland, and Nuclear Security Strategic Management… 17
• A Chemical, Biological, and Radiation Detection Requirements Analysis for the New World Trade Center Complex … 18
• Understanding Selective Field-Biased Gating of Biopolymers at Confining Nanopores … 19
• HPC Provides Quantitative Decision Support for Local Pandemic Response … 20
• Hazard Prediction and Assessment Capability Simulation Model… 21

Nuclear Weapons

• Keith Matzen, Director, Energy, Nuclear Weapons Science & Technology Programs… 22
• Stockpile Gravity Bomb-Spin Performance Using Computational Fluid Dynamics Modeling and Simulation… 23
• Magnetized Liner Inertial Fusion… 24
• Scaling Red Sky to Process Exascale Fluid Dynamics Research … 25
• Material Models of Equations of State… 26
• Testbeds: A look into the Future of HPC … 27

Laboratory Directed Research & Development Program

• Sheryl Martinez, Program Manager, Laboratory Directed Research and Development… 28
• Modeling Human Adult Neurogenesis … 29
• Molecular Modeling of the Attachment of the Dengue Virus to Cellular Membranes… 30
• A New Capability to Model Defects in Semiconductor Alloys… 31
• Network and Ensemble Enabled Entity Extraction in Informal Text … 32

High Peformance Computing Highlights

• Tom Klitsner, Computing Systems and Technology Integration …33
• Searching the Sky: Finding Anomalous Flights through Large Data … 34
• Uno, Sandia’s High-Throughput Computing Cluster … 35
• Sandia Deploys World’s Largest Enterprise Passive Fiber Optical Local Area Network… 36
• Sandia Collaborates to Expand 100G Access for New Mexicans… 36
• HPC User Support … 37

Systems Summary

• Mission Computing Systems… 38

Mission Excellence - From the Leadership (Mike Vahle)

MISSION EXCELLENCE FROM THE LEADERSHIP

Welcome! Leveraging the many years of support from the NNSA Advanced Simulation and Computing program, Sandia has developed a world-class high-performance computing environment supporting the nuclear weapons computational simulation needs. Sandia is now using this computational base to expand support to all Sandia programs run by Program Management Units (PMUs), including Energy and Climate, International, Homeland and Nuclear Security, Defense Systems and Assessments and Laboratory Directed Research and Development (LDRD) research programs. Collectively, the PMUs, LDRD, and the scientific computing organizations have become partners in providing Mission Computing to Sandia and our collaborators. This report introduces accomplishments supported by the Mission Computing program at Sandia during fiscal year 2013. This year, the report is focusing on the institutional computing aspect of Mission Computing.

In 2012, laboratory leadership established an Institutional Computing program charged with continuing the investment in high-performance computing systems, while fostering and encouraging the use of non-traditional computing assets by Sandia teams. The leadership team recognized that not all problems fit well within the historical scientific computing model and that systems designed to solve engineering and physics problems may be ill-suited to emerging fields such as graph analysis and information synthesis. The Institutional Computing program will engage all partners to identify and address these non-traditional needs in computing and plan for investments that will serve those needs.

The report you are reading is the first of a series of annual publications designed to provide the statistics of system use and some highlights of research activities conducted throughout Sandia. I trust this information will be useful and that it will spark interest in exploring additional services that can improve Sandia’s delivery of exceptional service in the national interest.

Mike Vahle Vice President, Chief Information Office & IT Services

Mission Excellence - From the Leadership (John Zepper)

Our personnel are the real source of our success, delivering a wide range of computing, telephony, network, and user support services to Sandia and NNSA’s other labs, 24 hours a day, 365 days a year. Enterprise Computing and Scientific Computing services also directly affect Sandians every day, whether through the desktop environment, email, web, mobile devices, or through delivery of scientific computational results. When Sandians enter the Tech Area, or receive on-site phone calls, they are using the infrastructure services our staff provides. When a Sandian or an outside collaborator encounters a computing issue using our systems, the Corporate Computing Help Desk helps resolve it. Our network and telephone support personnel assist staff through internal moves to re-establish connectivity and avoid work interruptions. Sandia’s pay and benefits are also processed and managed by our systems support personnel. The tools needed to plan, execute, and evaluate work are provided, licensed, and supported by professional, experienced, and dedicated staff members who strive to deliver customer service as needed, when needed, and where needed.

As a vital part of our service commitment to Sandia and in support of Sandia’s mission partners, Mission Computing supports scientific computing through the deployment and support of high performance computing (HPC) resources at many scales, management of big data, data transfer, and visualization. Our service offerings in this area are expanding and will continue to grow as we fully establish the Institutional Computing program during the next few years. While leveraging the existing expertise of our administrators and system engineering staff, our initial investments in cluster computer systems are being followed by more specialized platforms aligned with specific needs within Sandia programs. As computing needs grow beyond our traditional service space, we will develop the knowledge and skills of our staff to support these emergent needs.

We encourage our partners to engage us in discussions of mutual benefit through the newly established Mission Computing Council, a body that will improve communication of requirements and identify partner-specific priorities for utilization of corporate resources. We look forward to the collaboration potential and the focus on service that this new governance model will provide.

As you review the information in this report, please contact us with any questions or suggestions for improvement for the authors or our support teams. We are eager to engage with staff through the phone, email, consulting, or hallway conversations. We are here to address the computational needs of our users. Please help us improve our delivery of these services as we grow beyond our traditional areas. We have seen rapid change in recent years—think of the growth in HPC, visualization, Hadoop, clouds, data analytics—and we expect this trend to continue in areas such as emerging exascale computing. We look forward to learning what your Mission Computing requirements are and how well we are supporting your needs.

John Zepper Director, Computing & Network Services

Defense Systems & Assessments Overview

DEFENSE SYSTEMS & ASSESSMENTS

We deliver advanced science and technology solutions to deter, detect, track, defeat, and defend against threats to our national security. We analyze and exploit the vulnerabilities of our adversaries and develop innovative systems, sensors, and technologies for the defense and national security community.

The Defense Systems and Assessments (DS&A) Program Management Unit (PMU) exists to foster invention, innovation, maturation, and demonstration of technologies to enable future force capabilities. In addition to developing these capabilities across all domains (air, land, maritime, space, and cyber) we exploit these innovations and transition technology-enabled capabilities to the current force through our cooperative agreements with industrial partners. Implicit in this statement is the understanding that as a Federally Funded Research and Development Center, science and technology must serve the needs of the warfighter and provide our soldiers with new and improved capabilities to perform their missions against any current and/or future threat. National security threats remain many and varied, including those related to nuclear non-proliferation, cyber security, command and control, and preventing technological surprise.

In fiscal year 2013, DS&A relied heavily on HPC for modeling and simulation of cadmium zinc telluride materials used in radiation detection, for computational fluid dynamics modeling of advanced hypersonic systems to support the development of prompt global strike capabilities, and for coupled hydrodynamic/structural dynamic simulations for missile defense lethality predictions. In addition, our current program areas have active projects to support our customers on many other threats including network emulytics to understand network behavior and vulnerabilities, armor/anti-armor response to various threat scenarios, and understanding the expected signatures from a nuclear explosion to improve sensor monitoring. The DS&A PMU works strategically across these threat areas with all levels of government to solve the nation’s highest priority national security issues.

Anthony Thornton Deputy for Technology & Programs

Sidebar: National security threats remain many and varied, including those related to nuclear non-proliferation, cyber security, command and control, and preventing technological surprise.

Modeling Optical Signals from Urban Nuclear Explosions for Yield Determination

DEFENSE SYSTEMS & ASSESSMENTS MODELING OPTICAL SIGNALS FROM URBAN NUCLEAR EXPLOSIONS FOR YIELD DETERMINATION

The U.S. Government is enhancing its Technical Nuclear Forensics capability to determine both the design and materials in a terrorist nuclear device detonated in a city. Yield is an important input parameter to the forensics analysis; one of several methods of determining yield is measuring the optical output of the fireball and comparing this data to high-fidelity simulated optical signals.

The method is called “Yield Determination by Forward Simulation” (YDFS). This team is developing computational models for the three-dimensional evolution of a nuclear fireball in a city, its optical emission and propagation of the optical signal to sensors. If an actual terrorist attack were to occur, a computational search would be performed to find a yield and location whose simulated signals match observed signals. A proof-of-concept exercise has been completed in which a “red” team prepared simulated signals from a detonation and a “blue” team successfully determined the yield and location by varying inputs to Sandia simulation tools. A typical “moderate resolution” optical signal calculation takes about 10 hours on 1000 cores on Red Sky. To apply YDFS if a real explosion were to occur, dozens to hundreds of moderate-resolution simulations would be done to narrow the yield range of the explosion in a first-pass search on a large HPC machine. This would be followed by a few high resolution calculations to further refine the yield value, requiring 10’s of thousands of cores. The project is mainly supported by the Defense Threat Reduction Agency with some support from the National Nuclear Security Administration.

Caption: Computer simulation of a terrorist nuclear device detonated in a city. A computational model was used to calculate the three-dimensional evolution of a nuclear fireball (shown in red). A typical “moderate resolution” calculation takes about 10 hours on 1000 computer cores. High resolution calculations require 10’s of thousands of cores on a large HPC platform.

TEAM MEMBERS: Philip Dreike, PI Raymond Bell Mark Boslough Dennis Dinge Mehdi Eliassi William Hilbun Sue Phelps Joel Stevenson Marc Welliver Jason Wilke

PROCESSING HOURS: 449,580 Contact: Philip Dreike [email protected]

Investigation of Blast-Induced Traumatic Brain Injury

DEFENSE SYSTEMS & ASSESSMENTS INVESTIGATION OF BLAST-INDUCED TRAUMATIC BRAIN INJURY

Researchers at Sandia and the University of New Mexico compared supercomputer simulations of blast waves on the brain with clinical studies of veterans suffering from mild traumatic brain injuries (TBI) to improve the capacity to mitigate brain injury. Among many outcomes from the project, this work will aid in the development of new helmet designs to help protect military personnel from potential brain injury suffered from being near a detonated improvised explosive device. Future helmet designs could be based on research that identifies threshold levels of blast-induced stress and energy leading to brain injury, and how these factors are mitigated by the new helmet design. Based on this information, helmets can be improved through physical design and addition of embedded sensors to indicate whether a blast is strong enough to cause TBI.

By applying shock wave physics, Sandia researchers can predict the levels of wave stress and energy transmitted into the brain by a blast wave or blunt impact. Three-dimensional simulations computed on Red Sky demonstrate how sensitive brain tissue is affected by shock waves produced from roadside bombs within the first 5 to 10 milliseconds of exposure. This time interval lapses before a victim’s head moves any significant distance in response to the blast. (Typically, humans’ fastest reaction times are 75 to 100 milliseconds.) To date, this study has been the only TBI research that combines computer modeling and simulation of the physical effects of a blast on the human brain with analyses of clinical magnetic resonance images from veterans who have sustained such injuries.

Caption: These are stop-action plots of blast-generated pressure waves propagating through the head from the front (1st image), rear (2nd image), and laterally (3rd image) directions. The magnitude of the blast is 360 kilopascal (KPa), sufficient to cause mild traumatic brain injury. The included pressure bar provides a measure of the pressure wave magnitude as it transits the brain.

TEAM MEMBERS: Paul Taylor, PI John Ludwigsen Aaron Brundage Candice Cooper Andrei Vakhtin Corey Ford (University of New Mexico)

PROCESSING HOURS: 6,700 Contact: Paul Taylor [email protected]

Sandia Teams with Army in Launching Advanced Hypersonic Weapons

DEFENSE SYSTEMS & ASSESSMENTS SANDIA TEAMS WITH ARMY IN LAUNCHING ADVANCED HYPERSONIC WEAPONS

The advanced hypersonic weapons (AHW) program is part of Department of Defense’s Office of the Secretary of Defense (OSD) Conventional Prompt Global Strike (CPGS) effort to develop conventional weapon systems that can deliver a precision strike anywhere in the world within hours. Success means that the United States would have an alternative to nuclear weapons to prevent a crisis and it would decrease conventional military response time significantly.

Under the direction of the U.S. Army Space and Missile Defense Command/Army Forces Strategic Command, Sandia developed the three-stage booster system and glide vehicle used in the test flight; the thermal protection system development for the glide body was the responsibility of the U.S. Army Aviation and Missile Research Development and Engineering Center in Huntsville, Alabama. The test flight, launched from Sandia’s Kauai Test Facility, demonstrated that the AHW Hypersonic Glide Body could fly a non-ballistic glide trajectory at hypersonic speed.

HPC resources had a significant impact on this program. Most of the high-Mach-number flight control laws were based on computational fluid dynamics models, after benchmarking the computational results against available wind tunnel data. It would not be possible to meet the aggressive schedule without Sandia’s computing resources.

Many agencies are interested in the results of this effort. The Office of the Undersecretary of Defense (Acquisition, Technology, and Logistics), Assistant Secretary of Defense (Acquisitions), Strategic and Tactical Systems and Strategic Warfare are each seeking to develop and mature technologies and demonstrate capabilities that will enable transformational changes in the arena of global, time-critical conventional strike. The OSD CPGS technology development efforts will help inform an acquisition decision for a weapon system.

TEAM MEMBERS: Chris Bruner Justin Smith Ross Wagnild

PROCESSING HOURS: 2,993,209 Contact: Chris Bruner [email protected]

Absorption of Noble Gases into Porous Materials

DEFENSE SYSTEMS & ASSESSMENTS ABSORPTION OF NOBLE GASES INTO POROUS MATERIALS

Noble gases are used industrially in a multitude of applications including cryogenics, anesthetics, lighting, and lasers; however, because noble gases exhibit low chemical reactivity, they are difficult to extract. A joint experimental and simulation team from Sandia conducted simulations to determine which metal-organic frameworks (MOFs) would be most effective at separating noble gases from air. Atomistic and colloidal scale simulations can benefit experimentalists in two distinct ways: first, simulations can be used to explain the underlying chemical and physical properties that lead to experimentally observed phenomenon, and second, simulations can be used as a predictive tool to guide experiments to achieve improvements in cost, time, and safety.

This project used simulations as a predictive tool, beginning with validating Grand Canonical Monte Carlo simulations against experimental data. Additional simulations were then used to screen for materials likely to be significant so experimentalists could spend their time investigating the most promising candidate materials, thereby saving time and money.

One example illustrating how these simulations are useful is in understanding radon. Radon is a naturally occurring radioactive gas emitted during the radioactive decay of uranium contained in rocks, such as granite. Because of its radioactivity, approval to conduct experiments using radon is cumbersome. However, radon is harmless when modeled in a computer simulation! Our simulations have shown that certain MOFs show a high affinity for radon, and could potentially be used for home air purification or radon detection. Sandia’s capacity computers have enabled these simulations and large-scale screening efforts pivotal to this effort. This project is supported by the Department of Energy and has resulted in several peer-reviewed journal articles.

Caption: Model for the crystal structure of IRMOF-2.

TEAM MEMBERS: Mark Allendorf, PI Stephanie Teich-McGoldrick Marie Parkes Jeffery Greathouse

PROCESSING HOURS: 4,073,497 Contact: Stephanie Teich-McGoldrick [email protected]

Energy & Climate Overview

ENERGY & CLIMATE

Energy and Climate (EC) Program Management Unit (PMU) research programs are based on Sandia’s role as a national security laboratory—addressing the nation’s most daunting science and technology challenges within the national security context. EC goals and objectives seek to both leverage and enhance key competencies associated with Sandia’s nuclear weapons mission to amplify our contributions to broader national security in energy, climate, and infrastructure. EC work furthers Sandia engineering excellence with an emphasis on connecting deep science to engineering solutions. The EC PMU research programs work to: • Accelerate the development of transformative energy solutions that will enhance the nation’s security and economic prosperity. • Understand and prepare the nation for the national security implications of climate change. • Secure the nation’s critical infrastructure against natural or malicious disruption. • Provide a differentiating science understanding that supports the PMU and Sandia’s mission technologies now and into the future.

EC management chose these objectives to help the nation meet national security missions identified by the Department of Energy to reduce our dependence on foreign oil; increase use of low-carbon stationary power generation; understand risks and enable mitigation of climate-change impacts; increase security and resilience of critical infrastructures; and strengthen the nation’s science and technology base in energy, climate, and infrastructure.

A key element of EC success is how our programs fully integrate high-performance computing (HPC) into our research activities. Laboratory experiments and fieldwork provide a deeper understanding of physical systems that is subsequently incorporated into improved modeling and simulation codes. Simulation results often suggest where empirical research can foster a still deeper understanding that will improve the next generation of the model.

John Mitchiner Senior Manager, Energy & Climate

Sidebar: Energy and Climate work furthers Sandia engineering excellence with an emphasis on connecting deep science to engineering solutions

Atmosphere Component in Community Earth System Model

ENERGY & CLIMATE ATMOSPHERE COMPONENT IN COMMUNITY EARTH SYSTEM MODEL

Increasing the speed, scalability, and resolution of global climate models is a key step toward improving the ability to simulate regional details of climate change and improving our assessments of the impacts of climate change on extreme events. The Community Earth System Model (CESM) is a community, global climate model that provides computer simulations of the Earth’s past, present, and future climate states. This comprehensive model, developed in collaboration with National Science Foundation and Department of Energy (DOE) laboratories, couples state-of-the-art atmosphere, ocean, land, and ice model components. It is one of the primary tools used in DOE climate change assessments and in U.S. contributions to the United Nations Intergovernmental Panel on Climate Change assessment reports. Through the support of DOE Basic Energy Research, Sandia has been developing a spectral element dynamical core for the atmosphere component model in CESM. Sandia’s work removed the largest parallel scalability bottleneck in the CESM, allowing for faster simulation at higher resolutions. The improved model has the ability to capture hurricanes and other types of extreme weather not shown in lower-resolution simulations.

The highest resolution CESM simulations, using Sandia’s configuration of the atmosphere model running at 13 kilometer (km) resolution, also operates on DOE Leadership Computing Facilities at Oak Ridge and Argonne National Laboratories and scales out to 130,000 cores. The development work making these types of simulations possible was carried out primarily on Sandia’s Red Sky. This includes extensive low and moderate resolution simulations used for development, validation, and verification. Sandia also made use of the spectral elements unstructured grid support to perform global moderate resolutions simulations, which contain small continental-size regions of 13 km resolution, resulting in an efficient approach that enables scientists to use Red Sky to calibrate the global high-resolution model.

Caption: The snapshot above illustrates precipitable water simulated by the CESM running at 13 km resolution using the spectral element atmosphere dynamical core. The image shows a Category 5 hurricane in the Gulf of Mexico, with another tropical cyclone forming in the mid-Atlantic. At these resolutions, the model has the ability to capture hurricanes not present in lower resolution simulations providing critical information for climate change assessments.

TEAM MEMBERS: Mark Taylor, PI Laura Swiler Erika Roesler

PROCESSING HOURS: 1,882,957 Contact: Mark Taylor [email protected]

Computational Fluid Dynamics and Large-Scale Uncertainty Quantification for Wind Energy

ENERGY & CLIMATE COMPUTATIONAL FLUID DYNAMICS AND LARGE-SCALE UNCERTAINTY QUANTIFICATION FOR WIND ENERGY

A team of Sandia experts in aerospace engineering, scientific computing, and mathematics collaborated with researchers at Stanford University to study wind-energy design problems, using computational fluid dynamics (CFD) simulations and uncertainty analyses. The project developed new mathematical uncertainty quantification techniques and applied them, in combination with high-fidelity CFD modeling, to probabilistic wind-turbine design problems.

Quantifying uncertainty for this project required analyzing hundreds of simulation scenarios/three-dimensional high-fidelity computational runs to evaluate fluid dynamic flow past a wind turbine rotor using unstructured sliding meshes. This project contributed to developing the Nalu simulation code, a low-Mach-number CFD solver, scalable to problem sizes on the order of billions of elements running on hundreds of thousands of cores. The research team reviewed different computational models and discovered that using high-fidelity (CFD) and low-fidelity engineering models in combination to simulate the phenomena provides more reliable results at lower cost.

The analysis from this project provides the wind turbine industry with a new paradigm for design, relying more on high-fidelity simulation and less on simple empirical models. As the industry increasingly adopts high-fidelity modeling approaches, new design approaches will lead to greater certainty in wind-turbine performance and design load predictions, resulting in much more cost-effective wind turbines.

Caption: Computational Fluid Dynamics Simulation of a synthetic gust (inviscid Taylor Vortex) passing through a 3-bladed Vertical Axis Wind Turbine rotor.

TEAM MEMBERS: Michael Eldred, PI Matthew Barone Stefan Domino

PROCESSING HOURS: 350,000 Contact: Matthew Barone [email protected]

Atomistic Molecular Dynamics of Ion-Containing Polymers

ENERGY & CLIMATE ATOMISTIC MOLECULAR DYNAMICS OF ION-CONTAINING POLYMERS

In the quest to make better batteries, researchers at Sandia performed extensive, atomistic molecular dynamics (MD) simulations of a set of ionomer melts. Ionomers are polymers that typically have a neutral backbone with a small number of charged moieties that are covalently bound to the polymer backbone. Ionomer melts, liquids containing just ionomer molecules and their oppositely charged ions, show promise as safe, solvent-free, lightweight, single-ion-conducting battery electrolytes if their conductivity can be made high enough. However, without a solvent, ions aggregate and their dynamics slow, leading to conductivity too low for battery applications.

Sandia researchers have used molecular dynamics simulations to better understand the ionic aggregates formed in ionomer melts and also the effects of these aggregates on ion dynamics. Recent atomistic simulations have focused on the effects of different ionomer backbone designs and different cations on structures resulting from ionic aggregation. Atomistic molecular dynamics simulations are computationally expensive, making Red Sky invaluable for some of these simulations.

The simulations revealed that ionomers have a rich variety of ionic aggregate morphologies, often including long, “stringy” aggregates. This result contrasts the assumption in interpreting experimental scattering data; that of spherical aggregates with liquid-like order. These unsuspected morphologies nevertheless fit experimental X-ray data well. Simulations on Red Sky showed that an applied electric field can induce alignment of the aggregates, and also revealed a relaxation mechanism involving ions moving along the aggregate network. This enhanced understanding is important for designing ionomers with better ion conduction properties.

Caption: Snapshot of MD simulation showing “stringy” ionic aggregates consisting of lithium ions (yellow) and oxygen atoms (red). The rest of the ionomer is not shown for clarity.

TEAM MEMBERS: Amalie Frischknecht, PI Mark Stevens Sara Waters Jon Brown John McCoy

PROCESSING HOURS: 1,980,589 Contact: Amalie Frischknecht [email protected]

Geomechanical Modeling to Investigate the Cause of Oil Leaks in Wellbores at Big Hill Strategic Petroleum Reserve

ENERGY & CLIMATE GEOMECHANICAL MODELING TO INVESTIGATE THE CAUSE OF OIL LEAKS IN WELLBORES AT BIG HILL STRATEGIC PETROLEUM RESERVE

The U.S. Strategic Petroleum Reserve stores crude oil in 62 caverns located at four sites in Texas and Louisiana and currently contains over 700 million barrels. Most of the caverns were solution mined by the Department of Energy. Oil leaks have been found in two wellbores taken from caverns located at the Big Hill Strategic Petroleum Reserve in Texas. According to the multi-arm caliper survey performed by DM Petroleum Operations in 2009 and 2010, two instances of casing damage occurred at the depth of the interface between the caprock bottom and salt top. To help analyze the failures, Sandia researchers used high-performance computing resources to model the geomechanics of the wellbores.

A three-dimensional mesh that allowed each cavern to be configured individually was created to investigate wellbore damage at the interbed between the caprock bottom and the salt top as shown in the companion image. The mesh consists of 1,050,760 nodes and 1,012,932 elements with 37 element blocks, 5 node sets, and 28 side sets. Sandia-developed three-dimensional solid mechanics code, Adagio, was used in the analyses. The simulation was carried out on Sandia’s Red Sky, using 512 nodes (4,096 cores) and 51 hours of computer running time.

The analysis results indicated wellbore failures resulted from shear stress that exceeded the wellbore shear strength from the horizontal movement of the salt top relative to the caprock. Another wellbore failed from tensile stress created by the downward movement of the salt top from the caprock within the cavern. Computation images were constructed to enable comparisons of predictions to actual casing failures. These simulations matched the survey images well.

Caption: Overview of the finite element mesh of the stratigraphy and cavern field at Big Hill.

TEAM MEMBERS: Byoung Yoon Park, PI Steven Sobolik

PROCESSING HOURS: 3,053,634 Contact: Yoon Park [email protected]

Modeling the Degradation of Used Nuclear Fuel from First Principles

ENERGY & CLIMATE MODELING THE DEGRADATION OF USED NUCLEAR FUEL FROM FIRST PRINCIPLES

To ensure the safe long-term storage or disposal of used nuclear fuels (UNF), research is underway at Sandia to understand the degradation of UNF, specifically uranium dioxide (UO2), in geological repositories. Understanding the interfacial processes and the underlying interactions that occur between materials surfaces and their local environment is a prerequisite to controlling materials performance and degradation limits over time.

Metal oxides are particularly prone to corrosion and/or dissolution, which is pervasive and of critical significance for technological applications and environmental systems. Despite decades-long research efforts to explain the key factors controlling this ubiquitous process, understanding metal oxide dissolution at the molecular scale still remains an open challenge. A limited number of computational studies have explored the surface properties and reactivity of UO2 with respect to water and oxygen, using either empirical pair-potential or standard density functional theory (DFT) approaches.

Researchers at Sandia conducted first-principles calculations of the surface properties and chemistry of UO2 using the Vienna Ab-initio Simulation Package, beyond the standard DFT framework. The results revealed that some of the key electronic and chemical properties controlling UO2 surface reactivity with oxidizing agents are very sensitive to the strong electron correlation correction included in this study and could possibly affect the formation of the complex uranyl-based phases resulting from oxidative dissolution of the fluorite-structured UO2. The implications of these physicochemical changes are critical for the safe storage or disposal of used nuclear fuels, regardless of the repository environment.

Caption: Schematic representation of the degradation process of used nuclear fuel (UO2) to the corrosion product studtite ((UO2)O2(H2O)4) using atomistic models.

TEAM MEMBERS: Philippe F. Weck, PI David Sassani Carlos Jove-Colon Eunja Kim (University of Nevada, Las Vegas)

PROCESSING HOURS: 768,104 Contact: Philippe F. Weck [email protected]

International, Homeland, & Nuclear Security Overview

INTERNATIONAL, HOMELAND, & NUCLEAR SECURITY

The International, Homeland, and Nuclear Security (IHNS) Program Management Unit mission includes some of the newest program areas at Sandia, as well as several areas where the Laboratories have long provided support. Mission area responsibilities include advancing weapon of mass destruction (WMD) nonproliferation by supporting the development and implementation of arms control treaties and objectives and by securing and safeguarding WMD materials and facilities. This includes enhancing the security of nuclear weapons globally; countering, responding to, and recovering from WMD use by terrorists or others; ensuring the resilience of critical U.S. physical and cyber infrastructures; and reducing the risk to our nation from significant national incidents while maintaining and facilitating trade, travel, and personal freedoms.

IHNS supports a broad range of sponsors, including National Nuclear Security Administration; the Departments of Defense, Homeland Security, State, Health and Human Services, Treasury, and the Federal Aviation Administration. We understand that countering WMD and responding to other significant incidents requires working across dynamic, complex, interdependent systems and then creating solutions that best manage key risks across the entire system by employing a wide range of integrating and enduring Sandia capabilities, including: • Nuclear, radiological, biological, explosives, and chemical science and engineering • System analysis, engineering, and integration • Physical and cyber security methods, technologies, and systems • Predictive modeling and simulation of interdependent systems • Decontamination and restoration approaches and technologies • International security technologies and policy

By viewing the evolution of a problem from pre-crisis, through an incident, to post-crisis, we help the government understand how to effectively and efficiently anticipate and address the most important national security risks.

Holly Dockery Deputy for International, Homeland, and Nuclear Security

Sidebar: We help the government understand how to effectively and efficiently anticipate and address the most important national security risks.

A Chemical, Biological, and Radiation Detection Requirements Analysis for the New World Trade Center Complex

INTERNATIONAL, HOMELAND, & NUCLEAR SECURITY A CHEMICAL, BIOLOGICAL, AND RADIATION DETECTION REQUIREMENTS ANALYSIS FOR THE NEW WORLD TRADE CENTER COMPLEX

The new World Trade Center complex consists of many buildings including office towers, a transportation hub, retail shopping, a memorial, and a museum. Because of the high profile nature of this site, the Port Authority of New York and New Jersey worked with the Department of Homeland Security to enlist the help of Sandia, Argonne, and Los Alamos National Laboratories, along with the National Institute of Standards and Technology to conduct a chemical, biological, and radiation (CBR) detection requirements analysis. The Port Authority will use the results of this analysis to determine the types of detection systems necessary to achieve a desired level of performance and to maximize their CBR detection investment and response planning.

To conduct the analysis while accounting for uncertainties in release location, release size, building conditions, and ambient conditions, Sandia generated hundreds of thousands of plausible CBR attack scenarios. This library of scenarios, along with multiple models of evacuation responses, was used to measure performance of hundreds of different detector networks of varying size and sensitivity, each optimized to save the most lives possible. The interconnections between the World Trade Center complex facilities required the most detailed computational model ever used to track a contaminant as it moved between buildings, the outdoor environment, the Metropolitan Transit Authority subway system, and the Port Authority Trans-Hudson subways system. Due to the computational intensity of using this model, all scenario generation and detector optimization tasks were performed using Sandia’s high-performance computing (HPC) resource, Red Sky.

Caption: The new WTC complex, transportation hub, and memorial and museum occupy 16 acres in lower Manhattan and will eventually include four skyscrapers and a complex network of below-ground construction. Analysis includes event characterization and response planning.

TEAM MEMBERS: Garrett Barter, PI Nathaniel Gleason, PM Meghan Peterson Nerayo Teclemariam Trisha Miller David Franco

PROCESSING HOURS: 258,351 Contact: Garrett Barter [email protected]

Understanding Selective Field-Biased Gating of Biopolymers at Confining Nanopores

INTERNATIONAL, HOMELAND, & NUCLEAR SECURITY UNDERSTANDING SELECTIVE FIELD-BIASED GATING OF BIOPOLYMERS AT CONFINING NANOPORES

Proteins can selectively pass through nanoporous materials depending on the properties of the material, protein, solution, and applied voltage. “Selective field-biased gating” is used to create separation devices for sensors, especially field-deployable nanofluidic sensors for the detection of chemical and biological warfare agents.

This project contains both experimental and theoretical components. The theoretical component and particle-based simulations in particular can elucidate the energy barriers to the selective field-biased gating. The interaction of the protein with the material determines the voltage needed to drive the protein through the nanoporous material. The simulations also give insight into how the protein passes through the material, whether the protein remains folded or unfolds and threads through the material. The theoretical model derived from simulations can guide experiments and help find thresholds more quickly for the gating of a particular protein with a particular nanoporous material. This helps find the appropriate materials with desirable selectivities for use in sensors.

One part of the theoretical components involves computationally intensive many-body simulations suited for the Red Sky system. These simulations use the GROMOS united atom force field with implicit solvent to model polypeptides and nanoporous surfaces of interest. The model will eventually include applied voltages and inhomogeneous solutions where electrolyte and thus charge concentrations vary based on the proximity to charged proteins and porous materials. This phenomenon is known as concentration polarization.

Caption: Bovine serum albumin binding to an idealized polypropylene surface under the GROMOS united atom model with implicit solvent.

TEAM MEMBERS: Darin Pike, PI Susan Rempe, Co-PI Benjamin Schudel Junyu Mai Kerrie Wu David Rogers (University of South Florida) Mathias Anderson Ali Mani (Stanford University)

PROCESSING HOURS: 67,669 Contact: Darin Pike [email protected]

HPC Provides Quantitative Decision Support for Local Pandemic Response

INTERNATIONAL, HOMELAND, & NUCLEAR SECURITY HPC PROVIDES QUANTITATIVE DECISION SUPPORT FOR LOCAL PANDEMIC RESPONSE

Large-scale computer modeling is a vital component of national responses to pandemic disease outbreaks. Groups at national laboratories, federal agencies, and academic centers run sophisticated disease propagation models on high-performance computing (HPC) resources to estimate potential intensity and duration of projected outbreaks. Recent work at Sandia has focused on shifting from large, national-scale pandemic models to smaller, community-scale models, which better address the needs of local decision makers. To be effective for decision makers, such small-scale models must be sensitive to demographic variations, which characterize individual communities or neighborhoods.

Sandia’s HPC resources were leveraged to rigorously characterize how pandemic disease propagation models respond to these neighborhood-scale inputs. Red Sky’s computational power was used to conclusively demonstrate the potential applicability of off-the-shelf models to local-scale planning and response for decision makers.

The HPC-based work also generated two novel methods for transforming pandemic model outputs directly into quantitative rankings of intervention effectiveness for individual neighborhoods based on standard demographic data. These new methods, based on random forests machine learning techniques and Levenshtein edit distance metrics popular in genomic sequencing work, may eventually provide readily accessible decision analytics needed for micro-targeting of intervention strategies to specific local governmental or geographic entities.

TEAM MEMBERS: Patrick Finley, PI Greg Lambert Leland Evans Andrew Huff Rich Detry

PROCESSING HOURS: 10,299 Contact: Patrick Finley [email protected]

Hazard Prediction and Assessment Capability Simulation Model

INTERNATIONAL, HOMELAND, & NUCLEAR SECURITY HAZARD PREDICTION AND ASSESSMENT CAPABILITY SIMULATION MODEL

Sandia’s Hazards Assessment & Consequence Management Decision Analysis Capability (HazDAC) was developed to provide probabilistic end-to-end analysis and decision support for a diverse set of mitigation, risk, and threat responses to chemical, biological, and radiological attacks. A modular system-of-systems tool, this application provides multiple integrated modeling components and datasets in a single analysis center. This tool enables the capability to: analyze attack prevention or hardening measures, define threat, evaluate the effectiveness of countermeasures, define requirements, optimize detection system architectures, and provide modeling capability for decision support following an attack. Using real geometries and actual schedules, the domain data enables near actual planning. The attack module uses Monte Carlo methods to survey many threat possibilities. A dispersion module is designed to link indoor and outdoor dispersion codes: Hazard Prediction and Assessment Capability, indoor containment transport and diffusion model, and atmospheric transport and dispersion model: Hybrid Single-Particle Lagrangian Integrated Trajectory. The agent-based capability in the health effects module provides both detect-to-warn and detect-to-treat capability. The response module includes concept of operations, sensor siting, varied metrics, and reach-back analysis.

The Department of Homeland Security has applied this model to answer a variety of analysis questions including indoor detection analysis and optimization for the World Trade Center complex and bio-detection system requirements studies.

Red Sky was used to run the thousands of simulations required to span the threat and operational possibilities to properly answer analysis questions. While each individual run of HazDAC may take only minutes to hours to complete, running thousands to tens of thousands instances can be prohibitive without the use of Red Sky, which enables splitting the computational burden across hundreds of processors, thereby allowing faster analyses.

Caption: HazDAC uses the indoor contaminant transport and diffusion model, CONTAM, to simulate chemical, biological, and radiological attacks within a facility. The colored floor layout indicates areas of low, medium, and high contamination following release and dispersion.

TEAM MEMBERS: Meghan Peterson Nerayo Teclemariam Trisha Miller Nate Gleason Luke Purvis Ann Yoshimura

PROCESSING HOURS: 140,751 Contact: Meghan Peterson [email protected]

Nuclear Weapons Overview

NUCLEAR WEAPONS

Sandia’s Nuclear Weapons program manages the nation’s nuclear weapons stockpile, provides our customers with the scientific understanding needed to steward the stockpile, and manufactures specialized non-nuclear products and components for national defense and security applications. Computational simulation is playing an increasingly important role in the design and qualification of stockpile weapons and the annual assessment process that ensures their continual safety, surety, and reliability. Further, simulation is becoming more vital for the stockpile technical basis, scientific understanding, and resolution of anomalies, as well as sustaining the stockpile through support of life extension and technical maturation of improved safety and security capabilities.

Sandia’s mission demands a broad range of integrated application capabilities. Sandia has established focus areas that guide prioritization and stockpile impact, including applications to support radiation and electrical sciences, assured safety and security, delivery, and component performance. These applications are based on verification and validation studies, and supported by extensive uncertainty quantification and assurance capabilities.

The Advanced Simulation and Computing (ASC) program develops and applies advanced models, codes, and algorithms, in addition to developing and deploying high-performance computing (HPC) architectures, to meet design and qualification goals, while increasing the scientific underpinnings of the nation’s stockpile. Sandia develops and supports a broad suite of engineering and physics simulation codes, including the Sierra suite of structural, thermal, aero, and fluid mechanics capabilities, and the RAMSES suite of radiation, electromagnetic and electrical codes. Sandia’s portfolio comprises capabilities for shock physics and multiphysics, including the well-known CTH and Alegra codes, and advanced phenomenological models for engineering codes that encompass a wide range of physics, from turbulence to material decomposition and mechanical failure. These capabilities are founded on scalable parallel algorithms and libraries such as Trilinos, and enabled through state-of-the-art meshing capabilities in Cubit and the workflow capabilities of WorkBench and the CompSimUI.

Sandia is a leader in scalable architectures for extreme scale computing. We team with Los Alamos National Laboratory through the Alliance for Computing at Extreme Scales to design, deploy, and operate advanced technology platforms such as Cielo, and, in the next few years, Trinity. We are also exploring a wide range of future computing technologies through the Advanced Architecture Testbeds in order to shape the path toward exascale computing at the simulation resolutions required by long-term stockpile science challenges.

Sandia, through support of the National Nuclear Security Administration’s ASC program and our Institutional Computing program, operates and supports a broad range of high performance computing (HPC) platforms, providing multiple petaflops of computing capacity. In partnership with Los Alamos and Lawrence Livermore National Laboratories, Sandians have access to tens of petaflops of advanced computing capabilities. Our HPC team also supports a broad infrastructure including mass storage and high speed networks, and works with our partners to standardize the operating systems, tools, and user environment to provide a common responsive interface to the computational simulation user community.

Keith Matzen Director, Energy, Nuclear Weapons Science & Technology Programs

Sidebar: Simulation plays an increasing role in the stockpile technical basis, scientific understanding, and resolution of anomalies, as well as sustaining the U.S. nuclear stockpile

Stockpile Gravity Bomb-Spin Performance Using Computational Fluid Dynamics Modeling and Simulation

NUCLEAR WEAPONS STOCKPILE GRAVITY BOMB-SPIN PERFORMANCE USING COMPUTATIONAL FLUID DYNAMICS MODELING AND SIMULATION

Bomb-spin research, as part of the B61 Life Extension Program (B61-LEP) that ensures and extends the life of existing weapons systems, is a critical element of national security for the United States. Gravity bomb spin rocket motors (SRMs) fire and spin up the weapon system soon after being deployed. The spin up is a critical portion of the flight. During SRM firing, a large region of vortical flow is generated leading to interaction between the vortex and the fin. Simulating the vortex-fin interaction is an important element in predicting spin up and quantifying margins of uncertainty for the B61-LEP. Ongoing research at Sandia addresses the predictive capability for spin-to-arm aerodynamics, primarily the jet-fin interaction due to the firing of spin-rocket motors. Earlier research has shown that the current state-of-the-practice turbulence models cannot capture the dynamics of these jet-fin interactions accurately. While the class of models in this research require significantly more computer resources, their improved accuracy makes them more useful, and access to larger computers makes them more viable.

The Sierra/Conchas flow solver was used to validate the Detached Eddy Simulation (DES) model by comparison with experimental data obtained from a test designed and performed in Sandia’s wind tunnel. The test measured the pertinent flow features as a supersonic jet exhausted into a subsonic crossflow.

This research will create a more practical class of models that can reliably shed light on the flow physics of vortex/jet-fin interactions. Further, in addition to improving predictability for problems relevant to nuclear weapons, these models can enable the design of more efficient engines and newer transportation systems, and provide better understanding of industrial pollution and its effects on the environment.

Caption: The figure above is a turbulent flow field captured using Detached Eddy Simulation modeling and shows the DES results from Conchas; a simulation on 1,536 cores over 40 days of CPU time on Chama, to arrive at 0.1 seconds. This is a snapshot in time from an unsteady flow.

TEAM MEMBERS: Mary A. McWherter-Payne Srini Arunajatesan

PROCESSING HOURS: 376,614 Contact: Mary A. McWherter-Payne [email protected]

Magnetized Liner Inertial Fusion

NUCLEAR WEAPONS MAGNETIZED LINER INERTIAL FUSION

Fusion, a long-sought solution to humankind’s general energy problems, and a spinoff of nuclear research for peaceful purposes, has been studied since fission and fusion were first discovered in the 20th century. Fusion is the bringing together of two light nuclei while still destroying mass. The fusion content of a single cubic kilometer of seawater is equal to the energy content of all the world’s known remaining oil reserves. Unlike fission, the fusion nuclear process does not rely on weaponizable materials such as uranium or plutonium, so proliferation is not a concern, runaway reactions leading to meltdown are not possible, greenhouse gases are not produced, little (or no) radioactive waste is produced, the fuel supply is virtually inexhaustible, and the process itself can be extraordinarily efficient.

Sandia leverages the combined resources of HPC and the Z-Pinch Accelerator to devise and carry out the experiments that bring us closer and closer to achieving fusion. Physical experiments are first modeled using computations on HPC platforms that take many physical details and processes into account. This helps predict how the experiment will unfold and provides critical data for the engineers who build the target and conduct the experiment.

Sandia’s approach is to implode a target containing fusion fuel creating a tiny star that burns for a billionth of a second. Magnetized Liner Inertial Fusion, or MagLIF, experiments have begun in earnest at Sandia, in which solid beryllium liners containing magnetized and preheated deuterium fuel are imploded using magnetic pressure generated by the Z Accelerator. The MagLIF concept provides a path toward obtaining substantial fusion yields using the Z facility.

TEAM MEMBERS: Adam Sefkow Steve Slutz Kyle Peterson Joseph Koning

PROCESSING HOURS: 47,968,572 Contact: Adam Sefkow [email protected]

Scaling Red Sky to Process Exascale Fluid Dynamics Research

NUCLEAR WEAPONS SCALING RED SKY TO PROCESS EXASCALE FLUID DYNAMICS RESEARCH

Hydrodynamic instabilities and the transition to turbulence are century-old fundamental problems in fluid mechanics that still present formidable engineering challenges. Hydrodynamic instabilities produce pressure fluctuations, which significantly impact the vibrational loading of reentry vehicles, and strongly affect transition, which, in turn, impacts vehicle drag and heating. The ability to predict boundary-layer transition is essential for several applications. Predicting transition in gas flows requires a quantitative understanding of the relationships between molecular-level physical processes and hydrodynamic instabilities.

The direct simulation Monte Carlo (DSMC) method provides a high-fidelity representation of fundamental gas behavior by using computational molecules to simulate a gas. DSMC provides a novel way to investigate the relationship between molecular-level physics and macroscopic instabilities in a gas. DSMC represents a fundamental method to study gas behavior at the molecular level. The “statistical noise” inherent in DSMC simulations represent the fluctuations in real gases exactly. As a result, DSMC provides a pioneering method for rigorously studying hydrodynamic instabilities.

Until now, this inherent capability of DSMC has not been exercised because of the vast computational resources required to address such problems. As computers progress from petascale to exascale power, investigating hydrodynamic instabilities with DSMC is within reach. Sandia has developed a three-dimensional-DSMC code, called SPARTA, (for stochastic parallel rarified-gas time-accurate analyzer), capable of running scalably on current petascale platforms and future exascale architectures. Such machines, with millions of cores and accelerator options (graphics processing units, many-core threading, etc.), pose challenges for the design of efficient low-level data structures, partitioning of the grid, communication of particles, and load balancing. Solving those implementation problems in SPARTA, using Sandia’s high-performance computing systems, enables us to gain a deeper understanding of hydrodynamic instabilities.

Caption: Rayleigh-Taylor instability in gas mixing. A heavy (red) gas is accelerated toward a light one (green) giving rise to hydrodynamic instabilities. The mixing is not diffusive but characterized by these instabilities which in turn influence the quality of the mixing. This is a critical phenomenon in inertial confinement fusion, supersonic mixing, fuel injection in scramjets, and super nova core-collapse.

TEAM MEMBERS: Michail Gallis, PI Steve Plimpton John Torczynski William Rider Timothy Koehler

PROCESSING HOURS: 4,010,830 Contact: Michail Gallis [email protected]

Material Models of Equations of State

NUCLEAR WEAPONS MATERIAL MODELS OF EQUATIONS OF STATE

Material models, such as equations of state (EOS), electrical and thermal conductivity models, and opacity, tie various physics packages of multiphysics simulation codes together. Radiation magnetohydrodynamic (RadMHD) codes, such as ALEGRA, query the EOS for each material element every time step to determine how material states change over time. They also query electrical conductivity models to correctly calculate how magnetic fields behave. Thermal conduction models augment energy flow in a simulation. Radiation transport uses the opacity models to determine photon absorption and emission, which is another energy transfer mechanism. Small inaccuracies in material models can cause simulation codes to predict incorrect behavior. Material properties are difficult, and in some cases, impossible to measure in the regimes relevant to stockpile stewardship.

Operating on Red Sky, the quantum molecular dynamics code, Vienna Ab Initio Simulation Program (VASP), is being used to gain a quantitative understanding of materials. Simulations of several hundred atoms are run to calculate material properties based on quantum mechanics. VASP scales to between 1,000 and 2,000 cores, and large numbers of simulations on the supercomputers contribute to build databases of material properties that can then be used to develop wide-ranged, multi-phase material models employed by the RadMHD codes. The material models developed at Sandia have been applied across the NNSA laboratories and other customers.

When creating a material model, large sections of phase-space in temperature and density are mapped out. Calculations are made predicting the material response to shock and release compression and key thermodynamic properties such as the vapor-liquid critical point and phase transitions. Each of these requires between 20 and 80 simulations, taking from several days to weeks to complete. Conductivity and opacity models require 100 to 200 simulations for the same length of time or longer for required conductivity and opacity-specific parts of the calculations.

This year, Sandia researchers are building a new aluminum EOS to extend the work originally done in Desjarlais’ aluminum conductivity model: extending previous work in copper and gold conductivities, finishing work on glow discharge polymer (GDP), shock then reshock for carbon dioxide, calculating the opacity for air, and creating a new method for predicting the shock response of porous materials.

Caption: This graphic shows an iso-surface of the charge density of GDP at 1.05 g/cm^3 and 300 Kelvin. The red spheres represent carbon and the blue spheres are hydrogen. A characteristic hexagonal carbon structure indicative of benzene can be seen in the center left and another one at the top left.

TEAM MEMBERS: Kyle R. Cochrane (PI) Michael P. Desjarlais Martin French Rudolph J. Magyar Luke Shulenburger Thomas R. Mattsson

PROCESSING HOURS: 32,616,840 Contact: Kyle Cochrane [email protected]

Testbeds: A Look into the Future of HPC

NUCLEAR WEAPONS TESTBEDS: A LOOK INTO THE FUTURE OF HPC

Sandia Mission Computing hosts several advanced technology testbeds. These systems, primarily funded by the DOE/NNSA Advanced Simulation and Computing (ASC) program through the Nuclear Weapons Program Management Unit, contain leading edge computer technology that will likely appear in future HPC platforms. Access to advanced systems helps application and system software developers prepare for the disruptive computer architecture changes that have and will continue to emerge as HPC systems approach exascale computing. The testbeds, primarily pre-production prototypes to support exploration of a diverse set of architectural alternatives, are possible candidates for future pre-exascale systems. Testbeds allow for path finding explorations of 1) alternative programming models, 2) architecture-aware algorithms, 3) energy efficient runtime and system software, 4) advanced memory sub-system development, and 5) application performance, particularly using proxy applications. For more information on these testbed systems, see http://www.sandia.gov/asc/computational_systems/HAAPS.html.

Sandia also distributes open source code for modeling and simulation of parallel particles. The Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) is a modular dynamics simulator (http://lammps.sandia.gov) written in C++ with OpenMP and MPI for parallel particle simulation to model atoms or more generically, run simulations at the atomic, meso, or continuum scale. The code is easy to modify or extend with new functionality.

The image above shows the results of a study to understand potential application performance given new programming model alternatives and processor technologies. The study, run on several of the testbed systems, uses LAMMPS’ proxy application (miniMD) and new programming model libraries (e.g. Kokkos Array) to analyze new computer hardware architectural features.

Caption: As part of NNSA’s Advanced Simulation and Computing project, Sandia has acquired a set of advanced architecture test beds to help prepare applications and system software for the disruptive computer architecture changes that have begun to emerge. (Testbeds pictured: Compton, Curie, Shannon, Teller, Volta)

TEAM MEMBERS: Jim Laros, PI Char Arias Bob Ballance Jim Brandt Ann Gentile Sue Kelly Victor Kuhns Jason Repik

Contact: Sue Kelly [email protected]

Laboratory Directed Research & Development Program Overview

LABORATORY DIRECTED RESEARCH & DEVELOPMENT PROGRAM

As Sandia’s sole source of discretionary research and development funds, the Laboratory Directed Research and Development (LDRD) program functions as a catalyst for the genesis of innovative science and applied advancements in engineering and technology that serve the Department of Energy and other national security missions. Each year, following a competitive review process, the LDRD Program Office awards funding to Sandia’s scientists and engineers for approximately 400 projects (totaling $168 million in fiscal year 2013).

LDRD proposals outlining R&D work distinct from existing programs are reviewed and selected for funding by investment area team members in the following mission-relevant investment areas: Materials Science; Computing and Information Sciences; Engineering Sciences; Radiation Effects and High Energy Density Science; Nanodevices and Microsystems; Bioscience; Geoscience; Defense Systems and Assessment; Energy, Climate and Infrastructure Security; International, Homeland and Nuclear Security; Nuclear Weapons; Cyber security; Grand Challenges, and Research Challenges. Strategic Partnerships support the professional development of graduate students and new staff at Sandia through LDRD projects designated either as Campus Executive, Early Career R&D, or Truman Fellowships.

The integration of research activities and mission needs is fundamental to Sandia’s success. Expertise with and utilization of high performance computing (HPC) systems are evident throughout Sandia’s LDRD program, as demonstrated by the use of Red Sky, in roughly 15 percent of the LDRD portfolio. The insight, knowledge and science developed by LDRD projects like those highlighted on the following pages will enable technologies and capabilities that support our nuclear weapons and national security mission, while providing the reward of a technically challenging work environment for scientists and engineers at Sandia.

Sheryl Martinez Program Manager, Laboratory Directed Research and Development

Sidebar: The integration of research activities and mission needs is fundamental to Sandia’s success.

Modeling Human Adult Neurogenesis: The Role of Scale in Biologically Realistic Dentate Gyrus Networks

LABORATORY DIRECTED RESEARCH & DEVELOPMENT PROGRAM MODELING HUMAN ADULT NEUROGENESIS: THE ROLE OF SCALE IN BIOLOGICALLY REALISTIC DENTATE GYRUS NETWORKS

When new neurons are born into the adult brain, those neurons may affect how individuals think and feel. Results from computer modeling confirm that the birth of new neurons in the adult human brain could have a huge impact on its function. Specifically, the results indicate that new neurons may not only be vital for cognition, but perhaps even more important to the ability for humans to form complex “episodic” memories.

Until now, almost all neuroscience studies on adult-born neurons have focused on mice and rats, allowing questions about the relevance of new neurons to humans to persist. This study, the first of its kind, created realistically sized computational simulations of the region of the human brain that receives new neurons. Simulations of more than 10 million neurons and roughly 50 billion connections between neurons were run on Red Sky. The results simulated every neuron in the dentate gyrus region of the human brain.

Sandia research results show that the effects of scale are not trivial. Human brains cannot be thought of as simply bigger mouse brains. Thus, findings cannot be extrapolated from mouse-sized studies. This research has potentially broader impact than understanding the value of new neurons. One effect is a deepened understanding of when and how research on mouse-sized brains can be applied to humans. Importantly, the ability to build computational models at both human and mouse scales will continue to improve our ability to draw conclusions about the human brain from smaller brains. In the future, we plan to continue to use realistic-scale computational neuroscience modeling to improve our understanding of human memory and we hope that this work can help us use research from mice and rats more effectively. This work is also being applied to Beyond Moore’s Law computing research at Sandia through the potential to influence brain-inspired computing and pattern recognition in signal processing.

Caption: “The Brain is a Computer” – The dentate gyrus of a mouse hippocampus with adult-born neurons (green), seen through a microscope and represented as a computer model.

TEAM MEMBERS: Brad Aimone, PI Craig Vineyard Steve Verzi Mike Bernard

PROCESSING HOURS: 1,183,751 Contact: Brad Aimone [email protected]

Molecular Modeling of the Attachment of the Dengue Virus to Cellular Membranes

LABORATORY DIRECTED RESEARCH & DEVELOPMENT PROGRAM MOLECULAR MODELING OF THE ATTACHMENT OF THE DENGUE VIRUS TO CELLULAR MEMBRANES

Sandia theorists, in collaboration with experimentalists, are undertaking computational bioscience research to understand how viruses infect cells. Their findings may help determine how to block viral infections and prevent pandemics. The virus under study is Dengue virus (DENV). DENV is endemic to tropical regions and infects some 50 to 100 million individuals, accounting for 500,000 hospitalizations annually, according to the World Health Organization.

DENV infects a cell with help from a protein on the surface of the virus. The protein binds to the cell, leading to engulfment of the virus into vesicles (endosomes) within the cell. Infection culminates when the protein attaches to the endosomal membrane and brings the viral and endosomal membranes together for fusion and release of the viral genome into the cell cytoplasm. Sandia researchers are using atomistic and continuum-level simulation models run on Red Sky to understand how the viral fusion protein catalyzes membrane-membrane fusion. Experimental work determines attachment of the free fusion protein, separated from the virus, to membranes with different lipid compositions. Using theoretical models, researchers computed the free energy gained by protein attachment to the membranes and identified key chemical interactions between lipids and the viral protein that either facilitate or prevent membrane-to-membrane fusion and cell infection.

This work is the first molecular study of DENV infections to determine the free energy of protein binding to cell membranes and how that energy depends on membrane composition. The work also presents a novel pathway to computing that thermodynamic property. These results lay groundwork for developing an atomic-level understanding of viral protein infectivity that can aid the development of vaccines and therapeutics to block infection.

Caption: Hydration of the free viral fusion protein during insertion into cellular membranes with different lipid composition. Water density normal to the cell membrane surface (z, plotted on the horizontal axis) is plotted for each separation distance between the tip of the fusion protein and the membrane. Black lines trace the geometric center of the fusion protein during insertion. White lines outline regions that lack water in the hydrophobic layers of the membrane (dark blue) and regions of liquid water near the membrane surface (dark red). The simulation box is constructed so that half the membrane appears on left and right sides; periodic boundaries ensure that no interfaces with water vapor form. (Top) The membrane contains cholesterol (‘Chol’) and is uncharged. The cholesterol is in equal proportions with phosphatidyl choline (PC) and phosphatidylethanolamine (PE). (Middle) One-third of the membrane consists of anionic lipids (PG) and thus the membrane is charged. More water solvates the protein near the interface with the charged membrane compared to the uncharged membrane. Yet, little water is carried into the membrane by the fusion tip. Thus, hydration inside the membrane does not account for stronger attachment between the fusion protein and anionic membranes, compared to neutral membranes, that leads to membrane fusion and cell infection. (Bottom) Schematic showing hydrating waters (blue spheres) and ions (red spheres) at one separation distance during insertion of the viral fusion protein into an anionic cellular membrane.

TEAM MEMBERS: David Rogers Susan Rempe Mike Kent

PROCESSING HOURS: 1,432,249 Contact: Susan Rempe [email protected]

A New Capability to Model Defects in Semiconductor Alloys

LABORATORY DIRECTED RESEARCH & DEVELOPMENT PROGRAM A NEW CAPABILITY TO MODEL DEFECTS IN SEMICONDUCTOR ALLOYS

Nearly all modern electronic and opto-electronic devices are fabricated from crystalline semiconductor materials that contain point defects (point-like deviations from perfect crystalline order). These defects limit the performance of semiconductor devices. Device engineers need to know what defects are in the devices, how the defects influence device performance, and how they evolve over time. Experimental studies of this type are challenging. Density-functional theory (DFT), a quantum mechanical modeling method, is often used to obtain information about defects. However, DFT capabilities for studying defects are currently limited to elemental and binary semiconductors, whereas many devices of interest to Sandia and the commercial sector are fabricated from semiconductor alloys. This project is developing a new DFT capability that can be used to obtain information about defects in these technologically important semiconductor materials, potentially benefiting commercial technologies such as solid-state lighting and solar cells.

Due to the chemical disorder inherent in semiconductor alloys, the ability to obtain defect information in these materials depends heavily on the use of state-of-the-art high-performance computing capabilities at Sandia. For example, the computational effort needed to compute defect properties in an alloy such as indium gallium arsenide is roughly 200 times greater than the effort needed to compute the same defect properties in a binary compound, such as gallium arsenide. The new capability under development is able to obtain not only the stable-state properties of defects, but also their mobilities.

Caption: New DFT Capability: Diffusion processes of point defects in semiconductor alloys. The top graphic shows movement of an arsenide interstitial defect to a lower energy configuration through a saddle point. Below, atomic configurations of an arsenide interstitial defect corresponding to the configurations noted above.

TEAM MEMBERS: Alan Wright, PI Stephen Lee

PROCESSING HOURS: 12,343,429 Contact: Alan Wright [email protected]

Network and Ensemble Enabled Entity Extraction in Informal Text

LABORATORY DIRECTED RESEARCH & DEVELOPMENT PROGRAM NETWORK AND ENSEMBLE ENABLED ENTITY EXTRACTION IN INFORMAL TEXT

Before analysts can analyze data or “connect the dots,” they have to first identify “the dots” or entities of interest. This project used a novel combination of methods to improve the ability to automate entity identification. The first step in this process is the ability to accurately extract all information. This is straightforward for humans but it poses quite a challenge for computers. The effort is particularly difficult when computers attempt to identify information within informal text; for example, casual email or blogs that often have spelling mistakes, use abbreviations, jargon, or colloquial language. To accomplish this, analysts often turn to natural language processing (NLP) tools.

Using innovative algorithms, high-performance computing, and Sandia custom-developed code, researchers are able to improve the accuracy of entity extraction from informal text. This research combined ideas from two existing NLP methods: conditional random fields (CRFs) and ensemble methods. CRFs are supervised machine learning considered to be the best-known option for entity extraction in text. An ensembles method applies a powerful statistical inference meta-method, popular because it can make most supervised machine learning methods more accurate. Through new understanding of the fundamental character of CRFs, Sandia researchers were able to design CRF algorithms that integrate with ensembles, and outperform prior methods. Using these two methods together has provided researchers with more reliability and better results when extracting information from informal text.

Caption: Understanding CRF performance requires analysis and visualization tools for understanding the relationship of the attributes of the words in a text. Here we depict the Review of Kolmogorov-Smirnov statistical dissimilarity of those attributes for just one of the many links in the factor graph model underlying a CRF.

TEAM MEMBERS: Philip Kegelmeyer, PI Tim Shead Danny Dunlavy

PROCESSING HOURS: 480,755 Contact: Philip Kegelmeyer [email protected]

High Performance Computing Highlights Overview

HIGH PEFORMANCE COMPUTING HIGHLIGHTS

In 2012, Sandia’s leadership approved and established an ongoing Institutional Computing program charged with continuing investment in Sandia’s high performance computing (HPC) capabilities. In addition to traditional scientific computing platforms, this program will acquire and support other types of large-scale computing platforms optimized for fundamentally different types of problems, such as big-data/informatics and cloud/emulytics modeling that are part of existing or emerging Sandia mission areas. The Institutional Computing program and the Advanced Simulation and Computing (ASC) program—funded through NNSA as part of the NW program—complement and leverage each other. The Institutional Computing program makes high-performance computing capabilities available to Sandia programs outside of NW, while also providing additional capacity for NW modeling and simulation work. In addition, the two programs leverage the technical expertise within the Laboratories for systems and user support as well as modeling and simulation development and analysis. Where appropriate, the programs also share in providing the facilities, infrastructure, and computing environments necessary to house, maintain, and operate these HPC capabilities.

Before the establishment of the Institutional Computing program, the funding of institutional HPC systems was ad-hoc, resulting in problematic planning and management of these capabilities. With Laboratories leadership’s commitment to the Institutional Computing program, we can now engage in deliberate, priority driven investments, broaden the scope to include nontraditional systems, and take advantage of the large procurement activities planned and managed through the ASC program to optimize the value obtained from these investments.

The articles presented in the previous sections detail the results that Sandia’s scientists and engineers are delivering to their customers using our traditional high-performance computing resources. The following section of the Annual Report describes some of the new systems and capabilities that were stood up in fiscal year 2013. These examples give an indication of the types of future capabilities and programmatic impact the HPC programs will provide to enable execution of Sandia’s broad mission areas.

Tom Klitsner Senior Manager, Computing Systems and Technology Integration

Searching the Sky: Finding Anomalous Flights through Large Data Analysis

HIGH PERFORMANCE COMPUTING HIGHLIGHTS SEARCHING THE SKY: FINDING ANOMALOUS FLIGHTS THROUGH LARGE DATA ANALYSIS

There are well over 50,000 flights per day in U.S. airspace, nearly all of which are utterly ordinary. However, just knowing that is not enough.

Sandia researchers set out to solve the problem of identifying flights that are irregular or suspect. At 50,000 flights per day, any body of data large enough to contain more than a handful of anomalies takes weeks to process on even a well-equipped server. The analysis has much in common with database operations but involves trajectory and geometric computations that are foreign to database query languages.

Given the options of writing a database to run on supercomputers or writing computational primitives that run on a database, researchers opted for the latter, using an IBM Netezza Data Analytics System 1000, TwinFin, to store the data and adding computational geometry routines to the standard suite of database primitives. Databases are heavily optimized for generating summaries that allow quick characterization of normal behavior and finding trajectories that fall outside that profile. Although databases with spatial capability are not new, using them for trajectory analysis with sophisticated algorithms at large scale is a major innovation.

Caption: Two images above illustrate the results using air traffic near Midway and O’Hare airports in Chicago. The image on the left shows all the traffic. The image on the right highlights (in white) one particularly unusual flight identified from its geometric properties. Further research revealed that the aircraft in question was a survey plane equipped for LiDAR and overhead photography. Finding this kind of flight would normally take a stroke of luck or lengthy intensive searching for a human analyst but an algorithmic approach on a parallel system found it within seconds.

TEAM MEMBERS: Andy Wilson, PI Tom Otahal Jason Shepherd

Contact: Andy Wilson [email protected]

Uno, Sandia’s High-Throughput Computing Cluster

HIGH PERFORMANCE COMPUTING HIGHLIGHTS UNO, SANDIA’S HIGH-THROUGHPUT COMPUTING CLUSTER

Designing, building, and operating large-scale high-performance computing (HPC) resources have been the primary direction for innovation in scientific computing for the last two decades. Clusters are usually designed to run tightly coupled parallel jobs — jobs that run across many processors with frequent, high-bandwidth intercommunication. The current generation of parallel jobs also tends to be highly synchronous, so computation, communication, and file input/output occur in waves. Such parallel jobs run only as fast as the slowest component in the component’s partition. Because of this, most HPC clusters are designed and operated using a large number of identical components, balanced in speed and capabilities, with networking and file systems scaled to meet the demands of the largest jobs on the cluster.

Recently Sandia has seen an increase in the number of ensemble calculations—workloads that consist of hundreds or thousands of single-processor jobs—each of which runs independently from the others. This workload characteristic changes many of the design constraints for HPC clusters.

This year Sandia released Uno, a system designed for single node computing jobs. By design, Uno runs jobs that do not require either a substantial interconnection network or large file system bandwidth. Instead Uno’s interconnect and file systems are tuned to support many small asynchronous activities. Uno, and similar “high-throughput” clusters, allow scientists and engineers to conduct production runs of single node compute jobs, for a variety of end-purposes.

Uno supports several usage cases: running ensemble calculations, running small memory-intensive tasks, or experimenting with graphical processing units. Internally, Uno is provisioned with three different node types: dual-socket nodes for normal jobs, quad-socket jobs that support larger shared memory jobs, and accelerated nodes that contain graphical processing units. Altogether, Uno provides 72 teraflops of central processing unit-based computing.

Caption: UNO, high through-put cluster

TEAM MEMBERS: Steve Monk, PI Jay Livesay Kevin Kelsey Jeff Ogden Mark Schmitz Erik Illescas

Contact: Steve Monk [email protected]

Passive Optical LAN & 100G Access Expansion

HIGH PERFORMANCE COMPUTING HIGHLIGHTS

SANDIA DEPLOYS WORLD’S LARGEST ENTERPRISE PASSIVE FIBER OPTICAL LOCAL AREA NETWORK

Computer network researchers at Sandia have deployed the largest fiber optical local area network in the world. Hailed in Popular Science’s “Best of What’s New 2013” collection (http://www.popsci.com/bown/2013/product/sandia-national-laboratories-fiber-optic-network), the new gigabit passive optical network infrastructure reaches 265 buildings and 13,000 computer network ports that bring high-speed communication to some of the Laboratories’ most remote areas for the first time. Sandia replaced its conventional four-inch copper cable with a half-inch fiber-optic one that is capable of transferring voice, computer, and security data along a single line. The conversion will save an estimated $20 million over the next five years through a 65 percent energy savings and by not having to purchase replacement equipment. During this effort, Sandia has demonstrated “green computing” by diverting 7.2 tons of waste from landfills and waste streams through trading in decommissioned equipment.

TEAM MEMBERS (Passive Optical LAN): Steve Gossage, PI Jerry Davis Troy Holley Jimmie Wolf, PL Dan Garcia, PL Jim McPhee Janice Vaughan Marc Miller Ron Moody Roger Adams Joe Brenkosh Phil Ayala Joseph Maestas Monzy Merza

Contact: Steve Gossage [email protected] Caption: Steve Gossage, a senior engineer at Sandia National Laboratories, looks at fiber optics in a cable box that replaced heavier and bulkier copper cable for high-speed communications throughout much of the labs. Fiber offers more capacity and is more reliable than copper.

SANDIA COLLABORATES TO EXPAND 100G ACCESS FOR NEW MEXICANS

Sandia recently upgraded its connection to the Internet from 1 to 10 Gigabits/second (Gb/s). Along with its leading-edge desktop distribution network, Sandia is also pushing the boundaries of production Internet access throughout New Mexico. In collaboration with Los Alamos National Laboratory and New Mexico Internet provider, Century Link, Sandia is working to upgrade the communication infrastructure in New Mexico from a 10 Gb/s to a 100 Gb/s Dense Wavelength Division Multiplexing system.

An area of serious concern when increasing corporate Internet bandwidth, is the ability to perform cyber security functions at the increasing rates. Demonstrating the diverse strengths of Sandia, the networking group teamed with its cyber security and HPC departments to develop an HPC capture and analytics tool. The tool is capable of capturing, deep packet indexing, and storing 16 Gb/s of network traffic; improving search and retrieval of important events by an order of magnitude.

TEAM MEMBERS (100G Access): John Naegle, PI Robert Mason Luis Martinez Joseph Maestas Roger Adams Sil Han Uzoma Onunkwo Jim Schutt

Contact: John Naegle [email protected]

HPC User Support

HIGH PERFORMANCE COMPUTING HIGHLIGHTS HPC USER SUPPORT

Sandia’s user support team for high performance computing (HPC) provides “OneStop” service. A tiered support system of knowledgeable professionals provides responsive, as-needed support emphasizing quality and user success. Services include an HPC help desk, HPC account services, a OneStop web portal, and email notifications that deliver a wealth of information regarding platforms, systems status and outages, applications, “how-to’s,” user-specific job information, and access to support professionals. Our team provides customized in-depth support when needed to explore issues related to code porting, performance analysis, handling of large data, visualization—whatever it takes to help users get their HPC work done, particularly when help is needed for complex runs or for managing complicated workflows.

OneStop HPC support is implemented using Information Technology Infrastructure Library® (ITIL) as a framework for IT service management. This framework addresses the full life cycle of support from incident management, change management, and knowledge management through best practices. ITIL is also the foundation of ISO/IEC 20000 (previously BS15000), the International Service Management Standard for organizational certification and compliance.

The HPC OneStop portal is accessible on the Sandia Restricted Network at https://computing.sandia.gov. The OneStop service desk is available Monday through Friday from 8:30 a.m. to 4:30 p.m. MDT. The service desk can also be reached through the phone or email.

Contact Channels:

• Call HPC@CCHD: 845-2243 Select Option 8
• Call HPC Service Desk: 844-9328
• E-mail: [email protected]
• Web: https://computing.sandia.gov/help

Systems Summary - Part 1

SYSTEMS SUMMARY MISSION COMPUTING SYSTEMS

CHAMA USAGE: HPC PROGRAM: NW/ASC TFLOPS: 392 NODES: 1,232 CORES: 19,712 MEMORY/CORE: 2.0 GB* PROCESS HOURS/YR: 172,677,120 Description: Chama, along with Pecos, is a NW/ASC HPC system deployed in 2012 as part of the DOE/NNSA Tri-Labs TLCC–2 procurement. At 392 tflops, Pecos is a primary resource for NW/ASC users on the unclassified network. In October 2013, Chama, like Pecos, was upgraded to double its available memory on every compute node. This upgrade is expected to improve job throughput and reduce compute times by allowing applications to run more compactly within the same overall memory footprint.

CIELO DEL SUR USAGE: HPC PROGRAM: NW/ASC TFLOPS: 86 NODES: 556 CORES: 8,896 MEMORY/CORE: 2.0 GB PROCESS HOURS/YR: 77,928,960 Description: Cielo del Sur is a Cray XE6 system that supports users of classified computing as part of the National Security Computing Center operated by Sandia on behalf of DOE.

DARK BRIDGE USAGE: HPC PROGRAM: NW/ASC TFLOPS: 294 NODES: 924 CORES: 14,784 MEMORY/CORE: 4.0 GB PROCESS HOURS/YR: 129,507,840 Description: Dark Bridge is a TLCC–2 system (like Chama and Pecos) that supports users of classified computing as part of the National Security Computing Center operated by Sandia on behalf of DOE.

DARK NEBULA USAGE: Cloud PROGRAM: Institutional TFLOPS: NA NODES: 100 CORES: 1,600 MEMORY/CORE: Varies PROCESS HOURS/YR: 14,016,000 Description: Dark Nebula is an institutional cloud system that was acquired at the end of FY13 for use as a research cloud in FY14. The system is composed of 100 Dell r720 nodes having local disk and bound together with a highly configurable Ethernet fabric. An OpenStack cloud environment will support multiple research groups who are experimenting with “infrastructure as a service.”

DARK SAND USAGE: HPC PROGRAM: Institutional TFLOPS: 294 NODES: 924 CORES: 14,784 MEMORY/CORE: 4.0 GB PROCESS HOURS/YR: 129,507,840 Description: Dark Sand is a TLCC–2 system (like Chama and Pecos) that will support users of classified computing as part of the National Security Computing Center operated by Sandia on behalf of DOE.

EBIRD USAGE: HPC PROGRAM: Institutional TFLOPS: 5.5 NODES: 383 CORES: 766 MEMORY/CORE: 3.0 GB PROCESS HOURS/YR: 6,710,160 Description: Ebird is an institutional HPC system that was the last remaining section of the Thunderbird cluster. During FY13, it served a variety of customers on the open network. Ebird is scheduled for decommission in FY14, as the Red Mesa systems shifts into use by a wider audience.

GILA USAGE: HPC PROGRAM: Institutional TFLOPS: 10 NODES: 48 CORES: 768 MEMORY/CORE: 4.0 GB PROCESS HOURS/YR: 6,727,680 Description: Gila is a newly deployed Institutional HPC system destined for use in classified computing.

GLORY USAGE: HPC PROGRAM: NW/ASC TFLOPS: 38 NODES: 272 CORES: 4,352 MEMORY/CORE: 2.0 GB PROCESS HOURS/YR: 38,123,520 Description: Glory is a NW/ASC HPC system deployed in 2009 as part of the DOE/NNSA Tri-Labs TLCC–1 procurement. Now supplanted by the TLCC–2 systems, Glory ended FY13 as an NW-only resource and will transition to Institutional usage in FY14.

JEMEZ USAGE: HPC PROGRAM: Institutional TFLOPS: 95 NODES: 288 CORES: 4,608 MEMORY/CORE: 2.0 GB PROCESS HOURS/YR: 40,366,080 Description: Jemez is a new institutional HPC system acquired during FY13 for scheduled production during FY14. At 96 tflops peak, this system will be a major and cost-effective addition to the institutional resources available to users who are performing classified computing.

Systems Summary - Part 2

SYSTEMS SUMMARY (Continued)

MINI SEQUOIA USAGE: HPC PROGRAM: NW/ASC TFLOPS: 107 NODES: 512 CORES: 8,192 MEMORY/CORE: 1.0 GB PROCESS HOURS/YR: 71,761,920 Description: Mini Sequoia is a small version of the Sequoia system recently deployed at LLNL. The purpose of Mini Sequoia is to provide local code-development and checkout system for the application teams that support Sandia users on Sequoia.

MUZIA USAGE: HPC PROGRAM: NW/ASC TFLOPS: NA NODES: 20 CORES: 320 MEMORY/CORE: 2.0 GB PROCESS HOURS/YR: 2,803,200 Description: Muzia is a small-scale Cray XE6 system acquired as part of the Sandia/LANL partnership that manages the Cielo platform at LANL.

PECOS USAGE: HPC PROGRAM: NW/ASC TFLOPS: 392 NODES: 1,232 CORES: 19,712 MEMORY/CORE: 2.0 GB* (*Memory increased to 4 GB during September 2013) PROCESS HOURS/YR: 172,677,120 Description: Deployed in 2012 as part of the DOE/NNSA Tri-Labs TLCC–2 procurement, Pecos is a primary resource for NW/ASC users. Pecos was upgraded in October 2013 to double its available memory on every compute node. This upgrade is expected to improve job throughput and reduce compute times by allowing applications to run more compactly within the same overall memory footprint.

PLATO USAGE: Analytics PROGRAM: Institutional TFLOPS: NA NODES: 51 CORES: 816 MEMORY/CORE: 6.0 GB PROCESS HOURS/YR: 7,148,160 Description: Plato is a newly deployed Hadoop cluster which will enter production in FY14. The HP-based system runs out-of-the box Cloudera’s CDH enterprise product. Plato is intended to serve dual roles.

RED MESA USAGE: HPC PROGRAM: Institutional TFLOPS: 180 NODES: 1,920 CORES: 15,360 MEMORY/CORE: 1.5 GB PROCESS HOURS/YR: 134,553,600 Description: Red Mesa is an Institutional HPC system on the collaborative network. Red Mesa is based on the Red Sky platform architecture, and during FY13 was partially supported by the National Renewable Energy Laboratory (NREL). The Red Mesa collaboration shows how Sandia can partner effectively with outside organizations. In FY14, Red Mesa is scheduled to shift into wider use by ECIS and other Partners.

RED SKY USAGE: HPC PROGRAM: Institutional TFLOPS: 264 NODES: 2,823 CORES: 22,584 MEMORY/CORE: 1.5 GB PROCESS HOURS/YR: 197,835,840 Description: Red Sky is the unclassified component of the Red Sky cluster. Deployed in 2010, Red Sky and Red Sky (C) have been the workhorses of institutional HPC computing. Red Sky was developed in collaboration with Sun Microsystems, and was the first large-scale HPC system to deliver an Infiniband interconnect based on a Torus network.

TUNNISON USAGE: HPC PROGRAM: NW/ASC TFLOPS: 2.6 NODES: 190 CORES: 380 MEMORY/CORE: 1.0 GB PROCESS HOURS/YR: 3,328,800 Description: Tunnison is the last remaining 2003-vintage cluster at Sandia. It is currently used more for data transfer than for HPC, but continues to provide effective local computing for users at Sandia California.

TWINFIN USAGE: Analytics PROGRAM: Institutional TFLOPS: NA NODES: NA CORES: 0 MEMORY/CORE: 0.0 GB PROCESS HOURS/YR: 0 Description: TwinFin is an IBM/Netezza appliance for structured- and semi-structured search. TwinFin came online for friendly users early in FY13, and is now moving to full production. The system integrates proprietary hardware and software to accelerate structured search integrated with data analytics.

UNITY USAGE: HPC PROGRAM: NW/ASC TFLOPS: 38 NODES: 272 CORES: 4,352 MEMORY/CORE: 2.0 GB PROCESS HOURS/YR: 38,123,520 Description: Unity is a NW/ASC HPC system deployed in 2009 as part of the DOE/NNSA Tri-Labs TLCC–1 procurement. Now supplanted by the TLCC–2 systems, Unity ended FY13 as an NW-only resource and will transition to institutional usage in FY14.

UNO USAGE: HPC PROGRAM: NW/ASC TFLOPS: 8.64 NODES: 25 CORES: 400 MEMORY/CORE: 2.0-8.0 GB PROCESS HOURS/YR: 3,504,000 Description: Uno is the first high-throughput cluster deployed at Sandia. Based on a Dell compute node, Uno is designed to provide high-throughput and fast turnaround for single-node jobs. Systems designed to run large jobs have to provide a high-bandwidth, low-latency interconnect, and need to keep every node as similar as possible. These requirements stem from the fact that parallel jobs run at the speed of their slowest component. In contrast, Uno provides a variety of heterogeneous nodes (small and large memory, processors and accelerators) with its interconnect and file systems tuned for single-node activities.

WHITNEY USAGE: HPC PROGRAM: NW/ASC TFLOPS: 38 NODES: 272 CORES: 4,352 MEMORY/CORE: 2.0 GB PROCESS HOURS/YR: 38,123,520 Description: Whitney, located at Sandia California, is a NW/ASC HPC system deployed in 2009 as part of the DOE/NNSA Tri-Labs TLCC–1 procurement. Now supplanted by the TLCC–2 systems, Whitney will transition into alternative use in FY14.

Back Cover

SANDIA MISSION COMPUTING 2013 HPC ANNUAL REPORT

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND number: 2014-0803 P. Approved for unclassified, unlimited release.

Scanning this code with an iPhone or Android will provide access to an application that can be downloaded to a smart phone. This will enable readers to use their smart phone devices to scan images contained in this report and run simulations. This functionality is under development and will be completed by the third quarter of fiscal year 2014.