Sensors and instruments are the foot soldiers of science. They gather and generate the data that fuels investigation of phenomena ranging from the chemical reactions in rippling coastal waters to the energy rippling from quasars in deep space.
While the data provided by sensors and instruments are a boon, managing, processing, and storing the flood of data presents a challenge. NCSA is collaborating with the National Optical Astronomy Observatory (NOAO) to develop solutions for managing the tens to hundreds of gigabytes of data generated each night by its observatories. Using these solutions as first steps, the ultimate goal is to meet the needs of the Large Synoptic Survey Telescope (LSST); when LSST begins operation in 2013 it will generate an estimated 15 terabytes of raw data and more than 100 terabytes of processed data every night, 365 days a year. NCSA leads two of the three LSST data management teams and is responsible for managing the integrated work plans, tracking progress, reporting to the LSST Corporation, and coordinating a series of data challenges.
“Our NOAO/NCSA relationship provides very natural growth to LSST,” says Chris Smith, astronomer and manager of the Data Products Program at NOAO, the group in charge of developing and operating the NOAO data management system. “There are large datasets accumulating today, and really cutting-edge developments are being implemented and are benefiting scientists tomorrow rather than in 2013.”
Working closely with domain scientists is a hallmark of NCSA’s approach to the development of cyberenvironments—which encompass and integrate distributed computing and data resources, scientific application codes, workflow tools, and user-friendly interfaces into end-to-end scientific processes. NOAO brings scientific depth to the collaboration, while NCSA brings both technological expertise and experience in supporting astronomy research.
“We can talk to each other very well, and we complement each other very well,” Smith says.
“The great thing about this partnership with NOAO is we have a living, breathing community to work with,” says Ray Plante, a senior research scientist at NCSA and the leader of the center’s LSST effort.