A Vision for Cluster Security
NCSA is developing a software tool for visually monitoring the security of computing clusters
Clusters of computers are used by businesses and research centers because they are capable of performing trillions of calculations each second, storing trillions of bytes of data, and communicating quickly over high-speed networks.
Unfortunately, these same capabilities also make clusters an attractive target for malice and mischief, as hackers try to hijack the computing power to run their own calculations (such as password cracking), to launch denial-of-service attacks against Internet sites, or to warehouse unauthorized files (such as illicit copies of copyrighted material).
Cluster security is a prime concern at the National Center for Supercomputing Applications (NCSA), which is home to about 30 teraflops of computing power, much of it provided by massive clusters composed of hundreds of nodes.
As clusters become larger and larger, they become more and more complex to monitor. NCSA's Mercury Linux cluster, for example, comprises 1,776 processors, while the center's Tungsten Linux cluster contains more than 2,500 processors; a single system administrator simply can't manage such a large cluster without automation.
And while there are many tools for monitoring security of enterprise networks and a few tools for monitoring the performance of nodes in a cluster, there are no tools specifically designed for monitoring security on a cluster.
NCSA's Cluster Security (Cluster-Sec) research team is addressing this need by researching and developing a software tool to monitor cluster security. The team's work has been funded over the past 18 months by the Technology Research, Education, and Commercialization Center (TRECC), an initiative of the University of Illinois that is funded by the Office of Naval Research and administered by NCSA. (This work is also related to the NVisionIP security tool being developed by the NCSA SIFT team).
The Cluster-Sec team, led by NCSA senior systems security engineer Bill Yurcik, has developed NVisionCC (with the "CC" denoting "cluster computing"), an innovative software tool that collects and synthesizes data from heterogeneous sources and presents the information through an easy-to-understand visual interface. On a single screen, NVisionCC provides an overview of the cluster and generates alerts that pinpoint specific nodes where the data indicates a potential security breach.
A key step in the development of NVisionCC was the team's realization that the many nodes on a cluster actually fall into a small number of classes. Most of the nodes in a cluster are compute nodes (which are allocated to users to run serial or parallel jobs), some are head nodes (used to access the cluster, compile software, and submit and monitor jobs), some are storage nodes (to hold datasets), and some are management and monitoring nodes (which typically are accessible only to the cluster administrators). Instead of trying to focus on a large cluster consisting of hundreds or thousands of nodes, an administrator can instead easily comprehend a small number of node classes that are typically homogeneous.
Profiles can be set for each class of node, defining the parameters of acceptable, secure use for that type of node, including the processes that are allowable on that type of node, the ports that can be used, etc.
The individual nodes in a cluster can be separated into these classes, and NVisionCC then compares the incoming data on each node to the acceptable profile for that node category. In this fashion multiple compute nodes, for example, can be quickly compared to a single profile and any activity that falls outside the defined profile can be flagged for a system administrator to examine in more detail.
NVisionCC currently includes:
- a Process Monitor Module that tracks the processes running on each node,
- a Port Scanner Module that scans each node for open network ports,
- a File Integrity Module that validates the identity of disk files, particularly those files that hackers frequently try to alter for their own ends.
Real-time data from these three modules are compared to the configured profiles for each type of node and visualized on an interface plug-in extension of Clumon, a cluster performance monitoring tool developed at NCSA that is widely used on clusters worldwide.
As the team continues to develop NVisionCC, they plan to include a traffic analyzer, which will compare network traffic with the cluster communication pattern and will correlate the network traffic with the job scheduler, and a log analyzer, to analyze the cluster's system logs.
NVisionCC is installed on a 12-node cluster at TRECC's facility in West Chicago. The cluster security tool was previously tested on two larger NCSA production clusters (134-node "Titan" and 520-node "Platinum") that have recently been decommissioned.
Clumon users can contact Bill Yurcik for beta-testing opportunities. NVisionCC will also be on display in the NCSA booth at SC04 in Pittsburgh (Nov. 8-11).
Access Online | Posted 10-19-2004
