Sherlock Deploys UC’s first HIPAA-compliant Hadoop based Data Management System

Published October 30, 2017

The Health Cyberinfrastructure Division at the San Diego Supercomputer Center (SDSC) at the University of California San Diego, in partnership with University of California Office of the President (UCOP) Risk Services, has deployed a secure, HIPAA-compliant Hadoop-based Data Management System (DMS) within Sherlock Cloud.

This secure platform will provide a mechanism to collect, process, and transform UC Healthcare data from multiple sources and various formats into a single, integrated data set to enable UCOP Risk Services better manage UC’s liability program.

SDSC’s Health CI Division will support the UCOP Risk Services and UC Medical Centers initiative to deliver a consolidated data warehouse. This platform will facilitate access to data, reports, and advanced analytics, allowing for more informed decision-making to support quality and risk improvement programs in place throughout the UC Medical Centers. Not only will DMS improve how the data is collected, stored, and analyzed, but it will also mitigate claims costs and ensure the highest quality of care within the UC Medical Centers system-wide.   

When conceptualizing the DMS, the team decided on an alternative approach to the traditional data warehouse and opted to apply the “Big Data” paradigm to this system.  In doing so, the DMS infrastructure utilizes Cloudera’s Hadoop technology, which provides several benefits such as:

• Allowing raw data to be stored and transformed into a consumable form without the need to design a schema;
• Addressing data volume by scaling out as opposed to scaling up, which helps to preserve the efficiency with which data is processed and maintains costs;
• Facilitating storage of unstructured data.

“This platform presents an opportunity to demonstrate how best-of-breed Big Data technology solutions can be used to support large-scale data management and analytics for the varied types of protected data managed by UC, all operating within the secure boundaries of Sherlock Cloud,” said Sandeep Chandra, Executive Director of Sherlock Cloud. “The valuable experience we have gained in standing up this unique platform presents an opportunity for us to offer such capability to others in the research community. This initiative naturally builds on the successful partnership SDSC has had with UCOP Risk Services in supporting many of their business and technology needs, and we are excited to see it grow.”

About SDSC’s Health Cyberinfrastructure Division

SDSC’s Health Cyberinfrastructure Division focuses on providing innovative, secure information technology and data services for a wide range of initiatives for the UC system, academia, and state and federal government agencies. It is an SDSC Center of Excellence for secure HIPAA- and FISMA-compliant managed cloud hosting, and has recently added NIST CUI and CSF compliant managed cloud hosting to its repertoire. Launched under the brand Sherlock, its major services – Cloud, Compliance, Cybersecurity, and Data Lab – provide a secure foundation for a wide range of research and data collection initiatives. The Health CI Division supports a variety of entities including the Centers for Medicare and Medicaid Services (CMS), National Institutes of Health (NIH), and University of California Systems. For more information, please visit the Sherlock website.

About SDSC

As an Organized Research Unit of UC San Diego, SDSC is considered a leader in data-intensive computing and cyberinfrastructure, providing resources, services, and expertise to the national research community, including industry and academia. Cyberinfrastructure refers to an accessible, integrated network of computer-based resources and expertise, focused on accelerating scientific inquiry and discovery. SDSC supports hundreds of multidisciplinary programs spanning a wide variety of domains, from earth sciences and biology to astrophysics, bioinformatics, and health IT. SDSC’s petascale Comet supercomputer continues to be a key resource within the National Science Foundation’s XSEDE (Extreme Science and Engineering Discovery Environment) program.