How to run Neuron in Azure Synapse

Neuron is a dynamic data engine for automating insights into population health, equipped with capabilities for executing HEDIS, Stars, and Risk Adjustment measures.

‍

It's designed for seamless integration directly into your cloud infrastructure. Neuron supports deployment across various cloud platforms, including Azure, AWS, and GCP, providing flexible cloud runtime options. In today's discussion, we'll delve into utilizing Neuron within the Azure Synapse environment.

‍

Azure Synapse is an advanced analytics service from Microsoft, part of the Azure suite, crafted to unify big data and data warehousing technologies. This service enables organizations to efficiently process and analyze massive data volumes in real-time.

‍

I'll guide you through the process of harnessing Azure Synapse to apply Neuron on your population health data, allowing you to explore the insights within an Azure Data Lake. This walkthrough will cover manual setup via the Azure Portal, though the end objective is to embed Neuron into an automated workflow. This integration aims to streamline the delivery of daily updated metrics into a consolidated data repository.

‍

Let’s dive in ...

‍

1. Deploy Neuron to  your Azure Storage account

Neuron is packaged as a .jar file for straightforward deployment into your Azure account, utilizing an Azure Storage account for storage. This arrangement not only facilitates easy access by Azure Synapse for job execution but also leverages Azure's scalable storage solutions and global data center availability.

‍

The deployment process can also be enhanced through the use of Azure DevOps, offering a streamlined pipeline for continuous integration and delivery. This setup ensures that updates to Neuron are deployed efficiently, with Azure Active Directory managing access securely, allowing only authorized users to execute jobs.

‍

‍

2. Upload clinical data and mapping files

Neuron leverages clinical information sourced from both claim systems and Electronic Medical Records (EMR) to evaluate the quality of care offered by a healthcare organization. This clinical information can be assimilated through various methods, but in this session, we will guide you through the process of importing it using CSV files. These files encompass a range of data including claims, medical conditions, insurance coverage, patient encounters, medical procedures, provider specialties, among others.

‍

Neuron is equipped with a feature designed to import CSV files directly into your Data Lake. While there are alternative methods for data importation, such as utilizing an automated pipeline to regularly fetch data from a streaming service, today we will focus on a manual upload approach. We will manually transfer these files to an Azure Storage account, from which Neuron can retrieve and incorporate them into your Data Lake for subsequent analysis.

‍

‍

3. Create Neuron configuration file

Neuron is configured using YAML files, where you can set up details for the database, storage, staging, and measuring options. Let's dive into the key sections of the configuration and then present an example.

1. Database - In this section, you define how Neuron connects to your database by setting the driver and database name. It outlines where Neuron will fetch and store its data. The data source might be a Synapse Data Lake or a Snowflake Data Warehouse, among others.
2. Storage - This part deals with specifying your cloud storage provider, like Azure storage. Here, you store necessary files, including the engine jar, incoming clinical data, and various mappings like code value sets and risk adjustment factors, essential for the engine's computations.
3. Stage - Defines settings for Neuron's “Stage” tasks, translating how a CSV from the storage translates into a database table. It's about mapping data from one format to another.
4. Measure - Contains settings for the Neuron measure job, particularly highlighting the measure_keys property. This property lets you dictate which measures the engine should process, allowing for either comprehensive analysis or focusing on specific areas of interest.

‍

‍

4. Create Neuron jobs in Azure Synapse

Neuron leverages the well-known Apache Spark open-source framework for its underlying operations. This architecture empowers Neuron to handle vast datasets efficiently and rapidly. To deploy Neuron, it's necessary to set up an Apache Spark cluster within Azure Synapse.

‍

In this guide, we'll walk through the setup of four distinct Apache Spark tasks within Azure Synapse. These tasks can be initiated directly from the Azure Portal. Alternatively, for those preferring automation, these tasks can be incorporated into an automated data workflow, allowing for regular execution as per a set schedule.

‍

StageCSVTableJob

This task is designed to import CSV files from your storage solution into the designated database. It operates by reading a specified mapping of file paths to table names, setting the stage for Neuron to process the data at the database level.

‍

StageEventJob

This task is crucial for establishing the foundational event and event_value_set tables in the database, facilitating the creation of a unified large table. This preparatory step is vital, as these tables are central to the calculation processes of most metrics

‍

MeasureJob

Focused on executing the configured measures, this task allows for the flexible running of numerous measures. The processing engine executes each measure sequentially according to their arrangement in the configuration.

‍

MetricJob

This task is responsible for compiling the aggregate metric outcomes from the MeasureJob for every metric, including essential statistics like the total count of eligible patients, numerator, and necessary exclusions. It also calculates additional pertinent metrics related to each measure.

‍

‍

5. Run Neuron in an Azure Synapse pipeline

Neuron was built with automation as a foundational principle. It allows interaction via an automated data pipeline, which can be programmed to execute at regular intervals. Below is a demonstrate of how an Azure Synapse pipeline processes each of the Neuron jobs according to a schedule and in a specified sequence.

‍

‍

6. Visualize Neuron results in Azure Synapse Notebooks

Let's tap into the capabilities of an Azure Synapse Notebook to explore the quality metrics Neuron generates. We'll employ SQL to extract data from one of the metric result tables, revealing the total count of patients within the eligible population, those who comply with the numerator, those meeting the exclusion criteria, and the calculated rate.

‍

You have the flexibility to visualize this data through various formats such as bar charts or pie charts, among other visual representation tools offered by Synapse. Additionally, it's straightforward to export these findings to CSV or Excel formats for sharing. These results can also be saved to your storage account for future reference and are accessible for download whenever necessary.

‍

‍

Summary

Neuron emerges as a revolutionary tool for automating insights into population health analytics, boasting compatibility with HEDIS, Stars, and Risk Adjustment measures. It's engineered for straightforward integration into cloud infrastructures, supporting deployment on major platforms like Azure, AWS, and GCP.

‍

Today we focused on the utilization of Neuron within the Azure Synapse environment, an advanced analytics service by Microsoft that harmonizes big data and data warehousing. We outlined a step-by-step guide on deploying Neuron for population health data analysis in Azure Synapse, covering the manual setup process and aiming for integration into automated workflows for daily metrics updates.

‍

We discussed deploying Neuron as a .jar file in Azure Storage, importing clinical data, configuring Neuron with YAML files, and setting up Neuron jobs in Azure Synapse using Apache Spark. We elaborated on tasks like StageCSVTableJob, StageEventJob, MeasureJob, and MetricJob, each serving a pivotal role in preparing and processing the data.

‍

Finally, we touched on running an Neuron Azure Synapse pipeline for automated data processing and visualizing results with Synapse Notebooks, highlighting how SQL queries can extract and display crucial metrics, further enriched by various visualization tools available within Azure Synapse. This thorough walkthrough aims at enabling seamless delivery of updated metrics into a consolidated data repository, leveraging Neuron's capabilities within the Azure cloud environment.

‍

Contact us today to get a more in-depth demo and discuss how Neuron can help your organization with your value-based care needs.