ONE Data Lake & Splunk: Revolutionizing Network Data Analytics — Part 1

 

ONE Data Lake

In February, we introduced the ONE Data Lake as part of our ONES 2.1 release, highlighting its integration capabilities with Splunk and AWS. In this blog post, we’ll delve into how the Data Lake integrates specifically with Splunk.

A data lake serves as a centralized storage facility capable of accommodating large quantities of structured, semi-structured, and unstructured data on a significant scale. These are typically built using scalable distributed cloud-based storage systems, such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage.

A pivotal benefit of a data lake lies in its capacity to handle substantial amounts of data from diverse origins, offering a cohesive storage solution conducive to data exploration, analytics, and informed decision-making processes.

Aviz ONE-Data Lake functions as a platform facilitating the migration of on-premises network data to cloud storage. It encompasses metrics that capture operational data across the network’s control plane, data plane, system, platform, and traffic. Serving as an upgraded iteration of Aviz Open Networking Enterprise Suite (ONES), ONE-Data Lake stores the metrics previously utilized in ONES onto the cloud.

Why Splunk?

Splunk is highly significant for organizations across diverse industries for multiple reasons:

1. Operational Insight:

Splunk empowers organizations to obtain immediate insights from their operational data, facilitating the monitoring of system and application health and performance. This capability aids in promptly identifying and addressing issues, thereby reducing downtime and enhancing operational efficiency.

2. Security Surveillance and Threat Identification:

Splunk is extensively utilized for Security Information and Event Management (SIEM) objectives, aiding organizations in overseeing their IT environments for security threats and irregularities. By correlating data from diverse sources, it can efficiently identify and address security incidents, thereby bolstering the overall cybersecurity stance.

3. Regulatory Adherence and Oversight:

Splunk supports regulatory adherence and oversight by empowering organizations to gather, analyze, and report on data pertinent to regulatory requirements and industry standards. This capability is especially critical for sectors like finance, healthcare, and government, where stringent compliance mandates are in place.

4. IT Operations and DevOps:

Splunk aids in IT operations and DevOps practices by providing visibility into IT infrastructure, application performance, and deployment processes. This allows organizations to identify areas for optimization, streamline operations, and accelerate the development and delivery of software applications

5. Machine Learning and Predictive Analytics:

Splunk equips organizations with machine learning and predictive analytics functionalities, empowering them to uncover patterns, detect anomalies, and forecast outcomes from their data. This supports proactive resolution of issues, capacity planning, and efforts in risk management

6. Customer Satisfaction Administration:

Splunk can be utilized to assess customer interactions and feedback from various channels, enabling organizations to delve deeper into customer requirements and preferences. This information can then be utilized to tailor offerings, elevate customer satisfaction levels, and nurture brand loyalty

To sum up, Splunk is an essential tool for organizations to leverage data efficiently, promoting operational excellence, strengthening security measures, ensuring compliance, and achieving business objectives.

Integrating Splunk with ONES:

Steps involved to integrate the Splunk cloud service with ONES,

1. Mapping Splunk instance with the ONES server

To integrate the Splunk service with ONES, follow these steps:

• Configure Splunk Instances: Set up the Splunk instances on the ONES cloud page to start pushing metrics to the designated cloud endpoint.
• Provide Necessary Details: The following information is required for the integration:

→ Splunk URL: The URL of your Splunk instance.

→Unique Token: A unique token for authentication and secure data transmission.

→ Index: The specific index in Splunk where the pushed metrics will be stored.

By ensuring these details are accurately provided, you can successfully configure and integrate the Splunk service with ONES, enabling seamless metric collection and analysis.

Figure 1: Cloud Instance configuration page in ONES

Figure 2: Instance created and ready for data streaming

2. Managing the created Instance through ONES:

The cloud instance created within ONES offers several management options to enhance user experience and sustainability. Users can update the integration settings, pause and resume metric uploads to the cloud, and delete the created integration when needed. These features make it easy for users to maintain and manage their cloud endpoint integrations effectively.

Figure 3 : Updating the integration details

Figure 4: Option to pause and resume the metric streaming to cloud

Figure 5: Option to delete the integration created

3. User defined metric update:

The end user has the flexibility to select which metrics from their network monitored by ONES should be uploaded to the designated cloud service. This ONES 2.1 release supports various metrics, including Traffic Statistics, ASIC Capacity, Device Health, and Inventory. Administrators can choose and deselect metrics from the available list within these categories according to their preferences.

Figure 6 : Multiple options available for metric update on cloud

4. Multi vendor support

The metric update is not limited to any particular hardware or network operating system (NOS). ONE-Data Lake’s data collection capability extends across various network operating systems, including Cisco NX-OS, Arista AOS, SONiC, and Non-SONiC. Data streaming occurs via the gnmi process on SONiC-supported devices and through SNMP on OS from other vendors.

Figure 7: ONES inventory showing multiple vendor devices streaming

Splunk Analytical capabilities:

1.Event based Visualization:

Events within Splunk generally contain timestamped data alongside related metadata and content. Each event undergoes parsing and indexing separately, facilitating users to efficiently search, analyze, and visualize data. Splunk automatically extracts fields from events during indexing, streamlining filtering and correlation based on specific criteria.

Figure 8 — Inventory details from NX-OS is captured as events in Splunk

2. Chart representation:

This entails visually depicting data using charts or graphs, aiding users in comprehending patterns, trends, and relationships within the data more readily than analyzing raw data alone. These graphical representations encompass diverse types such as bar charts, line charts, pie charts, scatter plots, and others, each tailored to specific data types and analytical objectives

Figure 9 — Pie Chart in Splunk representing the data from different NOS vendors

Conclusion:

Aviz ONE-Data Lake functions as the cloud-based version of ONES, enabling the storage of network data in cloud repositories. It operates independently of any particular cloud platform and supports data streaming from leading network device manufacturers such as Dell, Mellanox, Arista, and Cisco. Network administrators have the freedom to specify the metrics they want to transfer to the cloud endpoint, granting customized control over the data storage procedure.