How to turn your SMF data into valuable insights without z/OS expertise

By: Thomas Drews, Giao Nguyen-Quynh, Khadija Souissi

Published: August 18th, 2022

Read time: 5 minutes

Introduction

System Management Facility (SMF) records represent a wealth of useful information that can be leveraged to extract insights into the activities of your z/OS systems. Novice system programmers, data scientists and data engineers might struggle to understand and interpret SMF data as they are often not yet acquainted with z/OS.

This blog describes how IBM SMF Explorer with Python can help you easily access and understand SMF data and unlock its value using state-of-the-art technologies, without the need to have deep z/OS skills to access and process data.

This new functionality is available for z/OS 2.5 via Data Gatherer PTF UJ07630 and requires Data Gatherer PTF UJ09235 and z/OSMF PTF UI82355.

What is IBM SMF Explorer with Python

IBM SMF Explorer with Python, or simply IBM SMF Explorer, is a data access and analysis toolkit designed to help you access SMF data and extract insights in an easy and modern way, leveraging Python and Jupyter Notebooks. Even with limited z/OS skills, you can now easily access, understand and interpret SMF data and unlock value from it.

Demystifying SMF data is important, especially if you would like to use it to develop data science or IT analytics solutions to bring valuable insights into your IT operations. Without needing unique z/OS skills to access and process this data, you are now able to develop this type of solution.

To access the SMF data, IBM SMF Explorer leverages REST API Services provided by the z/OS Data Gatherer, a data collector that is designed to collect and write specific z/OS IT data in raw format. It provides REST API data access for modern tools such as the SMF Explorer and existing or new industry-standard applications. The access method uses a request–response model that allows the user interface to access entire data sets in raw format. In other words, leveraging this approach for IBM SMF Explorer is key to providing easy, modern and scalable access to the SMF data.

Python is considered one of the most popular programming languages as it is easy to learn, thanks to its easy access and very readable code. IBM SMF Explorer allows you to write Python code in fewer lines than would be possible with many other programming languages to access data and gain insights by leveraging a rich set of libraries to visualize the data. Furthermore, IBM SMF Explorer includes Jupyter Notebook tutorials to guide you on how to access, process, and visualize the SMF raw data, allowing you to have quick prototyping and a straight-forward user experience.

What you can do with IBM SMF Explorer with Python – Use Cases

Based on the capabilities mentioned in the previous chapter, IBM SMF Explorer supports numerous use cases that deal with meaningful, easy and dynamic interpretation of SMF record types 70-79, 99 (subtypes 1, 2, 6, 12 and 14) and 113, and allow developers to preform quick prototyping. Following are some examples:

Data scientists and data engineers can use IBM SMF Explorer to access, understand, prepare and explore data needed for machine learning. Studies have shown that these data-centric activities often require up to 80% of the total machine learning workflow time. In order to build machine learning models, data scientists first need to understand the hidden patterns and trends in the data, go through feature engineering and predictor selection to identify which key performance indicators (KPIs) in the data can impact the result of the prediction, etc. When it comes to building models leveraging SMF data, data scientists can use IBM SMF Explorer to simplify and accelerate those steps, for example, by visualizing the data to identify patterns and define the important KPIs for their use cases.
Application programmers can make use of Python libraries for machine learning for pattern recognition based on data visualization to implement and extend home-grown system management tools.
Novice system administrators can better understand and interpret SMF data without prior z/OS knowledge.
z/OS system programmers can react faster to user complaints by better analyzing bottlenecks and performing root cause analysis.
z/OS performance analysts can quickly examine system health and better understand the impact of parameter tuning.
z/OS capacity planners can analyze current system utilization and do projections against future system and application growth.

The last two use cases are specific to capacity planning and performance analysis. With IBM SMF Explorer, you can leverage your own coding skills to develop Jupyter Notebook assets to extract insights based on your own needs. Refer to the section “What about existing IT Operational Analytics offerings?” to discover how this offering relates to existing capacity and performance analytics solutions.

In the future, we plan to support additional SMF record types to support a variety of use cases.

How you can use IBM SMF Explorer with Python

Thanks to the convenient interface to access SMF data using Python provided by IBM SMF Explorer, you can retrieve SMF data in tabular form and use that data for further processing tasks, such as data analysis and machine learning. As mentioned earlier, a prominent use case of IBM SMF Explorer is to employ Jupyter Notebooks for quick prototyping of reports and analyses. The Python ecosystem provides many powerful tools for this task. Further, Jupyter Notebooks are interactive, which is perfect for exploring and visualizing your data. To simplify your entry into SMF data analysis, in the IBM-SMF-Explorer GitHub repo in the “Notebooks” directory, you find a collection of tutorials and showcase notebooks for data analysis.

To get started, you first need to set up the SMF Explorer environment. To do this, refer to our documentation for a step-by-step guide on how to install the environment on your system. The documentation also provides details about the project structure and features. To start using IBM SMF Explorer, basic knowledge of Python is helpful, but a simple Python installation and the provided tutorials are enough to get going.

Note that in IBM SMF Explorer, user requests for SMF data are transformed into REST API calls to the Data Gatherer API. Therefore, both a local and a remote setup are needed. You must perform the remote setup on the z/OS system where your SMF data is located. In Chapter “Setting up z/OS Data Gatherer SMF REST services” of the Data Gatherer User’s Guide you find information how to set up and how to start the REST API. When the REST API is active on the remote system, you can retrieve data with the local SMF Explorer. After starting the SMF Explorer, you are prompted for the connection string. To connect the SMF Explorer you need to enter the same connection string that was provided when setting up the REST API.

Next, we would like to show you how easy it is to retrieve data from your remote server and visualize it with IBM SMF Explorer and Jupyter Notebooks. First, start your Jupyter Lab server, as described in our documentation. Open a new notebook and import the required packages:

import smfexplorer
from smfexplorer.fields import SMF99S1
import plotly.express as px

In the first line, we imported the IBM SMF Explorer package to our notebook. In the second line, we imported the SMF99S1 record subtype definition, that contains the data we need. Finally, we imported the Plotly package for data visualization. Retrieving the SMF fields related to system utilization is straightforward, as follows:

ctx = smfexplorer.new_context('YOUR.SMF.DATA')
df = ctx.request([SMF99S1.timestamp, SMF99S1.ziip_util, SMF99S1.cp_util, SMF99S1.zaap_util, SMF99S1.total_util]).run()

In the first line, we specified the data set from which we want to read. Subsequently, we made a request to the remote server to fetch the required data. As the data are fetched, we can plot the system utilization over time with Plotly:

plot = px.line(df,
               x="timestamp",
               y=["ziip_util", "cp_util", "zaap_util", "total_util"],
               title="System Utilisation",
               labels={"value": "%",
                       "variable": "Processor Type",
                       "timestamp": "Time"})
display(plot)

Figure: Plot output

As you can see, the tiring process of retrieving, understanding, post-processing, and visualizing SMF data can be done in just a few lines of code using IBM SMF Explorer. You can now leverage your data analytics skills to gain insights into the data without spending hours processing the raw SMF data.

Although IBM SMF Explorer initially only supports WLM, RMF and HIS (Hardware capacity, reporting, and statistics) records, IBM designed a plugin architecture that allows extensions with record types that serve the needs of your organization. This plugin system dynamically loads the information needed for other components. To define a new plugin for IBM SMF Explorer, you just need to define the SMF record structure, the required post processing functions, and the sample requests.

The provided Jupyter Notebook Tutorials are available in the IBM-SMF-Explorer GitHub repo. You are free to view and adapt them locally to meet the needs of your environment. We welcome all to join our community and contribute to our projects. For example, you can share your Jupyter Notebooks that demonstrate your analysis skills or add support for additional SMF record types. We are excited to see IBM SMF Explorer in action!

What about existing IT Operational Analytics offerings?

Today, there is a broad spectrum of offerings in the market that deal with SMF data analysis and visualization. Discover in this section how IBM SMF Explorer with Python positions itself in this crowded market.

As part of the z/OS Data Gatherer which is included in the z/OS base, IBM SMF Explorer does not require a separate license or additional components. In other words, you can use this capability as part of the operating system. Furthermore, SMF Explorer allows a wide range of data analysts, data savvy personas and novice mainframe users to quickly generate data visualization graphs on the fly with a small amount of coding and start getting value out of the SMF records.

As described in the use case section, IBM SMF Explorer can play an important role to simplify and speed up the data preparation and exploration phase of a machine learning project. This phase typically includes accessing the data, understanding it, uncovering hidden patterns and trends, as well as identifying the most important KPIs that can impact the result of the prediction. IBM SMF Explorer allows you to perform these data centric activities in an interactive way by using Jupyter Notebooks which directly refresh the results just after changing the code or the data. Furthermore, data scientists can leverage different data visualization capabilities and graphs to detect patterns. This interactivity, flexibility and simplicity of coding represent an added value compared to traditional reporting tools which might sometimes require you to rebuild the report if your data changes. In other words, SMF Explorer can help you shorten the time to value.

Once you have identified the metrics and relevant KPIs for your use case using IBM SMF Explorer, you can use those KPIs for modelling purposes or expose them to existing solutions, such as IBM Z Operational Log and Data Analytics, to perform deep-dive data analysis with focus on real time operational data flow.

Compared to turnkey solutions in the AI Operations area, IBM SMF Explorer is a toolkit with which you can apply your Python coding skills to build your analytics assets and address your own requirements in your own way, whereas the turnkey solutions don’t require any data analysis or data science skills to be deployed. Furthermore, those solutions usually use SMF and IT data to address specific AI use cases. They can bring value just after being installed and deployed, as they don’t require any further implementation overhead. These solutions include pre-built reports and predictive models that exploit the providers’ domain and data science expertise.

Note that IBM SMF Explorer does not require deep Python expertise. Even with limited Python skills, you can get value from the IBM SMF Explorer tutorials that help provide you with a smooth path to becoming acquainted with this programming language.

Summary

To summarize, IBM SMF Explorer is a z/OS non-priced offering that allows you to use modern, state-of-the-art technologies like Jupyter Notebooks and Python libraries to easily access and understand SMF data, apply data exploration and use AI methods to extract insights that go beyond traditional analytics tools.

About the authors

Khadija Souissi is the Product Manager responsible for AI for IBM zSystems solutions with a focus on z/OS. She holds a Masters Degree in Electrical Engineering, is a certified Master IT Specialist, and has more than 10 years of experience as a Subject Matter Expert in the area of Data and AI on IBM Z, delivering and designing AI and Analytics solutions for IBM Z Clients. In her current role, she acts as the interface between Development and clients; for example, she validates new AI solutions and initiatives with clients and takes their own pain points and current challenges into consideration when delivering relevant offerings to the market. Khadija has coauthored IBM Redbooks and IBM Redpaper publications and white papers. She has authored blogs on various sites such as worldofdb2.com related to AI and Analytics on IBM zSystems.

Giao Nguyen-Quynh is a software developer for z/OS Workload Manager and is enthusiastic about bringing artificial intelligence into existing systems. Giao holds an MS in Computer Science from the Karlsruhe Institute of Technology.

Thomas Drews is the Project Manager of several z/OS AI and data analytics projects in Böblingen. He has more than 20 years of experience in various roles and projects at IBM Z Systems such as System Automation, Linux on IBM Z and Hyper Protect Services.

Thanks to Anastasiia Didkovska, Dorian Czichotzki, Christopher Watson, and Frank Bellacicco who contributed to the editorial review.

Download this article

zinfo - ZOAU provides programmatic access to z/OS facts

Real-time network crypto enforcement with zERT