Real-time network crypto enforcement with zERT
This article explains how to design, configure and install zERT rules to enforce your organization's network security policies.
Since 2017, users have rapidly adopted z/OS® Encryption Readiness Technology (zERT) to gain a comprehensive view of their z/OS TCP/IP cryptographic protection. With z/OS V2R5, zERT goes to the next level by providing real-time enforcement of policy-based rules that are tailored to your system’s unique security requirements. This article reviews the various zERT features and explains how you can design, configure, and install zERT rules to enforce your organization’s network security policies.
Core zERT functions
zERT is a function of the z/OS TCP/IP stack that observes and reports on the cryptographic protection attributes of every TCP and Enterprise Extender (EE) connection that terminates on the stack. Because zERT is an endpoint function, it monitors only local traffic. Routed traffic, including that for zCX, is ignored.
zERT recognizes three heavily used cryptographic networking protocols: Transport Layer Security (TLS, evolved from old SSL protocol), Secure Shell (SSH), and IP Security (IPsec). If any of these protocols are used on a TCP connection or if IPsec is used on an EE connection (TLS/SSL and SSH can only be used on TCP connections), zERT records the important cryptographic protection attributes for that connection. If none of these protocols are observed, then zERT records that no recognized cryptographic protection is present for the connection. zERT can report on significant changes in cryptographic protection during the life of a specific connection when the IBM-provided protocol implementations are used (System SSL (including AT-TLS), ZERTJSSE, OpenSSH, and IPsec). For example, if a connection begins with TLS protection provided by System SSL and then TLS is terminated part way through the connection’s lifetime, zERT recognizes and records the loss of protection.
zERT monitoring and recording are enabled by parameters in the TCPIP profile data set. zERT collects information in two SMF formats. Each format can be written to SMF or to a z/OS Communications Server real-time Network Monitoring Interface (NMI).
SMF 119 subtype 11 “zERT Connection Detail” records are written at least once for every TCP and EE connection. These records describe the complete cryptographic protection history of each connection. They are best suited for real-time monitoring applications. The basic zERT monitoring function (zERT discovery) is enabled by specifying the ZERT parameter on the GLOBALCONFIG statement. The recording of these subtype 11 records is controlled by the SMFCONFIG ZERTDETAIL and NETMONITOR ZERTSERVICE statements.
SMF 119 subtype 12 “zERT Summary” records are gathered by the zERT aggregation function. Each record describes the repeated use of a single security session between the same client and server over a period of time called a “recording interval.” zERT aggregation is enabled by specifying the AGGREGATION subparameter on the GLOBALCONFIG ZERT parameter. The default recording interval is the system’s SMF interval. Additional subparameters of GLOBALCONFIG ZERT AGGREGATION allow you to specify recording intervals of up to 24 hours. The aggregated data recording into zERT Summary records is controlled by the SMFCONFIG ZERTSUMMARY and NETMONITOR ZERTSUMMARY statements.
Making use of zERT SMF data
There are many options to use the SMF data that zERT generates.
z/OS Communications Server provides a z/OSMF plug-in called the zERT Network Analyzer that reads SMF 119 subtype 12 zERT Summary records from an SMF dump data set, stores the data in a Db2® for z/OS database, and allows the user to formulate their own queries over the stored data. Query results can be viewed in the web UI or exported to a comma-separated value (CSV) file. The plug-in is a no-charge feature that z/OS system programmers can use to analyze the data zERT collects.
At least a dozen products from both IBM and other software vendors (including Black Hill Software, BMC, Broadcom, IntelliMagic, Merrill Consultants, Pacific Systems Group, and Vanguard Integrity Professionals) use zERT SMF records and render the data in a way that is relevant to each product’s operational context.
Because the SMF record formats are documented, you can also write your own custom programs to use the data as you see fit.
z/OS customers use zERT data to gain important insights to their network protection posture and to other aspects of their z/OS networking topology by using all the discussed approaches.
Going live! zERT policy-based enforcement
With z/OS V2R5, a new feature called zERT policy-based enforcement enables the z/OS TCP/IP stack to act on the data collected by zERT. The policy-based enforcement is based on customer-written rules that are created through the z/OSMF Network Configuration Assistant and installed in the TCP/IP stack by the z/OS Communications Server Policy Agent. zERT enforcement policy rules describe acceptable or unacceptable cryptographic protection attributes for TCP connections and any appropriate actions to take when a connection matches the rule. If the connection matches a rule, the actions associated with that rule are taken. Actions include silently permitting the connection to proceed, real-time notifications through log messages, auditing through SMF records, and even real-time defensive actions.
zERT rules are associated with a specific security protocol and are grouped into four rule sets. One rule set for each of the three security protocols (TLS, IPsec, and SSH) and one rule set for connections with no recognized protection. A single connection is evaluated against the zERT rules that govern the security protocols that are used for that connection. For example, if a connection is protected with IPsec, it is evaluated against the zERT rule set for IPsec. If a connection is protected with TLS, it is evaluated against the zERT rule set for TLS. If a connection is protected with both TLS and IPsec, it is evaluated against both the zERT rule set for TLS and the zERT rule set for TLS. Therefore, one connection can match multiple rules if the connection is protected with multiple security protocols. This is different from other policy agent technologies, such as AT-TLS, where a connection can match only one rule. If a connection does not match any zERT rule, no action is taken.
Some specific events such as a change in cryptographic protection of the connection drive zERT enforcement to reevaluate a connection against the rule set for new security protocol. For example, if an FTP session is protected by TLS and matches a zERT rule for TLS and then the user issues a CCC command to clear the protection on the control connection, zERT reevaluates the connection against the no-protection rule set at that point.
To specify conditions in a rule, use the zERT policy. There are conditions for connection attributes and protection attributes. You can describe the ‘connection attributes’ with local and remote IP addresses and ports, job name, userid, and connection direction like inbound and outbound. With ‘protection attributes’, you can go into cryptographic details such as security protocol versions (for example, TLSv1.3, SSHv2) and cryptographic algorithms for symmetric encryption, message authentication and hashing, and key exchange. You can also specify key lengths. zERT enforcement does not support digital signature algorithms in V2R5. We would like to add that in the future.
You can specify a zERT rule to take no action, take a reporting action, reset the connection, or take a combination of reporting and reset actions.
Figure 1: How the zERT policy-based enforcement works
The figure shows how the zERT policy-based enforcement works:
- The initial step is for a z/OS administrator to build a zERT policy using the Network Configuration Assistant plug-in in z/OSMF. Each zERT rule defines connection attributes, cryptographic protection attributes, and the actions to take when that rule is matched.
- When the zERT policy is complete in the Network Configuration Assistant, the generated policy file can be transferred to the target z/OS system.
- The Policy Agent on that system parses and installs the zERT policy into the zERT enforcement component of the TCP/IP stack.
- When new TCP connections are established inbound or outbound, zERT discovery is used to determine if any cryptographic protocols (TLS, SSH, or IPsec) are being used to protect the connection.
- When the connection protection attributes are determined, the zERT enforcement component compares the attributes of the connection to the zERT policy rules for each of the recognized security protocols. If no protection was found, then the connection is compared against the rules for no recognized protection. If the connection matches a rule, then the actions associated with that rule are enforced by zERT. The actions can include allowing the connection no action (default) to write a message to syslogd, write a message to the console, write an SMF record for that connection, or even terminate the connection. Multiple actions can be specified in a rule.
Because zERT enforcement uses the information that is collected by the zERT discovery function, the base zERT discovery function must be enabled through GLOBALCONFIG ZERT in the TCP/IP profile to use zERT enforcement.
Use Network Configuration Assistant to design and build zERT Enforcement rule sets
V2R5 Network Configuration Assistant (NCA) includes a new policy perspective for configuring zERT enforcement rules and rule sets. NCA’s new zERT technology provides an easy and logical flow for designing and creating zERT enforcement rules.
NCA zERT enforcement rules are organized into rule sets, which provide a hierarchical organization for a stack’s rules for a specific security protocol. A rule set is specific to a security protocol and can contain:
- A general rule: Specifies the set of attributes for a security protocol that meet your network’s security requirements. If present, this rule is evaluated first for the security protocol. This rule applies to all traffic and describes what is generally acceptable in your network and in most cases the action for a general rule is to allow the connection without taking any logging or auditing actions.
- Zero or more specific rules: Describe exceptions to the general rule that require special handling. If a connection does not match the general rule, the specific rules are examined in the order you specify them until a match is found. The action associated with that rule is taken. This action can vary according to your use case. If the specific rule is for traffic that allows an exception to the requirements of the general rule, the action can be the same as the general rule (for example, connections over a trusted network that don’t require protection). If the specific rule describes a protection attribute that is not acceptable, the action can be to generate an audit record, a log message, or even reset the connection (for example, connections protected by SSLv2 or SSLv3).
- A catch-all rule: Specifies how to handle connections that do not match your general rule or any specific rules. Generally, the action on this rule is to take a logging or auditing action because these are connections that do not meet the general requirements or any exceptions to those requirements. Connections that match these rules are the ones you need to follow up on to find out why they do not meet requirements or exceptions to decide what to do about them (for example, either upgrade the protection to meet requirements or create a new specific rule to account for the connections).
The concept of a rule set and the types of rules it contains is shown in the following scenario: A network requires TLS protection to be at TLS 1.2 or higher. However, because of the large number of clients in the network, TN3270 connections can continue to use TLS 1.1 until all the clients are updated. Any other connections that do not use at least TLS 1.2 should generate an audit record for later. The following figure describes a rule set that implements this requirement. When all the TN3270 clients are updated to use TLS 1.2, the specific rule can be removed from the rule set.
Figure 2: TLS rule set
The Network Configuration Assistant provides wizards and windows to guide you through creating your rule sets for zERT enforcement. The NCA allows one rule set per security protocol to be added to stack. You can create multiple rule sets for each security protocol and add them to different stacks depending on the security requirements of each stack.
You can do a deep dive into using NCA for zERT enforcement by earning the free zERT Policy Enforcement Configuration with NCA badge.
A closer look at zERT Enforcement actions
Let us look at each of the zERT enforcement actions in detail. The default action to silently allow the connection is used when none of the actions is specified.
- The first logging action is log to syslogd. When this action is specified, zERT enforcement logs a message about the connection matching the associated zERT rule to the syslog daemon. The messages are logged to syslogd using facility LOCAL5. The syslogd priority is determined by the LogLevel parameter. Valid values are in the range 0-7. The default is 4 (warning).
Note: The Traffic Regulation Manager Daemon (TRMD) must be active for any TCP/IP stack where zERT enforcement log to syslogd action is used. TRMD retrieves the zERT syslogd messages from the TCP/IP stack buffers and writes them to the syslog daemon.
Figure 3: Example of message EZZ8583I
- The first part of the message indicates that the connection was logged by zERT policy enforcement. The timestamp indicates the date and time when the connection was detected to match a zERT rule with log to syslogd action. The message contains TCP connection information, security information, and matching zERT rule information.
- The TCP connection information consists of the connection ID that uniquely identifies the connection, local IP address and port, remote IP address and port, transport protocol used (always TCP), job name of the application that is associated with this connection, userid that opened the socket for this connection, and connection direction (inbound or outbound).
- The security information displayed in the log message includes the security protocol that triggered the event, security protocol version, symmetric encryption, message authentication and hashing, and key exchange algorithms used to protect the connection. Some of these values are applicable only to certain security protocols. Security Protocol version and key exchange algorithms are applicable only to TLS and SSH. For connections protected by SSH, both inbound and outbound symmetric encryption and message authentication algorithms are included. Similarly, for connections protected by IPsec, both phase 1 and phase 2 tunnel encryption and authentication algorithms are included.
- The last part of the message indicates the names of the matching zERT enforcement rule and action.
- When both the log to syslogd and reset TCP connection actions are specified, the generated log message is EZZ8584I. The first part of the message indicates that the connection was reset. The rest of the information describing the connection is the same as EZZ8583I.
- The second logging action is log to console. When this action is specified, zERT enforcement logs a WTO message about the connection matching the associated zERT rule to the TCP/IP job log. The console messages are written to the TCP/IP job log. EZZ8551I is written when log to console is only specified and EZZ8562I is written when log to console and reset actions are specified.
Figure 4: Example of the message when log to console and reset actions are specified
The next zERT enforcement action is the audit record. When the audit record action is specified, an SMF 119 subtype 11 record with a new event type ‘zERT enforcement’ is written to SMF or NMI or both, depending on the configuration in the TCP/IP profile. In addition to the information available in the existing subtype 11 record, a new section called zERT policy-based enforcement, which contains the zERT rule names that are associated with the connection, is included. If a connection is protected by multiple security protocols (for example, IPsec and TLS), this section can include the matching zERT IPsec policy rule name and the matching zERT TLS policy rule name, depending on the event that caused the record to be written.
For these policy-driven subtype 11 records to be written to SMF, SMFCONFIG TYPE119 ZERTDETAILBYPOLICY parameter must be configured in the TCP/IP profile data set. This parameter is different from the existing ZERTDETAIL parameter that controls whether subtype 11 is written for TCP and EE connections on initiation, termination, and change in protection. Having separate controls allows for policy-based recording of SMF 119-11 records without having to enable ZERTDETAIL that writes out SMF records for all TCP and EE connections.
Similarly, for the policy-driven records to be written to the SYSTCPER NMI service, NETMONITOR ZERTSERVICEBYPOLICY parameter must be specified in the TCP/IP profile data set. If there is at least one zERT enforcement rule with an audit record action and neither ZERTDETAILBYPOLICY nor ZERTSERVICEBYPOLICY is enabled in the TCP/IP profile, error message EZZ8565I is written to the console.
The last zERT enforcement action you can take is to reset a TCP connection. The reset occurs after a connection is in the established state. zERT discovery detects a security protocol in use or detects that no recognized protection is in use and the connection is mapped to a rule that contains a reset action. It is important to understand that the reset action is reactive, not proactive. This action resets connections that are established. The action does not prevent connections from being established because of the need for the connection to establish zERT to learn its security characteristics. The reset action does not log a message automatically. It is recommended to specify a log action (log to syslogd or console) when the reset action is used. When a connection is reset, a new flag is set in the subtype 11 record for event types, connection termination, and short connection termination that indicates that the connection was reset by zERT enforcement. A similar flag is set in the subtype 2 TCP connection termination record as well. You can also use the Netstat ALL/-A report to see the matching zERT policy rules that were found for each connection. If a connection is protected by more than one protocol and has matching zERT rules for those protocols, Netstat shows all the matching zERT rule and action names.
Log suppression
The zERT enforcement log messages are written on a per-connection basis. To prevent flooding the syslogd and the console, zERT enforcement monitors and limits the number of messages that are written in 5-minute intervals. If the same zERT rule is matched more than 10 times or if 100 messages have been logged across all zERT rules within a 5-minute interval, then subsequent zERT messages are suppressed. An exception is made to ensure that at least one message is written for every unique zERT rule that is matched within the 5-minute interval. For example, if the 100 messages limit is reached and then a connection maps to a zERT rule that has not been matched during that interval, a message is written for that connection. Any subsequent matches to that rule during the 5-minute interval are suppressed.
After the 5-minute interval, the next time a connection matches a zERT rule with a log action, a message for that rule is written. A log-suppressed message is also written that indicates how many zERT messages were suppressed for that rule in the previous interval. Those log messages give you an idea of how many connections matched that rule for which the messages were suppressed.
Figure 5: Example of log-suppressed message
Log messages can only provide an awareness that connections are matching a particular ZERT rule. If a complete record of connections that match a zERT rule is needed, you can use the audit action.
Wrapping it up
To summarize, there are four major functions of zERT: zERT discovery, zERT aggregation, zERT Network analyzer, and zERT policy-based enforcement.
- zERT Discovery monitors, collects, and records cryptographic protection attributes on a per-connection basis. This is suitable for real-time monitoring applications.
- zERT Aggregation summarizes the repeated use of security sessions without compromising the reporting of essential security information about the network. This is suitable for historical reporting applications.
- zERT Network Analyzer is a good ease of use tool to analyze the zERT data. It helps you narrow down to the connections you are only interested in and find the needle in the haystack. There are also many other products from IBM and other software vendors that incorporate zERT data.
- zERT policy-based enforcement (new in z/OS V2R5) allows you tell the TCP/IP stack to act on the data collected by zERT per your policy in near real-time. The policy actions can include anything from simple notifications to termination of the connection and provide real-time monitoring, auditing, and defensive actions.
Many z/OS customers have discovered the valuable insights zERT can provide, not just about their network cryptographic protection, but also about their networks in general. zERT policy-based enforcement extends the value of zERT by adding real-time monitoring, notifications, and defensive actions. Give it a try!
About the authors
Mike Fox is a Senior Software Engineer in the IBM Enterprise Networking Software group, IBM zSystems. He is the product owner and architect for the IBM Configuration Assistant for z/OS Communications Server, and also specializes in routing and connectivity architecture for z/OS.
Chris Meyer, CISSP is an IBM Senior Technical Staff Member and the network security architect for IBM’s z/OS operating system. He has over 35 years designing and developing IBM operating systems, electronic payment systems and other enterprise software.
Navya Ramanjulu is a Senior Software Engineer in the IBM Enterprise Networking Software group, IBM zSystems. She is the product owner and architect for the z/OS Containers Networking project. She is responsible for the development and support of z/OS Encryption Readiness Technology, z/OS FTP and System Resolver. She has 11 years of experience in the z/OS domain.
Mary Fetterman contributed to the editorial review of this article.