Preparing RHEL for SAP Solutions Systems for SAP software installation with Ansible

[Prerna_Mohta]

Introduction

For running SAP HANA or SAP ABAP Platform on Red Hat Enterprise Linux, a Red Hat Enterprise Linux for SAP Solutions (RHEL for SAP Solutions) subscription is required; see SAP notes 3108316 (for RHEL 9) and 2772999 (for RHEL 8).

These SAP notes also contain further instructions on package installation and system configuration after the initial installation of RHEL for SAP Solutions.

Note: RHEL for SAP Solutions consists of standard Red Hat Enterprise Linux plus SAP specific repos and software packages as well as SAP specific support services. It is installed using the standard Red Hat Enterprise Linux image. On top of that, additional SAP specific software packages must be installed, and the system must be configured to optimize it for SAP workloads.

It is possible to perform these installation and configuration steps (as well as other necessary modifications laid out in other SAP notes) manually. However, the quickest, most reliable and recommended method is to use the SAP on RHEL preconfigured Ansible roles from the rhel-system-roles-sap (=RHEL System Roles for SAP) software package or from the Red Hat Automation Hub collection redhat.sap_install.

You can use the roles to perform a local configuration on each system. However, the recommended method is to use a central system as the control node on which the desired configuration of the SAP systems (the managed nodes) is stored and from which it is also initiated. 

The RHEL System Roles for SAP have made it much easier to prepare a RHEL system for SAP software installation compared to previous solutions. However, not all possible ways of using the Ansible roles can be predicted, and sometimes fixing usability problems can take some time.

This blog describes some pitfalls users might encounter when using the preconfigure Ansible roles (sap_general_preconfigure, sap_netweaver_preconfigure, sap_hana_preconfigure) for preparing their RHEL for SAP Solution systems. Finally, we will explain how to help yourself as much as possible when encountering any problems.

Note: The sample outputs shown here are from ansible-playbook runs with the environment variable setting ANSIBLE_STDOUT_CALLBACK=yaml or with the following line in the [defaults] section of the file /etc/ansible/ansible.cfg on the control node:
stdout_callback = yaml
Without this setting, multiline output is more difficult to read.

Symptoms and resolutions

Each of the sections below is divided into the following parts:

Problem: A very short description of the problem.

Symptom: Typical terminal outputs.

Root cause: What is causing the problem?

Verify the root cause: How can you verify that the symptom is caused by the mentioned root cause?

Solution: How to solve the problem?

Workaround: How to avoid the problem when a solution is not possible or too difficult to implement?

Installing software packages

Problem: The role fails when trying to install software packages.

Symptom: The task "Ensure that the required package groups are installed, RHEL 8 and RHEL 9" fails with a message like:

fatal: [host01]: FAILED! => changed=true
 cmd:
 - yum
 - install
 - '@server'
 - --nobest
 - --exclude=kernel*
 - -y
 delta: '0:00:00.886700'
 end: '2023-12-06 03:51:14.964584'
 msg: non-zero return code
 rc: 1
 start: '2023-12-06 03:51:14.077884'
 stderr: 'Error: There are no enabled repositories in "/etc/
 stderr_lines: <omitted>
 stdout: |-
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered with an entitlement server.
 stdout_lines: <omitted>

Root cause: If a RHEL System is not subscribed properly to a Red Hat repository source, e.g., a Red Hat Satellite server or the Red Hat Customer Portal, it will not have access to RHEL packages. As a consequence, the installation of additional software packages is impossible. In this case, the sap_general_preconfigure role will fail, just like running the command "dnf install ..." would also fail.

Verify the root cause: Try to install a software package (example: the Korn Shell) with the dnf command, as in:

# dnf install ksh

Solution: Before using the roles, make sure that the managed nodes are subscribed to a Red Hat repository source, e.g., a Red Hat Satellite server or the Red Hat Customer Portal. You can also use the rhc role from the RHEL System Roles package or from the redhat.rhel_system_roles collection to automate this step.

Error message related to the length of the hostname

Problem: The hostname is too long.

Symptom: The task Ensure that the length of the hostname is not longer than “sap_general_preconfigure_max_hostname_length" fails with:

fatal: [host01]: FAILED! => changed=false  
 assertion: 15 <= 13
 evaluated_to: false
 msg: The length of the hostname is 15 but must be less or equal to 13 (variable 'sap_general_preconfigure_max_hostname_length')!

Root cause: As per SAP note 611361, the length of the hostname must not exceed 13. So the role sap_general_preconfigure is verifying if this is the case.
Verify the root cause: Run the following command on the control node and observe that the output line shows the number 14 or higher:

# ssh root@host01 hostname -s | awk '{print length($0)}'

Solution: Either modify the host name so its length is less than or equal to 13, or set the variable sap_general_preconfigure_max_hostname_length to the desired value if you are aware of the possible consequences when installing SAP software. For example, for SAP HANA only systems, the length of the hostname can be longer.

Error message related to /etc/hosts entries and DNS

Problem: The role sap_general_preconfigure fails when checking /etc/hosts entries. The SAP notes mentioned above require the /etc/hosts entries for the SAP systems to meet certain requirements, so the role sap_general_preconfigure is verifying if these entries are correct.

Symptom: The task "Verify that variable sap_domain is set" fails with:

fatal: [host01]: FAILED! => changed=false
  assertion: not( (sap_domain is undefined) or (sap_domain is none) or (sap_domain | trim == '') )
  evaluated_to: false
  msg: Variable 'sap_domain' is undefined or empty. Please define it in defaults/main.yml or via --extra-vars!

Root cause: The variable sap_domain is set from the content of the variable ansible_domain, which is not always known to Ansible (e.g., if DNS has not been configured).
Verify the root cause: Run the following command (example for a host named host01) on the control node:

# ansible -m setup host01 | grep ansible_domain

And observe the following output:

     "ansible_domain": "",

On a system without this problem, the output will be similar to the following line:

     "ansible_domain": "example.com",

Solution: In your playbook or inventory (=file ./host_vars/host01 in our example), define the variable sap_domain to the DNS domain name of the managed node.

Error message related to SELinux

Problem: The role sap_general_preconfigure fails when performing SELinux configurations.

Symptom: The task "Configure '/usr/sap' SELinux file labels" fails with a message similar to:

ERROR! the role 'redhat.rhel_system_roles.selinux' was not found in redhat.sap_install:ansible.builtin:/root/ansible/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/root/ansible

Root cause: The role has dependencies, as also mentioned in the file README.md of the role. 

Verify the root cause: No further verification is necessary. If the output contains the message that the 'redhat.rhel_system_roles.selinux' role could not be found, then this role is not available.

Solution: Make the selinux role available on the control node, as also explained in the file README.md of the role, by running one of the following commands:

• When using the package rhel-system-roles-sap:

# dnf install rhel-system-roles

• Or, when using the collection redhat.sap_install:

# cd ~/.ansible/collections/ansible_collections/redhat/sap_install/roles/sap_general_preconfigure
# ansible-galaxy install -vv -r meta/collection-requirements.yml

The role sap_general_preconfigure fails only at the end

Problem: The role sap_general_preconfigure fails after executing all tasks, and the role sap_hana_preconfigure, called in the same playbook, is not executed.

Symptom: The task "Let the role fail if a reboot is required" fails with:

fatal: [host01]: FAILED! => changed=false  

 msg: Reboot is required!

Root cause: The preconfigure roles typically make changes to the system that require a reboot (e.g., kernel update, package installations, SELinux modifications). We do not want the reboot requirement to be overlooked, and we don't want to automatically reboot the affected nodes, so the default is to let the role fail in this case.

Verify the root cause: No further verification is necessary. If the output contains the message that the system needs to be rebooted, then this confirms the root cause.

Solution: There are two possible solutions:

1. Let the role reboot the managed nodes automatically (but only if necessary) by setting the applicable role variables (e.g. sap_general_preconfigure_fail_if_reboot_required, sap_general_preconfigure_reboot_ok).
2. Manually reboot the affected managed node(s).

Initial technical investigation

For identifying and solving problems when using the preconfigure roles, you could follow the path outlined below:

• Can the managed (remote) node(s) be accessed by the desired user via ssh without a password?
• Check the output of the first task, named "Display the role path". Does it contain the expected installation location of the role? If not, check your Ansible path settings so the desired roles can be found by Ansible.
• Read the README.md file of the role. Make sure that you have followed all the recommended steps.
• Does the last output section ("PLAY RECAP") contain "failed=0"? If not, one or more of the tasks of a role have failed. Note that when a dependent role which is called by the include_role Ansible module (e.g., selinux) is not found, the role will abort with a return code of 2, but the last output section will still show "failed=0".
• Have all the roles in the playbook been executed? If not, then one of the other roles has failed. Look backwards in the output to search for a failed task.
• Search the list of issues at https://github.com/sap-linuxlab/community.sap_install/issues?q=is%3Aissue to verify if your symptom has already been reported or discussed in the sap-linuxlab/community.sap_install community.

Conclusion

The preconfigure roles aim to make it easy to prepare your RHEL for SAP Solutions systems for the SAP HANA or SAP ABAP Platform installation,

• without maintaining your own list of SAP notes,
• without maintaining all configuration settings separately,
• without writing and maintaining your own set of scripts.

Feel free to share your experiences with the roles!

Thank you Bernd Finger for all your help and support :)!