SAP on Azure: Tomcat Clustering using Static Membe...

dennispadia · ‎09-04-2020

Purpose

The purpose of this blog is to provide guidelines to configure tomcat cluster in order to achieve high-availability for SAP BusinessObjects BI Platform web application servers on Azure.

SAP Note 2808640 provide detailed steps to configure tomcat cluster using multicast but in Azure multicast is not supported. Refer SAP Note 2764907 and FAQs - Azure Virtual Network.

So to configure tomcat cluster on Azure, we need to use Static Membership for cluster membership verification.

Overview

The deployment of SAP BusinessObjects BI Platform on Azure is similar to on-premise deployment. But in Azure, you can leverage some of their offerings to build the application which can reduces maintenance work for some application components like DBaaS (Azure SQL Database) for CMS Database, Azure Files or Azure NetApp Files (ANF) for File Repository Server, Azure Load Balancer or Application Gateway for load balancing traffic to web server, .

In below figure, SAP BusinessObjects BI Platform is installed on two Azure Virtual Machines (VM) along with tomcat. To load balance the traffic between two web servers, application gateway is used. The application gateway IP address (10.31.3.20) act as an entry point for the users, handles incoming TLS/SSL (HTTPS - TCP/443) connections, decrypt the TLS/SSL and passing on the un-encrypted request (HTTP - TCP/8080) to the servers (azusbosl1 or azusbosl2) in the backend pool. With in-built TLS/SSL termination feature, we just need to maintain only one TLS/SSL certificate i.e. on Azure Application Gateway which simplifies operations.

When one of the web server goes down, application gateway route all the traffic to other host(s). This way we can attain high availability of tomcat server at host level. But the problem here is that, we lose user session along with the host. So user needs to login again to access application via different tomcat server. To ensure user sessions remain intact during tomcat service disruption, we need to configure session replication in tomcat which replicates user session to other hosts that are member of the cluster group.

Traditionally in on-premise deployment, tomcat cluster is configured using multicast but as this is not supported on Azure (SAP Note 2764907), we need to configure tomcat cluster using Static Membership Interceptor.

Configuration Steps

To configure tomcat cluster on Azure, follow SAP Note 2808640 - Configure Tomcat Clustering for BI 4.2 with some changes in member verification option - instead of using McastService attribute we will be using StaticMember attribute for cluster membership. In this illustration, we are using two web servers azusbosl1 and azusbosl1 on Linux.

Copy forcereplicationvalve.jar from INSTALLDIR/enterprise_xi40/java/lib to TOMCATINSTALLDIR/tomcat/lib (in all tomcat nodes - azusbosl1 and azusbosl2)

Open INSTALLDIR/tomcat/conf/server.xml

Perform below action in azusbosl1

Replace



<!--

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>

-->



With below on azusbosl1 (10.31.0.8)



<Cluster 

        channelSendOptions="8" 

        channelStartOptions="3" 

        className="org.apache.catalina.ha.tcp.SimpleTcpCluster">

    <Manager 

        className="org.apache.catalina.ha.session.DeltaManager" 

        expireSessionsOnShutdown="false" 

        notifyListenersOnReplication="true"

    />

    <Channel className="org.apache.catalina.tribes.group.GroupChannel">

        <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />

        </Sender>

        <Receiver 

            address="10.31.0.8" 

            autoBind="0" 

            className="org.apache.catalina.tribes.transport.nio.NioReceiver" 

            maxThreads="6" 

            port="4000" 

            selectorTimeout="5000"

        />

        <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor" staticOnly="true"/>

        <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" />

        <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">

            <Member 

                className="org.apache.catalina.tribes.membership.StaticMember" 

                port="4000" 

                host="10.31.0.9" 

                uniqueId="{0,0,0,0,0,0,0,0,0,0,0,0,6,7,8,9}" 

            />

        </Interceptor>

        <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>

    </Channel>

    <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="" />

	<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

    <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener" />

	<Valve className="com.sap.customvalve.CustomReplicationValve"/>

</Cluster>

where org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor class include the information of all the other web server host that you want to be part of tomcat cluster. In this example, it includes information about azusbosl2. uniqueID is a universally unique ID for static member. The value must be 16 bytes in format {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}. For more information on attribute and option, refer Tomcat cluster guide.

Perform below action in azusbosl2

Replace:



<!--

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>

-->



With below on azusbosl2 (10.31.0.9):



<Cluster 

        channelSendOptions="8" 

        channelStartOptions="3" 

        className="org.apache.catalina.ha.tcp.SimpleTcpCluster">

    <Manager 

        className="org.apache.catalina.ha.session.DeltaManager" 

        expireSessionsOnShutdown="false" 

        notifyListenersOnReplication="true"

    />

    <Channel className="org.apache.catalina.tribes.group.GroupChannel">

        <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />

        </Sender>

        <Receiver 

            address="10.31.0.9" 

            autoBind="0" 

            className="org.apache.catalina.tribes.transport.nio.NioReceiver" 

            maxThreads="6" 

            port="4000" 

            selectorTimeout="5000"

        />

        <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor" staticOnly="true"/>

        <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" />

        <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">

            <Member 

                className="org.apache.catalina.tribes.membership.StaticMember" 

                port="4000" 

                host="10.31.0.8" 

                uniqueId="{0,0,0,0,0,0,0,0,0,0,0,5,6,7,8,9}" 

            />

        </Interceptor>

        <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>

    </Channel>

    <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="" />

	<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

    <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener" />

	<Valve className="com.sap.customvalve.CustomReplicationValve"/>

</Cluster>

where org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor class include the information of all the other web server host that you want to be part of tomcat cluster. In this example, it includes information about azusbosl1. uniqueID is a universally unique ID for static member. The value must be 16 bytes in format {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}. For more information on attribute and option, refer Tomcat cluster guide

In the web.xml of all web applications (azusbosl1 and azusbosl2) desired to have failover, the following tag must be added within <web-app> and </web-app> tags:
```
<web-app>

    

<distributable/>

</web-app>
```

Restart Tomcat. If there is no error then your tomcat server will start without any issue.You will find below message in catalina*.log file under INSTALLDIR/tomcat/logs directory, when tomcat service get started.

- On azusbosl1 (restarted first after changes)

org.apache.catalina.tribes.transport.ReceiverBase.bind Receiver 

Server Socket bound to:[/10.31.0.8:4000]

- On azusbosl2 (started after azusbosl1 is up and running).

org.apache.catalina.tribes.transport.ReceiverBase.bind Receiver 

Server Socket bound to:[/10.31.0.9:4000]



[Catalina-utility-1] org.apache.catalina.ha.tcp.SimpleTcpClust

er.memberAdded Replication member added:[org.apache.catalina.t

ribes.membership.StaticMember[tcp://10.31.0.8:4000,10.31.0.8,4

000, alive=0, securePort=-1, UDP Port=-1, id={0 0 0 0 0 0 0 0 

0 0 0 5 6 7 8 9 }, payload={}, command={}, domain={}]]

- On azusbosl1 log file, you will see that a cluster member is added as soon as tomcat on azusbosl2 is started.

INFO [Catalina-utility-1] org.apache.catalina.ha.tcp.SimpleTc

pCluster.memberAdded Replication member added:[org.apache.cat

alina.tribes.membership.StaticMember[tcp://10.31.0.9:4000,10.

31.0.9,4000, alive=0, securePort=-1, UDP Port=-1, id={0 0 0 0 

0 0 0 0 0 0 0 0 6 7 8 9 }, payload={}, command={}, domain={}]]

INFO [Catalina-utility-1] org.apache.catalina.tribes.group.in

terceptors.TcpFailureDetector.performBasicCheck Suspect membe

r, confirmed alive.[org.apache.catalina.tribes.membership.Sta

ticMember[tcp://10.31.0.9:4000,10.31.0.9,4000, alive=0, secur

ePort=-1, UDP Port=-1, id={0 0 0 0 0 0 0 0 0 0 0 0 6 7 8 9 },

 payload={}, command={}, domain={}]]

NOTE: To have these changes persist through wdeploy / install / upgrade / repair actions, step 4 must also be performed to the web.xml of the web applications stored in warfiles/webapps, eg: INSTALLDIR/enterprise_xi40/warfiles/webapps/BOE/WEB-INF/web.xml

Testing Tomcat Cluster

We have tomcat running on azusbosl1 and azusbosl2, but to simplify the test case we will first stop tomcat on azusbosl2 and make sure that our connection to CMC application happens through tomcat server running on azusbosl1.

Application Gateway URL: https://10.31.3.20/BOE/CMCNOTE: We have only stopped tomcat service on azusbosl2, not BOBI application. So it might happen that even though the tomcat session is established on azusbosl1, it get connected to CMS service running on azusbosl2. It is not the case here, but it might happen.

Now start tomcat on azusbosl2. It will immediately become part of tomcat cluster, which you can verify in azusbosl1 tomcat catalina*.log fileIn azusbosl2 catalina*.log file, you will see that server request session state from azusbosl1 (10.31.0.8)

Session states are transferred, now we will kill the tomcat process in azusbosl1

It takes few seconds to fail-over. But once the fail-over is completed, you can continue using CMC/BI application with the same user session. So you don't need to login again

Issue

When user try to access the application during tomcat service crash, it might happen that user will get 502 - Web server received an invalid response while acting as a gateway or proxy server error message. It is because session fail-over is still happening and it takes few seconds before user can connect to the same user session.

Resolution: Wait for few seconds and then click on any options in CMC/BI application. User will be able to access the application with the same session.

References

2764907 - Tomcat session failover using Azure ILB fails

2808640 - Configure Tomcat Clustering for BI 4.2 [How-to]

Tomcat Interceptor Architecture - Static Membership

Regards,

Dennis Padia.