Tune and Scale while you Develop

former_member695691 · ‎08-20-2020

Introduction

As a solutions architect, I have been doing application performance analysis for several years. Slow performing application is always a disruption to the business growth. I have seen many projects focus on application functionality in the build phase (development phase) and application performance is measured only in the performance test cycle. This approach will result in many performance issues in performance testing, and puts the development teams to do rework in the performance test phase.

I have been part of many war room sessions to resolve production performance issues, I hope its not a new thing for may of fellow architects.

This situation can be avoided by following the Left Shift Performance Tuning Paradigm.

Define, Measure and Scale

The build (or) development phase is the right time to define, measure, and scale application performance.

In SAP commerce applications, the below factors plays a major role in application performance and scalability

Tuning Database queries

Application Scalability

Note: Caching, infrastructure, configurations are also driving factors for application tuning, this article is focused on database usage and application scalability

How to tune database queries

In my experience, additional infrastructure may not help the application performance if the database queries are not properly handled. Improper use of read, update, insert queries will slow down the application on a peak load (promotional) day. Some times system can be unresponsive and can not be recovered. I have observed additional infrastructure puts the systems in unrecoverable state if the system is struggling with improper use of the database.

To avoid this situation, make sure to monitor the database query usage in the application development phase.

Perform the below tasks in the application development phase

Baseline database query count

This activity is to set the targeted number of queries in each flow (or) page.

How can I define a target? - It's simple, compare it with accelerator (B2B/B2C) database query count (or) based on your experience from other implementations. This number is to guide the development cycle.

Define Load profile

Load profile depicts the customer behavior on the site. Identify the critical areas of the application. Define a load profile based on the customer behavior. Especially consider the most used customer flows on a promotional day (peak load day). Create automated test scripts to execute the load profile.

Execute Test

Test the load profile with each build. Compare the database queries from each test with the baseline

Identify the difference between baseline and application queries and log a defect accordingly. This whole process to be automated so that every build is monitoring the database queries.

Tune Database Queries

Fix the issues which are identified as part of the automated test process.

This complete process to be automated and executed with each build. This process can be treated as a quality gate in the development cycle.

Application scalability

Application scalability is a potential factor for application growth. Application scalability check is one more major tuning exercise to measure the database tuning. This also helps the team to validate the database query target.

Steps to measure the application scalability

Define scalability SLAs based on application non-functional requirements

Benchmark single node test

Perform the actual system performance with a single application node and capture the SLAs of a single node. Capture concurrent users, transactions placed in a duration, response time of the site, memory consumption of the server.

Note: If the system is not meeting the SLAs, then reduce the concurrent users. When I define the benchmark for a single node I always check for a healthy system.

Recommendations to perform single node test:

- Remove the external system calls and perform the system test in an isolated mode. Implement stubs in the application for all external interfaces
- Stop batch jobs and backend processes
- Ramp up users based on system memory condition, response times. Always benchmark the concurrent users per node based on the healthy state of the application
- The test scenarios should reflect the customer behavior of a peak load. Ex: Promotional day the customer load is high

Sample matrix

Nodes	Concurrent users	Transactions	Response Time	Stub Enabled	Duration	CPU/Node	Heap Memory/Node	Errors
1	100	2000	2 sec	Yes	60 mins	4GB	2 GB	No

Scalability test

Perform the scalability test with more nodes. perform the test with 2, 4, 6, and more.

How to measure the successful scalability test? Transaction count and Concurrent user count should be increased proportional to the number of nodes. Response time, memory usage, and duration should stay constant. The numbers may not be proportional, but they should be close

Sample matrix with scalability test

Nodes	Concurrent users	Transactions	Response Time	Stub Enabled	Duration	CPU/Node	Heap Memory/Node	Errors
1	100	2000	2 sec	Yes	60 mins	4GB	2 GB	No
2	200	3950	2 sec	Yes	60 mins	4GB	2GB	No
4	400	7900	2 sec	Yes	60 mins	4GB	2GB	Np

Tune system

If the system is not meeting the scalability criteria then we need to tune the system. Consider the below aspects of the tuning based on application performance monitoring (APM) tools

- Validate Database calls
- Validate cache usage
- Code bottlenecks
- Repeat the cycle till we meet the benchmark

Conclusion

Application tuning and scalability is a critical part of the development. Many performance issues can be easily avoided in the performance test phase and in production if the teams understand the importance of this activity. Make sure to perform the below activities in the application development phase.

Define database query benchmarks for each application page and flow

Perform the scalability test to make sure the benchmarks are met

Tune the system based on benchmark

former_member663982 · ‎08-20-2020

Thanks Akhil for the article, it covers so many important aspects of performance tuning. The idea of doing performance tuning while doing implementation is really helpful.

former_member693557 · ‎08-20-2020

This document covers thorough steps that will be really helpful to consider the performance aspects during the development phase itself.

sdevarasetty · ‎08-22-2020

Good Coverage of the most important aspects of the performance tuning exercise. This helps a lot for the performance benchmark exercise and gives the direction to tackle the pain points first and get to the required degree of tuning. Liked the process defined to tune on single nodes and scale up the application further. Great explanation and very useful and perfect to share across the projects.