1. Introduction:
In this blog post, you will learn how to deploy PuLP based XSA applications in IBM PowerPC machines.
1.1. Background:
Emerging cloud technologies and on-premise runtime environments like HANA XSA dramatically help businesses not only build and run their analytical solutions in a secure and high-performance environment but also make solutions easily accessible to end-users like business analysts and sales managers.
HANA and cloud platforms provide runtime environments for a different choice of programming languages. Python is known in ML and AI world as one of the most used programming languages. The computational performance of Python is not comparable to many low-level programming languages, therefore, many optimization algorithms are written in C++ and delivered as compiled binary files, and optimization libraries like PuLP act as a wrapper around the core solver, and execute these binaries at the OS level to deliver results.
2. Problem statement
Even though Runtime environments are supposed to be independent of the OS, when it comes to optimization, binary files should execute at the OS level. Hence, the binary files must match the hardware architecture and the OS.
In this blog, how to resolve the OS mismatch error of the CBC solver binary file inside the python PuLP library and execute it on IBM PowerPC architecture (IBM ppc64le) will be demonstrated. This can be seen as an example of how to resolve this kind of issue in any Linux system.
If the binary file cannot be executed “OSError: [Errno 8] Exec format error” will be raised as depicted in Figure 1.
Figure 1 OSError
3. Solution
Below you can find steps on how to make the CBC work in the ppc64le environment:
3.1. Download and Build CBC on the platform:
Get the CBC solver from the link below and build it in the target Linux system with the “make” command.
https://www.coin-or.org/download/source/Cbc/
Navigate to the newly build CBC location, add "test.lp" to the directory and run the below command for testing the CBC solver.
>> ./cbc test.lp solve solu=sol.txtIf the result is similar to the below results, the CBC has been successfully built and everything is set for the next step.
Welcome to the CBC MILP Solver
Version: 2.10.5
Build Date: Sep 6 2021
command line - ./cbc test.lp solve solu=sol.txt (default strategy 1)
CoinLpIO::readLp(): Maximization problem reformulated as minimization
Coin0009I Switching back to maximization to get correct duals etc
Continuous objective value is 5.39478e+07 - 0.00 seconds
Cgl0002I 2 variables fixed
Cgl0003I 1 fixed, 0 tightened bounds, 702 strengthened rows, 4 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 456 strengthened rows, 1 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 260 strengthened rows, 0 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 74 strengthened rows, 0 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 58 strengthened rows, 0 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 9 strengthened rows, 0 substitutions
Cgl0003I 0 fixed, 0 tightened bounds, 3 strengthened rows, 0 substitutions
Cgl0004I processed model has 841 rows, 290 columns (290 integer (290 of which binary)) and 4352 elements
Cbc0038I Initial state - 0 integers unsatisfied sum - 1.44329e-15
Cbc0038I Solution found of -5.39478e+07
Cbc0038I Before mini branch and bound, 290 integers at bound fixed and 0 continuous
Cbc0038I Mini branch and bound did not improve solution (0.09 seconds)
Cbc0038I After 0.09 seconds - Feasibility pump exiting with objective of -5.39478e+07 - took 0.00 seconds
Cbc0012I Integer solution of -53947771 found by feasibility pump after 0 iterations and 0 nodes (0.09 seconds)
Cbc0001I Search completed - best objective -53947771.20000002, took 0 iterations and 0 nodes (0.09 seconds)
Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost
Cuts at root node changed objective from -5.39478e+07 to -5.39478e+07
Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
Result - Optimal solution found
Objective value: 53947771.20000002
Enumerated nodes: 0
Total iterations: 0
Time (CPU seconds): 0.10
Time (Wallclock seconds): 0.10
Total time (CPU seconds): 0.10 (Wallclock seconds): 0.113.2. Repackageing PuLP library with the new CBC solver
This section can be run from any machine, Windows or Linux as long as you have the python installed in that environment, the GitHub repository can be cloned, and the newly built CBC file can be moved to the machine.
Download pulp from the Github repository.
https://github.com/coin-or/pulp
Replace the recently built CBC binary file (from section 3.1) in the pulp directory below.
Please take note of the target hardware architecture. (32 or 64 bit)
~\pulp-master\pulp\solverdir\cbc\linux\64After manual replacement, it is time to repackage the PuLP project with the new CBC binary file, the PuLP version that is being downloaded is mentioned in the "constraints.py" of the PuLP package. Be aware that you need to create a custom version so that after repackaging, pip does not confuse it with the version available in the python library repository (PyPI). Personally, I add one zero as a prefix to the original version number but you can make your own choice, if the PuLP version is 2.5.0 you can change the version number to 0.2.5.0.
~\pulp-master\pulp\constants.pyBefore proceeding with the packaging, make sure the weel package is already installed on python. If it is not available you need to perform the next step.
python3.* -m pip install wheelNow, navigate back to the root of the PuLP master folder and run the below command:
python3.* -m pip install check-wheel-contentsIf all requirements are satisfied as figure 2, everything is set for the next level, and you can build the wheel out of pulp with the below command:
python3.* setup.py bdist_wheelBy running this command couple of folders will be created, in the “dist” folder the wheel can be found.
~\pulp-master\dist
Figure 2 Successful Build
3.3. Solver_test application with custom python libraries
The Solver_test XSA package is available in Github. Before pushing the application to XSA, all python libraries need to download and copy into the vendor folder, so during the push, XSA from the "vendor" folder installs python libraries rather than downloading them from the python public repository.
Using this method is a good choice if the XSA system is disconnected from the Internet too. To get all packages in the "vendor" folder, the below commands should be run in the target system (Linux, and in this case the IBM PowerPC).
After cloning the repository from GitHub, run the below command in the target system:
python3.* -m pip download -d ~/project_folder/vendor -r ~/project_folder/requirements.txtThe above command will create a new folder called "vendor" which includes all python packages mentioned in "requirements.txt". Place this folder at the root of the project, now you can deploy the package in XSA, it will install python libraries from the "vendor" folder. And the OS will pick the right CBC solver binary and execute it.
4. conclusion
IBM PowerPC architecture is compatible with the SAP HANA platform, but when it comes to the XSA python application installation, some libraries do not work in this environment, in this blog we learned how to re-build and re-package components of CBC solver to make it deployable on XSA on IBM PowerPC architecture.
5. Resources
GitHub repository for the sample code:
https://github.com/jahmadian/xsa_cbc_solver.git
The COIN-OR Foundation, Inc.
https://www.coin-or.org/
GitHub repository for python PuLP library:
https://github.com/coin-or/pulp
- SAP Managed Tags:
