DMO: downtime optimization by migrating app tables during uptime

[Boris_Rubarth]

This blog discusses a technique to further reduce the downtime of the Database Migration Option (DMO) procedure of Software Update Manager (SUM). Before reading this content, be sure to be familiar with the general concept of DMO, explained in Database Migration Option (DMO) of SUM - Introduction and in DMO technical background

[Update: blog has been updated on Sept 16th 2019 to reflect general availability]

Scope

Downtime-optimized DMO of SUM 2.0 can reduce the downtime of the DMO procedure. It integrates a technology to enable the migration of selected (large) application tables during uptime processing of DMO, thereby reducing the downtime migration time.

 

Technology

During uptime processing, the source system is still available for end users. End user activity in the system may change application tables, so if these tables have already been migrated to the target database (SAP HANA database), the changes have to be recorded and transferred to the target database as well. A dedicated technology offers the required procedure to set triggers on the respective application tables to create log entries, frequently analyze the logs, and transfer the delta to the target database.

Specific considerations compared to "standard" DMO

• A text file has to be created containing the application tables to be migrated during uptime (each table name in a separate line)


• The approach is available with SUM 2.0 (SP 06 and higher), which means:
  
  
  
  • The allowed target release of SAP_BASIS is 750 or higher
  
  
  • The source system has to be on Unicode already
  
  
  
  

 

Known limitations

• downtime-optimized DMO works for SAP Business Suite systems (not for SAP BW)


• downtime optimized DMO IS supported for the scenario System Conversion (targeting SAP S/4HANA) as well, if the source system is not yet on SAP HANA database


• [updated on February 26th 2020:] Restriction is removed with SUM 2.0 SP 07: downtime-optimized DMO IS now supported for targeting SAP S/4HANA 1909

 

No registration for customers any longer

With SUM 2.0 SP 06 and higher, the approach is generally available without the need of registration.

Abbreviations

• PAS: Primary Application Server (fka CI)


• PRD: Productive


• SHD: Shadow


• TGT: Target


• CRR: change record and reply technology

 

Technical background

 

The initial situation is like for the "standard" DMO:

Again, like in standard DMO, the shadow repository is created by the shadow instance:

 

The shadow repository is copied from the source database to the target database, the SAP HANA database.

Note that the shadow instance is still existing, although currently not used, but not deleted as in the standard DMO.

Now the trigger for the selected application tables is set up, and the initial transfer of the triggered tables starts.

The triggers are set by SUM internal technology.



Still in uptime, the delta transfer of the application tables is then executed. Therefore, a job starts the CRR replicator on the shadow instance to check for trigger logs, and transfer the delta to the Writer. For the Writer to write the data to the SAP HANA database, we need an additional instance

that uses the target version kernel for the SAP HANA database. This instance is called TMP instance (temporary).

Downtime starts, now the remaining delta of the application tables are migrated.

Now the remaining application tables (that have not been triggered) have to be migrated as in the standard DMO.

The target kernel is now applied to the PRD instance, the system is started to allow the update of the application tables. This is still business downtime.

Once the application tables are updated and the procedure is finished, the system is available again.

 

[former_member1232]

Hi Boris,

I hope all is well on your end. Just reading this wonderful blog of yours on the doDMO topic you wrote in 2014. I believe Downtime Optimized Database Migration(doDMO) of SUM 2.0 got a bunch of technical restrictions from SAP product development and it still exists as of today (2020), do you see any of these will go away from a technical perspective in near future?

• You cannot use downtime-optimized DMO with the scenario “DMO with System Move”
• You can combine the scenario DMO without Software Update with downtime-optimized DMO
• You must switch off other trigger-based technologies such as SAP Landscape Transformation (SLT) when using downtime-optimized DMO.
• Switch off the ABAP-based archiving during the downtime-optimized DMO.
• Tables added for Downtime Optimized handling must not be part of Table Comparison
• Non-Unicode start releases are not supported.
• Not applicable to the BW system.

  Thanks,
  Amit Lal

[former_member232760]

Hi Boris,

Thank you for sharing.
A bit update:

• Current restriction: downtime-optimized DMO is not currently supported for targeting SAP S/4HANA 1909

Restriction was removed starting with SUM 2.0 SP07.

Regards,

[Boris_Rubarth]

Hi Bobby,

thanks for pointing this out, I have update the respective sentence in the blog now.

Regards, Boris

[Boris_Rubarth]

Hi Amit Lal,

sorry for the delay ... actually at first sight, I am afraid I have no news on this list so far.

Thanks and regards, Boris

[former_member1232]

Thanks, Boris. I appreciate your inputs.

[former_member684719]

Hi Boris,

while performing DoDMO with below system we are facing an issue to upgrade the DB version to oracle 12.

Source : EHP7,Oracle 11g,AIX 7.1

Target : S4 Hana 1909 FPS00,SLES 15 SP1

SUM Tool :  2.0 SP8 PL1

as it is not mentioned for S4 1909 the shadow is created on target DB and the minimum required DB is oracle 11. we are not able to understand this case.

Do we have any limitation that when we use DoDMO the source db restrictions still exists?

[Boris_Rubarth]

Hi Shiva,

please create an incident for this, component BC-UPG-TLS-TLA.

Regards, Boris

[Boris_Rubarth]

Hi Shiva,

today we have updated the related notes (DMO and downtime-optimized DMO) with the requirement to have Oracle on version 12 or higher for downtime-optimized DMO.

Regards, Boris

[former_member1232]

Hi Boris,

With DMO with System Move(SUM 2.0 SP7) now, Windows is supported, but how to perform a DMO with system move and PARALLEL TRANSFER on Windows, knowing that RSYNC cannot be used. In Linux case, dmo2cloud.sh specifically calls the RSYNC tool.

Robocopy is feasible here? DMO guide did not cover insights on this option. Any inputs will be highly appreciated.

Regards,
Amit

[Boris_Rubarth]

Hi Amit,

good point. I am not a network expert and this is not SUM specific, so my hope would be that there are solutions and/or experts out there that have an idea.

Regards, Boris

[amitlalmicrosoft]

Thanks, Boris.

We are checking internally with SAP Maxattention to understand in detail.

But it looks like the parallel transfer option available only for Linux at present. DMO SUM guide and SAP SUM Note should add this point exclusively.

Also, I see SUM 2.0 binaries dmo2cloud.sh is renamed to dmosystemmove.sh, but can't find anything for windows in SUM repo.

Again, Thanks so much for your quick input.

Have a nice weekend!

Best Regards,
Amit Lal

 

 

[former_member684719]

Hi Boris,

apologies for delayed update.

we have created an incident before updating this here. we got the same response that it has been updated in the notes(DMO and downtime-optimized DMO- 2547309). but the note is still being updated.

The SUM 2.0 SP08(2882441) has been updated with this information.

Just FYI : due to some timelines constraint we have now proceed with standard DMO option.

Thanks a lot for your clarification.

Regards,

Shiva P

[former_member699455]

Hi Boris,

Can we adjust the transfer performance for the RSYNC script "dmosystemmove.sh"?

It seems the default parameter DMO_SYSTEMMOVE_NPROCS is set to 4. Would 6 or 8 be better if our export size is~ 760GB?

 

Thanks,

Jun

[Boris_Rubarth]

Hi Jun,

thanks for asking.

The script delivered with SUM is one way to handle the synchronization, you can adjust it, if you are experienced. Concerning the default parameter you mention, it is not as simple to judge on the export size to know the optimal number of processes. Nevertheless you can try to adapt the number by setting an environment variable. You do this by creating SAPup_add.par file (in SUM/abap/bin) and specify the variable with a line in that file, e.g.

/proc/userenv= DMO_SYSTEMMOVE_NPROCS=8

Regards, Boris

[former_member699455]

Thank you Boris. Will test that and let you know how it goes.

[former_member699455]

Hi Boris,

The environment variable is working and I can see the log shows the NPROCS is 8. But at the OS level, I can see only one rsync process. Is the NPROCS a sub-process of the rsync running in the background and can not be visible?

Also the new value of 8 seems not much difference for the data transfer speed. Would a higher number better? e.g. 16?

 

Thanks,

Jun

[Boris_Rubarth]

Hi Jun,

there are phases in which only one rsync is running, to check which files to synchronize. You can check the rsync process on OS level via "ps -ef" and should see it listed with parameter "--dry-run". If you should see that there is only one rsync even for the actual synchronization time (process with "--files-from"), that would be an issue.

Concerning the number of processes: as mentioned in my previous answer, there is no general guidance as it depends on various factors like bandwidth.

Regards, Boris

[former_member699455]

Hi Boris,

 

Yes, I saw that there are multiple rsync processes and ssh processes during the synchronization time and I could see the files were being transferred quite OK.

But then after half way, only one rsync and ssh left and running very slow. I could not the reason why that's case.

Should the rsync version must be identical between source and target?

Source is UNIX rsync  version 3.0.5  protocol version 30.

Target is Linux rsync  version 3.1.3  protocol version 31

Thanks,

Jun

[Boris_Rubarth]

Hi Jun,

I am not aware that the rsync version has an impact, as long as it is higher than 3.

Regards, Boris

[former_member699455]

Yeah, the requirement is to use rsync version 3 and above. I had raised an OSS to BC-UPG-TLS-TLA. You colleague is asking for a login to investigate the issue.

[former_member699455]

Hi Boris,

 

I have noticed the following issue in the latest test upgrade run over the weekend:

 

---

The upgrade triggers parallel file transfer via rsync. There were multiple rsync running and each one got its own sync list to transfer the files contained in the list. Once those parallel rsync are done, there’s always one rysnc left and it’s transferring files which were already done and exist in the target host. Once this single rsync is completed then a new parallel file transfer for some new files will be triggered and repeating the same process again (parallel rsync, then single rsync for old files).

---

 

That's the reason I am having very slow file transfer. Serial mode would be faster. Do you know why there's always a single rsync transferring the old file again. I checked those the files size in the target hostand there's no difference.

 

Thanks,

Jun

 

[former_member581265]

Hi Amit.
I have info, that at DMO with system move on Windows for RSYNC  one can use CYGWIN Linux environment. Possibly Linux subsystem on Windows Server can be used. But this is not officially supported by SAP.

[former_member581265]

Hi Boris.
Thank you very much for being responsive in article's comments.
I have 2 questions:

1. When downtime optimization will be available for System Move scenario? I believe technically this is feasible.

2.At SAP's 10Steps2S4 education uploading 13 GB data to migration server took 13 Min. This is 17 MB/s. And that was between systems in the same LAN.  Here (link below) on page 23 test shows the same number. It is visible, that bottleneck is SSH.
https://princetonuniversity.github.io/PUbootcamp/sessions/data-transfer-basics/PUBootCamp_20181031_D...
Patched HPN-SSH can saturate 1 Gbit WAN channel and give 95 to 150 MB/s depending  on server hardware. But it should be compiled and installed on source and target hosts. In this case we should not modify dmosystemmove.sh script.
I see alternative in using parallel copy tool BBCP (page 24 of the PDF doc above) and edit dmosystemmove.sh script to adopt it. Looks BBCP functionality allows usage in SUM DMO parallel mode. Looks with BBCP can saturate any WAN channel. And we should not care about this limit more.

http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm

How are you going to solve this bottleneck in RSYNC/SSH?

Update:
I've made a test on 2 SLES hosts connected with 1 Gbit LAN on 2 GB directory and also one 8 GB file. Effective speed for bbcp  и для rsync.  For uncompressed directory rsync outperformed bbcp (76 vs 71 MB/s). Large file transfer gave near-line effective speed 99,4 MB/s. Possibly saturation for rsync will occur somewhere at higher speeds - maybe 130 MB/s or because of disk subsystem. 

[Boris_Rubarth]

Hi Andrei,

thank you for your contribution.

1) For "DMO with system move", using the parallel mode is the downtime optimization that we offer, if possible with continuous synchronization. No plans to support e.g. downtime-optimized DMO.

2) In general, we deliver the script dmosystemmove.sh as a kind of proposal and easy possibility to have a continuous synchronization. The script may be adapted. As this part of the scenario is not SUM specific, any optimizations should be discussed by network experts, and may be applied on each project specifically.

Looks like you have some ideas, so hopefully the SAP Community will join the discussion and comment these.

Regards,
Boris

[sudip_saha1]

Hi Boris,

This blog is for standard” DMO. Do you have something for Downtime-Optimize DMO?

 

Thank you