XS Advanced: Update to Patch Level 13 failed

[fabian_krger]

Hello,

I just tried to update my XS Advanced from Patch Level 9 (from initial installation of Rev. 110) to Patch Level 13 (latest available) after installing HANA Rev. 111.

Basically everything went OK, but in the end I got an error:

6:05:38.321 - INFO:   ---------------------------------------------------------
16:05:38.321 - INFO:   Calling postInstall event handler
16:05:38.321 - INFO:   ---------------------------------------------------------
16:05:38.321 - INFO:     isUpdate=1
16:05:38.321 - INFO:     -------------------------------------------------------
16:05:38.321 - INFO:     Starting system...
16:05:38.321 - INFO:     -------------------------------------------------------
16:05:38.322 - INFO:       Starting instance on host 'hdb' (worker, xs_worker)...
16:05:38.322 - INFO:       Parameters: instance number = 00, user = hdbadm
16:05:38.326 - INFO:         Instance is already running
16:05:38.326 - INFO:       Instance on host 'hdb' (worker, xs_worker) started
16:05:38.326 - INFO:     -------------------------------------------------------
16:05:38.326 - INFO:     END: Start system
16:05:38.326 - INFO:     -------------------------------------------------------
16:05:38.326 - INFO:     Configuring software...
16:05:38.326 - INFO:       6 Package(s) installed
16:05:38.326 - INFO:       baseVersion = 1.00.00.262519
16:05:38.326 - INFO:       Write parameters to secure store
16:05:38.379 - INFO:     Starting external program /hana/shared/HDB/xs/installation-scripts/installation/storeParameters
16:05:38.380 - INFO:       Command line is: /hana/shared/HDB/xs/installation-scripts/installation/storeParameters
16:05:38.584 - INFO:       Output line 1: Initially filling HANA Secure Store with parameters
16:05:38.634 - INFO:       Output line 2: RSecSSFs: Entering function "RSecSSFsGetRecord" [/bas/745_REL/src/krn/rsec/rsecssfs.c 1433]
16:05:38.634 - INFO:       Output line 3: RSecSSFs: Using explicitly set configuration data [/bas/745_REL/src/krn/rsec/rsecssfs.c 7811]
16:05:38.634 - INFO:       Output line 4: RSecSSFs: SSFS-1440: File "/usr/sap/HDB/HDB/SYS/global/security/rsecssfs/data/SSFS_HDB.DAT" cannot be opened in mode "rb": No such file or directory (errno = 2) [/bas/745_REL/src/krn/rsec/rsecssfs.c 1794]
16:05:38.634 - INFO:       Output line 5: RSecSSFs: SSFS-4182: Data file "/usr/sap/HDB/HDB/SYS/global/security/rsecssfs/data/SSFS_HDB.DAT" does not exist (this is not an error per se: a non-existing data file is a valid situation and treated equally as one without entries) [/bas/745_REL/src/krn/rsec/rsecssfs.c 4908]
16:05:38.634 - INFO:       Output line 6: RSecSSFs: SSFS-4187: Record with key "XSA/CORE/RUNTIME_DB_USER_NAME" not found in secure storage [/bas/745_REL/src/krn/rsec/rsecssfs.c 4911]
16:05:38.634 - INFO:       Output line 7: RSecSSFs: Exiting function "RSecSSFsGetRecord" with return code 1 (message: SSFS-4187: Record with key "XSA/CORE/RUNTIME_DB_USER_NAME" not found in secure storage <-- SSFS-4182: Data file "/usr/sap/HDB/HDB/SYS/global/security/rsecssfs/data/SSFS_HDB.DAT" does not exist (this is not an error per se: a non-existing data file is a valid situation and treated equally as one without entries) <-- SSFS-1440: File "/usr/sap/HDB/HDB/SYS/global/security/rsecssfs/data/SSFS_HDB.DAT" cannot be opened in mode "rb": No such file or directory (errno = 2)) [/bas/745_REL/src/krn/rsec/rsecssfs.c 1522]
16:05:40.916 - INFO:       Output line 8: Exception in thread "main" com.sap.security.nw.SecStoreFS.SecStoreFSException: SSFS-1560: Could not open lockfile; open(/usr/sap/HDB/HDB/SYS/global/security/rsecssfs/data/SSFS_HDB.LCK) == 2 (No such file or directory) [17=EEXIST,13=EACCES,2=ENOENT,22=EINVAL,20=ENOTDIR] (RC = -3)
16:05:40.916 - INFO:       Output line 9: at com.sap.security.nw.SecStoreFS.SecStoreFS.putRecord(Native Method)
16:05:40.917 - INFO:       Output line 10: at com.sap.xs2rt.installation.impl.util.InstallationParameterWriterSecureStoreImpl.initParameters(InstallationParameterWriterSecureStoreImpl.java:235)
16:05:40.917 - INFO:       Output line 11: at com.sap.xs2rt.installation.impl.util.InstallationParameterWriterSecureStoreImpl.<init>(InstallationParameterWriterSecureStoreImpl.java:213)
16:05:40.917 - INFO:       Output line 12: at com.sap.xs2rt.installation.impl.util.InstallationParameterWriterSecureStoreImpl.createFromStdIn(InstallationParameterWriterSecureStoreImpl.java:142)
16:05:40.917 - INFO:       Output line 13: at com.sap.xs2rt.installation.impl.util.InstallationParameterWriterSecureStoreImpl.main(InstallationParameterWriterSecureStoreImpl.java:121)
16:05:40.922 - INFO:       Program terminated with exit code 1
16:05:40.922 - INFO:     Failed to write parameters to secure store.
16:05:40.922 - ERR :     Custom event postInstall failed
16:05:40.922 - INFO:   ---------------------------------------------------------
16:05:40.922 - INFO:   END: postInstall event handler (start: 16:05:38.321 duration: 00:00:02.601)
16:05:40.922 - INFO:   ---------------------------------------------------------
16:05:40.923 - INFO: -----------------------------------------------------------
16:05:40.923 - INFO: END: Installing SAP HANA XS RUNTIME (start: 16:05:17.192 duration: 00:00:23.730)
16:05:40.923 - INFO: -----------------------------------------------------------
16:05:40.922 - ERR : Cannot install
16:05:40.923 - ERR : error installing
16:05:40.923 - ERR : Installation failed
16:05:40.923 - INFO: Summary of critical errors
16:05:40.923 - ERR :   Installation failed
16:05:40.923 - ERR :     error installing
16:05:40.922 - ERR :       Cannot install
16:05:40.922 - ERR :         Custom event postInstall failed

Of course the file /usr/sap/HDB/HDB/SYS/global/security/rsecssfs/data/SSFS_HDB.DAT doesn't exist since it is at /usr/sap/HDB/SYS/global/security/rsecssfs/data/SSFS_HDB.DAT (there is one "/HDB" too much in the path).

It looks like installation was nearly complete, but now xscontroller, xsexecagent and xsuaaserver are not running (status red via studio). There are no errors related to those 3 processes in trace folder, but I found out that the number of instances have been set to 0 in daemon.ini configuration file.

What am I supposed to do now? Should I try setting the number of instances to 1? Is this an error in PL 13?

Any hints appreciated,

Fabian

[ansteiner · Accepted solution]

Hi Fabian,

the problem with the path is a known issue and will be fixed with the next patch. It's because of the SID you used (HDB), so it will work with another SID than HDB without a problem.

About the new problem with the access token, this is related to the XSA update (xscontroller). Please do a logout or a new login with the API URL to get a new valid token.

Kind regards,

Andreas

[fabian_krger · Accepted solution]

Hi Andreas,

thanks for clarification.

Unfortunately I couldn't solve the token issue (login, password change; nothing helped) so I ended up restoring from snapshot and not doing the update... I will wait for PL14 and try again then.

Best Regards,

Fabian

[ansteiner · Accepted solution]

Hi Fabian,

a new login -a <API_URL> should renew the access token. If this login will also fail, I'm really interested to get the error message.

In addition you can open a support ticket on BC-XS to follow up on that. Because I don't think this will change with an other update like PL14.

Kind regards,

Andreas

[fabian_krger · Accepted solution]

Hi Andreas,

I tried "xs api [...]" and after that "xs login" without success. I did not try "xs logout" though since I already restored the instance from snapshot... so unfortunately I can't test this any more.

I tested the login from different clients as well as on the instance itself - everywhere the same result. "xs oauth-token" was displaying a token but somehow I couldn't get it working.

Anyhow I would let you know if this issue comes up again some day.

Thanks for your help,

Fabian

[fabian_krger · Accepted solution]

Hi Andreas,

PL17 still has the same issue

So PL17 is not "the next patch"?

Best Regards,

Fabian

[ansteiner · Accepted solution]

Hi Fabian,

unfortunately the fix about the HDB SID problem didn't made it into the current patch level (1.0.17). Or are you facing again a problem with the access token?

If it is still the HDB SID problem and critical for you, please let me know. We can then provide it with the next patch level in the next days. Otherwise it will take some weeks until we will provide the next regular update.

Kind regards,

Andreas

[fabian_krger · Accepted solution]

Hello Andreas,

in the meantime I tried the symlink option again.

I did a reinstall without deleting the xs folder then.

Basically it's working now (all processes running without issues), only the installer is still stuck:

As long as I don't encounter any errors I will probably leave it as it is now.

xs tool is working this time, no problems with access token...

Interesting fact: xs version gives me still 1.0.9, xs system-info 1.0.17 instead:

Would you recommend trying to fix this by either deleting xs folder and dirty reinstall or going back to snapshot and making the symlink before first installation attempt?

[ansteiner · Accepted solution]

Hi Fabian,

as not only the xs client is on the older version, also the controller is on the old level. But the execution agent is already on the new version. You should go back to the snapshot, create the symlink and do a clean fresh installation with 1.0.17.

Because there are already user and worker roles created in the HANA, deleting the xs folder isn't enough and can end up in installation problems again.

Kind regards,

Andreas

[thomas_jung]

I've had this happen as well.  I can offer advice based upon my experience but it isn't an official answer from SAP. I encourage you to enter a support ticket.

What I've done to get around this error is to create a dummy HDB folder to match the incorrect structure the installer is looking for. I then copy the content contents of rsecssfs into this duplicate/dummy directory structure.  Now if you just rerun the installer, it will fail towards the end on a an access error since its trying to update the installation that never finished.  I find that if I delete the entire /hana/shared/<SID>/xs folder and all its contents and then rerun the installer it will perform a clean install (but now with the dummy folder structure in place so it finds the rseccssfs). However I can't speak to the safety of manually deleting your xs folder.  It worked on several systems I've built, but I'm building throwaway test and development systems so I took the risk.  You mileage may very -- perform this action at your own risk, don't sue me, etc.

[fabian_krger]

Thanks Thomas for your reply.

If I understand correctly, why can't i just create a symlink?

ln -s /usr/sap/HDB /usr/sap/HDB/HDB

Re-installation was successful with this symlink. (but the number of instances was not set to 1 after installation, I had to do this manually)

From the server it looks good now, BUT xs command-line tool is not working any more. Of course certificate is untrusted but after login I always get

FAILED: Could not decode access token: Bad credentials.

I also double checked the credentials, but if I really input a wrong password I get a different error:

Authentication failed: Name or password mismatch

At first I thought it might be a mismatch between xs tool (1.0.9) and xs controller (1.0.13) but I get the same error on the instance itself with command-line tool 1.0.9...

Do you have any idea what might be the cause for the access token issue? Otherwise I will probably re-install XSA again tomorrow...

Best Regards,

Fabian

[thomas_jung]

Nope sorry, never hit that next error.  And much better idea on the symlink than a copy. You can tell my Linux/Unix skills are rusty. I resort back to what I know.