SAP ASE Backup issue with NFS mount. write access Error

[darshinee]

Hi , I am taking backup through SAP ASE DB Application using dump command.
I am taking the backup on NFS mount.
But it is giving me below errors. please help in this case. We are qualifying SAP-ASE for our customers
1> dump database test_DB to '/mnt1_nfs/sapase-stu/1.bak'

2> go

Backup Server: 4.172.1.4: The value of 'allocated pages threshold' has been set

to 40%.

Backup Server session id is: 6. Use this value when executing the

'sp_volchanged' system stored procedure after fulfilling any volume change

request from the Backup Server.

Backup Server: 4.41.1.1: Creating new disk file /mnt1_nfs/sapase-stu/1.bak.

Backup Server: 4.141.2.87: [11] The 'lockf' call failed for database/archive

device while working on stripe device '/mnt1_nfs/sapase-stu/1.bak' with error

number 5 (Input/output error). Refer to your operating system documentation for

further details.

Backup Server: 6.53.1.1: OPERATOR: Volume on device '/mnt1_nfs/sapase-stu/1.bak'

cannot be opened for write access. Mount another volume.

[c_baker · Accepted solution]

Resolution

1. Determine if the dump is hanging on the file locking by using:

tusc on HP-UX
truss on Solaris
strace on Linux

1. Start the Backup Server
   
2. Run this command: tusc -f -o <output file> -p <pid>
   
3. Start dump database.

If the trace output contains a line like shown below then most likely there is an issue with file locking on the NFS server.

lockf(0, 0x2, 0) .........................................
[sleeping]

If dumping to NFS device, file locking must be enabled or it will hang. If file locking is enabled and the hang still occurs then the NFS server may need to be rebooted.

[darshinee · Accepted solution]

Thanks Chris. The locking was not enabled .
when I mounted nfs share , the local_lock=all option was not enabled.
That's why the issue was coming.
When I enabled that option, the Backup worked.

10.198.165.177:/data/col1/sapase-stu on /mnt1_nfs/sapase-stu type nfs (rw,nosuid,nodev,noexec,relatime,sync,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.198.165.177,mountvers=3,mountport=2052,mountproto=udp,local_lock=all,addr=10.198.165.177,user)
Thanks a lot everyone for timely replies.
Regards,
Punya

[bart_van_kuijk]

Hello Punyatoya

This is not an ASE error or something that can be resolved from ASE side.

The message clearly references an OS error :

The 'lockf' call failed for database/archive device while working on stripe device '/mnt1_nfs/sapase-stu/1.bak'
with error number 5 (Input/output error). Refer to your operating system documentation for further details.

With a further clue in the next message

Backup Server: 6.53.1.1: OPERATOR: Volume on device '/mnt1_nfs/sapase-stu/1.bak'
cannot be opened for write access. Mount another volume.

An input/output is quite a general error category. If you need more help, refer to your OS team or web search it. Most likely, in the context of what you report, it indicates that the Backup cannot access this mount due to permissions not set correctly for the user that run the Backup Process , which is typically the 'sybase' OS user.

Kind regards

Bart

[darshinee]

Hi Bart,
Thanks a lot for your response. I tried to directly write the file to this mount point without using DB application, I see no issue.
But when I use ASE to perform Backup to this mount point, It gives an access error.

I saw one KB article released by ASE where this error occurs with Commvault for NFS mount.

I am unable to access that article. if you could check and see , that would be a great help.
I am stuck because the ASE backup works with fuse mount, other mount points but fails with NFS mount point.
Thanks and Regards,
Punya

[darshinee]

This is the KB Article I am referring to. I am unable to access it. if you can check that and help me with the steps to resolve it. That would be of great help.

https://userapps.support.sap.com/sap/support/knowledge/en/2211841

[bart_van_kuijk]

I had seen that KBA before I replied and that KBA states the same as I replied with. There is nothing specific in that KBA for Commvault or general NFS. Quoting from that KBA :

Cause

The permission to the directory is not correctly set to dump.

Resolution

Abort the hang session by executing sp_volchanged with ‘ABORT’ action.

Consult your system engineer to solve the permission problem.

Redo the dump.

------ end KBA

This is really something you need to look at OS level. When you say

"I tried to directly write the file to this mount point without using DB application,"

Is this 'I' as your Unix login or the 'sybase' user (or which ever the user is that is running the Backup Server ? You may want to deploy some OS tracing. Before you call this backup, start a trace on BS process, using -f as it will need to capture the spawned sybmultbuf process.

strace -fvtTo /tmp/strace.out -s 1024 -p <backupserver_pid#)

You may want to do same for one of the working dumps and compare where / what call exactly fails and share that with OS side.

[darshinee]

Hi Bart,

I found one KB article https://userapps.support.sap.com/sap/support/knowledge/en/1924291.
In my case also Backup server hangs with NFS mount. I don't see error with in strace logs

1924291 - Backup server hangs when using NFS mounted disk - SAP ASE

Could you please provide me the details about the Resoulation provided in this Article.
Regards,
Punya

[former_member990744]

The file gets created and locked to avoid that it gets overwritten by a concurrent dump. The first to check is if there isnt another session, maybe left around from a previous dump that is locking the file. See if there are symbultbuf processes around.

[former_member990744]

Does it fail no matter the file name?

[darshinee]

Hi Carlos,
Thanks a lot for your response.
yes, it fails no matter the file name. I checked there is no other session going on.
I saw one KB article released by ASE where this error occurs with Commvault for NFS mount.
I am unable to access that article. if you could check and see , that would be a great help.
Thanks and Regards,
Punya

[darshinee]

This is the KB Article I am referring to. I am unable to access it. if you can check that and help me with the steps to resolve it. That would be of great help.
https://userapps.support.sap.com/sap/support/knowledge/en/2211841
Thanks a lot,
Punya