Stability issues post 1808 - Are you also being affected.

[Obuone]

Hello,

I'm trying to evalutate if GSK's intermittent outage situation is specific to us or if it is affecting other customers.

Here is the background:

After past weekend upgrade to 1808p20, we at GSK were affected by an outage of our learning platform on Monday.

SAP had confirmed that this was a DC2 specific outage that required a fix to be applied on monday evening and it was affecting all customers on DC2

The official communication mentions the fix was successfully applied on DC2 

However, we continue to suffer from intermittent outages where the system is completely unaccessible, throwing a "BAD GATEWAY" error for periods of 10 minutes.

Eventuallu the system comes back up and goes down a few minutes/hours later.

This is happening now for the past 2 days (Tuesday and Wednesday) and is heavily impacting our users.

Are there any other customers facing similar issues ?

Thanks for your reply

[Brabham_Marla]

We have had that as well, but it only happened for one day for about 5 minutes, I was in the system working and clicked on a link and got the "Bad Gateway" message; the issue that we have had, has been more around the slowness of going from one screen to another in the LMS.

[Obuone]

Thanks.

Has it been resolved now or is it still happening, on a random basis ?

If resolved, do you have any idea what did SAP do ?

Regards

[Ibrosseau]

Since the upgrade, we had several issue with the Database UTC convsersion, access to curricula etc.

Also every week-end since, a downtime is planned and incident occurs. (DC8)

This was our first upgrade, we are hoping this year is an exception, else we doubt SuccessFactor Stability.

[PKetnouvong]

We are having the same issue with the Item and Curriculum Connectors.  It works in our vStaging environment with ANSI and PROD (pre upgrade). 

I have a ticket with SAP.

We also noticed our notifications are off.  Learner fields are not being populated.

[PKetnouvong]

Does anyone else have issues with their Item and Curriculum Connector?  We have an integration from our document management system to the LMS.  It's causing a delay in when items are updated and assignments in the curriculum.

[Obuone]

Hi, 

I can confirm all of our connectors are now running much slower than it used to before HANA.

This is causing also a lot of disruption in our operations !

[may_thang]

We are experiencing the same issues. Its slower hence it impacted our item connector. It was scheduled to run every hour and since it was delayed, that hour's file was not processed and the next hour's file had overwritten it since it had the exact same file name.

[Obuone]

I wanted to provide an update to the stability issues:

1. SAP have now suggested they would use a HANA DB feature to flag the labels associated to our inactive items & curricula so that they are not indexed. They believe this would help regain some of the lost performance

2. They have also identified 2 patches that could address both a general product bug and a GSK specific bug. No ETA for those patches to be made available.

3. SAP is asking us to start working on a solution to archive our inactive labels (archive = get rid of them from the system) as it seems it is causing the perf. issues.

On #3, has any other customer ever been asked by SAP to archive data (in the sense : offload from the live environment) ? If so I would be intrested to hear experience and solution put in place.

Thanks.

[Obuone]

Just wanted to close off this thread with a last update:

The database flagging + deployment of the 2 patches have stabilized our production instance.

We don't have intermittend outage any longer.

However, it is clear that there are a lot of HANA / Upgrade erros hitting us (and other customers) and we're not quite happy with the quality of this release.

Finally, it seems the archiving recommendation from SAP is still going to be our next step forward, even though we find it strange that lables on inactive objects could be the cause of user issues that only look at active items.

[JustinChlan]

We are hosted in DC12 and have not had any performance related issues with respect to the upgrade or our move to HANA. Based on feedback it seems more related to the datacenter than the application (exclusively). Hope the issues resolve quickly.

[Obuone]

Thank you all for your feedback so far.

There is clearly something related to DC2 but also it seems that tweaks may be required at HANA level to stabilize our system at peak usage hours.

If anyone else experiences issues, I'd be glad if those could be posted here.

[Wick_Orley]

We, too, have experienced intermittent issues with Bad Gateways/downtimes. We are on DC8 for Learning and were moved to HANA in January. We have also recently noticed issues with our reports created through ORD that contain learning data where sporadic data is missing when the report is distributed through email. Data is there when the report is run/exported manually through Analytics, but when emailed some data is missing. Overall, I would say our instance has not been stable since Dec2018 (Learning and overall BizX).

[Obuone]

Hi Wick,
That's very intresting as this is exactly the same experience that we have since Monday (post the upgrade).
Reports also have issues with data missing. For this piece though, SAP has stepped forward (prior to the upgrade) and provided updated version of the reports tuned for HANA. However our internal testing wasn't always successful and some of those have to be reworked.

[may_thang]

We are on DC8 and we are experiencing the same issue however the "BAD GATEWAY" is very short. By the time the users ping me and I log on, it went away. I've experienced it myself and have been clicking on the Home button which re-logs me in and the issue went away. In addition the Learning Administration page has been intermittently not loading and I find myself constantly clicking around to refresh the screen. I'm not sure if this is all related but wonder if you experience the same thing. This issue has come up after this weekend's upgrade as well. I wonder if this has anything to do with being migrated to HANA.

[AF21]

We are experiencing the same issues on DC8 for reporting timing out and Bad Gateway. Noticed this gets to be worse during the day when more users are in and pulling reports. 

[Obuone]

That's also similar to us : reports are timing out or being queued without being processed.
Then the Bad Gateway is a second manifestation of the issue.
It seems it occuers more often when people come online early moning EU hours then at the start of the US day.

Did you get any feedback from SAP ?

[AF21]

Our tickets into Support were not handled well. By the time they responsed there were no more issues on Monday. We have seen few issues on Tuesday and Wednesday but still experiencing report slowness and abort message on reports that were not an issue last week. 

We did finally hear from our CSM that an an internal memo overnight indicating there was a 12 minute disruption while accessing the LMS on Monday during Monday indicated for DC8, Pool 75 but no RCA or confirmation from support or Operations. 

We have had to schedule regular calls with Operations since we have had so many service disruption issues on our GMP system. 

[Obuone]

Thank you all for contributing to this thread.
To summarize :
* Several customers are experiencing Stability issues if short duration. Root cause is yet unknown
* Other issues have been observed such as Connectors running for longer or other operational issues
SAP is dealing with these mostly through tickets and not a quickly as customers would hope.
In our case, we finally got SAP MCC involved hoping this would accelerate the process. We also were told that our issues are due to the size of our database ... surprisingly as HANA should be able to handle extremely large volumes !

[Obuone]

Hello,
This is exactly what is happening to us:
Repeaded short outages. Usually no longer than 10 - 15 minutes, but on multiple occasions during the day.
And this is affecting both end-user side and admin side.
We're being told by SAP that it could have to do with some sub-optimized queries, but no further details shared.