SAP HTTP/2 call results in 404 error from Kyma Gateway

[SeanKilleen]

Hi all,

Background:

• We are running an API application within BTP Kyma.
• We expose the application through a Kyma API rule.
• I've confirmed that this application, when hit directly (in its docker container locally, or when deployed in Kyma and exposed via the Kyma API rule) is able to receive and respond to HTTP/2, HTTP/1.1, and HTTP/1.0 requests.
• We've worked out (as far as I can tell) the certificate trust between our client's SAP environment and our Kyma instance. HTTP/1.x requests work as expected.

Problem:

• When making an HTTP request from within SAP to the Kyma-deployed API endpoint, SAP appears to try to "help". We make an HTTP/1.1 request, but the environment (Kyma gateway/istio-envoy) supports HTTP2, and therefore (based on what I understand from the trace logs) SAP attempts to make the request via HTTP/2 to the Kyma gateway.
• For some reason, this HTTP/2 request receives a 404 response. I've confirmed the request appears to be for the exactly correct URL, correct body contents, correct content type, etc.
• If we disable HTTP/2 within the SAP environment, the calls succeed. It's as though the HTTP/2 "wrapper" SAP is using is somehow the cause of the problem itself, but I'm not sure how to prove that out.
• When these 404 errors occur, I don't see them in our application logs. For any requests I make, even incorrect ones, I see them in our application logs. So it seems like something is rejecting these requests before they make it to our app somehow.

My concern is: We don't control our client's environments and so cannot ask them to disable HTTP/2. I am unable to find any known issues about HTTP/2 causing 404 errors.

I'm at a bit of a loss. Thank you for any help in diagnosing!

[gabbi]

Hi Sean,

AFAIK, HTTP/2 protocol should not cause any issues. I see it being used in browser developer tools.

Does the call reach your app with HTTP/2 protocol?

I suspect not. I would recommend try enabling istio proxy sidecar logs and observe the istio-proxy container logs

istioctl -n <NS> proxy-config log <pod> --level "http:debug"

You will need istioctl for that.

Ref: https://istio.io/latest/docs/reference/commands/istioctl/

BR

Gaurav

[bchwila]

Hello Sean,

Please have a look at Kyma/Istio troubleshooting guide on getting 404 errors with Istio.
There might be a conflict in terms of the Istio Gateways you are using, so if you could share how exactly your setup looks in that terms (for example from the output of istioctl x internal-debug configz command) it would be very helpful for us to recommend a solution.
As Guarav mentioned before, for running the command you need to have the istioctl CLI tool you can download from Istio release page.
Best regards,

Bartosz

[SeanKilleen]

@Gaurav -- thanks, I'll try to enable that and see what I see. Just to be clear, I'm not using istio directly -- I'm using a Kyma API Rule CRD which it appears does some additional things behind the scenes, like creating a an istio VirtualService. I am unaware of any istio sidecars that are running, but will look for them.

@Bartosz -- I did check the troubleshooting guide. But what is unclear to me is: all the Kyma API rules that I've defined appear to utilize the same istio gateway. There is one "kyma-gateway" out of the box, and my understanding is that I'm using Kyma API rules and thus not touching istio directly at all. Since the API rules use the same Kyma gateway, the "multiple gateways" 404 issue shouldn't apply I don't think.

Thanks to you both for the tip on the CLI tools -- I'll check those and report back.

[SeanKilleen]

Hi Gaurav / Bartosz,

The istio debug configuration can be found in the attachments per request.

I also updated the http and http2 log levels to debug in the istio-ingressgateway-* pods set up by Kyma. (Those pods are pulling from europe-docker.pkg.dev/kyma-project/prod/external/istio/proxyv2:1.19.3-distroless, for what it's worth; that's all done by Kyma).

I'm in the process of narrowing the logs to look into the differences between failure and success scenarios. I hope this will hold the key.

I will also revisit the article around 404s with multiple gateways, though as far as I can tell kyma-gateway still seems to be the only gateway and these other items are proxies (I may just be misunderstanding though; happy to be corrected!)

____

Some more information in case it helps, (and to help me keep it straight)

| Status |          Source          | Protocol |          Destination          |
| ------ | ------------------------ | -------- | ----------------------------- |
| ✅      | Local Machine (Postman)  | HTTP/1.1 | app in local docker container |
| ✅      | Local Machine (Postman)  | HTTP/1.1 | app in Kyma                   |
| ✅      | Local Machine (Insomnia) | HTTP/1.0 | app in local docker container |
| ✅      | Local Machine (Insomnia) | HTTP/1.1 | app in local docker container |
| ✅      | Local Machine (Insomnia) | HTTP/2   | app in local docker container |
| ✅      | Local Machine (Insomnia) | HTTP/1.0 | app in Kyma                   |
| ✅      | Local Machine (Insomnia) | HTTP/1.1 | app in Kyma                   |
| ✅      | Local Machine (Insomnia) | HTTP/2   | app in Kyma                   |
| ✅      | SAP                      | HTTP/1.1 | app in Kyma                   |
| ❌      | SAP                      | HTTP/2   | app in Kyma                   |
| ✅      | Local Machine (Apidog)   | HTTP/1.1 | app in Kyma                   |
| ❌      | Local Machine (Apidog)   | HTTP/2   | app in Kyma                   |
| ✅      | Local Machine (Apidog)   | HTTP/2   | app in local docker container |

[bchwila]

Hey Sean,

On Gateway and VirtualService side, your configuration looks OK, so I would suspect it might be the tooling from SAP side that has some issue with handling the response, since it is not happening for tools like Postman.
Do you see the requests passing to the workload when doing a request with either SAP or Apidog with HTTP/2 (for example logging happens)? If so, the infrastructure configuration should be correct.

By the way, one of your APIRules is in ERROR state, make sure to update the path to use regex.
BR,

Bartosz

[SeanKilleen]

@Bartosz -- yes, sorry, that was a temporary issue during troubleshooting and I've since fixed it.

----

OK, after looking at some successful and failed requests, I think I've narrowed it down.

Successful calls are showing up in istio/envoy with an authority header of, for example, redacted.redacted.kyma.ondemand.com.

For some reason, when the calls that ultimately fail show up (whether from Apidog locally or from SAP), they have an authority header in the istio logs of:

redacted.redacted.kyma.ondemand.com,redacted.redacted.kyma.ondemand.com

It's duplicated and separated by a comma.

So my theory is that perhaps the istio (created/managed by Kyma) is matching based on the authority and that the match is an exact match, and thus it doesn't find any available routes, and thus 404s.

So the thing I have to figure out is:

• Is that duplicated authority header actually valid to send?

• Why is it being sent that way? Is it being sent that way by those clients, or is something within our Kyma networking processing it incorrectly for some reason and thus creating that duplicate value?

• Do I have any control over Istio via my Kyma settings to turn that match from an exact match into a prefix match, to alleviate the symptom?

----

Lastly, worth noting: I installed Kiali in the Kyma cluster via a Helm chart so I could visualize the istio bits better, and I'm not seeing anything strange there either.

[SeanKilleen]

I have posted more detailed issues on both the Kyma and Istio repositories because I'm unsure in which realm the issue lies. But at this point, all that seems sure is that there are multiple `authority` header values and that seems to cause an issue with alpn being able to find routes.

• Istio GitHub issue: https://github.com/istio/istio/issues/48088
• Kyma GitHub issue: https://github.com/kyma-project/kyma/issues/18447

[SeanKilleen · Accepted solution]

I now have an answer that includes a workaround.

Cause #1: A Bug Within Envoy

I confirmed that this appears to be an issue in the Envoy proxy used by Istio, which is then used by BTP Kyma:

https://github.com/envoyproxy/envoy/issues/31118#issuecomment-1834164342

Any client that sends both the ":authority" pseudo-header and the "Host" header will run into this problem with Envoy it seems, as Envoy will incorrectly concatenate into the ":authority" field which winds up with an authority that Istio then doesn't find any routes for, because it's looking for a single instance of the name.

Cause #2: NetWeaver Sending Both ":authority" and "Host" headers

NetWeaver sends both ":authority" and "Host" headers even when those values are the same. While allowed by the HTTP/2 spec, this is still slightly unusual behavior for a client (curl, Insomnia, and others appear to interpret the HTTP/2 spec the same way that Envoy did initially).

The only workaround until Envoy releases a fix, it seems, is to get clients (such as NetWeaver) to only send one of the ":authority" or "Host" headers as part of the request. This will result in a request that does not encounter the Envoy bug with Kyma, and thus the request can complete successfully.

(Slightly Hacky) Workaround: Remove hostname from request_url

In our SAP code that utilizes the NetWeaver client to make the HTTP request, we we modify the request_url field to remove the hostname prior to sending the request to Kyma. It's missing the Host header when it goes out, but it has the :authority header already set. Because of that, the header isn't duplicated, and therefore it doesn't hit the Envoy bug.