NetWeaver HTTP/2 request duplicating ":authority" pseudo-header?

[SeanKilleen]

Hi all,

I'm trying to troubleshoot an issue where I'm unsure if it originates on the Kyma side or the calling client side (NetWeaver in this case); this is the companion question to an SAP forum question in the Kyma area and the question I raised on the Kyma repo (which has more technical information including logs and examples.)

Background

• I have an API running within SAP BTP Kyma, using a Kyma API Rule to expose it.
• I can make calls to this fine from many applications. HTTP/1.1 calls via Postman, HTTP/2 calls via Insomnia or Curl.
• For SAP calls to this API, NetWeaver is upgrading our HTTP/1.x calls to HTTP/2, which seems like it should be fine.

The Problem

• Calls from SAP NetWeaver are receiving a 404 back from Kyma when HTTP/2 is enabled. When we disable HTTP/2 (an option for our environment but not our customers), the problem is resolved.
• When I look in the Kyma logs, I see something strange -- all the failed requests have an ":authority" pseudo-header where the hostname is duplicated. If our hostname is "xyz.com", the ":authority" header contains "xyz.com,xyz.com"
• This causes Kyma's Istio instance to return a 404 during ALPN filtering, I think because it's looking for an exact match on the authority.

The Question

I'm unsure how to proceed. This seems to be an issue only with NetWeaver calls (though I was able to reproduce it using another Windows client app, Apidog, but that app's HTTP/2 support is only experimental). So the questions I have are:

• Is there any way to confirm whether NetWeaver is sending this heading in this way? Or better, any way to stop it?
• Is there any way to ask NetWeaver not to upgrade a given request to HTTP/2 short of disabling HTTP/2 entirely? (I'm assuming disabling HTTP/2 won't be an option our customers' systems)

[SeanKilleen · Accepted solution]

I now have an answer that includes a workaround.

Cause #1: A Bug Within Envoy

I confirmed that this appears to be an issue in the Envoy proxy used by Istio, which is then used by BTP Kyma:

https://github.com/envoyproxy/envoy/issues/31118#issuecomment-1834164342

Any client that sends both the ":authority" pseudo-header and the "Host" header will run into this problem with Envoy it seems, as Envoy will incorrectly concatenate into the ":authority" field which winds up with an authority that Istio then doesn't find any routes for, because it's looking for a single instance of the name.

Cause #2: NetWeaver Sending Both ":authority" and "Host" headers

NetWeaver sends both ":authority" and "Host" headers even when those values are the same. While allowed by the HTTP/2 spec, this is still slightly unusual behavior for a client (curl, Insomnia, and others appear to interpret the HTTP/2 spec the same way that Envoy did initially).

The only workaround until Envoy releases a fix, it seems, is to get clients (such as NetWeaver) to only send one of the ":authority" or "Host" headers as part of the request. This will result in a request that does not encounter the Envoy bug with Kyma, and thus the request can complete successfully.

(Slightly Hacky) Workaround: Remove hostname from request_url

In our SAP code that utilizes the NetWeaver client to make the HTTP request, we we modify the request_url field to remove the hostname prior to sending the request to Kyma. It's missing the Host header when it goes out, but it has the :authority header already set. Because of that, the header isn't duplicated, and therefore it doesn't hit the Envoy bug.

[SeanKilleen · Accepted solution]

FWIW, this issue has now been resolved in the latest Kyma production release, which uses a newer Istio release that includes the Envoy proxy fixes.