SAP Developer Challenge - APIs - Task 12 - Use the access token to call the API endpoint

[qmacro]

(Check out the SAP Developer Challenge - APIs blog post for everything you need to know about the challenge to which this task relates!)

Well done for making it to the final task of this SAP Developer Challenge on APIs! You're going to finish on a high, by finally calling the API endpoint in the Core Services for SAP BTP API package. But not without a little diversion in the road on the way there 🙂 Let's get to it!

Background

So, at this point, you have completed steps 1, 2 and 3 in this group of tasks. And after a slight detour into JWTs in the previous task, you're now back on track, on the home straight, ready to complete step 4.

1. create an instance of the SAP Cloud Management Service, with a plan that contains the appropriate scope(s) that you need ✅
2. create a service key based on that instance ✅
3. use the details in the service key to request an access token ✅
4. use the access token thus obtained to authenticate a call to the API endpoint

To complete this task you're going to need to recall bits and pieces from prior tasks:

• the GUID of the directory that you created in Task 7
• the service key data that you obtained in Task 9
• the JSON object containing the access token and related metadata that you requested and received in Task 10

How are you going to use each of these pieces of information?

Well, you'll need the directory GUID to replace the {directoryGUID} placeholder in the actual API endpoint that you're going to be calling (the endpoint detail was also mentioned in Task 9). In other words:

GET /accounts/v1/directories/{directoryGUID}

You'll need information from the service key data to know what the base URL of the Accounts Service API to use, because this /accounts/v1/directories/{directoryGUID} API endpoint belongs to that Accounts Service API, remember? Recall that the service key data looks like this (heavily redacted in the .credentials.uaa section for brevity):

{
  "credentials": {
    "endpoints": {
      "accounts_service_url": "https://accounts-service.cfapps.eu10.hana.ondemand.com",
      "cloud_automation_url": "https://cp-formations.cfapps.eu10.hana.ondemand.com",
      "entitlements_service_url": "https://entitlements-service.cfapps.eu10.hana.ondemand.com",
      "events_service_url": "https://events-service.cfapps.eu10.hana.ondemand.com",
      "external_provider_registry_url": "https://external-provider-registry.cfapps.eu10.hana.ondemand.com",
      "metadata_service_url": "https://metadata-service.cfapps.eu10.hana.ondemand.com",
      "order_processing_url": "https://order-processing.cfapps.eu10.hana.ondemand.com",
      "provisioning_service_url": "https://provisioning-service.cfapps.eu10.hana.ondemand.com",
      "saas_registry_service_url": "https://saas-manager.cfapps.eu10.hana.ondemand.com"
    },
    "grant_type": "user_token",
    "sap.cloud.service": "com.sap.core.commercial.service.central",
    "uaa": {
      "apiurl": "https://api.authentication.eu10.hana.ondemand.com",
      "clientid": "...",
      "clientsecret": "...",
      "...": "..."
    }
  }
}

So you will need the value of the .credentials.endpoints.accounts_service_url property from your service key data.

Finally, you'll need of course the access token you obtained in Task 10, i.e. the value of the access_token property in the JSON object that looks like this:

{
  "access_token": "eyJhbGciOiJSUzI1NiIs...",
  "token_type": "bearer...",
  "id_token": "eyJhbGciOiJSUzI1NiIs...",
  "refresh_token": "e72b61a9a9304dde963e...",
  "expires_in": 43199,
  "scope": "cis-central!b14.glob...",
  "jti": "579fea14a1cf47d7ab9e..."
}

"But wait!" I hear some of you cry. "Task 10 was last week. That's, err, more than 43199 seconds ago, right? What's going to happen?"

Well let's find out!

Attempting a call with an expired access token

Let's assume for this experiment that:

• the GUID of your directory is 57675710-7b16-43ec-b64a-ab14660c1b24
• you have your service key data from Task 9 in a file called sk.json
• your access token data from Task 10 in a file called tokendata.json

Let's also assume that the access token data in tokendata.json was indeed obtained on Friday last week, when Task 10 was published.

Using curl (but we'd see the same effect using any HTTP client, of course), let's see what the actual call to the API endpoint would look like:

curl \
  --verbose \
  --header "Authorization: Bearer $(jq -r .access_token tokendata.json)" \
  --url "$(jq -r .credentials.endpoints.accounts_service_url sk.json)/accounts/v1/directories/57675710-7b16-43ec-b64a-ab14660c1b24"

Invoking this returns something interesting, but not entirely unexpected. Here's part of the verbose output from that curl invocation:

> GET /accounts/v1/directories/57675710-7b16-43ec-b64a-ab14660c1b24 HTTP/2
> Host: accounts-service.cfapps.eu10.hana.ondemand.com
> user-agent: curl/7.74.0
> accept: */*
> authorization: Bearer eyJhbGciOiJSUzI1Ni...
>
< HTTP/2 401
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< date: Sat, 26 Aug 2023 09:30:06 GMT
< expires: 0
< pragma: no-cache
< www-authenticate: Bearer error="invalid_token", error_description="An error occurred while attempting to decode the Jwt: Jwt expired at 2023-08-22T20:30:22Z", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-vcap-request-id: 7a7c0c79-f3f7-4b19-651f-6c9b8dd2b013
< x-xss-protection: 1; mode=block
< content-length: 0
< strict-transport-security: max-age=31536000; includeSubDomains; preload;

Ooh! Let's examine the content of that WWW-Authenticate HTTP response header:

error="invalid_token"
error_description="An error occurred while attempting to decode the Jwt: Jwt expired at 2023-08-22T20:30:22Z"
error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"

The error description pretty much gives it to us straight. Our JWT, i.e. the access token, has expired! Now, this example is from my context, where I'd obtained an access token earlier last week. This is why the expiry in this example is on 22 Aug. In fact, let's double check.

Checking when the access token expires

Some of you may have been wondering about the expires_in property in the access token JSON object. This is a simple number of seconds (in this case 43199, to be precise) and represents the lifetime of the token. But how does this relate to actual dates and times? It is of course the lifetime starting from whenever the token was generated, so you may end up calculating the actual expiry date and time by adding those number of seconds on to the exact date and time you obtained the access token. But that can be cumbersome.

You can probably guess that the JWT contains expiry information too. In other words, expiry information is also contained within the access token itself, along with the list of scopes, and other things, that you discovered in the previous task. In fact, not only is the exact expiry date and time in there, but also the date and time when the token was issued. The eagle-eyed amongst you may have spotted the iat and exp properties in the payload part of the JWT in the previous task:

{
  "header": {
    "alg": "...",
    "jku": "https://c2d7b642trial-ga.authentication.eu10.hana.ondemand.com/token_keys",
    "kid": "default-jwt-key-1281344942",
    "typ": "...",
    "jid": "iaVmTleRBCIVnVE7veQ9opMtlHnk+3DvKWWsjpsm542="
  },
  "payload": {
    "...": "...",
    "grant_type": "password",
    "user_id": "965a393a-dc96-422f-87ac-9f3d8bb25142",
    "origin": "sap.default",
    "iat": 1692693022,
    "exp": 1692736222,
    "...": "..."
  },
  "signature": "ZVe_aqyLAyXwToCvG...",
  "input": "eyJhbGciOiJSUzI1NiIsI..."
}

These properties are standard registered claim names, defined in the JSON Web Token (JWT) RFC7519. Specifically, they are:

• the "iat" (Issued At) Claim which conveys when the token was issued
• the "exp" (Expiration Time) Claim which conveys when the token expires

The values of these claims (1692693022 and 1692736222) are UNIX epoch values, i.e. the number of seconds since the UNIX epoch (01 Jan 1970), a standard way to measure time.

Let's examine these a little closer, using the power of the command line and a bit of jq, because why not. Getting the value of the actual access token from the JSON object in tokendata.json file, getting it parsed into its component JWT parts (using the jwt command line tool that we learned about in the previous task), and then taking the exp and iat values from the payload part of the JWT, subtracting one from the other:

jq \
  --raw-output \
  '.access_token' \
  tokendata.json \
  | jwt --output=json \
  | jq '.payload | .exp - .iat'

This emits, rather beautifully:

43200

The --raw-output (which can be shortened to -r) tells jq to emit the raw string, rather than try to always emit valid JSON. So if the value is the string 'hello', then the raw version is hello whereas a valid value as far as JSON is concerned is "hello". Yes, a double-quoted string, all on its own, is syntactically valid JSON. See https://www.json.org/json-en.html for more details.

What about the values themselves? Well, if you're running a standard UNIX style environment with the normal tools (such as in a Dev Space in the SAP Business Application Studio) you can use the standard date command to convert from an epoch value.

Starting almost the same as before, let's first emit the two epoch values:

jq \
  -r \
  '.access_token' \
  tokendata.json \
  | jwt --output=json \
  | jq '.payload | .iat, .exp'

This produces:

1692693022
1692736222

We can then feed those into the date command, using the --date (or -d) parameter to display the date and time denoted by the value that follows it, which will be the epoch time preceded with an @ sign to symbolize "this value is the number of seconds since the epoch":

jq \
  -r \
  '.access_token' \
  tokendata.json \
  | jwt --output=json \
  | jq -r '.payload | "@\(.iat)", "@\(.exp)"' \
  | while read -r epochvalue; do
      date -d "$epochvalue";
  done

This produces:

Tue Aug 22 08:30:22 UTC 2023
Tue Aug 22 20:30:22 UTC 2023

It was last Tuesday morning that I requested and received this access token. And we can see that it expired exactly 12 hours (43200 seconds) later, at Tue Aug 22 20:30:22 UTC 2023. And lo and behold, this is precisely the date and time given in the error description returned in the response where we got an HTTP 401 (UNAUTHORIZED) status code:

error_description="An error occurred while attempting to decode the Jwt: Jwt expired at 2023-08-22T20:30:22Z"

So what are we going to do?

Of course, the sledgehammer approach would be to request another access token. But that's bad practice, because the Resource Owner Password Grant grant type requires the resource owner's credentials, and for the consumer script to hold onto them for such purposes is (or should be) frowned upon, and for the script to re-request them from the resource owner each time is an anti-pattern.

Instead, we can request a new token via the Refresh Token grant type, which "allows clients to continue to have a valid access token without further interaction with the user". Isn't OAuth lovely?

Using the refresh token grant type

So we have everything we need already to request a fresh token. This is what must be supplied in such a call:

• the client's identity (the client ID and secret)
• the grant type, which must be refresh_token
• the actual refresh token itself

The refresh token itself was supplied along with the original access token, in the JSON object returned from the call in Task 10, which we saw briefly earlier in this section:

{
 "access_token": "eyJhbGciOiJSUzI1NiIs...",
 "token_type": "bearer...",
 "id_token": "eyJhbGciOiJSUzI1NiIs...",
 "refresh_token": "e72b61a9a9304dde963e...",
 "expires_in": 43199,
 "scope": "cis-central!b14.glob...",
 "jti": "579fea14a1cf47d7ab9e..."
}

And of course, we still have the client ID and client secret in the service key data (stored in the tokendata.json file).

Where do we send such a call? To the same Authorization Server endpoint as before, i.e. to the /oauth/token endpoint we've used already.

So a token refresh call looks very similar to the previous request when the grant type was "password". Here's the structure:

curl \
  --user '<clientid>:<clientsecret>' \
  --data 'grant_type=refresh_token' \
  --url 'authorizationserver/oauth/token'

Note that the resource owner credentials are conspicuous by their absence here. They are not needed, and should not be required!

Here's an actual call. Values are needed from two places - the service key data and the access token JSON object.

From the service key data, the <clientid>:<clientsecret> construction is achieved with a bit of jq inside a command substitution expansion, joining the .clientid and .clientsecret values (found within the .credentials.uaa value, which is an object) with a colon. And the retrieval of the actual authorization server details is also done in a similar way, taking the value of the .credentials.uaa.url property in the service key data.

And from the access token JSON object, the refresh_token value is taken.

curl \
  --user "$(jq -r '.credentials.uaa|[.clientid,.clientsecret]|join(":")' sk.json)" \
  --data 'grant_type=refresh_token' \
  --data-urlencode "refresh_token=$(jq -r .refresh_token tokendata.json)" \
  --url "$(jq -r .credentials.uaa.url sk.json)/oauth/token" \
  | tee tokendata.json

The venerable UNIX command tee is used here to save the output (the new access token and corresponding metadata in a JSON object) in a file (tokendata.json) as well as letting it pass through to STDOUT so we see it too. It's called tee because it's like using a tee pipe fitting in a plumbing context.

And what do you know? We get a freshly minted access token to use, with 12 more hours on the clock:

{
  "access_token": "eyJhbGciOiJSUzI1Niq2...",
  "token_type": "bearer...",
  "id_token": "eyJhbGciOiJSUzI1NiJS...",
  "refresh_token": "e72b61a9a9304dae263e...",
  "expires_in": 43199,
  "scope": "cis-central!b14.glob...",
  "jti": "579fea14a1cf47d7ab9e..."
}

Nice!

It's useful to know as well that you can refresh your token before the expiry. This gives you a chance to build in a robust token management system into your client, and avoid the risk of falling between the gaps between validity periods.

Using a valid access token

Now that we have a fresh, valid access token, we can complete the journey and make the call to the API endpoint. There's nothing special here, so let's get right to trying it out. In fact, the call is going to be exactly the same as before; the only thing that is different now is that the access token is still valid:

curl \
  --silent \
  --header "Authorization: Bearer $(jq -r .access_token tokendata.json)" \
  --url "$(jq -r .credentials.endpoints.accounts_service_url sk.json)/accounts/v1/directories/57675710-7b16-43ec-b64a-ab14660c1b24"

The --silent parameter here is used to suppress the "progress bar" that curl shows while retrieving a resource.

And the call is successful, emitting ...

Well.

That would be giving the task away, wouldn't it!

Your task

Your task is to ensure you have a valid access token. Ideally, you should work through the process above, using your old (and expired) access token to make a first call to the API endpoint, using the GUID of your directory that you created.

You should see the HTTP 401 status code and look at the value of the WWW-Authenticate header in the HTTP response. You should embrace all that this entails and enjoy matching up the expiration date and time stated in that header, and try to match it up with the value of the exp claim in the payload of the JWT that is your old access token.

Then you should run through the process of requesting a new token, using the Refresh Token grant type explained above.

And with this fresh access token, you should make the call again to the API endpoint, to get the details of the directory that you created way back in Task 7.

Once you have this directory detail, which will be in the form of a JSON object, you should take the value of two of the properties in that detail, join them together with a colon, and send them to the hash service. Then, as always, and as described in Task 0, you should reply to this discussion thread with the hash that's returned.

You need to take the values from these two properties:

• displayName
• directoryType

And don't forget to concatenate them with a colon.

That's it ... you've done it!

1. create an instance of the SAP Cloud Management Service, with a plan that contains the appropriate scope(s) that you need ✅
2. create a service key based on that instance ✅
3. use the details in the service key to request an access token ✅
4. use the access token thus obtained to authenticate a call to the API endpoint ✅

Great work.

Hints and tips

Most of what you'll need has already been covered this time in the narrative within the Background section. And you've all worked so hard with these tasks over the month that you're now well prepared for working with APIs in the SAP universe, dealing with OAuth, endpoints, service key information and more.

Well done!

For discussion

In the directory details that were returned from your successful call to the API endpoint, did you also spot the label information that you added during the directory's creation in Task 7? Where was it? What else did you find interesting about the data returned?

[johna69]

I have in interesting issue with :

curl \  --user "$(jq -r '.credentials.uaa|[.clientid,.clientsecret]|join(":")' service_key)" \  --data 'grant_type=refresh_token' \                                              
  --data-urlencode "refresh_token=$(jq -r .refresh_token tokendata.json)" \
  --url "$(jq -r .credentials.uaa.url service_key)/oauth/token" \
  | tee tokendata.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   125    0    86  100    39    234    106 --:--:-- --:--:-- --:--:--   342
{"error":"invalid_request","error_description":"refresh_token parameter not provided"}

Changing tee to write to a different file, e.g. tokendata2.json, I get the refreshed token, Something strange afoot here for my next coffee break.

[johna69]

Something to be aware of, if you add "Allow Entitlements" to the directory, the directoryType will also change.

[johna69]

Thanks for the fun challenge, still need to go back and finish 4,5 and 6. Enjoyed every minute of it.

[qmacro]

Thanks @johna69 !

[Ashok459]

12f500dd6109d5cf61bed6a7a85a06e6472c877c626de0061a0960af832073de

[martinstenzig]

9b7687d38d069d9a461418e0dbb65dacd5449add0442896df85f2900e0043729

[martinstenzig]

Is there an advantage of refreshing a token in a shell script vs. always creating a fresh one?

I usually go the latter route. 

[qmacro]

Is your question focusing on usage of a shell script, or on refresh vs obtaining fresh access token?

I'm going to guess the latter, and say that refreshing, when needed, and when possible (not all grant types have a refresh aspect), because it's good practice. See the discussion earlier in this thread here for details: https://groups.community.sap.com/t5/application-development-discussions/sap-developer-challenge-apis...

Or have I misunderstood your question?

[martinstenzig]

414fa51ac4364d580a1551e9154ae9dbab7fb97dc295cb74fc053aadd46c5746

[nex]

e51a3244c70b099f676d20fa5cd075c1a7781b9e27a82e456924add67e391d18

[sabarna17]

b73408b06b28289ae2bb404c79bcb05036e76229fd650c74fd837bec650a1802

[sabarna17]

this-is-the-year-of-the-api...
Surely it is. @qmacro - You have created a awesome, interactive, learning experience with lots of easter eggs...
Kudos to SAP Community. The Community is brilliant with lots of ideations and explorations. Here is a thread from X to all of you. 

Feel free to have a look and comment in the solutions using Node-RED for these Tasks in this Git-Repo.

[qmacro]

That's great, thank you very much for putting this together @sabarna17 and sharing it with the community 👍

[szeteng00]

2085008e1cb0cc27d3d9226a9ab560f5f13cffb39104033ec03ce481dc4ca48a

[PriyankaChak]

3245b53695a197069d74c8e540cb37cdf4a51bedfa20b21b3a4a1437b2ebd970

[PriyankaChak]

Thank you for this wonderful series of challenges. Learned a lot 🙂

[qmacro]

Great - and you're welcome @PriyankaChak 

[former_member136915]

74ecc36ae6e13bd30f7c63f7c833d5e492992350743e2069aedaaecf61f57bf0

[choujiacheng]

e944b62905c359b515fad786dfd642ceceb79ca5d5ab8f95740b52e98109c043

[choujiacheng]

The label is found under the labels.task in the response JSON. I also noticed the parent GUID and the global account GUID are the same, perhaps since there is not much in my trial account to begin with, as well as having a state message indicating that the directory created.

[qmacro]

Thanks for sharing. Yep, that sounds about right!

[tobiasz_h]

b454f2b94532934c13672abf1feeb827b8a1e0cefd754d0fe14c2f008496298b

[RaulVega]

b39dc2a9723cc2122b679c80193b228a7bda66a19b1eff14a3b04ce2fbdd4c04

[cdias]

bc7dc3771edb65f50cad8e5746b05056af4c1eaf15a2aac4c8ca220e48909dac

[berserk]

9b730436e1cf0da3e6861fd7463f4000a56a2b51a7e8155829595bd18cdec805

[flo_conrad]

36c3453b8cefbabf98cf81982980ccb6b7e1c86c4a0b61916a3530994c45724a

[qmacro]

Hey everyone! The challenge to which this task belongs is now officially closed. Head over to the original blog post SAP Developer Challenge – APIs to check out the closing info and final statistics, and to see your name in lights! 🎉  And we thank you all for participating, you made this challenge great!

[satya-dev]

557d9573235c908f4b6213dde3752fdbd6adc06752f7caca084360b857cb3d96

[andrew_chiam]

05f81fa7eaf15abcb2e6915a33e8ffd4cd754249e1f597df356dab5e11b2a58c

[buz]

8cff12709c856de6d2c7f28580fe61b4aaeba10808b3dab9b4a6dc4ec488a0ef