Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
cancel
Showing results for 
Search instead for 
Did you mean: 
TiaXu
Product and Topic Expert
Product and Topic Expert




Introduction


The domain model in CAP (Cloud Application Programming) is crucial for defining domain entities using CDS (Core Data Services), allowing seamless integration with external services or databases. CAP, along with its associated tools, automates the translation of CDS models into database-supported schemas. CAP provides native support for various databases such as SAP HANA (Cloud), PostgreSQL, SQLite, and H2. To learn more about CAP's database support, please refer to CAP - Database Support.


This post focuses on utilizing the PostgreSQL database in the CAP framework.





  • For Java, CAP Java SDK is tested on PostgreSQL 15 and supports most of the CAP features.




  • For Node.js, before we use the CAP-community-provided adapter cds-pg in combination with cds-dbm to consume PostgreSQL. Since cds 7, CAP Node.js has natively supported PostgreSQL by releasing new database services and its implementation @cap-js/postgres. @cap-js/postgres provides the functionalities to translate the incoming requests from CDS model to PostgreSQL during runtime, and analyze the delta between the current state of the database and the current state of the CDS model, deploy the changes to the database, load CSV files, etc.




 

The structure of the following content:

  • Run and Deploy SAP CAP (Node.js) with PostgreSQL using @cap-js/postgres

    • Feature - Schema Evolution

    • Prepare your CAP Project

    • Connect to a PostgreSQL Locally

    • Build your Application

    • Deploy to Cloud Foundry and Consume SAP BTP PostgreSQL, Hyperscaler Option

    • Connect to PostgreSQL Instance Directly in Local



  • Run and Deploy CAP (Java) with PostgreSQL


 

Run and Deploy SAP CAP (Node.js) with PostgreSQL using @cap-js/postgres


With cds 7, new database services for SQLite and PostgreSQL are released (new SAP HANA Service will follow soon), which are based on an entirely new database service architecture. The new services are implemented in new open-source packages as follows:
























Database Implemented In Learn More
SQLite @cap-js/sqlite New SQLite Service
PostgreSQL @cap-js/postgres New PostgreSQL Service



Note: We strongly encourage you to start migrating to and using the new database services as soon as possible. We were able to keep breaking changes to a minimum, mostly affecting undocumented behaviour. We tested them thoroughly, also with customers' test suites. Nevertheless, they're in their very first release, of course... carefully read the migration guides for that reason.



Feature - Schema Evolution


When redeploying after you changed your CDS models, like adding fields, automatic schema evolution is applied. Whenever you run cds deploy (or cds-deploy) it executes these steps:





  1. Read a CSN of a former deployment from table cds_model.




  2. Calculate the delta to current model.




  3. Generate and run SQL DDL statements with:





    • CREATE TABLE statements for new entities




    • CREATE VIEW statements for new views




    • ALTER TABLE statements for entities with new or changed elements




    • DROP & CREATE VIEW statements for views affected by changed entities






  4. Fill in initial data from provided .csv files using UPSERT commands.




  5. Store a CSN representation of the current model in cds_model.





You can switch of automatic schema evolution, if necessary, by setting cds.requires.db.schema_evolution = false.



Prepare your CAP Project


Create a new project using cds init



cds init sample-cap-postgresql-nodejs-cap-js-postgres

Open the project in VS Code



code sample-cap-postgresql-nodejs-cap-js-postgres


Note: VS Code CLI on macOS needs extra setup, please read https://code.visualstudio.com/docs/setup/mac.



Add your domain model and services. For more details, please read: https://cap.cloud.sap/docs/.


Execute cds watch to run it locally, which automatically bootstraps an SQLite in-process and in-memory database by default.



cds watch

Access http://localhost:4004/ in your browser:



Click Books to check the sample data:




Connect to a PostgreSQL Locally


Add and setup local PostgreSQL database. Simply create a docker-compose.yml file in the root folder of the project and insert the following data:



version: '3.1'

services:
 db:
   image: postgres:alpine
   restart: always
   environment:
     POSTGRES_PASSWORD: 'postgres'
   ports:
     - '5432:5432'
 adminer:
   image: adminer
   restart: always
   ports:
     - 8080:8080

Run the container:



docker compose up


To create the database, just open the browser and access the adminer interface at http://localhost:8080. Login with the following credentials (these will also be required later):





  • Server: db (this is the name of PostgreSQL service in the docker-compose.yml file)




  • User: postgres




  • Password: postgres




In the adminer interface, create a new database and give it a name. And now we are ready to go.




With cds 7, new database services for SQLite and PostgreSQL are released (new SAP HANA Service will follow soon), which are based on an entirely new database service architecture. The new services are implemented in new open source packages as follows:
























Database Implemented In Learn More
SQLite @cap-js/sqlite New SQLite Service
PostgreSQL @cap-js/postgres New PostgreSQL Service



Note: We strongly encourage you to start migrating to and using the new database services as soon as possible. We were able to keep breaking changes to a minimum, mostly affecting undocumented behaviour. We tested them thoroughly, also with customers' test suites. Nevertheless, they're in their very first release, of course... carefully read the migration guides for that reason.



Upgrade the version of @Sap/cds to 7 in the package.json:



  "dependencies": {
  "@sap/cds": "^7",
  "express": "^4"
},

Install and add dependency @cap-js/postgres:



npm add @cap-js/postgres


You can see the new dependency is added with latest version:



"dependencies": {
"@cap-js/postgres": "^1.0.1",
"@sap/cds": "^7",
"express": "^4"
},


Add the local database information in the package.json:



    "cds": {
      "requires": {
          "db": {
              "kind": "postgres",
              "impl": "@cap-js/postgres",
              "credentials": {
                  "host": "localhost",
                  "port": 5432,
                  "database": "bookshop-tia",
                  "user": "postgres",
                  "password": "postgres"
              }
          }
      }
  },


The credentials can be configured in ~/.cdsrc.json or .env file.


For example, in .env:



cds.requires.db.credentials.host = localhost
cds.requires.db.credentials.port = 5432
cds.requires.db.credentials.user = postgres
cds.requires.db.credentials.password = postgres
cds.requires.db.credentials.database = bookshop-tia

For example, in ~/.cdsrc.json:



{
  "requires":{
    "db":{
        "credentials":{
          "host":"localhost",
          "port":5432,
          "user":"postgres",
          "password":"postgres",
          "database":"bookshop-tia"
        }
    }
  }
}


Check the cds environment:



cds env requires.db




Deploy your cds model to your PostgreSQL database:



cds deploy

Check the tables and views created in your database:



Also, you can check your sample data in the table and view:



Execute cds watch to check the result again:



cds watch


You can see your application is started with the connection to your PostgreSQL:




Try to post new data with the odata API http://localhost:4004/odata/v4/catalog/Books, and check with it in your PostgreSQL database. Before that, remove the @readonly annotation in the cat-service.cds file.


For example:





Build your Application


In order to connect in a secure way to Cloud, please add the following dependencies:



npm add passport
npm add @sap/xssec

Build your application:



npx cds build


If it returns error:



Add one more dependency and install it:



"dependencies": {
.....
"@sap/cds-dk": "^7"
},


After that, you can see a folder gen is generated



To deploy the cds model to PostgreSQL when deploying to Cloud Foundry, we need to prepare a simple app as a DB deployer. You can construct the app as follows:





  1. Create a new folder named gen/pg:



    mkdir -p gen/pg/srv



  2. Generate a precompiled cds model:



    cds compile '*' > gen/pg/srv/csn.json



  3. Add required .csv files, for example:



    cp -r db/data gen/pg/srv



  4. Add a package.json to gen/pg with this content:



    {
    "engines": {
      "node": "^18"
    },
    "dependencies": {
      "@sap/cds": "*",
      "@cap-js/postgres": "^1.0.1"
    },
    "scripts": {
      "start": "cds-deploy"
    }
    }


    Note: the dash in cds-deploy, which is required as we don't use @cds-dk for deployment and runtime, so the cds CLI executable isn't available.





You can also include all command in one shell and execute it automatically in build time.



Deploy to Cloud Foundry and Consume SAP BTP PostgreSQL, Hyperscaler Option


You can deploy your application either via MTA-based deployment or Cloud Foundry manifest.


Execute the following command to add mta file:



cds add mta

Add the PostgreSQL, Hyperscaler Option instance:



resources:
- name: devtoberfest-db
  type: org.cloudfoundry.existing-service


For more details on how to create a PostgreSQL, Hyperscaler Option instance:




Of course, you can create a new database instance at the same as deploying your application to Cloud Foundry. You should modify the mta.yaml file accordingly.



Add the service binding for the database instance to the srv application:



modules:
- name: sample-cap-postgresql-nodejs-cap-js-postgres-srv
  type: nodejs
  path: gen/srv
  parameters:
    buildpack: nodejs_buildpack
  build-parameters:
    builder: npm-ci
  provides:
    - name: srv-api # required by consumers of CAP services (e.g. approuter)
      properties:
        srv-url: ${default-url}
  requires:
    - name: <postgresql-instance-name>

Add the module definition for the DB deployer application:



  - name: sample-cap-postgresql-nodejs-cap-js-postgres-db-deployer
  type: nodejs
  path: gen/pg
  parameters:
    no-route: true
    no-start: true
    disk-quota: 1GB
    memory: 256MB
    tasks:
    - name: deploy-to-postgresql
      command: npm start
      disk-quota: 1GB
      memory: 256MB
  build-parameters:
    ignore: ["node_modules/"]
  requires:
    - name: <postgresql-instance-name>

Change the build command according to your setup, for example:



build-parameters:
before-all:
  - builder: custom
    commands:
      - npx cds build
      - ./pg-build.sh

Build your application:



mbt build

Deploy your application to Cloud Foundry:



cf deploy mta_archives/sample-cap-postgresql-nodejs-cap-js-postgres_1.0.0.mtar

Connect to PostgreSQL Instance Directly in Local


In terminal 1, enable SSH for your app:



cf enable-ssh <app-name>
cf restart <app-name>

To establish SSH access to your service instance, you must create a service key that contains information for configuring your SSH tunnel:



cf create-service-key <postgresql-instance-name> <access-key-name>
cf service-key <postgresql-instance-name> <access-key-name>


Get the dbname, hostname, username, password and port from the service key. Or, you can get the same information from the environment variables of your application by executing command:



cf env <app-name>


Configure an SSH tunnel to your service instance using cf ssh:



cf ssh -L 63306:<postgresql-instance-hostname>:<postgresql-instance-port> <app-name>

Install CLI for postgreSQL according to your preference, for example:



brew install PostgreSQL

In terminal 2, access to your db service instance using psql client to make sure you are able to establish direct command-line access to your service instance:



psql -d <postgresql-instance-dbname> -U <postgresql-instance-username> -p 63306 -h localhost


Enter the password.



Then, you can execute SQL sentence in the terminal.


Export data: Export Data from PostgreSQL Service Instance.



Run and Deploy CAP (Java) with PostgreSQL


For Java, CAP Java SDK is tested on PostgreSQL 15 and supports most of the CAP features. For more details, please read CAP (Java) with PostgreSQL and limitations.

17 Comments
mike_zaschka
Active Participant

Hi tiaxu,

great post with step-by-step instructions on how to use cap-js/postgres in a local environment as well as in combination with Cloud Foundry.
Since this is some kind of spiritual successor, I also linked your post directly from my older blog posts covering the now deprecated cds-pg and cds-dbm libraries to use CAP with PostgreSQL.

Kind Regards,

Mike

PS: I also found your secret reference to my older posts by using devtoberfest-db as the PostgreSQL instance name.  😀

TiaXu
Product and Topic Expert
Product and Topic Expert
Hi Mike,

Nice to have your comment! Thanks for your great contribution to the topic : )

BR,

Tia
jravnik
Participant
In case somebody is following this tutorial and is - like me - stuck at deploying the cds model to PostgreSQL:

Executing the command
cds compile '*' > gen/pg/srv/csn.json

failed with the following message: Couldn't find a CDS model for ''*'' in C:\dev\cap\my-bookshop


 

However, executing the command without the single quotes, i.e.
cds compile * > gen/pg/srv/csn.json

worked for me.
kondiiq
Explorer
0 Kudos
Hi

I've got problem. I binded service postgresql locally and when try to use cds deploy

/> deployment to postgres_db failed

TimeoutError: ResourceRequest timed out
at ResourceRequest._fireTimeout (/home/user/projects/WieGehts/node_modules/generic-pool/lib/ResourceRequest.js:62:17)
at Timeout.bound (/home/user/projects/WieGehts/node_modules/generic-pool/lib/ResourceRequest.js:8:15)
at listOnTimeout (node:internal/timers:559:17)
at processTimers (node:internal/timers:502:7)
Active connections:0

May someone have similary problem ?

Root package json

"@cap-js/db-service": "^1.1.0",

    "@cap-js/postgres": "^1.1.0",

    "@sap/cds": "^7",

    "@sap/cds-dk": "^7",

    "@sap/cds-odata-v2-adapter-proxy": "^1.9.21",

    "dotenv": "^16.3.1",

    "express": "^4"

ritushree
Explorer
0 Kudos
Hi ,

 

I have added pg-build.sh file in the project path and added the lines - 

#!/bin/bash
mkdir -p gen/pg/db
# Works only the first time until https://github.com/cap-js/cds-dbs/issues/100 is fixed
# cp -r db/data gen/pg/db
cds compile '*' > gen/pg/db/csn.json
cp pg-package.json gen/pg/package.json
cp package-lock.json gen/pg/package-lock.json

and executing the the mbt build , I get the permission denied to execute pg-build.sh. 

 

Thanks in advance

Ritushree

 
avik_saha0104_28
Explorer
0 Kudos

Hello tiaxu Hello mike_za 

 

Permission denied error while executing the pg-build.sh file in BAS.

 

And from local system getting below error.

 

[2023-08-19 16:11:26] INFO executing the "./pg-build.sh" command...
[2023-08-19 16:11:26] ERROR the "before-all"" build failed: could not execute the "./pg-build.sh" command: fork/exec ./pg-build.sh: %1 is not a valid Win32 application.
make: *** [Makefile_20230819161122.mta:28: pre_build] Error 1
Error: could not build the MTA project: could not execute the "make -f Makefile_20230819161122.mta p=cf mtar= strict=true mode=" command: exit status 2

 

Thanks in advance

Avik

avik_saha0104_28
Explorer
0 Kudos

Hello tiaxu   Hello mike_za 

 

If I deploy the application without pg-build.sh file.

In the BTP when I open the srv link I get the below internal server error.

 

<code>500</code>
<message>Internal Server Error</message>
</error>
And in DB deployer the below errors
Errors by cds.compile
in db/csn.json:1:1 — Error: Invalid JSON: Unexpected token � in JSON at position 0
at throwWithError (/home/vcap/deps/0/node_modules/@sap/cds-compiler/lib/base/messages.js:496:13)
at compileDoX (/home/vcap/deps/0/node_modules/@sap/cds-compiler/lib/compiler/index.js:483:3)
at /home/vcap/deps/0/node_modules/@sap/cds-compiler/lib/compiler/index.js:163:12
at async Object.to (/home/vcap/deps/0/node_modules/@sap/cds/lib/dbs/cds-deploy.js:24:48)
at async /home/vcap/deps/0/node_modules/@sap/cds/lib/dbs/cds-deploy.js:384:10
Regards
Avik
TiaXu
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Avik,

 

Thanks for reaching out.

For the permission denied problem, please try to check the permission of the file by executing a command like: chmod +x

For the local problem, as per the error message that you are using windows OS to execute your shell script (mine is Mac), please try to change the command in Windows way, like bash pg-build.sh

 

BR,

Tia
TiaXu
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Ritushree,

Thanks for reaching out.

For the permission denied problem, please try to check the permission of the file by executing a command like: chmod +x

BR,

Tia
TiaXu
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi Avik,

Please try to put the DB artifacts in the folder of gen/pg/srv.

For more details, please read the Build your Application part in this blog carefully.

BR,

Tia

 
DSree
Explorer
0 Kudos
Hello tiaxu

 

Thank you for the detailed steps to take advantage of postgres plugin for CDS. I have been successful in testing this locally. However, I do have 3 questions:

  1. The documentation suggests that auto-wiring (cds-plugin) will automatically get the default deploy and runtime parameters of localhost:5432 postgres/postgres/postgres. And this can be overridden by multiple options. If overriding, Is it required to override all parameters or is it possible to override database only. So localhost:5432 , uid/pwd : postgres/postgres works. Only the DB name needs to be changed. Does that work ?

  2. It appears that the database to which this connects cannot have any other table definitions. If they exist the deploy fails. Is the expectation that each application has its own "database" under the Postgres Service ? Would the evolution work if multiple applications are deployed to same database? This was the behavior I noticed. I used a database that had other tables and deploy failed with "cannot delete". When a new database was created without any content and added to default-env.json, the deploy was successful.

  3. When deploying to BTP cloud Postgres hyperscaler option? How is the database name determined? Does the deploy automatically take care of it? How does this work for multiple application scenario from question 2. Would each application need to be deployed in a separate database for evolution to work? Is it possible for the developer to choose a database name?


Thank you for your valuable feedback.
0 Kudos
Hi .. my 10 cents...

Have this up and running on Trial which gave me some issues you would probably only find on Trial. Below are my main problem-stones I had to sweep away. (Mac)

  1. @sap/cds-dk was missing on Trial - I had to make it a dependency. The cds service could not start. (sh cds not found)

  2. For building MTA and the /gen/pg folder aong with it, i found the below ... before-all: .. mta builder to work fine.

    ... mta.yaml fragment ...

    build-parameters:

    before-all:

    - builder: custom

    commands:

    - npm install --production

    - npm run build:cf

    - mkdir -p gen/pg/db

    - npm run pg:mta:buildstep

    - cp -r ./db/data gen/pg/db/data

    - cp config/package-pg.json gen/pg/package.json


    ...
    Last step above i copy the below package.json into the gen/pg folder per the official guide requirements.

    {

    "dependencies": {

       "@sap/cds": "*",

       "@cap-js/postgres": "*"

    },

    "scripts": {

       "start": "cds-deploy"

       }

    }


    ... package-json fragment ...

    "build:cf": "npm run build:cds && npm run cleanup:ts && npm run build:ts",

    "build:ts": "tsc",

    "build:cds": "cds build --production",

    "cleanup:ts": "npx rimraf gen/srv/**/*.ts",

    "pg:mta:buildstep": "cds compile './*' > gen/pg/db/csn.json",


    ...

  3. Deployer App module in MTA.yml
    Just use the example from the article.
    This part failed initially because the .csn was not copied properly to the /gen/pg folder per the mta builder script. I had to move the scripts to package.json - they could, for some reason, not run "inline" from the .mta. (Files were not generated).

DSree
Explorer
0 Kudos
Hello,

Hoping you can help in with a problem I am facing when I use above when the application is deployed to BTP. I have two applications following similar guidelines. Both are working successfully in local platform (VS Code). When deployed in BTP cloud, one is successful and other fails.

 

What is different between the two applications ? The one that works does not have any nested transactions. The one that fails has nested database transaction.
    await cds.tx(async () => {
refreshStatus = await SELECT.from(???Entity Name Here???);
});

Here is the error screenshot:
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR [cds] - TimeoutError: ResourceRequest timed out
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR at ResourceRequest._fireTimeout (/home/vcap/app/node_modules/generic-pool/lib/ResourceRequest.js:62:17)
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR at Timeout.bound (/home/vcap/app/node_modules/generic-pool/lib/ResourceRequest.js:8:15)
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR at listOnTimeout (node:internal/timers:569:17)
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR at process.processTimers (node:internal/timers:512:7)
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR Active connections:0
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR {
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR id: '1287074',
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR level: 'ERROR',
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR timestamp: 1694150939395
2023-09-08T01:28:59.39-0400 [APP/PROC/WEB/0] ERR }

 

This does work locally but fails when deployed in BTP. I was initially getting the same error locally also. As per documentation, at https://cap.cloud.sap/docs/node.js/databases#databaseservice-configuration, cap-js does not have the default pool configuration. So I added this to cds.requires:

 
      "db": {
"kind": "postgres",
"pool": {
"acquireTimeoutMillis": 5000,
"min": 1,
"max": 100,
"fifo": true
}
}

 

Any guidance or suggestions please?

Thank you
DSree
Explorer
0 Kudos
the nested transaction with "select" may not be that useful. But I do have another transaction that is an update transaction and needs a nested transaction enforced.
hash47
Explorer
0 Kudos
Hi @tiaxu,

 

Can you please share the '/pg-build.sh'? I did not see that in the blog.

 

Kind Regards,

Harshad
Jay2
Product and Topic Expert
Product and Topic Expert
0 Kudos
Hi tiaxu -

 

Thank you for sharing this blog really helpful.  I am getting the following error while creating the mta build. Can you please help with this?

 

command: fork/exec ./pg-build.sh: no such file or directory

 

Kr,

Jay
skymu89
Advisor
Advisor

Hi @kondiiq  @DSree 

I also faced the same error of message "TimeoutError: ResourceRequest timed out" recently by following CAP doc default configurations under evironment of MacOS also using latest cds-dk version as below.

@cap-js/cds-types: 0.2.0
@cap-js/postgres: 1.5.1
@cap-js/sqlite: 1.5.1
@sap/cds: 7.7.0
@sap/cds-compiler: 4.7.6
@sap/cds-dk: 7.7.0
@sap/cds-dk (global): 7.7.1
@sap/cds-fiori: 1.2.3
@sap/cds-foss: 5.0.0
@sap/cds-hana: 2.0.0
@sap/cds-mtxs: 1.16.0
@sap/eslint-plugin-cds: 2.6.5
Node.js: v18.19.1

Managed to solve it by debugging into deeper noticed that it's connecting with ipv6 refused issue with error messge "Error: connect ECONNREFUSED ::1:5432

Found a solution by testing which is to change db host configuration value from localhost to 127.0.0.1, then will work correctly because it's treated as ipv4. See my complete .cdsrc-private.json file content as below:

{
  "requires": {
    "db": {
      "kind": "postgres",
      "pool": {
        "acquireTimeoutMillis": 5000,
        "min": 1,
        "max": 100,
        "fifo": true
      },
      "credentials": {
        "host": "127.0.0.1", // <- Not default value localhost
        "port": 5432,
        "user": "postgres",
        "password": "postgres",
        "database": "postgres"
      }
    }
  }
}

BR, Sky