ABAP: Fast JSON Serialization

[TimoStark]

tldr; Install https://github.com/timostark/abap-json-serialization and enjoy the fastest possible JSON serialization. The result will be a 10x faster JSON serialization and deserialization compared to /UI2/CL_JSON at the same quality. Be warned though: Read the limitation section first. 

Oh no - another JSON Serialization Blog Post? Hey - At least no blog about excel exports 🙂

So why are we in need of a "new" way for JSON serialization? The reason is simple: Runtime! Especially when working with custom REST services with a big payload you will notice a lot of runtime getting lost in JSON serialization. Loosing 30% of your runtime in JSON serialization makes me very unhappy (when I just optimized my more difficult business class).

So, what are our goals:

1. Fast


2. Support Camel-Case


3. Support real booleans and numbers


4. Not need to be easy or generic (I will accept a bad life as a developer if it is fast and reliable).

There are already multiple solutions out there - just to mention the most important ones:

• /UI2/CL_JSON (plain ABAP)


• CALL TRANSFORMATION  (Kernel solution, using XSLT or Simple Transformation Mapping)


• CL_TREX_JSON_SERIALIZER (plain ABAP)


• XCO_CP_JSON  (S/4 Only, not stable, extremely slow and crashes - therefore let's keep that one out for the moment)

So how are they behaving from runtime perspective. Let's take a very simple example and serialize 5.000 lines of SFLIGHT lines and a very complex and deep structure:

So what does that tell us?

Not really surprisingly the only feasible solution on a ABAP stack is the usage of CALL TRANSFORMATION - as this is executed directly in the Kernel, thus not depending on slow ABAP String concat and/or field-symbol traversal.

It might be strange but always remember: Building up strings using concats and traversing over field-symbols inside a structure is very slow in ABAP compared to native languages --> Where possible Kernel Modules like Simple Transformations are preferable performance wise.

There are however quality problems when using CALL TRANSFORMATION ID:

1. No Camel-Case


2. No real "booleans" (instead 'X' is printed.. tell that somebody outside of the SAP world)


3. No real NUMC (instead leading 0s are printed)

There is one solution which was already mentioned in a blog post, using a custom ABAP transformation to at least support camel case. Unfortunately, that throws away the performance benefit as the fast kernel module has to go up to the ABAP stack for a simple "to-camel-case" transformation.

My suggested solution is, that we use CALL TRANSFORMATION for what it is actually thought: to transform data using ST transformations (Simple Transformation). Remark: CALL TRANSFORMATION can also be used for XSLT Transformation (which are much more powerful but also slower - see remark by @Sandra Rossi), but this is simply not required here. This means we are creating an own Simple transformation for the structure/table-type we want to serialize (nested structures are of course possible).

Let's see an example transformation for the table SFLIGHT (shortened):

Nobody wants to write that code (and for sure nobody with a right mind will want to keep that transformation up to date) - but let's first see the runtime impact.

==> The solution is around 10 times faster than /UI2/CL_JSON, while having the same quality as a result.

As already said of course nobody wants to write these ST mappings - especially for deeply nested structures this is horrible.

Therefore, I've published a small helper program ZJSON_TO_XSLT under MIT license to GitHub which allows you to directly create those transformation for any structure/table

 

Output (next to the generated transformation).

Execute the transformation using normal CALL TRANSFORMATION call:

DATA(lo_writer_json) = cl_sxml_string_writer=>create( type = if_sxml=>co_xt_json ).

CALL TRANSFORMATION ZSFLIGHT SOURCE root = lt_flights RESULT XML lo_writer_json.

DATA(lv_json) = cl_abap_codepage=>convert_from( lo_writer_json->get_output( ) ).

In my customer projects I am using the API called in the program in a regular job (including a mapping-table) which updates the transformations on the development system in a regular manner. If you want to spend a lot of time you could even create the transformations "live" as local objects on the first access. I personally do not like the approach of local development objects though.

==> Using JSON serialization with fixed transformations, you can get an extremely fast JSON serialization and deserialization while still having high quality.

Limitations:

• A big word of warning: The solution is thought for performance critical development. The solution comes with very high costs: you have to think of an additional development object (the transformation). Even if it updates automatically, it can get out-of-date, you can forget it or it can get corrupted.


• Summed up: If you do not have a problem (i.E. customer complains about slow type-ahead, where you need response times in ms) do not create additional problems using a more complex solution mentioned here.


• The solution works as long as you know the exported JSON types upfront ( i.E. have static data-types). For dynamic data-structures this solution will not work.

[abo]

Love the irony of not-another-Excel blogpost 🙂

[Sandra_Rossi]

Excellent! Your blog post is the first one I see to inform about the efficiency of Transformations for high data volume versus SAP classes.

Thank you for the tool, it will be very helpful. Until now, if the people wanted to use transformations for JSON, they had to understand both ST and JSON-XML format (elements object, str, etc.) and so had good reasons to not use them.

Note that the language you used is not XSLT, it's Simple Transformation AKA ST. XSLT is much more powerful but is much slower (ratio roughly 1 to 10 based on personal tests ~10 years ago). ST is also very powerful compared to XSLT as it's able to transform in the 2 directions (ABAP to XML/JSON and XML/JSON to ABAP) with the same code (with complex XML/JSON, some ST rules are to be indicated to apply specifically to either serialization or deserialization).

NB: both transactions STRANS or XSLT_TOOL are the same and can be used to maintain transformations in both languages, better use STRANS so that people stop confusing XSLT and Simple Transformation.

[TimoStark]

Thanks Sandra. You are absolutly right, that XSLT mapping is the wrong word. I've replaced it with "Simple-Transformation" everywhere (besides in the code - will do that over time 🙂 ), based on your feedback.

I tested XSLT initially (there are some hacks on stackoverflow to actually get camelCase running without a hard-coded transformation using plain xslt), but it is just much slower as the hardcoded approach (as you said).

[Michael_Keller]

I've added some irony about "Excel read/write requirement in ABAP" in my new article on heise, too 🙂

[Attila]

Hello Timo,

thanks for sharing. To my knowledge this is the same approach when You activate an SEGW service. During that based on the model (entitytype) it is generating a transformation. I always wanted to have a feature or tool like this, but never had the possibility to understand it in the deep jungle of the standard code. This tool can be used in many context, to speed up integration developments.

This is great,  thank You!

Best regards,Attila

[Andre_Fischer]

Hi Timo,

I would be interested to learn in more detail which problems you encountered with the xco json libraries.

I am using this library in my open source project since quite a while to generate and read (small) json files without any problems on 2021, 2022 and on SAP BTP ABAP Environment.

And what I like with the approach is the very comprehensible coding when it comes to generate a json file

DATA(lo_json_builder) = xco_cp_json=>data->builder( ).

lo_json_builder->begin_object(

  )->add_member( 'SessionId' )->add_string( '7cd44fff-036a-4155-b0d2-f5a4dfbcee92'

  )->add_member( 'UserName' )->add_string( 'John Doe'

  )->add_member( 'IsPremiumUser' )->add_boolean( abap_true

  )->end_object( ).

Since you are reporting chrashes I am wondering whether you have opened a ticket ?

If so, please send me the number directly.

Kind regards,

Andre

[TimoStark]

andre.fischer

Hi Andre,

Thanks for your comment - After rechecking with our latest S/4 Version - the xco_cp_json class is not crashing anymore - and yes I love the builder pattern which just creates nice and readable code.

Anyways: The purpose of the blog is to show a possibily to create a JSON serializer / deserializer which is at least comparable performance-wise to the JSON serializer of a normal web-browser.

When writing the blog I was very frustrated, that I've optimized a very complex HANA native query "to the end", just to still loose >200ms for simple data transformation. The business use case for the code above was a type ahead functionality which had to run on a very complex data source. In such cases it is simply a major difference if the preview is returning after 50, 200 or 500ms.

If I execute the following code on a SAP S/4HANA 2022 i get runtimes of >5 seconds for serialization and >5 seconds for deserialization.

  SELECT * FROM sflight INTO TABLE @DATA(lt_sflight). "5000 lines

  DATA(lv_json_string) = xco_cp_json=>data->from_abap( lt_sflight )->to_string( ).



  CLEAR: lt_sflight.

  xco_cp_json=>data->from_string( lv_json_string )->write_to( EXPORTING ia_data = REF #( lt_sflight ) ).

Maybe I am missing some magic performance boost in the xco framework. If not the XCO framework is hardly usable for high-performant/real-time end-user applications where more than a few lines are transferred as it is even much slower than /UI2/CL_JSON, which already has a horrible performance...

Kind Regards,

Timo

PS: Very generally I am still confused why I can not find a very easy to use JSON.stringify / JSON.parse Kernel functionality in ABAP. Not having a native parser for JSON is a major drawback of ABAP (if we are talking about cloud enablement).

[TimoStark]

Small addition:  When hardcoding the transformation using the builder pattern (which is also not nice) the performance of serializing 5k lines of SFLIGHT is over 8 seconds.

As said, maybe I am doing something wrong, but I am pretty convinced that an ABAP based serialization / de-serialization logic just can not be fast enough to be comparible to kernel based solutions.

 

  DATA(lo_builder) = xco_cp_json=>data->builder( ).

  DATA(lo_array) = lo_builder->begin_array( ).

  LOOP AT lt_sflight ASSIGNING FIELD-SYMBOL(<ls_sflight>).

    lo_array->begin_object(

      )->add_member( 'CARRID' )->add_string( <ls_sflight>-carrid

      )->add_member( 'CONNID' )->add_string( <ls_sflight>-connid

      )->add_member( 'FLDATE' )->add_string( <ls_sflight>-fldate

      )->add_member( 'PRICE' )->add_number( <ls_sflight>-price

      )->add_member( 'CURRENCY' )->add_string( <ls_sflight>-currency

      )->add_member( 'PLANETYPE' )->add_string( <ls_sflight>-planetype

      )->add_member( 'SEATSMAX' )->add_number( <ls_sflight>-seatsmax

      )->add_member( 'SEATSOCC' )->add_number( <ls_sflight>-seatsocc

      )->add_member( 'PAYMENTSUM' )->add_number( <ls_sflight>-paymentsum

      )->add_member( 'SEATSMAX_B' )->add_number( <ls_sflight>-seatsmax_b

      )->add_member( 'SEATSOCC_B' )->add_number( <ls_sflight>-seatsocc_b

      )->add_member( 'SEATSMAX_F' )->add_number( <ls_sflight>-seatsmax_f

      )->add_member( 'SEATSOCC_F' )->add_number( <ls_sflight>-seatsocc_f )->end_object( ).

  ENDLOOP.

  lo_array->end_array( ).



  DATA(lv_str) = lo_builder->get_data( )->to_string( ).

[Andre_Fischer]

Hi Timo,

thank you very much for your detailed examples. I will discuss this with my colleagues from development.

Kind regards,

Andre