SYSTEM_DATA_ALREADY_FREE when overwriting freed itab reference in secondary key

[cmilkau]

The following code runs fine, although LT_ORDER contains a line with a freed itab reference (LR_CHOCOLATE). If however, you uncomment the marked line (so that dangling itab reference is replaced by a valid reference), you get a SYSTEM_DATA_ALREADY_FREE at the end. I would expect the opposite at worst (a shortdump when the freed ref is in the key, but no shortdump when the freed reference is gone). The weirdest thing is the shortdump is triggered when referencing the OTHER order (for ice cream).

What is happening here? Why do I get a shortdump when there is no dangling reference, and how do I avoid it?

Note 1: This code example is brutally simplified, of course.
Note 2: Replacing references by keys is not an (easy) option because in the real example, the keys do not yet exist.

form RUN.
  types: TPRODUCT type STRING,
         TORDER   type ref to TPRODUCT.
  data: LT_PRODUCTS type standard table of TPRODUCT with empty key,
        LT_ORDERS   type standard table of TORDER   with empty key
                                                    with unique hashed key PRODUCT
                                                         components TABLE_LINE.
  insert `chocolate` into table LT_PRODUCTS reference into data(LR_CHOCOLATE).
  insert `ice-cream` into table LT_PRODUCTS reference into data(LR_ICE_CREAM).
  insert LR_CHOCOLATE into table LT_ORDERS.
  insert LR_ICE_CREAM into table LT_ORDERS.
  data(LR_CHOCOLATE_ORDER) = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_CHOCOLATE ] ).
  "uncomment this line (removes reference to LT_PRODUCTS[ 1 ])
  "LR_CHOCOLATE_ORDER->* = new #( `improved chocolate` ).
  delete LT_PRODUCTS index 1. "LR_CHOCOLATE is now invalid
  "this line crashes only when above is uncommented
  data(LR_ICE_CREAM_ORDER) = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_ICE_CREAM ] ).
endform.

[matt]

Ew. A form... How very 1990s.

I ran this:

REPORT.
  types: TPRODUCT type STRING,
         TORDER   type ref to TPRODUCT.
  data: LT_PRODUCTS type standard table of TPRODUCT with empty key,
        LT_ORDERS   type standard table of TORDER   with empty key
                                                    with unique hashed key PRODUCT
                                                         components TABLE_LINE.
  insert `chocolate` into table LT_PRODUCTS reference into data(LR_CHOCOLATE).
  insert `ice-cream` into table LT_PRODUCTS reference into data(LR_ICE_CREAM).
  insert LR_CHOCOLATE into table LT_ORDERS.
  insert LR_ICE_CREAM into table LT_ORDERS.
  data(LR_PRODUCT) = ref #( LT_PRODUCTS[ 1 ] ). " = LR_CHOCOLATE
  data(LR_ORDER) = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_PRODUCT ] ). " -> LTORDERS[ 1 ]
  "uncomment this line
  LR_ORDER->* = new #( |improved { LR_PRODUCT->* }| ). " Uncommented
  delete LT_PRODUCTS index 1.
  LR_PRODUCT = ref #( LT_PRODUCTS[ 1 ] ). " = LR_ICE_CREAM
  "this line crashes only when comment above is uncommented
  LR_ORDER = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_PRODUCT ] ). " -> LTORDERS[ 2 ]<br>

In debug, checking what was going on in each variable. And it didn't dump. Then I single stepped through in debug without examining variable contents... and it dumped. I put a breakpoint at the line after the delete to see what had happened to the variables.

Your DELETE frees LR_CHOCOLATE, and that frees the whole of LT_PRODUCT (one of its elements is freed so the whole table must be, it seems). Similarly LT_ORDERS contains two undefined entries. The final line tries to read the second record of LT_ORDERS which is undefined, and so you get a dump.

Try it yourself.

[cmilkau]

I ran this in debugger many times, on different kernels and in different code variations, before deciding to post it. I did not experience a behavior like you describe in any of these cases. In fact, the crashing programs and the non-crashing programs looked identically in the debugger (save the actual change of LT_ORDERS[ 1 ]).

All debuggers displayed this behavior: The delete removes the first item of LT_PRODUCTS (as it should by definition). That invalidates LR_CHOCOLATE, but LR_ICE_CREAM remains valid and now points to the first entry (which is now `ice-cream`). The contents of LT_ORDERS reflect this, as they contain exactly these two references (unless overwritten by `improved chocolate`, of course).

It is a form because that is the syntactically shortest way to produce a subprogram. This is an example code. Global variables, in theory, can behave a little differently than local variables in edge cases, and I am only interested in the behavior of local variables.

[FredericGirod]

If I used the code of matthew.billingham, and I stop before the "improved" :

(run in french, but you could easily find the same on your sapgui)

In Clé (key) I select the second index, I see the content.

If I run just the "improved" line, look at the result:

The second index is dead.

I don't know if there is a statement to rebuild index, maybe sandra.rossi knows it (as she knows all the doc by heart 😉 ).

[cmilkau]

frdric.girod I assume by "dead index" you mean "stale secondary key". That confirms that part of my suspicion. I just did a key access to force a rebuild (see comments on the question itself).

[matt]

It is a form because that is the syntactically shortest way to produce a subprogram

It wasn't though. As I demonstrated. 😉 Even so, I post using local methods simply to counteract the persistence of obosolet programing.

[matt]

I don't know why my comment is red!

[FredericGirod]

you certainly turn to the dark side, did you dream about Java ? or JavaScript ?

[cmilkau]

I suspect ABAP delays the (ABAP-internal) update of the secondary key PRODUCT until it is actually used. When the secondary key update is finally triggered, the stale key data still contains the (now dangling) LR_CHOCOLATE reference (invisible to the ABAP user code). That might catch the ABAP kernel by surprise. The following code forces a secondary key update before invalidating LR_CHOCOLATE, and it does not crash:

form RUN.
  types: TPRODUCT type STRING,
         TORDER   type ref to TPRODUCT.
  data: LT_PRODUCTS type standard table of TPRODUCT with empty key,
        LT_ORDERS   type standard table of TORDER   with empty key
                                                    with unique hashed key PRODUCT
                                                         components TABLE_LINE.
  insert `chocolate` into table LT_PRODUCTS reference into data(LR_CHOCOLATE).
  insert `ice-cream` into table LT_PRODUCTS reference into data(LR_ICE_CREAM).
  insert LR_CHOCOLATE into table LT_ORDERS.
  insert LR_ICE_CREAM into table LT_ORDERS.
  data(LR_CHOCOLATE_ORDER) = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_CHOCOLATE ] ).
  "uncomment this line (removes reference to LT_PRODUCTS[ 1 ])
  LR_CHOCOLATE_ORDER->* = new #( `improved chocolate` ).
  "this line forces a secondary key update
  data(LR_ICE_CREAM_ORDER) = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_ICE_CREAM ] ).
  delete LT_PRODUCTS index 1. "LR_CHOCOLATE is now invalid
  "this line crashes only when secondary key is stale
  LR_ICE_CREAM_ORDER = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_ICE_CREAM ] ).
endform.

[Sandra_Rossi]

Situations of Delayed Update and Lazy Update are documented in the ABAP documentation. If you don't find it, please revert back so that I find the reference for you.

[cmilkau]

I am aware, but there is nothing there about delayed/lazy update causing a crash. As I understand it, these are supposed to change performance, but not behavior of your program.

[raymond_giuseppi]

Look at methods such as FLUSH_ITAB_KEY (a Specific Secondary Key) and FLUSH_ITAB_KEYS (All Secondary Keys) of the class CL_ABAP_ITAB_UTILITIES to rebuild secondary indexes.

cl_abap_itab_utilities=>flush_itab_key(EXPORTING keyname = 'PRODUCT' CHANGING  itab = LT_ORDERS).

ref: online help

[FredericGirod]

Nice one Raymond !

[cmilkau]

Flush sounds like dropping a cache, rather than actually rebuilding it. It's already stale at that point, the workaround needs it to be actually constructed before the delete (I don't want to say the bad word but that seems like a bug in ABAP-internal key construction).

Have you run this example with that addition yet?

EDIT: seems to work. This code runs without crash:

form RUN.
  types: TPRODUCT type STRING,
         TORDER   type ref to TPRODUCT.
  data: LT_PRODUCTS type standard table of TPRODUCT with empty key,
        LT_ORDERS   type standard table of TORDER   with empty key
                                                    with unique hashed key PRODUCT
                                                         components TABLE_LINE.
  insert `chocolate` into table LT_PRODUCTS reference into data(LR_CHOCOLATE).
  insert `ice-cream` into table LT_PRODUCTS reference into data(LR_ICE_CREAM).
  insert LR_CHOCOLATE into table LT_ORDERS.
  insert LR_ICE_CREAM into table LT_ORDERS.
  data(LR_CHOCOLATE_ORDER) = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_CHOCOLATE ] ).
  "uncomment this line (removes reference to LT_PRODUCTS[ 1 ])
  LR_CHOCOLATE_ORDER->* = new #( `improved chocolate` ).
  "update key PRODUCTS
  CL_ABAP_ITAB_UTILITIES=>FLUSH_ITAB_KEY( exporting KEYNAME = 'PRODUCT'
                                          changing  ITAB = LT_ORDERS ).
  delete LT_PRODUCTS index 1. "LR_CHOCOLATE is now invalid
  "this line crashes only when key PRODUCTS is stale
  data(LR_ICE_CREAM_ORDER) = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_ICE_CREAM ] ).
endform.

[FredericGirod]

  "uncomment this line
  LR_ORDER->* = new #( |improved { LR_PRODUCT->* }| ). " Uncommented
  cl_abap_itab_utilities=>flush_itab_key( EXPORTING keyname = 'PRODUCT' CHANGING  itab = LT_ORDERS ).
  delete LT_PRODUCTS index 1.
  LR_PRODUCT = ref #( LT_PRODUCTS[ 1 ] ). " = LR_ICE_CREAM
  LR_ORDER = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_PRODUCT ] ). " -> LTORDERS[ 2 ]<br>

[raymond_giuseppi]

It was only during a development phase (I was trying to guess what the previous developer had failed to achieve) that the final program was rewritten from scratch and no longer needed this trick.

In case of doubt, look at itab - Updating Secondary Table Keys

• The methods FLUSH_ITAB_KEY and FLUSH_ITAB_KEYS of the class CL_ABAP_ITAB_UTILITIES can be used to update individual secondary keys or all secondary keys of an internal table explicitly in exceptional circumstances. These methods can be used for analysis and test purposes. It might also make sense to use them after making changes if the next access does not take place immediately afterwards to handle possible exceptions there and then.

Seems your code didn't explicitly trigger the update of the index.

[cmilkau]

My code did trigger a key update, but only after the delete. For some reason, the key update (by key use or by FLUSH_ITAB_KEY) itself then crashes. If I ensure the key is fresh before the delete (by either method), the crash goes away. I don't know what exactly happens in FLUSH_ITAB_KEY but for some reason it can't deal with dead references in the stale data, even if those references are just discarded anyway.

[cmilkau]

The crash is caused when rebuilding the secondary key PRODUCT of internal table LT_ORDERS (I don't think this is intended behavior of ABAP, but I don't know for sure). The problem can be avoided by rebuilding the secondary key before invalidating the reference LR_CHOCOLATE (rather than after). The following code example proves these claims:

form RUN.
  types: TPRODUCT type STRING,
         TORDER   type ref to TPRODUCT.
  data: LT_PRODUCTS type standard table of TPRODUCT with empty key,
        LT_ORDERS   type standard table of TORDER   with empty key
                                                    with unique hashed key PRODUCT
                                                         components TABLE_LINE.
  insert `chocolate` into table LT_PRODUCTS reference into data(LR_CHOCOLATE).
  insert `ice-cream` into table LT_PRODUCTS reference into data(LR_ICE_CREAM).
  insert LR_CHOCOLATE into table LT_ORDERS.
  insert LR_ICE_CREAM into table LT_ORDERS.
  data(LR_CHOCOLATE_ORDER) = ref #( LT_ORDERS[ key PRODUCT TABLE_LINE = LR_CHOCOLATE ] ).
  "uncomment this line (removes reference to LT_PRODUCTS[ 1 ])
  LR_CHOCOLATE_ORDER->* = new #( `improved chocolate` ).
  "this line never crashes
  "CL_ABAP_ITAB_UTILITIES=>FLUSH_ITAB_KEY( exporting KEYNAME = 'PRODUCT'
  "                                        changing  ITAB = LT_ORDERS ).
  delete LT_PRODUCTS index 1. "LR_CHOCOLATE is now invalid
  "this line crashes only when key PRODUCT is stale
  CL_ABAP_ITAB_UTILITIES=>FLUSH_ITAB_KEY( exporting KEYNAME = 'PRODUCT'
                                          changing  ITAB = LT_ORDERS ).
endform.