Hello all,
For some customers Archiving seems not be working properly.
If we took a a closer look these customers have millions of entities which shall - and can - be archived. But it does not happen.
But why?
You need to take a closer look into how the selection is working:
- In the beginning it simply depends on the LastChangeDate (see this blog).
- It collects all "old" records and expect that all of these can be archived. If there are still some left after the daily run, it is expected that the first ones are archived.
So, the backlog will be burned down. - Then we experience dumps due to the large number of entities which fulfill the selection criteria.
So, we limit the daily number to a maximum number, currently one million.
Then we get complains that the oldest ones are not being archived, whilst younger ones were.
- Therefore we introduced a sorting in the selection.
- By this we make sure, that the oldest one are the first in the processing.
This means we archive along the timeline from "old" to "new".
Unfortunately, there are customers for which these oldest ones are not archivable (based on the business checks). That means the first assumption "the first ones are archived" is wrong. Here the selection selects the same entities each day. 😒
- Consequently, we introduce with the new release a paging factor.
- It the selection has not reached the maximum number for the selection no paging is needed.
- Else it adds to the current paging factor the number of instances which have not been archived in the current run.
- The selection starts at this paging factor, so it "eats" through the millions of entities fulfilling the selection criteria until it comes to the end while ignoring the already processed entities.
- Then the paging factor is reset and it starts from the beginning: "oldest" ones first again.
We have already proved with some customers that now we are able to handle also millions of entities fulfilling the selection criteria with only some of them are archivable by business checks.
