Delta log information

[GrahamNewport]

A change (or addition/deletion) is logged in the SAP Hana log. So row/LOB table change gets logged but also Row store changes.

I have looked everywhere for a better understand of the following detail but am struggling

Does a column store change get logged in both the Log file and the delta log and of so where is the delta log kept? just in memory? in the data area? or the log area?

As I understand it delta areas do not get persisted. Does this mean on restart the deltas always get recreated from the delta log? and similarly is the delta log just used to roll out of a delta merge failure?

[dennispadia]

Hello Newport,

Provided little background as well. You can directly read the highlighted part and skip the rest if you want.

The goal of HANA is the integration of transactional and analytics workloads within the same data management system. In the transactional workload of the SAP business applications, more than 80% of all statements are read access. The remaining data write operations consists mainly of inserts, a few updates and very rare deletes. The analytical workloads are even more dominated by reads. So HANA's relational in-memory columnar store is tuned for such enterprise workloads.

In-memory processing reconciles the traditionally row-oriented transactional database design with the innovative column-oriented design that is optimal for analytics. This allows both types of workloads to be executed on the same operational data. The trade-off favors the large majority of read accesses through the choice of a columnar database. The performance impact on the row-oriented smaller transactional workload is acceptable due the fast in-memory access.

HANA optimizes its utilization of the available memory by implementing extensive compression technologies within its columnar store. Compression pays off best when data does not change, in which case the computational effort spent to compress data is leveraged over a longer duration. The data structure optimal for storing and processing compressed data are however not update-friendly as they do not come in order and the data needs to be re-sorted/re-organized all the time. Due to this data processing dilemma which is seen as a conflict when optimizing for both the analytical and the transactional workloads.

HANA resolves above conflict by using two table fragments per columnar table, with different store organization: Main and Delta.

The main fragment is reader-friendly, it contains most of the data, and it changes rarely; it uses sorted dictionaries, N-bit and other compression techniques. The delta fragment is writer-friendly; it contains the remaining smaller part of the data, uses only non-sorted dictionaries and N-bit compression.

For both the main and the delta fragments, snapshot isolation is implemented separately of the data, using dedicated MVCC structure. You can read how MVCC work, here. Queries access both the delta and main fragments. All new data is inserted into the delta fragment, and the MVCC structure are modified accordingly. When the delta fragment becomes too large, it is merged into the main fragment and a new empty delta is created. The delta merge process is non-blocking to readers and writers.

The purpose of the delta merge operation is to move changes collected in the delta storage to the read-optimized main storage. After the delta merge operation, the content of the main storage is persisted to disk. Its compression is recalculated and optimized, if necessary.

A further result of the delta merge operation is the truncation of the delta log. The delta storage structure itself exists only in-memory and is not persisted to disk. The column store creates its logical redo log entries for all operation executed on the delta storage. This log is called the delta log. If a system restart occurs, the log entries are replayed to rebuilt the in-memory delta storage. After the changes in the delta storage are merged into the main storage, the delta log file is truncated by removing those entries that were written before the merge operation.

NOTE: You can get more information in HA200 which is available in SAP Learning Hub.

Regards,

Dennis

[GrahamNewport]

Thank you for your clarification Dennis.

I teach HA200 but the information is not clear. It states that each transaction is recorded in the redo log but you are stating it is recorded in the delta log. Are these the same thing? Or is it recorded twice? or is the delta log part of the log file?

This is key information to me in the architecture that is not present in the documentation or the FAQ

[dennispadia]

Hello Newport,

As mentioned, all the write transaction in column store happens in delta storage. The delta storage exists only in main memory. So when we talk about the redo log entries that has been recorded in delta storage, it is termed as delta log. Only delta log entries are written to the persistence layer when delta entries are inserted.

So, delta log is the subset of the term redo log. It can only be used when we talk about changes that has been recorded in delta storage. So delta logs can be referred as redo log, but not all redo logs are termed as delta log.

In my above answer, if you see the line "The column store creates its logical redo log entries for all operation executed on the delta storage. This log is called the delta log." We generally refer redo log, when we talk about changes made to the table. But here delta storage table (own structure) is part of column store, so in order to refer redo log to this specific part of delta storage table, delta log term is being used.

Regards,

Dennis.