Taking huge heap memory by SAP JCO library

[polireddy]

we are seeing java.lang.OutOfMemoryError: Java heap space in SAP Java connector. After analyzing heap dumps, we came to know is, for 278 MB file, jco is taking around 2GB heap memory. 

For 400 mb file, it taking around 3.2GB.

1. We need to know if the Idoc file is having x size , how much approximate memory required (2x, 4x,6x,8x,10x) to process this file using java program.

For example, if the Idoc file is 100 MB, do we need around 400/600/800 MB of java heap  memory to process the file.

2. Is there anyway by which java application will get to know the file size using SapJCO api before start processing/reading file?

Error:

One instance of “com.sap.conn.rfc.data.ARFCSDATA” loaded by “com.test.$Loader @ 0x60129d7a8” occupies 2,790.48 MB (34.17%) bytes. The memory is accumulated in one instance of “char[][]”, loaded by “<system class loader>”, which occupies 2,790.47 MB (34.17%) bytes.

[Ulrich_Schmidt]

Not sure, what you mean by "IDoc file", because if you use JCo, the IDoc is not transferred as a file, but as an internal table. The memory consumption of this table can be calculated easily: the table is of type EDI_DD40, which has 7 fields with a total length of 1063 characters. So in a Unicode system, the memory consumption is 2126 bytes per line. (Roughly speaking, one line in the table corresponds to one segment in the IDoc.)

Next we need to know, that the data is transferred via tRFC protocol, which means that JCo needs to have two copies of the data in memory: first for the DATA table containing the recorded function call (ARFC_DEST_SHIP function module), and then for the SDATA table (type EDI_DD40) of the IDOC_INBOUND_ASYNCHRONOUS function module. The required memory for the DATA table is not easy to calculate, because it contains partially compressed data, which can be processed in a kind of "streaming mode", but I guess, if we just take a factor of two, we are on the safe side.

So a good rule of thump to calculate the size requirement of an IDoc in JCo/Java is:
4k x number of segments

If that 2790 MB char[][] corresponds to one SDATA table of one IDoc, this would mean that this IDoc (package) had around 1.4 million segments?! Can this be true?? I don't think that any single IDoc can be that big, so more likely, you have defined the partner profile outbound parameter for this IDoc to "collect" several IDocs into one package? In that case, you should make the package size smaller. (See transaction WE20: in the detail view of the Partner Profile corresponding to the Receiver & IDoc type, there is a field named "Pack. Size". You could experiment a bit with it and cut it down, until the IDoc packages reach a "manageable" size in terms of total number of segments.)