For details on how to optimally set the Java heap size for RPE: Sizing the Java Heap
A larger heap size helps with both generating larger documents but can also improve the performance of RPE. By default RPE uses a 1GB heap size and this should suffice in most cases. However if you run into Out Of Memory Issues you can try to enhance this value ( see above guideline).
You can control the Java Heap size for RPE through the settings in rpe-launcher.ini and rpe-studio.ini files, located in the launcher and studio folders from the RPE installation folder.
-vm ..\jre\bin --launcher.library C:\Program Files (x86)\IBM\Rational\Publishing Engine\126.96.36.199\studio -vmargs -Xms56m -Xmx1024m