Memory usage for repository export to XML may be unnecessarily high

[Sat-Ing]

If I export a repository to XML via org.eclipse.emf.cdo.server.CDOServerExporter.XML#exportRepository, I need a very high system memory, otherwise I run into out-of-memory exceptions. The required memory correlates to the size of the repository.

On the other hand, the import via org.eclipse.emf.cdo.server.CDOServerImporter.XML#importRepository, seems not to be dependent on the repository's size and the memory usage is always quite low.

Now the question is: could we somehow reduce the required memory for repository exports? This would greatly benefit the operational usage.

[estepper]

I implemented both the exporter and the importer with scalability in mind. Though, it's possible that I overlooked something. Do you have a concrete suspicion where memory could leak? Perhaps you can use a profiler to inspect your scenario?

[Sat-Ing]

@estepper the memory pretty much linearly increases until the export is complete. Binary/XML exports are affected in a similar way. I did not have a look at the CDO code yet. This is a heap dump for a binary export:

[estepper]

Thanks for the additional info! I fear I'll have to create a huge repo and try it out myself. Is 15 million objects approx. the size of your repo?

[Sat-Ing]

Thats a good question ;). The h2 database size is ~4GB. I need ~8GB of RAM to export it. I am using a repository with no branching and no audits.

[estepper]

H2 and non-auditing are good infos, too. My guess of >15 million objects came fro the instance countin your heap dump for CDOIDObjectLongImpl ;-)

[estepper]

Just to be sure, you haven't customized the server in any way? I mean, no custom listeners or whatever could hold onto Java objects in addition to CDO's shipped code...

[Sat-Ing]

No, that should not be the case. This is basically what we do during the export routine. Nothing else is done before this code:

        File h2db = new File(getRepoPath());
        JdbcDataSource dataSource = new JdbcDataSource();
        dataSource.setURL("jdbc:h2:" + h2db + File.separator + REPOSITORY_NAME);

        IMappingStrategy mappingStrategy = CDODBUtil.createHorizontalMappingStrategy(false);
        mappingStrategy.getProperties().put(Props.QUALIFIED_NAMES, "true");
        IDBAdapter dbAdapter = new H2Adapter();
        IDBConnectionProvider dbConnectionProvider = DBUtil.createConnectionProvider(dataSource);
        IStore store = CDODBUtil.createStore(mappingStrategy, dbAdapter, dbConnectionProvider, new HashMap<>());

        Map<String, String> props = new HashMap<String, String>();
        props.put(IRepository.Props.OVERRIDE_UUID, REPOSITORY_NAME);
        props.put(IRepository.Props.SUPPORTING_AUDITS, "false");
        props.put(IRepository.Props.SUPPORTING_BRANCHES, "false");
        props.put(IRepository.Props.ENSURE_REFERENTIAL_INTEGRITY, "false");
        props.put(IRepository.Props.COMMIT_INFO_STORAGE, "NO");
        IRepository repository = CDOServerUtil.createRepository(REPOSITORY_NAME, store, props);

        XML xmlExporter = new XML(repository);
        try (OutputStream os = new FileOutputStream(outputFile)) {
            xmlExporter.exportRepository(os);
        }