CG overhead limite exceed

Added by Cátia Saraiva about 1 year ago

When I try to transform my Land Cover data (159220 instanses) the software cant do it and an error appears.
What can I do?

hale_error.png (13.2 kB)


Replies (15)

RE: CG overhead limite exceed - Added by Thorsten Reitz about 1 year ago

Dear Cátia,

you could increase memory by modifying a parameter in the hale.ini file as follows:#

-Xmx1000m to -Xmx4000m

If that doesn't help, you've potentially hit this bug:

https://github.com/halestudio/hale/issues/225

We're working on resolving it. There are several workarounds, but to know which one could be applied, we'd need to look at your mapping project.

All the best,

Thorsten

--
Thorsten Reitz
Founder

E-Mail:
Web: www.wetransform.to

wetransform GmbH
Fraunhoferstr. 5
64283 Darmstadt

RE: CG overhead limite exceed - Added by Cátia Saraiva about 1 year ago

Thank you for the help and the quick reply.

I did use the merge fuction. I'll leave a file with my project.

Best Regards,

Cátia

RE: CG overhead limite exceed - Added by Thorsten Reitz about 1 year ago

Dear Cátia,

I looked at your project. In this case, usage of the Merge makes sense. The thing that leads to problems is the aggregation of the 150k geometries using Rename, which results in one gigantic feature that is very hard to manage for hale. What you can do to resolve that issue is:

Since for that target FeatureType, you only need an EX_Extent, you can build a custom groovy function that uses a collector to iteratively built the extent. This would be very efficient. Here is more information on Groovy Scripts and on Collectors:

http://help.halestudio.org/latest/topic/eu.esdihumboldt.cst.doc.functions/functions/eu.esdihumboldt.cst.functions.groovy.html?resultof=%22%43%6f%6c%6c%65%63%74%6f%72%22%20%22%63%6f%6c%6c%65%63%74%6f%72%22%20

All the best,

Thorsten

RE: CG overhead limite exceed - Added by Cátia Saraiva about 1 year ago

Is there some alternative to Groovy Scripts? Since we want to create a WFS with the GML file through Geoserver, but based on a presentasion that I saw the Groovy fuction is not suported by GeoServer.

RE: CG overhead limite exceed - Added by Thorsten Reitz about 1 year ago

No, a groovy script doesn't work on Geoserver, but neither would a merge of 150k+ geometries to get the EX_EXTENT geometry. I assume that would also crash GeoServer when you try to display or deliver that feature.

An alternative would be to use a Geographic Description element - where you can just assign a fixed value string such as "Portugal" to extent > EX_Extent > description > CharacterString > CharacterString.

Another alternative would be to do two-step processing, which is a pretty common approach. You first use hale to add any information you can't generate usign app-schema (such as this extent) and store it in a separate file or in the database, which you then add to the source for the app-schema transformation.

All the best,

Thorsten

RE: CG overhead limite exceed - Added by Cátia Saraiva about 1 year ago

Dear Thosten,

I tried the first alternative and I still can't transform the data.

Is there any more detailed post about the second alternative? I'm new to the software and data harmonization.

Thank you for the help.

Best Regards,

Cátia

RE: CG overhead limite exceed - Added by Simon Templer about 1 year ago

Hi Cátia,

if what you want to do is using the GeoServer AppSchema export, then transforming the whole data in hale isn't your goal anyway. It makes sense to have a subset of the data in hale, to have some data when working with the mapping (e.g. using "Instance sampling"). But what you will export to GeoServer is the mapping itself, from which a configuration for GeoServer is created.

More information on how to use hale with GeoServer can be found in the help, including information on what restrictions there are.

I don't think an aggregation of the geometry like that is supported in that context. What you could try to do is let the database do that job for you, and create a corresponding database view that you use in your mapping.

Best,
Simon

RE: CG overhead limite exceed - Added by Steven Schreuder about 1 year ago

Hi,

I also got the CG overhead limit error. I am not using the merge function, but probably i am doing something wrong. I build a simple mapping, and when exporting to gml I get the error. (When exporting to xls I don't get it).

project.zip (27.8 kB)

RE: CG overhead limite exceed - Added by Thorsten Reitz about 1 year ago

Hi Steven,

I just checked your project, that is indeed very simple. No obvious things amiss. Could you please either provide the data for repoduction purposes (you can send an email to with a link if it is a lot of data), or provide us with a hale log file so that we can investigate? The log files reside in the sub-folder \workspace\.metadata in your HALE installation.

All the best,
Thorsten

RE: CG overhead limite exceed - Added by Steven Schreuder about 1 year ago

Hi Thorsten,
Thanks for the quick reaction.

This is a shapefile I used as input data.
First I tried with a connection to quite a big PostGIS database, so I tries smaller data, unfortunately it failed as well.

Regards,
Steven

RE: CG overhead limite exceed - Added by Thorsten Reitz about 1 year ago

Hi Steven,

I quickly tested with that data set and was able to transform and export as a GML data set (see attachment - it still fails to validate since some properties are missing and some unique contraints are violated). Could you tell me which export setting exactly you've used?

All the best,

Thorsten

test.zip (49.3 kB)

RE: CG overhead limite exceed - Added by Steven Schreuder about 1 year ago

Hi Torsten,
Good to se that it worked on your side.
I used: Export -> transformed data -> GML feature collection with the basic settings.
I tried on different machines, on one i encountered the mentioned error, the other a java heap space error.

Regards, Steven

RE: CG overhead limite exceed - Added by Thorsten Reitz about 1 year ago

Hi Steven,

OK, the only thing I might have done differently was to use the Export called "GML (WFS 2.0 FeatureCollection)", which is the current default for INSPIRE.

If that is what you also used, I will need a bit more information to progress though. This error description does sound systematic, so I'd need you full system information - operating system, java version, hale version, and ideally the log files. You can send those via email to in case you're not comfortable with sharign them here.

All the best,

Thorsten

RE: CG overhead limite exceed - Added by Simon Templer about 1 year ago

Hi Steven,

as this seems to be a heap space problem, it would also be good to know what the status of the heap space is before you start your export. There is a heap space bar displayed at the bottom of hale studio, in the status bar.

Best,
Simon

RE: CG overhead limite exceed - Added by Steven Schreuder about 1 year ago

Hi Torsten and Simon,

Thanks for all the input. I will list my specs and send to them you.
At the moment, the setting Torsten used (WFS 2.0 Feature collection) worked!

Steven

(1-15/15)