How to trouble shoot “Out of memory exception:

Here is how it works:
JVM Heap: The size you specify via -Xmx flag. This is the Memory that JVM uses to allocate objects created from your Application
Native Memory: This is outside of JVM Heap. And the Maximum limit is enforced only by the OS. This is the memory that JVM uses for it's internal operations.
So, when you find out the Memory used by your JVM process from the OS level (using TOP or ps), the amount you see is "JVM Heap + Native Memory". That's why this number is always higher than the Heap Usage.
So, when the JVM quits stating "out of memory" error, it does not necessarily mean it has run out of Heap (in your case it definitely is not running out of Heap). But it could mean it runs out of Native Memory.
Now, as I mentioned earlier Native Memory is limited by the OS itself. We'll need to find out what is the maximum size a process can grow in Solaris 10 64bit (In 32 Bit OSs, it cannot grow beyond 2GB). With Solaris 10, I know you can set this "Maxsize" for individual processes/projects.
So, find out what is the memory utilized by the process itself (Not JVM heap). Use TOP to find out what's the "SIZE" of the process. Let's see if it always break after reaching certain size.
As far -Xms and -Xmx, I normally wouldn't recommend setting this to all the Apps. It makes the JVM work harder during GCs because it needs to go through the entire 2.5GB. Generally I recommend setting this to half of the Max heap initially. If your memory utilization is always lingering close to 2GB,then you can set both -Xms and -Xmx the same.