Don't know about Java ( because I have not coded in Java since 1998 ).

However using an Intel i7-1355U (mobile i7 CPU, 1.7Ghz base speed) with 16GB of memory, I was able to sort 200 million DWORDs ( 4 bytes unsigned integer ) in 20.563 secs using Shellsort ( normally slower than Quicksort ). The codes were compiled using Visual Studio's C++.

So your implementation is definitely flawed ( unoptimised somewhere ).

Microsoft would not allow me to sort 300 million records so cannot tell you anything beyond this limitation. Sometimes I could sort 250 million records.