Repository navigation
Size constraint for cudaMallocHost #54
Description
Not sure if this has been brought up before, but I ran into an issue while trying to allocate host memory with size larger than MAX_INT. E.g. running:
JCuda.setExceptionsEnabled(true);
JCuda.cudaMallocHost(pointer, 2384040000L);
gives:
java.lang.IllegalArgumentException: capacity < 0: (-1910927296 < 0)
at java.base/java.nio.Buffer.createCapacityException(Buffer.java:256)
at java.base/java.nio.Buffer.<init>(Buffer.java:220)
at java.base/java.nio.ByteBuffer.<init>(ByteBuffer.java:281)
at java.base/java.nio.ByteBuffer.<init>(ByteBuffer.java:289)
at java.base/java.nio.MappedByteBuffer.<init>(MappedByteBuffer.java:90)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:158)
at jcuda.runtime.JCuda.cudaMallocHostNative(Native Method)
at jcuda.runtime.JCuda.cudaMallocHost(JCuda.java:4313)
After digging around this seems to be caused by the fact that although cudaMallocHost takes a long size argument (which corresponds to the native cuda size_t), Java Buffers only have integer capacity. When it tries to create a Java DirectByteBuffer with the JNI NewDirectByteBuffer call the size is cast back to an int. The capacity then ends up overflowing to a negative value, causing the error.
Currently the workaround for me seems to be splitting the allocation into multiple chunks, so as to keep each one below the limit. Some options I can think of to deal with this issue in JCuda are (in increasing order of difficulty):
- Update the documentation to list as a known limitation
- Check for values larger than the limit and give a more descriptive error/warning
- Avoid using Java Buffers, instead use some custom class which wraps the pinned memory and supports indexing with a long value etc
Please let me know if I've just made a silly mistake somewhere. Thanks for your time, I appreciate all the work that goes into this :)
Edit: just found this reply to a forum post, indicating that you might be aware of the issue
Activity
jcuda commentedon Aug 3, 2022
I think the linked forum question mainly refers to
cudaHostAllochaving some limits on the CUDA side. Even if it accepts alongvalue, and even if you have 32GB of RAM, this does not necessarily mean that you can allocate 16GB at once. (The exact limit is not specified, IIRC, but that thread is a few years old, and I might need a refresher here).Beyond that, I'm generally aware of the difficulties related to "memory sizes" and the different sizes of
size_tandint. I usually try to handle that insofar that asize_torlongis usually translated to a Javalong(which should cover most memory sizes).But there still are some caveats in the interoperation with Java. And what you just mentioned is such a caveat. The first options (add documentation, and do a sanity check) are things that I'll certainly consider.
What you suggested as o 8934 ption 3 could be broken down into two steps:
sun.misc.Unsafe#allocateMemory: It could return thelong addressfrom the internal allocation, and users could have fun withUnsafeand manipulate the data directly, as they wantlong address, there could be some sort of convenience class to access this memoryNow, one reason for me to have used the
ByteBufferwas exactly that this is such a "convenience class". Currently, the main (or rather the only) way to access the data that is allocated withcudaMallocHostis by obtaining the byte buffer viaPointer#getByteBuffer. And I think that having the option to writepointer.getByteBuffer().asFloatBuffer().put(myJavaFloatArray)is important.
It is true that this limits the size of the host allocation, due to the limitation of
ByteBufferthat you linked to in the JDK code, and in its interface in general. But every change that would avoid the use ofByteBufferwould require some larger restructuring here. (It might be possible to solve that in a backward compatible way, but it may not be entirely trivial).I might consider to expose the "address" from
cudaMallocHostmore directly. (It could just be somepublic static long cudaMallocHostAndReturnAddress, FWIW....). But would hesitate to start writing any sort of "convenience class" on top of that. It would either be "inconvenient" (similar toUnsafe), or an attempt to emulate the roughly 100...Buffer...classes in thejava.niopackage that offer all the sorts of "convenience" that users may want...If you strongly need that 'raw address', I could probably add such a function in the next release. But for now, the workaround of breaking the allocation into chunks probably makes more sense.
LlamaLamp commentedon Aug 3, 2022
Yep I was mainly referring to your brief aside at the bottom of the reply that I linked, where you mentioned the fact that Java arrays etc are indexed with integers.
But yes I agree with what you're saying, I can see some merit in being able to access the address directly and using
Unsafefrom there, however at this point I think I will just stick with chunking the allocation.ByteBufferand its related classes are definitely convenient to use, and trying to reinvent them to work with long indexing would be a big task, for rather limited gain.Happy to stick with the workaround for now, just thought I would point out the issue in case I was missing some easy way to fix it. Thanks again for your time!
jcuda commentedon Aug 3, 2022
Websearches indicate that there are some attempts to provide a
BigByteBufferwithlongindices, but there doesn't seem to be a standard solution. There's also the question of what one could do with such a buffer on the Java side. Iterating over them with a good oldfor-loop can hardly be the goal. So there would have to be some sort of "bulk" operations that either copy their contents into arrays or buffers that are then processed further, or (preferably) methods that offer views on the data with the standardByteBufferinterface - as inByteBuffer chunk = bigByteBuffer.slice(longStartIndex, longEndIndex);In any case, the point about checking the limit in
cudaMallocHostand documenting it is a valid one. I'll try to do that for the next release.LlamaLamp commentedon Aug 4, 2022
Yeah it looks like there were some
BigByteBufferproposals in the past that didn't end up being brought to life unfortunately.For my current use case I am in fact just iterating over the buffer with a
forloop to directly generate the data to send to the GPU, as I have no need to hold that data in a Java array first (so I'm saving memory space, and time by avoiding the extra memory copy). So theforloop method of access is certainly useful. The bulk operations and views on the data that you mention would definitely make life easier in some cases, but I wouldn't say they are absolutely essential.But yes for now the documentation/limit checking would be great thanks :)