I am using a script provided by microsoft (bitonic_start_sort_cs.hlsl) but it gives me the following error:
ambiguous type for bit shift; use a type suffix on literal values, like 'L' or 'U', or a cast [-Werror,-Wambig-lit-shift] (line 39, line 53)
It is not documented anywhere, as far as I know, and I have no idea how to fix it.
The script:
(The errors can be seen on line 39 and line 53)
#include "cs.hlsli"
#include "bitonic_sort_rs.hlsli"
#include "indirect.hlsli"
ConstantBuffer<bitonic_sort_cb> cb : register(b0);
StructuredBuffer<uint> counterBuffer : register(t0);
RWStructuredBuffer<D3D12_DISPATCH_ARGUMENTS> dispatch : register(u0);
uint nextPowerOfTwo(uint i)
{
uint mask = (1 << firstbithigh(i)) - 1; // ERROR 1 HERE
return (i + mask) & ~mask;
}
[numthreads(22, 1, 1)]
[RootSignature(BITONIC_SORT_RS)]
void main(cs_input IN)
{
if (IN.groupIndex >= cb.maxNumIterations)
{
return;
}
uint listCount = counterBuffer[cb.counterOffset];
uint k = 2048 << IN.groupIndex; // ERROR 2 HERE
// We need one more iteration every time the number of thread groups doubles.
if (k > nextPowerOfTwo((listCount + 2047) & ~2047))
{
listCount = 0;
}
uint offset = IN.groupIndex * (IN.groupIndex + 1) / 2;
// Generate outer sort dispatch arguments.
for (uint j = k / 2; j > 1024; j /= 2)
{
// All of the groups of size 2j that are full.
uint completeGroups = (listCount & ~(2 * j - 1)) / 2048;
// Remaining items must only be sorted if there are more than j of them.
uint partialGroups = ((uint)max(int(listCount - completeGroups * 2048 - j), 0) + 1023) / 1024;
D3D12_DISPATCH_ARGUMENTS arguments = { completeGroups + partialGroups, 1, 1 };
dispatch[offset] = arguments;
++offset;
}
// The inner sort always sorts all groups (rounded up to multiples of 2048).
D3D12_DISPATCH_ARGUMENTS arguments = { (listCount + 2047) / 2048, 1, 1 };
dispatch[offset] = arguments;
}
I tried removing the 2 lines that gave me the error but it gave more errors.
I also tried changing the C++ Language Standard in Visual Studio but that didn't fix it (It's set to C++ 17)
I also tried compiling the code both in Visual studio 2022 and Visual Studio 2019, but that didn't change anything either.
And changing the compiling mode (Release, Debug, 32bit, 64bit) did nothing as well.
There's not much I could have done as I never had this error before and I can't find any documentation or forum question that has had it too, so I have no idea how to fix it.
I'm pretty new to shaders and hlsl, so of course I have no idea what the cause and the fix for this is. All I did was use Microsoft's own script they provided in their example.
I have been having trouble converting from a single character to an integer while in the host function of my CUDA program. After the line -
token[j] = token[j] * 10 + (buf[i] - '0' );
I use cuda-gdb check the value for token[j], and I always get different numbers that do not seem to have a pattern. I have also tried simple casting, not multiplying by ten (which I saw in another thread), not subtracting '0', and I always seem to get a different result. Any help would be appreciated. This is my first time posting on stack overflow, so give me a break if my formatting is awful.
-A fellow struggling coder
__global__ void rread(unsigned int *table, char *buf, int *threadbytes, unsigned int *token) {
int i = 0;
int j = 0;
*token = NULL;
int tid = threadIdx.x;
unsigned int key;
char delim = ' ';
for(i = tid * *threadbytes; i <(tid * *threadbytes) + *threadbytes ; i++)
{
if (buf[i] != delim) { //check if its not a delim
token[j] = token[j] * 10 + (buf[i] - '0' );
There's a race condition on writing to token.
If you want to have a local array per block you can use shared memory. If you want a local array per thread, you will need to use local per-thread memory and declare the array on the stack. In the first case you will have to deal with concurrency inside the block as well. In the latter you don't have to, although you might potentially waste a lot more memory (and reduce collaboration).
I'm working on Solaris 5.8, C++, using the Json parser.
The problem is: while parsing a file of size greater than 700 MB, the process crashes with core dump error.
It roughly occurs at below code point -
int printbuf_memappend(struct printbuf *p, char *buf, int size)
{
char *t;
if(p->size - p->bpos <= size)
{
int new_size = json_max(p->size * 2, p->bpos + size + 8);
if (!(t = realloc(p->buf, new_size)))
return -1;
p->size = new_size;
p->buf = t;
}
memcpy(p->buf + p->bpos, buf, size); // CORE DUMP HERE
p->bpos += size;
p->buf[p->bpos]= '\0';
return size;
}
Could you please help to identify the problem? The core dump file contain only the data being copied. Can increase of RAM be a solution ? Or do I need to limit the file size to 700MB ?
If crash happened in memcpy, you have two variants
something wrong with input or output.
To test the second variant add memset after realloc:
int new_size = json_max(p->size * 2, p->bpos + size + 8);
if (!(t = realloc(p->buf, new_size)))
return -1;
p->size = new_size;
p->buf = t;
memset(p->buf + p->bpos, 0, size);
On Linux (depend on configuration) possible to allocate not existing virtual memory.
The real allocation happens after the first usage. May the same happens on your Solaris? relloc return ok, but system really have no enought memory? memset should give answer to this question.
This is a printing thread that prints the statistic of my currently running program
void StatThread::PrintStat(){
clock_t now = 0;
UINT64 oneMega = 1<<20;
const char* CUnique = 0;;
const char* CInserted = 0;;
while((BytesInserted<=fileSize.QuadPart)&&flag){
Sleep(1000);
now = clock();
CUnique = FormatNumber(nUnique);
CInserted = FormatNumber(nInserted);
printf("[ %.2f%%] %u / %u dup %.2f%% # %.2fM/s %.2fMB/s %3.2f%% %uMB\n",
(double)BytesInserted*100/(fileSize.QuadPart),
nUnique,nInserted,(nInserted-nUnique)*100/(double)nInserted,
((double)nInserted/1000000)/((now - start)/(double)CLOCKS_PER_SEC),
((double)BytesInserted/oneMega)/((now - start)/(double)CLOCKS_PER_SEC),
cpu.GetCpuUtilization(NULL),cpu.GetProcessRAMUsage (true));
if(BytesInserted==fileSize.QuadPart)
flag=false;
}
delete[] CUnique; //would have worked with memory leak if commented out
delete[] CInserted; // crash at here! heap corruption
}
This is FormatNumber that returns a pointer to a char array
const char* StatThread::FormatNumber(const UINT64& number) const{
char* result = new char[100];
result[0]='\0';
_i64toa_s(number,result,100,10);
DWORD nDigits = ceil(log10((double)number));
result[nDigits] = '\0';
if(nDigits>3){
DWORD nComma=0;
if(nDigits%3==0)
nComma = (nDigits/3) -1;
else
nComma = nDigits/3;
char* newResult = new char[nComma+nDigits+1];
newResult[nComma+nDigits]='\0';
for(DWORD i=1;i<=nComma+1;i++){
memcpy(newResult+strlen(newResult)-i*3-(i-1),result+strlen(result)-i*3,3);
if(i!=nComma+1){
*(newResult+strlen(newResult)-4*i) = ',';
}
}
delete[] result;
return newResult;
}
return result;
}
What is really weird was that it crashed only in release mode because of a heap corruption, but run smoothly in debug mode. I've already checked everywhere and found no obvious memory leaks, and even Memory Leak Detector said so too.
Visual Leak Detector Version 2.2.3 installed.
The thread 0x958 has exited with code 0 (0x0).
No memory leaks detected.
Visual Leak Detector is now exiting.
The program '[5232] Caching.exe' has exited with code 0 (0x0).
However, when run in release mode,it threw an error that said my program stop working and I clicked on debug, it pointed to the line that caused the heap corruption.
The thread 0xe4c has exited with code 0 (0x0).
Unhandled exception at 0x00000000770E6AE2 (ntdll.dll) in Caching.exe: 0xC0000374: A heap has been corrupted (parameters: 0x000000007715D430).
If I commented out this line, it worked fine but Memory Leak Detector would have complained about memory leak! I don't understand how to cause a heap corruption when there was no memory leaks, (at least that's what the Leak Detector said). Please help, Thank you in advance.
Edit:
Heap corruption was fixed, because in the very last iteration, I still copied 3 byes to the front instead of whatever is leftover. Thank you all for helps!
const char* StatThread::FormatNumber(const UINT64& number) const{
char* result = new char[100];
result[0]='\0';
_ui64toa_s(number,result,100,10);
DWORD nDigits = (DWORD)ceil(log10((double)number));
if(number%10==0){
nDigits++;
}
result[nDigits] = '\0';
if(nDigits>3){
DWORD nComma=0;
if(nDigits%3==0)
nComma = (nDigits/3) -1;
else
nComma = nDigits/3;
char* newResult = new char[nComma+nDigits+1];
DWORD lenNewResult = nComma+nDigits;
DWORD lenResult = nDigits;
for(DWORD i=1;i<=nComma+1;i++){
if(i!=nComma+1){
memcpy(newResult+lenNewResult-4*i+1,result+lenResult-3*i,3);
*(newResult+lenNewResult-4*i) = ',';
}
else{
memcpy(newResult,result,lenNewResult-4*(i-1));
}
}
newResult[nComma+nDigits] = '\0';
delete[] result;
return newResult;
}
return result;
}
Sorry to be blunt, but the code to "format" a string is horrible.
First of all, you pass in an unsigned 64-bit int value, which you formatted as a signed value instead. If you claim to sell bananas, you shouldn't give your customers plantains instead.
But what's worse is that what you do return (when you don't crash) isn't even right. If a user passes in 0, well, then you return nothing at all. And if a user passes in 1000000 you return 100,000 and if he passes in 10000000 you return 1,000,000. Oh well, what's a factor of 10 for some numbers between friends? ;)
These, along with the crash, are symptoms of the crazy pointer arithmetic your code does. Now, to the bugs:
First of all, when you allocate 'newResult' you leave the buffer in a very weird state. The first nComma + nDigits bytes are random values, followed by a NULL. You then call strlen on that buffer. The result of that strlen can be any number between 0 and nComma + nDigits, because any one of the nComma + nDigit characters may contain the null byte, which will cause strlen to terminate prematurely. In other words, the code is non-deterministic after that point.
Sidenote: If you're curious why it works in debug builds, it's because the compiler and the debug version of the runtime libraries try to help you catch bugs by initializing memory for you. In Visual C++ the fill mask is usually 0xCC. This made sure that the bug in your strlen() was covered up in debug builds.
Fixing that bug is pretty simple: simply initialize the buffer with spaces, followed by a NULL.
char* newResult = new char[nComma+nDigits+1];
memset(newResult, ' ', nComma+nDigits);
newResult[nComma+nDigits]='\0';
But there's one more bug. Let's try to format the number 1152921504606846975 which should become 1,152,921,504,606,846,975. Let's see what some of fancy pointer arithmetic operations give us:
memcpy(newResult + 25 - 3 - 0, result + 19 - 3, 3)
*(newResult + 25 - 4) = ','
memcpy(newResult + 25 - 6 - 1, result + 19 - 6, 3)
*(newResult + 25 - 8) = ','
memcpy(newResult + 25 - 9 - 2, result + 19 - 9, 3)
*(newResult + 25 - 12) = ','
memcpy(newResult + 25 - 12 - 3, result + 19 - 12, 3)
*(newResult + 25 - 16) = ','
memcpy(newResult + 25 - 15 - 4, result + 19 - 15, 3)
*(newResult + 25 - 20) = ','
memcpy(newResult + 25 - 18 - 5, result + 19 - 18, 3)
*(newResult + 25 - 24) = ','
memcpy(newResult + 25 - 21 - 6, result + 19 - 21, 3)
As you can see, your very last operation copies data 2 bytes before the beginning of the buffer you allocated. This is because you assume that you will always be copying 3 characters. Of course, that's not always the case.
Frankly, I don't think your version of FormatNumber should be fixed. all that pointer arithmetic and calculations are bugs waiting to happen. Here's the version I wrote, which you can use if you want. I consider it much more sane, but your mileage may vary:
const char *StatThread::FormatNumber(UINT64 number) const
{
// The longest 64-bit unsigned integer 0xFFFFFFFF is equal
// to 18,446,744,073,709,551,615. That's 26 characters
// so our buffer will be big enough to hold two of those
// although, technically, we only need 6 extra characters
// at most.
const int buflen = 64;
char *result = new char[buflen];
int cnt = -1, idx = buflen;
do
{
cnt++;
if((cnt != 0) && ((cnt % 3) == 0))
result[--idx] = ',';
result[--idx] = '0' + (number % 10);
number = number / 10;
} while(number != 0);
cnt = 0;
while(idx != buflen)
result[cnt++] = result[idx++];
result[cnt] = 0;
return result;
}
P.S.: The "off by a factor of 10" thing is left as an exerise to the reader.
At the line
DWORD nDigits = ceil(log10((double)number));
you need three digits for 100 but log 100 = 2. This means that you allocating one too few characters for char* newResult = new char[nComma+nDigits+1];. This means that the end of your heap cell is being overwritten which is resulting in the heap corruption you are seeing. Debug heap allocation may be more forgiving which is why the crash is only in debug mode.
Heap corruption is usually caused by overwriting the heap data structures. There is a lot of use of "result" and "newResult" without good boundary checking. When you do a debug build, the whole alignment changes and by chance the error doesnt happen.
I would start by adding checks like this:
DWORD nDigits = ceil(log10((double)number));
if(nDigits>=100){printf("error\n");exit(1);}
result[nDigits] = '\0';
Two things in your StatThread::PrintStat function.
This is a memory leak if the loop body executes more than once. You would reassign these pointers without calling delete[] for the previous values.
while((BytesInserted<=fileSize.QuadPart)&&flag){
...
CUnique = FormatNumber(nUnique);
CInserted = FormatNumber(nInserted);
...
}
Is this supposed to be an assignment = or a comparison ==?
if(BytesInserted=fileSize.QuadPart)
flag=false;
Edit to add:
In your StatThread::FormatNumber function this statement adds a null terminator to the end of the block but the previous chars may contain garbage (new doesn't zero allocated memory). The subsequest calls to strlen() may return an unexpected length.
newResult[nComma+nDigits]='\0';
Good morning, I am trying to write a cMemoryMappedFile class that deals with large files that cannot be mapped to one view in Win32. The code is at the pastebin url.
In GetPointer(), we are trying to carefully map and unmap file regions as they are needed. We are encountering an access violation in GetPointer() on the line marked with the comment // crashes here if adjusted ptr is bad ptr.
The test program is:
int TotalKeys = 2709783;
try {
mmapFile = new cMemoryMappedFile(TotalKeys * 54 + 1);
}
catch (cException e) {
printf("catch cMemoryMappedFile\n");
throwl
}
for (i = 0; i < 50000; i++) {
mmapFile->GetPointer(i * 54);
}
When i equals 28521 and i * 54 = 1540134, we get an access violation reading location 0x0024000. The MEM_BLOCK_SIZE is on the 64K alignment required by MapViewOfFile, MEM_BLOCK_SIZE = 65536 * 2. We are running Windows XP Professional Version 2002 Service Pack 3, 32 bit version. What is puzzling is that we are trying the move the map view within the file mapping. Please suggest the cause of the access violation if you wish.
Thank you.
This calculation seems overly complicated.
char * cMemoryMappedFile::GetPointer(int n){
unsigned int baseoff;
if( n < MEM_BLOCK_SIZE / 2)
{
baseoff = 0;
}
else
{
baseoff = ((n + MEM_BLOCK_SIZE / 4) &
(~(MEM_BLOCK_SIZE / 2 - 1))) - MEM_BLOCK_SIZE / 2;
}
...
As I understand, n is the desired offset into the file, and baseoff is the first address we'll actually map. We want the largest baseoff <= n, where baseoff is a multiple of MEM_BLOCK_SIZE. So how about:
unsigned int baseoff = (n / MEM_BLOCK_SIZE) * MEM_BLOCK_SIZE;
Later:
MapPtr = (char*)::MapViewOfFile(hMapping, FILE_MAP_WRITE | FILE_MAP_READ, 0,
baseoff, mappedlength);
You should check the result of MapViewOfFile.
It's not clear what you want the return value to be. I'm assuming adjustedptr + n should point to the originally requested byte. But then you try to access *adjustedptr, which is not guaranteed to be part of the view when the base offset is larger than MEM_BLOCK_SIZE. I think you want the printf to use *(adjustedptr + n) or adjustedptr[n]. Or maybe you're trying to access the first byte actually mapped, which would be adjustedptr[baseoff].
I fixed the access violation and the Microsoft operator delete error. The modified code is in the www.pastebin.com url.<iframe src="http://pastebin.com/embed_iframe.php?i=4dsVfkzg" style="border:none;width:100%"></iframe> > I wish to thank Adrian
McCarthy and Andre Caron for their help.