A WRITEBLK command fails when the item reaches 2GB in size (item is truncated to 2147483647 bytes).
Using cat I was able to create an item larger than 2GB in the same directory, but opening it in UV gave a corrupt (negative) value for STATUS<4> (Number of bytes available to read).
uv 11.1.4
64bit Linux on a VM
64BIT_FILES = 1
You can make the universe files 32 or 64 bit (regardless of the OS). So you can do a FILEINFO call to see if the file is actually 64bit (even if the account is 64bit).
My guess is that there is an File system limitation on the file size. in the Rocket UniVerse documentation (page 927) it says:
If the device runs out of disk
space, WRITEBLK takes the ELSE clause and returns –4 to the STATUS
function.
Generally only 32 bit systems would be the hard limit on 2 GB, but maybe there is some kind of 32 bit process running in our 64 bit virtual machine that is producing the same effect. See here for a few leads: https://unix.stackexchange.com/questions/274380/file-size-limit
Related
Computer:
Processor: Intel Xeon Silver 4114 CPU # 2.19Ghz (2 processors)
Ram: 96 Gb 2666 Hz: 12 - 8 Gb sticks
OS: Windows 10
GPU: None
Hard drive: Samsung MZVLB512HAJQ-000H2 - 512GB M.2 PCIe NVMe
IDE:
Visual Studio 2019
I am including what I am doing in case it is relevant. I am running a visual studio code where I read data off a GSC PCI SIO4B Sync Card 256K. Using the API for this card (Documentation: http://www.generalstandards.com/downloads/GscApi.1.6.10.1.pdf) I read 150 bytes of data at a speed of 100Hz using the code below. That data is then being split into to the message structure my device. I can’t give info on the message structure but the data is then combined into the various words using a union and added to an integer array int Data[100];
Union Example:
union data_set{
unsigned int integer;
unsigned char input[2];
} word;
Example of how the data is read read:
PLX_PHYSICAL_MEM cpRxBuffer;
#define TEST_BUFFER_SIZE 0x400
//allocates memory for the buffer
cpRxBuffer.Size = TEST_BUFFER_SIZE;
status = GscAllocPhysicalMemory(BoardNum, &cpRxBuffer);
status = GscMapPhysicalMemory(BoardNum, &cpRxBuffer);
memset((unsigned char*)cpRxBuffer.UserAddr, 0xa5, sizeof(cpRxBuffer));
// start data reception:
status = GscSio4ChannelReceivePlxPhysData(BoardNum, iRxChannel, &cpRxBuffer, SetMaxBytes, &messageID);
// wait for Rx operation to complete
status = GscSio4ChannelWaitForTransfer(BoardNum, iRxChannel, 7000, messageID, &amount);
if (status)
{
// If we have an error, "bytesTransferred" will contain the number of bytes that we
// actually transmitted.
DisplayErrorMessage(status);
printf("\n\t%04X bytes out of %04X transferred", amount, SetMaxBytes);
}
My issue is that this code works fine and keeps up for around 5 minutes then randomly it stops being able to keep up and the FIFO (first in first out) register on the PCI card begins to fill up faster than the code can process the data. To me this seems like a memory leak issue since the code works fine for a long time, then starts to slow down when nothing has changed as all the code is doing it reading the data off the card. We used to save the data in a really large array but even after removing that we had the same issue.
I am unsure how to figure out exactly what is happening and I'm hopping for a way to determine if there is a memory leak and how to fix it if there is.
It being a data leak is only a guess though and it very well could be something else that is the problem so any out of the box suggestions for diagnosing the problem are also appreciated.
Similar to Paul's answer, but I like to strategically place two (or more) _CrtMemCheckpoint followed by _CrtMemDifference, to cut down the noise.
Memory leaks can be detected and reported on (in Debug builds) by calling the _CrtDumpMemoryLeaks function. When running under the debugger, this will tell you (in the output tab) how many allocations you have at the time that it is called and the file and line number that each was allocated from.
Call this right at the end of your program, after you (think you) have freed all the resources you use. Anything left over is a candidate for being a leak.
I am using FreeRTOS v 10.1.0 , in addition I have downloaded FreeRTOS+FAT from the labs area (160919 release)
I am using an Altera Cyclone V evaluation board and have succesfully ran FreeRTOS projects on the board using the Demo project and the available port for my board as the basis for my own applications.
I have also succesfully mounted a partion on my SD Card and read files from the SD Card and also written files to the SD Card.
My problems begin when I try to read a file bigger than 2K. I am using the following ff_fread command to read from a file I have previous opened and I know to be 5777 bytes long:
ff_fread( &byteBuffer[0],1,5777, pxSourceFile );
What I find is the byte buffer is repetedly populated with the same 2048 bytes, up to the maximum of 5777 bytes. So byteBuffer[0] to byteBuffer[2047] are what I expect but then this data is repeated.
I have also tried to read the data in 512 byte chunks and also in 2048 byte chunks in case the issue was related to a sector boundary (512 byte sector) or a cluster boundary (4 sectors per cluster).
My suspicion is that the issue is in FreeRTOS + Fat as opposed to the Altera code for interfacing with the SD Card. This is becasue when I put a break point in the following function I see that the FreeRTOS+Fat api does actually seem to jump back to the first sector after it has successfully read 4 sectors of data. So it would seem that the Altera Api is returning the data requested by FreeRTOS + FAT.
static int32_t prvReadSd( uint8_t pucDestination,
uint32_t ulSectorNumber,
uint32_t ulSectorCount,
FF_Disk_t pxDisk )
{
int32_t errorCode = alt_sdmmc_read(pucDestination,
ulSectorNumber * 512,
ulSectorCount * 512);
return errorCode
}
Any insights anyone can offer into the issues I am having will be greatly appreciated.
Ok, I have resolved my issue. My apologies for blaming FreeRTOS+FAT, I will explain the issue below just in case others have the same issue.
I had created a 1MB partition on my SD Card which I believed to be Fat16. After trying various things I decided to reformat my sd card using the following command in Linux.
sudo mkdosfs -F 16 /dev/sdc4
linux gave the following warning
WARNING: Not enough clusters for a 16 bit FAT! The filesystem will be
misinterpreted as having a 12 bit FAT without mount option "fat=16".
This prompted me to enable Fat12 support in the FreeRTOS+Fat config file and this fixed my issue.
why GlobalMemoryStatusEx() gives huge total virtual memory.Does it take into account all the page files that can be created?
System details:
Windows 8.1, 64 bit Process, x64 Processor
int main()
{
MEMORYSTATUSEX mex;
mex.dwLength = sizeof (mex);
GlobalMemoryStatusEx(&mex);
std::cout<<mex.ullTotalVirtual<<" "<<mex.ullAvailVirtual;
}
140737488224256 140737478111232
EDIT:
I got same result on Windows 10.I am interested in knowing how this 127 TB figure comes up.Why does the system not take into account that i don't have 127 tb space on my disk?
A 32 bit process on (x64 system) shows only 2gb which is the accessible address limit of a 32 bit process for user mode.Why does it not take into account page files in case of 32 bit process?
Yes. From MSDN:
You can use the GlobalMemoryStatusEx() to determine how much memory your application can allocate without severely impacting other applications.
I am using windwos xp and i want to append data (about 4 MB each time) to a file.
Using 'WriteFileGather', i increase the offset's value of the overlapped struct all the time.
Since the the system call is a 32 bit, the offset value(DWORD) can not exceed 4GB value.
Code sample:
LARGE_INTEGER size;
if (!GetFileSizeEx(_hFile, &size))
{
return;
}
overlapped.Offset = size.QuadPart;
Using the ftell and fseek is not working for me.
Apparently, 'WriteFileGather' requires the overlapped.offset value to be assigned.
What is the way to exceed the 4G limit?
WriteFileGather takes an OVERLAPPED structure from which it gets the offset within the file at which writing starts. This offset is split into a high and a low doubleword (Offset and OffsetHigh), so it's really a 64-bit offset.
So, if you want to use WriteFileGather (personally, I wouldn't), this is no hindrance. You cannot write more than 4GB in one go (but you wouldn't want to do that anyway!), though the total file length or the offset into the file can be a lot larger than that.
It's a late answer, but 64 bit versions of ftell and fseek exist for Windows and Linux. For Windows, it's _ftelli64 and _fseeki64. For Linux, it's ftello64 and fseeko64.
I have this snippet of code:
if ((shmid = shmget(key, 512, IPC_CREAT | 0666)) < 0)
{
perror("shmget");
exit(1);
}
Whenever I set the number any higher than 2048, I get, an error that says:
shmget: Invalid argument
However when I run cat /proc/sys/kernel/shmall, I get 4294967296.
Does anybody know why this is happening? Thanks in advance!
The comment from Jerry is correct, even if cryptic if you haven't played with this stuff: "What about this: EINVAL: ... a segment with given key existed, but size is greater than the size of that segment."
He meant that the segment is already there - these segment are persistent - and it has size 2048.
You can see it among the other ones with:
$ ipcs -m
and you can remove your segment (beware: remove your one only) with:
$ ipcrm -M <key>
After that you should be able to create it larger.
man 5 proc refers to three variables related to shmget(2):
/proc/sys/kernel/shmall
This file contains the system-wide limit on the total number of pages of System V shared memory.
/proc/sys/kernel/shmmax
This file can be used to query and set the run-time limit on the maximum (System V IPC) shared memory segment size that can be created. Shared memory segments up to 1GB are now supported in the kernel. This value defaults to SHMMAX.
/proc/sys/kernel/shmmni
(available in Linux 2.4 and onward) This file specifies the system-wide maximum number of System V shared memory segments that can be created.
Please check you violated none of them. Note that shmmax and SHMMAX are in bytes and shmall and SHMALL are in the number of pages (the page size is usually 4 KB but you should use sysconf(PAGESIZE).) I personally felt your shmall is too large (2**32 pages == 16 TB) but not sure if it is harmful or not.
As for the definition of SHMALL, I got this result on my Ubuntu 12.04 x86_64 system:
$ ack SHMMAX /usr/include
/usr/include/linux/shm.h
9: * SHMMAX, SHMMNI and SHMALL are upper limits are defaults which can
13:#define SHMMAX 0x2000000 /* max shared seg size (bytes) */
16:#define SHMALL (SHMMAX/getpagesize()*(SHMMNI/16))
/usr/include/linux/sysctl.h
113: KERN_SHMMAX=34, /* long: Maximum shared memory segment */