linux c++ synchronization method both inter and intra process - c++

Question

The question is brought up when I developing a registry system (c/c++, 2.6.32-642.6.2.el6.x86_64 #1 SMP) used to bookmark information for each database, which requires locking for both inter and intra process. Normally, lockf(), flock(), fcntl() are obvious candidates for the inter process locking, but then I find out that they do not work as expected for intra-process locking(multi threads in same process).
I tested it using the following program:
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <fcntl.h> /* For O_RDWR */
#include <unistd.h> /* For open(), creat() */
#include <errno.h>
int counter = 0;
void* counterThread(void* ptr)
{
int lockfd = 0;
int tmpCounter = 0;
lockfd = open("/tmp/lockfile.txt", O_CREAT|O_WRONLY, 0666);
if(lockfd == -1)
{
printf("lockfile could not be created, errno:%d\n", errno);
return NULL;
}
if(lockf(lockfd, F_LOCK, 0) == -1)
{
printf("lockfile could not be locked, errno:%d\n", errno);
return NULL;
}
counter++;
tmpCounter = counter;
if(lockf(lockfd, F_ULOCK, 0) == -1)
{
printf("lockfile could not be unlocked, errno:%d\n", errno);
return NULL;
}
close(lockfd);
printf("counter is %d, lockfile is %d\n", tmpCounter, lockfd);
}
int main()
{
int threadNum = 30000;
pthread_t threads[30000];
int i = 0;
int rv = 0;
for(; i < threadNum; i++)
{
rv = pthread_create( &threads[i], NULL, &counterThread, NULL);
if(rv != 0)
{
printf("failed to create pthread %d\n", i);
return -1;
}
}
for(i = 0; i < threadNum; i++)
pthread_join(threads[i], NULL);
return 0;
}
The output would be:
counter is 1, lockfile is 4
counter is 2, lockfile is 3
counter is 3, lockfile is 5
counter is 4, lockfile is 6
counter is 7, lockfile is 4
...
counter is 29994, lockfile is 3
counter is 29995, lockfile is 3
counter is 29996, lockfile is 3
counter is 29997, lockfile is 3
counter is 29998, lockfile is 3
The output sequence is random and sometimes missing some numbers inside, meaning there is definitely a race condition happening. I think the reason is probably that fd opened for the same file in the same process is somehow optimized to be reused. Because all these locking mechanism is implemented in granularity of fd, the locking does not work in this case.
Given the background, I would like to ask the following question:
Is there any means I could force open to return different fd for different threads to same process to make the locking works?
Is there any good practice or convenient API in Linux to do both inter and intra process locking? What I could think of is the following means to implement it(not verified yet), but I would like to know some easier ways:
(1) Implement mutex and semaphore to serialize the access to all these lockfile APIs for the critical resources
(2) shm_open a shared memory, mmap it in different processes and add semaphore/mutex inside to lock the critical resources
Thanks in advance:)