Linux: Is writeback related to number of files? - c++

I am trying to do some experiments of pagecache and writeback mechanism on linux 5.4.81. I have set dirty_ratio to be 50, dirty_background_ratio 45, dirty_expire_centisecs 500000000, dirty_writeback_centisecs 500000000. When I write a big file of 250GB using write(), I can find that writeback begin to work when dirty_pages reach 45% of total memory, which is in line with expectation.
But when I write 125 small files of 2GB, the writeback begin to work when dirty_pages only reach about 25%. I wonder why this happen?
The total memory of my platform is 256G.
#include <stdio.h>
#include <string>
#include <sys/time.h>
#include <sstream>
#include <fstream>
#include <iostream>
#include <fcntl.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/stat.h>
using namespace std;
int main()
{
struct timeval start, end;
long long total_time = 0;
ofstream outfile("write_time.txt",ios::trunc);
int loop=1;
while(loop--){
for(int i=0; i<125; i++){
stringstream ss;
string num;
ss<<i;
ss>>num;
string a = "test";
string b = ".dat";
string fileName = a+num+b;
const char*tmp=fileName.c_str();
int fp = open(tmp,O_CREAT|O_RDWR,S_IRUSR|S_IWUSR);
// int fp = open(tmp,O_WRONLY|O_TRUNC);
int pos = 0;
char data[1024] = "ab\n";
gettimeofday( &start, NULL );
while (1)
{
write(fp, data, 1024);
pos++;
if (pos >= 2*1024*1024)
break;
}
gettimeofday( &end, NULL );
int timeuse = 1000000 * ( end.tv_sec - start.tv_sec ) + end.tv_usec - start.tv_usec;
printf("write_time: %d us\n", timeuse);
total_time +=timeuse;
outfile<<timeuse<<endl;
}
}
outfile<<total_time;
outfile.close();
return 0;
}

Related

How to benchmark my system using multiple threads in c++?

I have simplified my code, and it compiles, but it doesn't do anything. It doesn't error out though either. I am trying to get 7 threads (on my 8-core processor) in this example to write to a variable to benchmark my system. I would like to do this with multiple threads to see if it's faster. It's based off other code that worked before I added multithreading. When I run, it just terminates. It should show progress each second of how many total iterations all the threads have done together. Some of the includes are there from other code I am working on.
I would like to also gracefully terminate all 7 threads when Ctrl-C is pressed. Help would be appreciated. Thanks!
//Compiled using: g++ ./test.cpp -lpthread -o ./test
#include <stdio.h>
#include <string>
#include <iostream>
#include <time.h>
#include <ctime>
#include <ratio>
#include <chrono>
#include <iomanip>
#include <locale.h>
#include <cstdlib>
#include <pthread.h>
using namespace std;
using namespace std::chrono;
const int NUM_THREADS = 7;
const std::string VALUE_TO_WRITE = "TEST";
unsigned long long int total_iterations = 0;
void * RunBenchmark(void * threadid);
class comma_numpunct: public std::numpunct < char > {
protected: virtual char do_thousands_sep() const {
return ',';
}
virtual std::string do_grouping() const {
return "\03";
}
};
void * RunBenchmark(void * threadid) {
unsigned long long int iterations = 0;
std::string benchmark;
int seconds = 0;
std::locale comma_locale(std::locale(), new comma_numpunct());
std::cout.imbue(comma_locale);
auto start = std::chrono::system_clock::now();
auto end = std::chrono::system_clock::now();
do {
start = std::chrono::system_clock::now();
while ((std::chrono::duration_cast < std::chrono::seconds > (end - start).count() != 1)) {
benchmark = VALUE_TO_WRITE;
iterations += 1;
}
total_iterations += iterations;
iterations = 0;
cout << "Total Iterations: " << std::setprecision(0) << std::fixed << total_iterations << "\r";
} while (1);
}
int main(int argc, char ** argv) {
unsigned long long int iterations = 0;
int tc, tn;
pthread_t threads[NUM_THREADS];
for (tn = 0; tn < NUM_THREADS; tn++) {
tc = pthread_create( & threads[tn], NULL, & RunBenchmark, NULL);
}
return 0;
}

Trim an audio file that is labelled with a timestamp c++

So i have a program that records snippets of audio on a microphone array, it then timestamps the end time of the file. My problem is that the process of the program starting the recording takes a random amount of time to start and so the audio is a random length. I want a way to read the size of the file (kilobytes) then trim the file by a certain number of kilobytes so that the recordings are always the same length.
#include <stdio.h>
#include <time.h>
#include <math.h>
#include <stdlib.h>
#include <chrono>
#include <thread>
#include<signal.h>
#include<unistd.h>
#include <sys/time.h>
#include <unistd.h>
#include <sys/wait.h>
#include <sys/types.h>
#include "zlib.h"
long long int rem;
long long int ms1;
long long int unixtime;
using namespace std;
const char* Filenames;
int main(void) {
int system(const char *command);
int x;
struct timeval tp1;
rem = 5;
while (rem != 0) {
gettimeofday(&tp1, NULL);
ms1 = tp1.tv_sec * 1000ll + tp1.tv_usec / 1000ll;
rem = ms1 % 10000;
}
for (x=0; x<3; x++){
pid_t pid=fork();
if (pid==0){
execl("/home/pi/odas/bin/odaslive", "/home/pi/odas/bin/odaslive", "-vc", "/home/pi/odas/config/odaslive/matrix_creator.cfg", (char *)NULL);
exit(127);
} else {
std::this_thread::sleep_for(std::chrono::milliseconds(15000));
kill(pid, SIGINT);
}
gettimeofday(&tp1, NULL);
unixtime = tp1.tv_sec + tp1.tv_usec / 1000000ll;
std::string name1 = "/home/pi/matrix_creator_explore/postfiltered/postfiltered1_";
std::string name2 = ".raw";
std::string result1;
result1 = name1 + std::to_string(unixtime) + name2;
const char *cstr = result1.c_str();
rename ("/home/pi/matrix_creator_explore/postfiltered.raw", cstr);
std::string name3 = "/home/pi/matrix_creator_explore/tracked/tracked1_";
std::string name4 = ".raw";
std::string result2;
result2 = name3 + std::to_string(unixtime) + name4;
const char *cstr1 = result2.c_str();
rename ("/home/pi/matrix_creator_explore/tracked.raw", cstr1);
struct stat buf;
stat( cstr,&buf);
printf ("\n %i \n", buf.st_size);
}
}

Segmentation fault (core dumped), storing char * to string vector of struct

#include <iostream>
#include <fstream>
#include <string.h>
#include <string>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <semaphore.h>
#include <sys/shm.h>
#include <sys/sem.h>
#include <sys/ipc.h>
#include <vector>
#include <sstream>
#define SHMSIZE 1024
using namespace std;
namespace patch
{
template < typename T > std::string to_string( const T& n )
{
std::ostringstream stm ;
stm << n ;
return stm.str() ;
}
}
struct process
{
int r;
string name;
vector<string> lines;
};
int main(int argc, char * argv[])
{
int firstRun = 1; //Skipping First Line of Assign-1.ip.
int quantum = 0; //For taking input of quantum.
int count = 0; //For number of processes.
int pchtoint;
string c;
char * pch; //For tokenization.
string reading_file; //Reading a line from file.
char * readarr; //Converting "reading_file" to readarr for tokenization.
process * p;
//=== Quantum Input ===//
cout<<"Enter Quantum size [1-1000]: ";
cin>>quantum;
while(quantum < 1 || quantum > 1000)
{
cout<<"Wrong input!!! Enter Again [1-1000]: ";
cin>>quantum;
}
//=====================//
//===Filing===//
ifstream read("Assign-2.ip");
if(read.is_open())
{
while(!read.eof())
{
getline(read, reading_file);
readarr = new char[reading_file.size() + 1];
for(int i = 0; i < reading_file.length(); i++)
{
readarr[i] = reading_file[i];
}
if(firstRun > 1)
{
int countingline = 0; //counting the number of lines in a process.
pch = strtok (readarr," ,");
while (pch != NULL)
{
c = pch[1];
pchtoint = atoi(c.c_str());
p[pchtoint-1].r++;
p[pchtoint-1].lines.push_back(pch);
for(int i = 0; i < p[pchtoint-1].lines.size(); i++)
cout<<p[pchtoint-1].name<<"=="<<p[pchtoint-1].lines.at(i)<<endl;
pch = strtok (NULL, " ,");
}
}
else
{
pch = strtok (readarr,",.-");
while (pch != NULL)
{
count++;
pch = strtok (NULL, ",.-");
}
p = new process[count];
string s = "p";
for(int i = 0; i < count; i++)
{
s = s + patch::to_string(i+1);
p[i].name = s;
s = s[0];
}
firstRun++;
}
}
}
else
{
cout<<"Cannot open file!!!"<<endl;
}
read.close();
return 0;
}
Enter Quantum size [1-1000]: 2
p1==p1-l1
p2==p2-l1
p3==p3-l1
p1==p1-l1
p1==p1-l2
p2==p2-l1
p2==p2-l2
p3==p3-l1
p3==p3-l2
p1==p1-l1
p1==p1-l2
p1==p1-l3
p3==p3-l1
p3==p3-l2
p3==p3-l3
p1==p1-l1
p1==p1-l2
p1==p1-l3
p1==p1-l4
Segmentation fault (core dumped)
I am reading data from a cvs file. and storing it in struct that is p here. but I don't know why it is giving segmentation fault. I am compiling it on ubuntu terminal.
The input file contains data:
P1, P2, P3,
p1-l1, p2-l1, p3-l1
p1-l2, p2-l2, p3-l2
p1-l3, , p3-l3
p1-l4, ,

XDR serializing variable lenght array of string

I am serializing a packet over XDR but i do not understand how to provide vector of string. I have here a small fully working serialization / deserialization for a std::vector of uint64_t. Here my code:
Serializer:
#include <stdio.h>
#include <iostream>
#include <rpc/rpc.h>
#include <vector>
#define MAX_LENGTH_ 100
int main(void)
{
XDR xdr;
xdrstdio_create(&xdr, stdout, XDR_ENCODE);
std::vector<uint64_t> ids; // vector i want to send
ids.push_back(1);
ids.push_back(2);
ids.push_back(3);
// serializing the vector
uint64_t *_ids = &ids[0];
uint32_t size = ids.size();
xdr_array(&xdr,(char**)(&_ids), &size, MAX_LENGTH_,sizeof(uint64_t),(xdrproc_t)xdr_u_long);
return 1;
}
Deserializer:
#include <stdio.h>
#include <iostream>
#include <rpc/rpc.h>
#include <vector>
#define MAX_LENGTH_ 100
int main(void)
{
XDR xdrs;
xdrstdio_create(&xdrs, stdin, XDR_DECODE);
uint64_t *ids_ = new uint64_t[MAX_LENGTH_];
uint32_t size;
bool status = xdr_array(&xdrs,(char**)(&ids_), &size, MAX_LENGTH_,
sizeof(uint64_t), (xdrproc_t)xdr_u_long);
std::vector<uint64_t> ids(ids_,ids_+size);
for(std::vector<uint64_t>::iterator it = ids.begin(); it != ids.end(); ++it)
{
std::cout << *it <<std::endl;
}
return 1;
}
The following code works... running ./serializer | ./deserializer i obtain 1 2 3. Now I do not know how to handle having to serialize std::vector<std::string>. A single string works well using xdr_string.
http://linux.die.net/man/3/xdr_array
Any help would be very much appreciated!
EDIT:
I have tried the following:
Serializer:
#include <stdio.h>
#include <iostream>
#include <rpc/rpc.h>
#include <vector>
#include <algorithm>
#include <cstring>
#define MAX_VECTOR_LENGTH_ 100
#define MAX_STRING_LENGTH_ 50
char *convert(const std::string & s)
{
char *pc = new char[s.size()+1];
std::strcpy(pc, s.c_str());
return pc;
}
int main(void)
{
XDR xdr;
xdrstdio_create(&xdr, stdout, XDR_ENCODE);
std::vector<std::string> messages; // vector i want to send
messages.push_back("this is");
messages.push_back("my string");
messages.push_back("vector test");
// transform the vector to c style
std::vector<char*> messagesCStyle;
std::transform(messages.begin(), messages.end(), std::back_inserter(messagesCStyle), convert);
// serializing the vector
char **_messages = &messagesCStyle[0];
uint32_t size = messages.size();
xdr_array(&xdr,(char**)(&_messages), &size, MAX_VECTOR_LENGTH_ * MAX_STRING_LENGTH_,sizeof(char),(xdrproc_t)xdr_string);
return 1;
}
Deserializer:
#include <stdio.h>
#include <iostream>
#include <rpc/rpc.h>
#include <vector>
#define MAX_VECTOR_LENGTH_ 100
#define MAX_STRING_LENGTH_ 50
int main(void)
{
XDR xdrs;
xdrstdio_create(&xdrs, stdin, XDR_DECODE);
std::vector<char*> messagesCStyle_;
uint32_t size;
bool status = xdr_array(&xdrs,(char**)(&messagesCStyle_), &size, MAX_VECTOR_LENGTH_,
MAX_STRING_LENGTH_, (xdrproc_t)xdr_string);
for(std::vector<char*>::iterator it = messagesCStyle_.begin(); it != messagesCStyle_.end(); ++it)
{
std::cout << *it <<std::endl;
}
return 1;
}
I am pretty sure the code for the Serializer is not best but at least it seams to work. However the deserializer does not!! I think the problem is related to the fact that i do not know how much memory to allocate before calling the xdr_array. Any help?
I made it work:
Encoder:
#include <stdio.h>
#include <iostream>
#include <rpc/rpc.h>
#include <vector>
#include <algorithm>
#include <cstring>
#define MAX_VECTOR_LENGTH_ 100
#define MAX_STRING_LENGTH_ 50
char *convert(const std::string & s)
{
char *pc = new char[s.size()+1];
std::strcpy(pc, s.c_str());
return pc;
}
int main(void)
{
XDR xdr;
xdrstdio_create(&xdr, stdout, XDR_ENCODE);
std::vector<std::string> messages; // vector i want to send
messages.push_back("this is");
messages.push_back("my string");
messages.push_back("vector test");
messages.push_back("this is a relatively long string!!!");
// transform the vector to c style
std::vector<char*> messagesCStyle;
std::transform(messages.begin(), messages.end(),
std::back_inserter(messagesCStyle),
[](const std::string & s){
char *pc = new char[s.size()+1];
std::strcpy(pc, s.c_str());
return pc;
});
// serializing the vector
char **_messages = &messagesCStyle[0];
uint32_t size = messages.size();
xdr_array(&xdr,(char**)(&_messages), &size, MAX_VECTOR_LENGTH_ * MAX_STRING_LENGTH_,sizeof(char*),(xdrproc_t)xdr_string);
return 1;
}
Decoder:
#include <stdio.h>
#include <iostream>
#include <rpc/rpc.h>
#include <vector>
#define MAX_VECTOR_LENGTH_ 100
#define MAX_STRING_LENGTH_ 50
int main(void)
{
XDR xdrs;
uint32_t size;
char** buffer = NULL;
xdrstdio_create(&xdrs, stdin, XDR_DECODE);
bool status = xdr_array(&xdrs, (char**) &buffer, &size, MAX_VECTOR_LENGTH_,
sizeof(char*), (xdrproc_t)xdr_string);
std::cout << "status: " << status << std::endl;
std::cout << "size: " << size << std::endl;
std::vector<std::string> stringMessages_(buffer, buffer + size);
for(std::vector<std::string>::iterator it = stringMessages_.begin(); it != stringMessages_.end(); ++it)
{
std::cout << *it <<std::endl;
}
for (int i = 0; i < size; i++) {
free(buffer[i]);
}
free(buffer);
return 1;
}

In memory Binary Search Vs. Disk based Binary Search

In this program, I am reading "key.pc.db" file and printing its mid value.
#include <fstream>
#include <conio.h>
#include <stdio.h>
#include <time.h>
#include <sys/time.h>
#include <sys/stat.h>
using namespace std;
int main( int argc, char *argv[] )
{
ifstream fp;
int mval;
int sizek;
struct stat filek;
int min, max, mid;
if(stat("key.pc.db", &filek) ==0 )
sizek=filek.st_size;
sizek=sizek/sizeof(int);
min=0;
max=sizek-1;
mid=(min+max)/2;
printf("mid %d ",mid);
fp.open( "key.pc.db", ios::in | ios::binary );
fp.seekg(mid, ios::beg);
fp.read( (char *) &mval, (int) sizeof( int ) );
printf("%d mval ", mval);
getch();
return 1;
}
In this program also I am reading the same file but I am storing the value of the file in an array and then printing the mid value. The mid index for both the program shows the same but the value comes out to be different. Why so?
#include <fstream>
#include <conio.h>
#include <stdio.h>
#include <time.h>
#include <sys/time.h>
#include <sys/stat.h>
using namespace std;
int main( int argc, char *argv[] )
{
ifstream fp;
int index;
int sizek;
int kval;
struct stat filek;
int min, max, mid;
int i=0;
if(stat("key.pc.db", &filek) ==0 )
sizek=filek.st_size;
sizek=sizek/sizeof(int);
int k[sizek];
fp.open( "key.pc.db", ios::in | ios::binary );
fp.read( (char *) &kval, (int) sizeof( int ) );
while( !fp.eof() )
{
k[i++]=kval;
fp.read( (char *) &kval, (int) sizeof( int ) );
}
min=0;
max=sizek-1;
mid=(min+max)/2;
printf(" index %d ", mid);
printf(" kmid %d ", k[mid]);
getch();
return 1;
}
You're seeking to the wrong point in the file in the first case. You should be seeking to mid*sizeof(int) rather than mid.