I have written some codes to load 1025 images with OpenCV to process them;these codes are in two versions :
single thread and multi threads; problem is that the results of codes have confused me; because single thread version is faster than multi threads.
What do you think about it?? what's wrong ?
my codes are below.
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <iostream>
#include "ctpl.h"
using namespace std;
using namespace cv;
#define threaded
void loadImage(int id, int param0) {
stringstream stream;
stream << "/home/me/Desktop/Pics/pic (" << param0 << ").jpg";
Mat x = imread(stream.str(), IMREAD_REDUCED_COLOR_8);
}
int main() {
#ifdef threaded
ctpl::thread_pool p(8);
for (int i = 1; i <= 1025; i++) {
p.push(loadImage,i);
}
// for (int i = 0; i < 1025; ++i) {
// pthread_join(threads[i], NULL);
// }
#else
for (int i = 1; i <= 1025; i++) {
stringstream stream;
stream << "/home/me/Desktop/Pics/pic (" << i << ").jpg";
Mat x = imread(stream.str(), IMREAD_REDUCED_COLOR_8);
}
#endif
return 0;
}
Related
Im trying to make multithreaded proxy checker in c++, when I start the threads and lock it all threads wait till the request is finished. I tried to remove the locks but that doesn't help either. Im using the cpr library to make the requests, the documentation can be found here: https://whoshuu.github.io/cpr/advanced-usage.html.
Reproduceable example:
#include <stdio.h>
#include <pthread.h>
#include <iostream>
#include <queue>
#include <mutex>
#include <cpr/cpr.h>
#include <fmt/format.h>
#define NUMT 10
using namespace std;
using namespace fmt;
std::mutex mut;
std::queue<std::string> q;
void* Checker(void* arg) {
while (!q.empty()) {
mut.lock();
//get a webhook at https://webhook.site
string protocol = "socks4";
string proxyformatted = format("{0}://{1}", protocol, q.front());
auto r = cpr::Get(cpr::Url{ "<webhook url>" },
cpr::Proxies{ {"http", proxyformatted}, {"https", proxyformatted} });
q.pop();
mut.unlock();
}
return NULL;
}
int main(int argc, char** argv) {
q.push("138.201.134.206:5678");
q.push("185.113.7.87:5678");
q.push("5.9.16.126:5678");
q.push("88.146.196.181:4153");
pthread_t tid[NUMT]; int i;
int thread_args[NUMT];
for (i = 0; i < NUMT; i++) {
thread_args[i] = i;
pthread_create(&tid[i], NULL, Checker, (void*) &thread_args);
}
for (i = 0; i < NUMT; i++) {
pthread_join(tid[i], NULL);
fprintf(stderr, "Thread %d terminated\n", i);
}
return 0;
}
Thanks in advance.
I suggest to implement a wrapper class for your queue that will hide the mutex.
That class can provide push(std::string s) and bool pop(std::string& s) that returns true and populate s if the queue wasn't empty or false othervise. Then your worker threads can simply loop
std::string s;
while(q.pop(s)) {
...
}
I have simplified my code, and it compiles, but it doesn't do anything. It doesn't error out though either. I am trying to get 7 threads (on my 8-core processor) in this example to write to a variable to benchmark my system. I would like to do this with multiple threads to see if it's faster. It's based off other code that worked before I added multithreading. When I run, it just terminates. It should show progress each second of how many total iterations all the threads have done together. Some of the includes are there from other code I am working on.
I would like to also gracefully terminate all 7 threads when Ctrl-C is pressed. Help would be appreciated. Thanks!
//Compiled using: g++ ./test.cpp -lpthread -o ./test
#include <stdio.h>
#include <string>
#include <iostream>
#include <time.h>
#include <ctime>
#include <ratio>
#include <chrono>
#include <iomanip>
#include <locale.h>
#include <cstdlib>
#include <pthread.h>
using namespace std;
using namespace std::chrono;
const int NUM_THREADS = 7;
const std::string VALUE_TO_WRITE = "TEST";
unsigned long long int total_iterations = 0;
void * RunBenchmark(void * threadid);
class comma_numpunct: public std::numpunct < char > {
protected: virtual char do_thousands_sep() const {
return ',';
}
virtual std::string do_grouping() const {
return "\03";
}
};
void * RunBenchmark(void * threadid) {
unsigned long long int iterations = 0;
std::string benchmark;
int seconds = 0;
std::locale comma_locale(std::locale(), new comma_numpunct());
std::cout.imbue(comma_locale);
auto start = std::chrono::system_clock::now();
auto end = std::chrono::system_clock::now();
do {
start = std::chrono::system_clock::now();
while ((std::chrono::duration_cast < std::chrono::seconds > (end - start).count() != 1)) {
benchmark = VALUE_TO_WRITE;
iterations += 1;
}
total_iterations += iterations;
iterations = 0;
cout << "Total Iterations: " << std::setprecision(0) << std::fixed << total_iterations << "\r";
} while (1);
}
int main(int argc, char ** argv) {
unsigned long long int iterations = 0;
int tc, tn;
pthread_t threads[NUM_THREADS];
for (tn = 0; tn < NUM_THREADS; tn++) {
tc = pthread_create( & threads[tn], NULL, & RunBenchmark, NULL);
}
return 0;
}
This question already has answers here:
c++ multiple definitions of a variable
(5 answers)
multiple definition error c++
(2 answers)
What exactly is One Definition Rule in C++?
(1 answer)
Closed 2 years ago.
I am attempting to compile my c++ code, and I continue getting the error:
/tmp/ccEsZppG.o:(.bss+0x0): multiple definition of `mailboxes'
/tmp/ccEZq43v.o:(.bss+0x0): first defined here
/tmp/ccEsZppG.o:(.bss+0xc0): multiple definition of `threads'
/tmp/ccEZq43v.o:(.bss+0xc0): first defined here
/tmp/ccEsZppG.o:(.bss+0x120): multiple definition of `semaphores'
/tmp/ccEZq43v.o:(.bss+0x120): first defined here
collect2: error: ld returned 1 exit status
Here is my code:
addem.cpp
#include <stdio.h>
#include <iostream>
#include <stdlib.h>
#include <semaphore.h>
#include <pthread.h>
#include "mailbox.h"
using namespace std;
void *sumUp(void *arg);
int main(int argc, char *argv[]) {
int numThreads, minThreads, maxInt, minInt;
if (argc < 3) {
cout << "Error: Need three arguments" << endl;
return 1;
}
numThreads = atoi(argv[1]);
maxInt = atoi(argv[2]);
minThreads = 1;
minInt = 1;
if (numThreads < 1) {
cout << "Cannot work with less than one thread\n"
<< "It's okay but do better next time!\n"
<< "We'll work with 1 thread this time.\n";
numThreads = minThreads;
} else if (numThreads > MAXTHREAD) {
cout << "Sorry, the max for threads is 10.\n"
<< "We'll work with 10 threads this time.\n";
numThreads = MAXTHREAD;
}
if (maxInt < 1) {
cout << "What do you want me to do? I can't count backwards!\n"
<< "I can barely count forwards! Let's make the max number\n"
<< "be 1 to save time\n";
maxInt = minInt;
}
struct msg outgoingMail[numThreads];
int divider = maxInt / numThreads;
int count = 1;
//initialize arrays (mailboxes, semaphores)
for (int i = 0; i < numThreads; i++) {
sem_init(&semaphores[i], 0, 1);
outgoingMail[i].iSender = 0;
outgoingMail[i].type = RANGE;
outgoingMail[i].value1 = count;
count = count + divider;
if (i = numThreads - 1) {
outgoingMail[i].value2 = maxInt;
} else {
outgoingMail[i].value2 = count;
}
}
for (int message = 0; message < numThreads; message++) {
SendMsg(message+1, outgoingMail[message]);
}
int thread;
for (thread = 0; thread <= numThreads; thread++) {
pthread_create(&threads[thread], NULL, &sumUp, (void *)(intptr_t)(thread+1));
}
struct msg incomingMsg;
int total = 0;
for (thread = 0; thread < numThreads; thread++) {
RecvMsg(0, incomingMsg);
total = total + incomingMsg.value1;
}
cout << "The total for 1 to " << maxInt << " using "
<< numThreads << " threads is " << total << endl;
return 0;
}
void *sumUp(void *arg) {
int index,total;
index = (intptr_t)arg;
struct msg message;
RecvMsg(index, message);
message.iSender = index;
message.type = ALLDONE;
total = 0;
for (int i = message.value1; i <= message.value2; i++) {
total += i;
}
SendMsg(0, message);
return (void *) 0;
}
mailbox.cpp
#include <stdio.h>
#include <iostream>
#include "mailbox.h"
using namespace std;
int SendMsg(int iTo, struct msg &Msg) {
if (safeToCall(iTo)) {
cout << "Error calling SendMsg" << endl;
return 1;
}
sem_wait(&semaphores[iTo]);
mailboxes[iTo] = Msg;
sem_post(&semaphores[iTo]);
return 0;
}
int RecvMsg(int iFrom, struct msg &Msg) {
sem_wait(&semaphores[iFrom]);
if (safeToCall(iFrom)) {
cout << "Error calling RecvMsg" << endl;
return 1;
}
mailboxes[iFrom] = Msg;
sem_post(&semaphores[iFrom]);
return 0;
}
bool safeToCall(int location) {
bool safe = !(location < 0 || location > MAXTHREAD + 1);
return safe;
//return true;
}
mailbox.h
#ifndef MAILBOX_H_
#define MAILBOX_H_
#define RANGE 1
#define ALLDONE 2
#define MAXTHREAD 10
#include <semaphore.h>
#include <pthread.h>
struct msg {
int iSender; /* sender of the message (0 .. numThreads)*/
int type; /* its type */
int value1; /* first value */
int value2; /* second value */
};
struct msg mailboxes[MAXTHREAD + 1];
pthread_t threads[MAXTHREAD + 1];
sem_t semaphores[MAXTHREAD + 1];
int SendMsg(int iTo, struct msg &Msg);
int RecvMsg(int iFrom, struct msg &Msg);
bool safeToCall(int location);
#endif
I am compiling the code with the command
g++ -o addem addem.cpp mailbox.cpp -lpthread
I have tried commenting out all of the function bodies in the source code to leave them as stub functions, and the same error occurs. The only way I have been able to compile the file is if I comment out the function bodies, and remove
#include "mailbox.h"
From at least one of the files. I feel it has to do with how I am initializing the arrays? But I cannot figure out a workaround.
I am having a problem in trying to serialize an array of unsigned char into file with GZIP compression using protobuf while playing with the library.
I think the problem might have to do with some of my syntax or misuse of API.
I have also tried std::fstream.
FYI, Windows 8.1 & VS2013 is the building environment.
scene.proto
syntax = "proto3";
package Recipe;
message Scene
{
repeated int32 imageData = 1 [packed=true];
}
source.cpp
#include <iostream>
#include <fstream>
#include <ostream>
#include <istream>
#include <string>
#include <cstdint>
#include "Scene.pb.h"
#include <google\protobuf\io\zero_copy_stream_impl.h>
#include <google\protobuf\io\gzip_stream.h>
int const _MIN = 0;
int const _MAX = 255;
unsigned int const _SIZE = 65200000;
unsigned int const _COMPRESSION_LEVEL = 10;
void randWithinUnsignedCharSize(uint8_t * buffer, unsigned int size)
{
for (size_t i = 0; i < size; ++i)
{
buffer[i] = _MIN + (rand() % static_cast<int>(_MAX - _MIN + 1));
}
}
using namespace google::protobuf::io;
int main()
{
GOOGLE_PROTOBUF_VERIFY_VERSION;
Recipe::Scene * scene = new Recipe::Scene();
uint8_t * imageData = new uint8_t[_SIZE];
randWithinUnsignedCharSize(imageData, _SIZE);
for (size_t i = 0; i < _SIZE; i++)
{
scene->add_imagedata(imageData[i]);
}
std::cout << "scene->imagedata_size() " << scene->imagedata_size() << std::endl;
{
std::ofstream output("scene.art", std::ofstream::out | std::ofstream::trunc | std::ofstream::binary);
OstreamOutputStream outputFileStream(&output);
GzipOutputStream::Options options;
options.format = GzipOutputStream::GZIP;
options.compression_level = _COMPRESSION_LEVEL;
GzipOutputStream gzipOutputStream(&outputFileStream, options);
if (!scene->SerializeToZeroCopyStream(&gzipOutputStream)) {
std::cerr << "Failed to write scene." << std::endl;
return -1;
}
}
Recipe::Scene * scene1 = new Recipe::Scene();
{
std::ifstream input("scene.art", std::ifstream::in | std::ifstream::binary);
IstreamInputStream inputFileStream(&input);
GzipInputStream gzipInputStream(&inputFileStream);
if (!scene1->ParseFromZeroCopyStream(&gzipInputStream)) {
std::cerr << "Failed to parse scene." << std::endl;
return -1;
}
}
std::cout << "scene1->imagedata_size() " << scene1->imagedata_size() <<std::endl;
google::protobuf::ShutdownProtobufLibrary();
return 0;
}
You seem to have a typo in your code. Compression level is according to documentation in range 0-9. You set incorrectly compression level to 10.
Your example is working for me when corrected to:
unsigned int const _COMPRESSION_LEVEL = 9;
I wrote the following structure to implement a simple single producer / multi consumer synchronization. I'm using two integers available_index and consumed_index, access to consumed_index is protected by the condition variable cv. Here's the code:
#include <iostream>
#include <mutex>
#include <condition_variable>
#include <vector>
#include <thread>
struct ParserSync {
std::mutex worker_lock;
std::condition_variable cv;
int consumed_index = -1;
int available_index = -1;
bool exit_flag = false;
int consume_index() {
int ret = -1;
// get worker_lock
std::unique_lock<std::mutex> w_lock(worker_lock);
// wait for exit_flag or new available index
cv.wait(w_lock, [this] { return exit_flag || available_index > consumed_index; });
if (available_index > consumed_index) {
consumed_index++;
ret = consumed_index;
}
// Unlock mutex and notify another thread
w_lock.unlock();
cv.notify_one();
return ret;
}
void publish_index() {
available_index++;
std::cout << "before" << std::endl;
cv.notify_one();
std::cout << "after" << std::endl;
}
void set_exit() {
exit_flag = true;
cv.notify_all();
}
};
I tested my implementation using the following code (just a simple example to show the problem):
void producer(ParserSync &ps){
for (int i=0;i<5000;i++){
ps.publish_index();
std::this_thread::sleep_for(std::chrono::milliseconds(1));
}
ps.set_exit();
std::cout << "Producer finished!" << std::endl;
}
void consumer(ParserSync &ps){
while (true){
int idx = ps.consume_index();
if (idx == -1)
break;
std::this_thread::sleep_for(std::chrono::milliseconds(4));
}
std::cout << "Consumer finished!" << std::endl;
}
int main() {
ParserSync ps{};
const int num_consumers = 4;
std::vector<std::thread> consumer_threads(num_consumers);
// start consumers
for (int i = 0; i < num_consumers; ++i) {
consumer_threads[i] = std::thread{consumer, std::ref(ps)};
}
// start producer
std::thread producer_thread = std::thread{producer, std::ref(ps)};
for (int i = 0; i < num_consumers; ++i) {
consumer_threads[i].join();
}
producer_thread.join();
std::cout << "Program finished" << std::endl;
return 0;
}
I would expect that producer thread produces 5000 indices and exits afterwards, but unfortunately, it gets stuck at some random iteration. I used print statements to find the code line that blocks and tracked it down to cv.notify_one();. This is the (shortened) console output:
...
before
after
before
after
before
Does anyone know why the call to cv.notify_one(); blocks?
I'm using MinGW (x86_64-6.2.0-posix-seh-rt_v5-rev1) on Windows 10.
Thanks in advance!
EDIT:
When compiling the exact same code with Visual Studio, the program works as expected and doesn't lock itself up. Unfortunately, I need to use MinGW for other reasons.