I am using [dynamoDBLockClient][1] to choose the leader host
Expectation :
If host is dead new host should become leader in next 12 seconds
Followed this:
https://aws.amazon.com/blogs/database/building-distributed-locks-with-the-dynamodb-lock-client/
https://github.com/amazon-archives/dynamodb-lock-client
Reality
If leader host is dead other host/non leader wont try to become leader. How do i set DDB lock in such a way that if leader is dead other host must become leader in next 10 seconds
#Provides
public AmazonDynamoDBLockClient getLeaderSelectionLockClient(AmazonDynamoDB dynamoDB) {
final AmazonDynamoDBLockClient client = new AmazonDynamoDBLockClient(
AmazonDynamoDBLockClientOptions.builder(dynamoDB, LEADER_SELECTION_DDB_TABLE)
.withTimeUnit(TimeUnit.SECONDS)
.withLeaseDuration(10L)
.withHeartbeatPeriod(3L)
.withOwnerName(EC2MetadataUtils.getInstanceId())
.withCreateHeartbeatBackgroundThread(true) //If true then leader will continue to be leader till it die
.build());
return client;
}
#Provides
public AcquireLockOptions getAcquireLockOptionsForEnchanter() {
final String keyToLock = "LEADER";
AcquireLockOptions acquireLockOptions = AcquireLockOptions
.builder(keyToLock)
.withRefreshPeriod(11L)
.withTimeUnit(TimeUnit.SECONDS)
.build();
return acquireLockOptions;
}
public void competeForLeadership() {
final String lockSuccessMessage =
"Acquired lock! If I die, my lock will expire in 10 seconds Otherwise, I will hold it until I stop "
+ "heart beating." + EC2MetadataUtils.getInstanceId();
try {
final Optional<LockItem> lockItem = dynamoDBLockClient.tryAcquireLock(acquireLockOptions);
if (lockItem.isPresent()) {
log.info(lockSuccessMessage);
} else {
log.error("Failed to acquire lock!");
}
} catch (Exception e) {
log.error("Leader Selector is down");
}
}
As per document https://www.mvndoc.com/c/com.amazonaws/dynamodb-lock-client/com/amazonaws/services/dynamodbv2/AcquireLockOptions.AcquireLockOptionsBuilder.html#withAdditionalTimeToWaitForLock-java.lang.Long-
withRefreshPeriod should have solved this problem but it is not solving.
How to do set up correctly?
Looks like tryAcquire is non blocking call since I can see control is coming back to code after failed to get lock with message "Failed to acquire lock!"
Is it possible to keep on retrying till it get lock
Thanks
Jk
Related
I cannot really find an example on how to use this.
Right now, I'm doing like this:
// Request 10 connections.
ListConnectionsResponse response = peopleService.people().connections()
.list("people/me")
.setRequestSyncToken(true)
.setPageSize(10)
.setPersonFields("names,emailAddresses")
.execute();
I make some changes to my contacts (adding, removing, updating), then I do this:
// Request 10 connections.
ListConnectionsResponse response2 = peopleService.people().connections()
.list("people/me")
.setSyncToken(response.getNextSyncToken())
.setPageSize(10)
.setPersonFields("names,emailAddresses")
.execute();
But it seems like I cannot get the changes I've done earlier, not even if I do them directly from the UI. I'm pretty sure I'm using the sync token in the wrong way.
Update (19/02/2020): In this example I call the API requesting the sync token in the first request (I successfully get the contacts), pause the execution (by breakpoint), delete a contact and update another one (from the web page), resume the execution and then I call the API again with the sync token that I extracted from the previous call. The result is that no change was made for some reason:
// Build a new authorized API client service.
final NetHttpTransport HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
PeopleService peopleService = new PeopleService.Builder(HTTP_TRANSPORT, JSON_FACTORY, getCredentials(HTTP_TRANSPORT))
.setApplicationName(APPLICATION_NAME)
.build();
// Request 10 connections.
ListConnectionsResponse response = peopleService.people().connections()
.list("people/me")
.setPageSize(10)
.setPersonFields("names,emailAddresses")
.setRequestSyncToken(true)
.execute();
// Print display name of connections if available.
List<Person> connections = response.getConnections();
if (connections != null && connections.size() > 0) {
for (Person person : connections) {
List<Name> names = person.getNames();
if (names != null && names.size() > 0) {
System.out.println("Name: " + person.getNames().get(0)
.getDisplayName());
} else {
System.out.println("No names available for connection.");
}
}
} else {
System.out.println("No connections found.");
}
// CORRECT: 2 CONTACTS PRINTED
// CORRECT: THE SYNC TOKEN IS THERE
String syncToken = response.getNextSyncToken();
System.out.println("syncToken = "+syncToken);
// I SETUP A BREAKPOINT BELOW, I DELETE ONE CONTACT AND EDIT ANOTHER AND THEN I RESUME THE EXECUTING
// Request 10 connections.
response = peopleService.people().connections()
.list("people/me")
.setPageSize(10)
.setPersonFields("names,emailAddresses")
.setSyncToken(syncToken)
.execute();
// Print display name of connections if available.
connections = response.getConnections();
if (connections != null && connections.size() > 0) {
for (Person person : connections) {
List<Name> names = person.getNames();
if (names != null && names.size() > 0) {
System.out.println("Name: " + person.getNames().get(0)
.getDisplayName());
} else {
System.out.println("No names available for connection.");
}
}
} else {
System.out.println("No connections found.");
}
// WRONG: I GET "NO CONNECTIONS FOUND"
Something I've found out is that, when requesting or setting a sync token, you must iterate the entirety of the contacts for the nextSyncToken to be populated.
That means that as long as there is a nextPageToken (wink wink setPageSize(10)), the sync token will not be populated.
You could either:
A) Loop over all the contacts using your current
pagination, doing whatever you need to do at every
iteration, and after the last call retrieve the populated
sync token.
B) Iterate over all the contacts in one go, using the max
page size of 2000 and a single personField, retrieve the
token, and then do whatever you need to do. Note that if
you are expecting a user to have more than 2000
contacts, you will still need to call the next pages using
the nextPageToken.
Here is an exemple of a sync loop, adapted from Synchronize Resources Efficiently. Note that I usually use the Python client, so this Java code might not be 100% error free:
private static void run() throws IOException {
Request request = people_service.people().connections()
.list("people/me")
.setPageSize(10)
.setPersonFields("names,emailAddresses");
// Load the sync token stored from the last execution, if any.
// The syncSettingsDataStore is whatever you use for storage.
String syncToken = syncSettingsDataStore.get(SYNC_TOKEN_KEY);
String syncType = null;
// Perform the appropiate sync
if (syncToken == null) {
// Perform a full sync
request.setRequestSyncToken(true);
syncType = "FULL";
} else {
// Try to perform an incremental sync.
request.setSyncToken(syncToken);
syncType = "INCREMENTAL";
}
String pageToken = null;
ListConnectionsResponse response = null;
List<Person> contacts = null;
// Iterate over all the contacts, page by page.
do {
request.setPageToken(pageToken);
try {
response = request.execute();
} catch (GoogleJsonResponseException e) {
if (e.getStatusCode() == 410) {
// A 410 status code, "Gone", indicates that the sync token is
// invalid/expired.
// WARNING: The code is 400 in the Python client. I think the
// Java client uses the correct code, but be on the lookout.
// Clear the sync token.
syncSettingsDataStore.delete(SYNC_TOKEN_KEY);
// And anything else you need before re-syncing.
dataStore.clear();
// Restart
run();
} else {
throw e;
}
}
contacts = response.getItems();
if (contacts.size() == 0) {
System.out.println("No contacts to sync.");
} else if (syncType == "FULL"){
//do full sync for this page.
} else if (syncType == "INCREMENTAL") {
//do incremental sync for this page.
} else {
// What are you doing here???
}
pageToken = response.getNextPageToken();
} while (pageToken != null);
// Store the sync token from the last request for use at the next execution.
syncSettingsDataStore.set(SYNC_TOKEN_KEY, response.getNextSyncToken());
System.out.println("Sync complete.");
}
Consider this context:
Having a group of threads doing some work (that work is in a infinite loop, embedded project) where the number of threads (and some parameters) depends from a Database result.
What I need is to remove or create threads from that group when there´s a change in the database.
Here is the code:
for (result::const_iterator pin = pinesBBB.begin(); pin != pinesBBB.end(); ++pin)
{
string pinStr = pin["pin"].as<string>();
boost::thread hiloNuevo(bind(WorkPin, pinStr));
Worker.add_thread(&hiloNuevo);
}
Where result is pqxx::result from pqxx library.
This piece of code iterates a table from an SQL query result and creates a thread for every record found.
After that, there´s this code that checks the same table every a couple of minutes:
`
void ThreadWorker(boost::thread_group *worker, string *pinesLocales)
{
int threadsVivosInt = worker->size();
string *pinesDB;
int contador;
for (;;)
{
contador = 0;
sleep(60);
try
{
result pinesBBB = TraerPines();
for (result::const_iterator pin = pinesBBB.begin(); pin != pinesBBB.end(); ++pin)
{
pinesDB[contador] = pin["pin"].as<string>();
contador++;
}
thread hiloMuerto
}
catch (...)
{
sleep(360);
}
}
}
`
What I want to do is access this thread_group worker and remove one of those threads.
I´ve tryed using an Int index like worker[0] and with thread´s ID boost::thread::id
I can remove a thread using a native_handle and then using an plattform specific like pthread_cancel but I can´t get the thread from the thread group.
Any ideas? Thanks!
boost::thread_group::remove_thread() removes the specified thread from a given thread_group. Once you've done this, you're now responsible for managing the thread.
Im basically facing a blocking problem.
I have my server coded based on C++ Boost.ASIO using 8 threads since the server has 8 logical cores.
My problem is a thread may face 0.2~1.5 seconds of blocking on a MySQL query and I honestly don't know how to go around that since MySQL C++ Connector does not support asynchronous queries, and I don't know how to design the server "correctly" to use multiple threads for doing the queries.
This is where I'm asking for opinions of what to do in this case.
Create 100 threads for async' query sql?
Could I have an opinion from experts about this?
Okay, the proper solution to this would be to extend Asio and write a mysql_service implementation to integrate this. I was almost going to find out how this is done right away, but I wanted to get started using an "emulation".
The idea is to have
your business processes using an io_service (as you are already doing)
a database "facade" interface that dispatches async queries into a different queue (io_service) and posts the completion handler back onto the business_process io_service
A subtle tweak needed here you need to keep the io_service on the business process side from shutting down as soon as it's job queue is empty, since it might still be awaiting a response from the database layer.
So, modeling this into a quick demo:
namespace database
{
// data types
struct sql_statement { std::string dml; };
struct sql_response { std::string echo_dml; }; // TODO cover response codes, resultset data etc.
I hope you will forgive my gross simplifications :/
struct service
{
service(unsigned max_concurrent_requests = 10)
: work(io_service::work(service_)),
latency(mt19937(), uniform_int<int>(200, 1500)) // random 0.2 ~ 1.5s
{
for (unsigned i = 0; i < max_concurrent_requests; ++i)
svc_threads.create_thread(boost::bind(&io_service::run, &service_));
}
friend struct connection;
private:
void async_query(io_service& external, sql_statement query, boost::function<void(sql_response response)> completion_handler)
{
service_.post(bind(&service::do_async_query, this, ref(external), std::move(query), completion_handler));
}
void do_async_query(io_service& external, sql_statement q, boost::function<void(sql_response response)> completion_handler)
{
this_thread::sleep_for(chrono::milliseconds(latency())); // simulate the latency of a db-roundtrip
external.post(bind(completion_handler, sql_response { q.dml }));
}
io_service service_;
thread_group svc_threads; // note the order of declaration
optional<io_service::work> work;
// for random delay
random::variate_generator<mt19937, uniform_int<int> > latency;
};
The service is what coordinates a maximum number of concurrent requests (on the "database io_service" side) and ping/pongs the completion back onto another io_service (the async_query/do_async_query combo). This stub implementation emulates latencies of 0.2~1.5s in the obvious way :)
Now comes the client "facade"
struct connection
{
connection(int connection_id, io_service& external, service& svc)
: connection_id(connection_id),
external_(external),
db_service_(svc)
{ }
void async_query(sql_statement query, boost::function<void(sql_response response)> completion_handler)
{
db_service_.async_query(external_, std::move(query), completion_handler);
}
private:
int connection_id;
io_service& external_;
service& db_service_;
};
connection is really only a convenience so we don't have to explicitly deal with various queues on the calling site.
Now, let's implement a demo business process in good old Asio style:
namespace domain
{
struct business_process : id_generator
{
business_process(io_service& app_service, database::service& db_service_)
: id(generate_id()), phase(0),
in_progress(io_service::work(app_service)),
db(id, app_service, db_service_)
{
app_service.post([=] { start_select(); });
}
private:
int id, phase;
optional<io_service::work> in_progress;
database::connection db;
void start_select() {
db.async_query({ "select * from tasks where completed = false" }, [=] (database::sql_response r) { handle_db_response(r); });
}
void handle_db_response(database::sql_response r) {
if (phase++ < 4)
{
if ((id + phase) % 3 == 0) // vary the behaviour slightly
{
db.async_query({ "insert into tasks (text, completed) values ('hello', false)" }, [=] (database::sql_response r) { handle_db_response(r); });
} else
{
db.async_query({ "update * tasks set text = 'update' where id = 123" }, [=] (database::sql_response r) { handle_db_response(r); });
}
} else
{
in_progress.reset();
lock_guard<mutex> lk(console_mx);
std::cout << "business_process " << id << " has completed its work\n";
}
}
};
}
This business process starts by posting itself on the app service. It then does a number of db queries in succession, and eventually exits (by doing in_progress.reset() the app service is made aware of this).
A demonstration main, starting 10 business processes on a single thread:
int main()
{
io_service app;
database::service db;
ptr_vector<domain::business_process> bps;
for (int i = 0; i < 10; ++i)
{
bps.push_back(new domain::business_process(app, db));
}
app.run();
}
In my sample, business_processes don't do any CPU intensive work, so there's no use in scheduling them across CPU's, but if you wanted you could easily achieve this, by replacing the app.run() line with:
thread_group g;
for (unsigned i = 0; i < thread::hardware_concurrency(); ++i)
g.create_thread(boost::bind(&io_service::run, &app));
g.join_all();
See the demo running Live On Coliru
I'm not a MySQL guru, but the following is generic multithreading advice.
Having NumberOfThreads == NumberOfCores is appropriate when none of the threads ever block and you are just splitting the load over all CPUs.
A common pattern is to have multiple threads per CPU, so one is executing while another is waiting on something.
In your case, I'd be inclined to set NumberOfThreads = n * NumberOfCores where 'n' is read from a config file, a registry entry or some other user-settable value. You can test the system with different values of 'n' to fund the optimum. I'd suggest somewhere around 3 for a first guess.
I have this webmethod(called from android app)
[WebMethod]
public bool addVotes(string username,string password,int votes)
{
bool success= false;
if (Membership.ValidateUser(username, password) == true)
{
DbContext context = new DbContext();
AppUsers user = context.AppUsers.Where(x => x.Username.Equals(username)).FirstOrDefault();
if (user != null)
{
user.Votat += votes;
context.SaveChanges();
success = true;
}
}
return success;
}
This web service will be called from 80 users(probably) in the same period of time(within two or three hours). I am afraid that there can occur a deadlock while reading or updating data in the database. Could you tell me weather there is a possibility of a deadlock and if there is such a possibility how can I prevent it with EF or sql or whatever.
With this code : you can't
AppUsers user = context.AppUsers.Where(x => x.Username.Equals(username)).FirstOrDefault();
This line will wait for a readlock but eventually it will aquire one, so no deadlock possible.
context.SaveChanges();
This line will try and update your user table. It will wait for a writelock but it will eventually get one and then move on.
You can only get a deadlock while inserting / deleting / ... over multiple tables and usually it happens during a cursor iteration. I have yet to bump in a situation where EF ends up in a deadlock, so i wouldn't worry about it too much.
Maybe you'll find this article usefull : http://blogs.msdn.com/b/diego/archive/2012/04/01/tips-to-avoid-deadlocks-in-entity-framework-applications.aspx
So the mongo c++ documentation says
On a failover situation, expect at least one operation to return an
error (throw an exception) before the failover is complete. Operations
are not retried
Kind of annoying, but that leaves it up to me to handle a failed operation. Ideally I would just like the application to sleep for a few seconds (app is single threaded). And retry with the hopes that a new primary mongod is established. In the case of a second failure, well I take it the connection is truly messed up and I just want to thrown an exception.
Within my MongodbManager class this means all operations have this kind of double try/catch block set up. I was wondering if there is a more elegant solution?
Example method:
template <typename T>
std::string
MongoManager::insert(std::string ns, T object)
{
mongo::BSONObj = convertToBson(object);
std::string result;
try {
connection_->insert(ns, oo); //connection_ = shared_ptr<DBClientReplicaSet>
result = connection_->getLastError();
lastOpSucceeded_ = true;
}
catch (mongo::SocketException& ex)
{
lastOpSucceeded_ = false;
boost::this_thread::sleep( boost::posix_time::seconds(5) );
}
// try again?
if (!lastOpSucceeded_) {
try {
connection_->insert(ns, oo);
result = connection_->getLastError();
lastOpSucceeded_ = true;
}
catch (mongo::SocketException& ex)
{
//do some clean up, throw exception
}
}
return result;
}
That's indeed sort of how you need to handle it. Perhaps instead of having two try/catch blocks I would use the following strategy:
keep a count of how many times you have tried
create a while loop with as terminator (count < 5 && lastOpSucceeded)
and then sleep with pow(2,count) to sleep more in every iteration.
And then when all else fails, bail out.