I am writing a program using AWS Java SDK to create EC2 instances and do some processing once they are launched.
I have written the following code but I know there could be a better way to do this:
public static void main(String[] args) {
// Some code initialization here
Instance ec2instance;
do {
try {
Thread.sleep(sleep_cyle);
} catch (InterruptedException ex) {
System.out.println(ex.getMessage());
}
} while(ec2instance.getState().getCode() != 16);
//Proceed with processing after instance is running
}
Thank you.
Since there is nothing reporting back to your application related to status changes, the only way to identify changes is to poll every 15-30 seconds.
There are rare cases where an instance may simply fail to launch. Your code should expect the possibility of going from pending to terminated.
Related
We have a deadlock situation which occured because of this heavy load on the microservice (Say A) causing multiple requests from different client services (B,C). So these calls from B and C come for the same clientId(key) and are served by different instances of A and they try to update the same clientId data in database at same time causing below error.
CannotAcquireLockException is thrown,
(SQL Error: 60, SQLState: 61000..
ORA-00060: deadlock detected while waiting for resource
We have decided to implement sharding at load balancer(haproxy) level which will ensure same instance of A will always serve the requests from B and C for a specific key(clientId), so we dont have multiple instances processing the request for same key(clientId).
Now we get into the mode of everything in single jvm as we have made sure requests from B and C for a specific clientId always come to same instance of A.
With this its still possible that requests from B and C services come for same clientId with difference in time of nanoseconds. Any then multiple threads will again try to update the same clientId data in database at same time causing same error again.
To improve this we are looking for possible solutions and one solutions is ReentrantReadWriteLock which should take care of this based on the concepts.
We are using spring data jpa and have a save being done which looks like
clientJpaRepository.save(ClientObject);
Now is it possible to use something like below.
public void save(Client clientObject) {
String clientId = clientObject.getClientId();
try {
boolean isLockAcquired = writeLock.tryLock(100, TimeUnit.MILLISECONDS);
if (isLockAcquired) {
clientJpaRepository.save(clientObject);
}
} catch (InterruptedException e) {
log.error("exception occured trying to acquire lock for clientId={}", clientId);
} finally {
writeLock.unlock();
}
}
I am not very sure how its going to deal with the keys. As in i don't want any threads to block if they are wanting to update for different key(clientId 2).
Also, other thing to note is there could be reads happening as part of other API calls for this data from database. They would not be waiting too long hopefully and i hope i don't need to make any changes there for the reads.
Sorry for the long question, Hope i will hear from someone soon.
Thanks.
I have this scenario where I have a WebApi and an endpoint that when triggered does a lot of work (around 2-5min). It is a POST endpoint with side effects and I would like to limit the execution so that if 2 requests are sent to this endpoint (should not happen, but better safe than sorry), one of them will have to wait in order to avoid race conditions.
I first tried to use a simple static lock inside the controller like this:
lock (_lockObj)
{
var results = await _service.LongRunningWithSideEffects();
return Ok(results);
}
this is of course not possible because of the await inside the lock statement.
Another solution I considered was to use a SemaphoreSlim implementation like this:
await semaphore.WaitAsync();
try
{
var results = await _service.LongRunningWithSideEffects();
return Ok(results);
}
finally
{
semaphore.Release();
}
However, according to MSDN:
The SemaphoreSlim class represents a lightweight, fast semaphore that can be used for waiting within a single process when wait times are expected to be very short.
Since in this scenario the wait times may even reach 5 minutes, what should I use for concurrency control?
EDIT (in response to plog17):
I do understand that passing this task onto a service might be the optimal way, however, I do not necessarily want to queue something in the background that still runs after the request is done.
The request involves other requests and integrations that take some time, but I would still like the user to wait for this request to finish and get a response regardless.
This request is expected to be only fired once a day at a specific time by a cron job. However, there is also an option to fire it manually by a developer (mostly in case something goes wrong with the job) and I would like to ensure the API doesn't run into concurrency issues if the developer e.g. double-sends the request accidentally etc.
If only one request of that sort can be processed at a given time, why not implement a queue ?
With such design, no more need to lock nor wait while processing the long running request.
Flow could be:
Client POST /RessourcesToProcess, should receive 202-Accepted quickly
HttpController simply queue the task to proceed (and return the 202-accepted)
Other service (windows service?) dequeue next task to proceed
Proceed task
Update resource status
During this process, client should be easily able to get status of requests previously made:
If task not found: 404-NotFound. Ressource not found for id 123
If task processing: 200-OK. 123 is processing.
If task done: 200-OK. Process response.
Your controller could look like:
public class TaskController
{
//constructor and private members
[HttpPost, Route("")]
public void QueueTask(RequestBody body)
{
messageQueue.Add(body);
}
[HttpGet, Route("taskId")]
public void QueueTask(string taskId)
{
YourThing thing = tasksRepository.Get(taskId);
if (thing == null)
{
return NotFound("thing does not exist");
}
if (thing.IsProcessing)
{
return Ok("thing is processing");
}
if (!thing.IsProcessing)
{
return Ok("thing is not processing yet");
}
//here we assume thing had been processed
return Ok(thing.ResponseContent);
}
}
This design suggests that you do not handle long running process inside your WebApi. Indeed, it may not be the best design choice. If you still want to do so, you may want to read:
Long running task in WebAPI
https://blogs.msdn.microsoft.com/webdev/2014/06/04/queuebackgroundworkitem-to-reliably-schedule-and-run-background-processes-in-asp-net/
I have a MVC app where I am trying to capture all the incoming requests in a ActionFilter. Here is the logging code. I am trying to log in a fire and forget model.
My issue is if I execute this code synchronously by taking out the Task.Run Elmah does send out an email. But for the code shown below I can see the error getting logged to the InMemory logger in elmah.axd but no emails.
public void Log(HttpContextBase context)
{
Task.Run(() =>
{
try
{
throw new NotImplementedException(); //simulating an error condition
using (var s = _documentStore.OpenSession())
{
s.Store(GetDataToLog(context));
s.SaveChanges();
}
}
catch (Exception ex)
{
ErrorSignal.FromCurrentContext().Raise(ex);
}
});
}
Got this answer from Atif Aziz (ELMAH Lead contributor) on the ELMAH google group:
When you use Task.Run, the HttpContext is not transferred to the thread pool thread on which your action will execute. When ErrorSignal.FromCurrentContext is called from within your action, my guess is that it's probably failing with another exception because there is no current context. That exception is lying with the Task. If you're on .NET 4, you're lucky because you'll see the ASP.NET app crash eventually (but possibly much after the fact) when the GC will kick in and collect the Task and its exception will go “unobserved”. If you're on .NET 4.5, the policy has been changed and the exception will simply get lost. Either way, your observation will be that mailing is not working. In fact, logging won't work either unless you use Elmah.ErrorLog.GetDefault(null).Log(new Error(ex)), where a null context is allowed. But that call only logs the error but does not do any mailing. ELMAH's modules are connected to the ASP.NET context. If you detach from that context by forking to another thread, then you cannot rely on ELMAH's modules. You can only use Elmah.ErrorLog.GetDefault(null).Log(new Error(ex)) reliably to log an error.
I am looking for an efficient way to find whether a given application (say app.exe) is single instance or not? I thought of these following sols:
Do CreateProcess() twice and check whether there are two or more instance running of that application? If no, it is single instance application. But, this is not efficient.
Do CreateProcess() and wait for 1-2 sec. If this instance is killed (because there is already an instance running for it), it will be single instance app.
But I am not convinced with both above sol. Is there any other efficient way of doing that in windows?
Please note that I don't to kill or make any modifications to an already running (if any) instance of that application.
Think about it the other way: When you write a program, how do you specify whether it is single-instance or multiple-instance? Is there a way that some other program can get that information out of your program without running it? (Once you answer this question, then you have the answer to your question.)
This problem is not solvable in general because single-instance/multiple-instance-ness is determined at runtime and can be based on runtime conditions. For example, some applications are "sometimes multiple instance, sometimes single": If you run the application to open document X, and then document Y, you will get two instances. But if you open document X, and then document X again, the two instances will fold into one. Other applications may have a configuration switch that lets you select whether they are single-instance or multiple-instance. Or maybe they decide to flip a coin and decide to be single-instance if tails and multiple-instance if heads.
The best way is via using synchronization object called Mutex (Mutually exclusive). You may google it.
I think the following code may help to.
//---------------------------------------------------------------------------
WINAPI _tWinMain(HINSTANCE, HINSTANCE, LPTSTR, int)
{
try
{
HANDLE hMutex=OpenMutex(MUTEX_ALL_ACCESS,0,"SIns");
if (!hMutex) {
//Mutex doesn’t exist. This is the first instance so create the mutex.
//in this case app name is SIns (Single Instance)
hMutex=CreateMutex(0,0,"SIns");
Application->Initialize();
Application->MainFormOnTaskBar = true;
Application->CreateForm(__classid(TfMain), &fMain);
Application->Run();
ReleaseMutex(hMutex);
}
else{
//This is not single. The prev instance is already running
//so informing about it
//remember that if it finds prev instance we're activating it here
//you may do whatsoever here ...... e.g. you may kill process or stuff like this:)
ShowMessage("The program is already running. Switching to ...");
HWND hWnd=FindWindow(0,"SIns");
SetForegroundWindow(hWnd);
}
}
catch (Exception &exception)
{
Application->ShowException(&exception);
}
catch (...)
{
try
{
throw Exception("");
}
catch (Exception &exception)
{
Application->ShowException(&exception);
}
}
return 0;
}
//---------------------------------------------------------------------------
There is no way to do this at all. What happens if the application checks a mutex then makes a messagebox to tell the user an instance is already running and only when the user dismisses it does it kill the application? There are many different ways to ensure mutual exclusion via some shared resource, mutex, shared file, even maybe setting some registry key, the methods are unlimited.
The usual solution is to use some sort of a locking file. Under
traditional Unix, for example, the application will start by creating a
file (which will succeed even if the file exists), then try to create a
link to it (an atomic action); if that fails, the application will
immediately kill itself. Under Windows, the share mode of CreateFile
can be used to the same effect: open a file with share mode 0, and if
that fails, quit. (The Unix solution will leave the lock if the process
crashes, requiring it to be cleaned up manually. The Windows solution
will remove the lock if the system crashes.)
you may use mutexes... I do such check with following code:
bool insureApplicationUniqueness(HANDLE& mutexHandle)
{
mutexHandle=CreateMutexW(NULL,true,UNIQUE_INSTANCE_MUTEX_NAME);
if( mutexHandle&&(ERROR_ALREADY_EXISTS==GetLastError()))
{
CloseHandle(mutexHandle);
return false;
}
return true;
}
but this is for application which source code is yours and which checks is another instance of itself running.
The problem with the notion is that in common environments, there is no explicit static data that determines whether an application is single-instance. You only have behavior to go on, but you cannot fully test behavior.
What if you have an app that is multi-instance, but will fail to open a file that's already open? If you test it twice with the same, valid filename, it would create only a single process, but any other command line argument would cause two processes to exist. Is this a single-instance program?
You could even argue that "single instance" isn't a well-defined catageory of programs for this reason.
I have a server application which I am debugging which basically parses scripts (VBscript, Python, Jscript and SQl) for the application that requests it.
This is a very critical application which, if it crashes causes havoc for a lot of users. The problem I am facing is how to handle exceptions so that the application can continue and the users know if something is wrong in their scripts.
An example: In the SQL scripts the application normally returns a set of values (Date, Number, String and Number). So the scripts have to have a statement at the end as such:
into dtDate, Number, Number, sString. These are values that are built into the application and the server application knows how to interpret these. These fields are treated in the server app as part of an array. The return values should normally be in a specific order as the indexes for these fields into the array are hardcoded inside the server application.
Now when a user writing a script forgets one of these fields, then the last field (normally string) throws an IndexOutofBoundsException.
The question is how does one recover from exceptions of this nature without taking down the application?
Another example is an error in a script for which no error parsing message can be generated. These errors just disappear in the background in the application and eventually cause the server app to crash. The scripts on which it fails don't necessarily fail to execute entirely, but part of it doesn't execute and the other parts do, which makes it look fairly odd to a user.
This server app is a native C++ application and uses COM technologies.
I was wondering if anyone has any ideas on what the best way is to handle exceptions such as the ones described above without crashing the application??
You can't handle problems like this with exceptions. You could have a top-level catch block that catches the exception and hope that not too much state of the program got irrecoverably munched to try to keep the program alive. Still doesn't make the user happy, that query she is waiting for still doesn't run.
Ensuring that changes don't destabilize a critical business app requires organization. People that sign-off on the changes and verify that they work as intended before it is allowed into production. QA.
since you talk about parsing different languages, you probably have something like
class IParser //parser interface
{
virtual bool Parse( File& fileToParse, String& errMessage ) = 0;
};
class VBParser : public Parser
class SQLParser : public Parser
Suppose the Parse() method throws an exception that is not handled, your entire app crashes. Here's a simplified example how this could be fixed at the application level:
//somewhere main server code
void ParseFileForClient( File& fileToParse )
{
try
{
String err;
if( !currentParser->Parse( fileToParse, err ) )
ReportErrorToUser( err );
else
//process parser result
}
catch( std::exception& e )
{
ReportErrorToUser( FormatExceptionMessage( err ) );
}
catch( ... )
{
ReportErrorToUser( "parser X threw unknown exception; parsing aborted" );
}
}
If you know an operation can throw an exception, then you need to add exception handling to this area.
Basically, you need to write the code in an exception safe manner which usually uses the following guidelines
Work on temporary values that can throw exceptions
Commit the changes using the temp values after (usually this will not throw an exception)
If an exception is thrown while working on the temp values, nothing gets corrupted and in the exception handling you can manage the situation and recover.
http://www.gotw.ca/gotw/056.htm
http://www.gotw.ca/gotw/082.htm
It really depends on how long it takes to start up your server application. It may be safer to let the application crash and then reload it. Or taking a cue from Chrome browser run different parts of your application in different processes that can crash. If you can safely recover an exception and trust that your application state is ok then fine do it. However catching std::exception and continuing can be risky.
There are simple to complex ways to baby sit processes to make sure if they crash they can be restarted. A couple of tools I use.
bluepill http://asemanfar.com/Bluepill:-a-new-process-monitoring-tool
pacemaker http://www.clusterlabs.org/
For simple exceptions that can happen inside your program due to user errors,
simply save the state that can be changed, and restore it like this:
SaveStateThatCanBeAlteredByScript();
try {
LoadScript();
} catch(std::exception& e){
RestoreSavedState();
ReportErrorToUser(e);
}
FreeSavedState();
If you want to prevent external code from crashing (possible untrustable code like plugins), you need an IPC scheme. On Windows, I think you can memory map files with OpenFile(). On POSIX-systems you can use sem_open() together with mmap().
If you have a server. You basically have a main loop that waits for a signal to start up a job. The signal could be nothing and your server just goes through a list of files on the file system or it could be more like a web server where it waits for a connection and executes the script provided on the connection (or any thing like that).
MainLoop()
{
while(job = jobList.getJob())
{
job.execute();
}
}
To stop the server from crashing because of the scripts you need to encapsulate the external jobs in a protected region.
MainLoop()
{
// Don't bother to catch exceptions from here.
// This probably means you have a programming error in the server.
while(job = jobList.getJob())
{
// Catch exception from job.execute()
// as these exceptions are generally caused by the script.
try
{
job.execute();
}
catch(MyServerException const& e)
{
// Something went wrong with the server not the script.
// You need to stop. So let the exception propagate.
throw;
}
catch(std::exception const& e)
{
log(job, e.what());
}
catch(...)
{
log(job, "Unknown exception!");
}
}
}
If the server is critical to your operation then just detecting the problem and logging it is not always enough. A badly written server will crash so you want to automate the recovery. So you should write some form of heartbeat processes that checks at regular intervals if the processes has crashed and if it has automatically restart it.