I'm working on an online reservation system (the details aren't really relevant to my issue). It's mostly developped in C# with framework 4.0. I have a DB which contains, among other things, the products and the availability of these products. The DB is accessed by a web service, the latter being queried by a web application, which in turn sends the information to be displayed to the browser. Nothing too fancy in term of architecture, but I don't have much experience with that kind of system. This projet is quite old; it's been in development for a few years and I just hopped in six months ago, having no web programming experience before.
I'm currently experiencing concurrency problems. If I open two browsers window (any browser), enter the same query in both windows and post on both pages as fast as humanly possible, I get unhandled exception in my web service. One of the client gets the data, the other receives an error. EDIT : Also, I tested using two different computers simultaneously, and I still could get the unhandled exception.
I was able to trace the problem back to a cache system that was implemented almost a year ago. There's is no locking mecanism, and the cache uses a Dictionary (which is not thread-safe). A few Stack Overflow questions helped me in pinpointing the problem and how I should correct it (this one and that one for example).
That being said, I'm trying to reproduce the problem without having to manually click in two browser windows and hoping to get the right timing to access the same cache simultaneously. The reason I'm trying to do that is that I don't like implementing a solution that should work when I can't reliably prove it actually worked. Also, there's a lot of places where the cache has been (badly) implemented, and I'm not sure I can test all those places by "guessing" the right timing manually.
So, I downloaded Apache JMeter, and using Fiddler2 hooked up to my two browsers, I could find all the HTTP requests I needed to create a Test Plan that sends exactly the same requests.
But to my dismay, JMeter couldn't reproduce the problem, even if I set my Thread Group to create 40 concurrent users! Each one of them receives the data correctly, and none triggers the unhandled exception I am getting by doing it manually. I know it's not a cookie issue, nor a difference in user-agent, as I tested with multiple browsers and with disabled cookies. As the HTTP requests were created directly from Fiddler2, it shouldn't be an issue with hidden fields or URL rewriting, and there's no dynamic field names or content involved. I know Fiddler2 has a "replay" function, but even that couldn't reproduce the problem. This baffles me to no end.
So, my questions are : why can't I reproduce the problem with JMeter? Is there a better tool than JMeter (which is quite complete for a free software) to do that? What are the browsers doing that Jmeter doesn't do (apart from rendering the page, which is really not the issue here)?
Any thought would be much appreciated!
When you say 2 Windows are you sure they use 2 sessions, depending on the browser and the way you open the second window you won't get 2 sessions.
Are you sure you reproduced all the trafic between browser and server ?
If you have more than 1 request then you should Check if it's not 2 parallel Ajax requests from the browser , if so you must know That Without some custom scripting jmeter does not reproduce those 2 parallel request for same session(except for resources download), That might be your issue .
Regards
Philippe M.
http://www.ubik-ingenierie.com
Related
We're a small 4-man team working on an indie games marketplace.
Like several previous posters before us, we are having trouble with the Windows Defender Smartscreen blocking our app. The most common message is "File is not commonly downloaded" but sometimes it also reads something like "Windows Protected your PC."
We've received a number of complaints from our users about the Smartscreen warning message, and it's badly hurting our efforts with user acquisition.
In accordance with previous threads we found on this topic, we have tried
Code Signing - we sign our code through DigitCert, and have done so for nearly 5 months; the message has not gone away. As we use AWS for our servers, we don't have any way to use an EV Code Signing hardware token.
Microsoft Code Review - We submitted our files to Microsoft for Malware analysis. The analysis came back clean, even with a note saying they could not reproduce the warning, and yet, both our own testing and our users confirms the Windows Defender message is still happening.
Becoming a Microsoft Partner - Unfortunately, even after getting approval and confirmation, absolutely nothing has changed.
Everything I can find online about getting our app whitelisted is about whitelisting specific programs from the user side -- not very useful for convincing people we're safe to download. There are two threads on Stack Overflow that link to a blog article which has since been removed. Microsoft, Amazon, and DigitCert support have all been less than helpful.
My questions are:
1. Is Amazon CloudHSM a substitute for EV Code Signing? As stated above, we can't use a hardware token, but I have to imagine that AWS has some kind of substitute. We've never re-issued the serial on our current code signing cert, but I can't imagine 5 months is a normal turn-around time.
2. Does hosting the download link on our own domain make a huge difference? Currently, the download link leads to an AWS bucket. One website said that could potentially slow things?
3. What else are we supposed to do? We're not a large team and we're kind of stretched thin as it is. If anybody has any suggestions we haven't tried yet, I'm all ears.
UPDATE
It appears this issue is caused by a bug related specifically to using Axis2 with ColdFusion and we have been able
to replicate the issue in our production environment on two different servers by
switching between Axis1 and Axis2. My original tests to compare the
two were apparently thwarted by an override in an Application.cfc
which forced Axis2.
We ran into a memory leak today which forced us to speed up the resolution to this issue. It resembled the leak
discussed here though we aren't sure if it is the exact same
problem
(https://www.hass.de/content/coldfusion-10-webservice-leaking-memory-trusted-cache-leaks-memory).
Our primary webservices are in Axis1 and we only switched to Axis2 for
this new set of webservices because we needed document literal style
for SalesForce and with Axis1 an invalid wsdl was being created (did
not properly describe all object types in arrays). So now we have it as
Axis1 and using a manually manipulated wsdl. Not entirely sure if it
will work out with SalesForce but as far as a general fix this works.
I am investigating an issue with our coldfusion based soap webservices in our production environment. It appears that the time between the return statement in the webservices method code and actually receiving a response can be significant and appears to directly correspond to the size of the response and/or number of objects.
In development a particular request that returns 1000 records takes about 6 seconds to return. However in production that same hit takes 50+ seconds to return. I added some timing code and found that the actual function code takes less than 1 second to run at the start of the request, meaning that generating the response is taking coldfusion about 50 seconds in production. Hitting the webservice with simple http request does not have the same slowness so seems to be soap/axis specific. The resulting xml is about 1MB which I have compared and found no differences. I also copied out settings from cfadmin in both environments to compare and could find no performance related setting differences.
Both environments are at the same CF 10 update level. The server monitor shows no significant memory usage. I also ran the request from in the server to make sure there wasn't some slow connection issues or https slowing things down but the results are the same.
Any suggestions or solution would be appreciated.
Additional notes...
CPU sits at about 17% for most of the time of the request which is a lot of work to be doing. Something is happening very inefficiently
I tried switching instance to Axis1 and back again followed by an instance restart and additional tests with no change in results
One possibility is that you have them throttled - check the "request tuning" in your CF administrator. By default the setting for "number of simultaneous web service requests" is 10. Are you looping and hitting the server? In production is there more traffic?
In server monitor enable profiling and monitoring, then click on "statistics". On the far right there is a little chart icon. click on it and you will see a chart and a counter legend in the top right. Then run your code. Does the "web services running" reach a threshold and cross into "web services queued" - if so you need to increase that threshold.
One more clue - in the server monitor do NOT run the "memory profiling for more than a few seconds - say 30. If you don't you will have performance problems for sure.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I had to "sporadically" work with Heimdal / MIT Gssapi for kerberos authentication over past couple of years. I had to build an application that was to be used as a web-service running on a Linux box, and serve client applications like browsers, running on Windows and/or Linux Desktops and Workstations. Surely not the easiest of beasts to tame. Eventually when summarizing my work, I could record that the difficulties emanated due to challenges in multiple dimensions. Getting started with gssapi programming is truly a challenge just because of poor documentation, and practically non-existant tutorials. Googling mostly results in either some theoretical discussion on what's kerberos, or leads to content written with presumption that you already know everything besides some particular semantic issue.
Some really good hacks around here contributed to help me, I therefore suppose it would be a good idea to summarize the stuff, from a developer's perspective, and share it here as some sort of a wiki, to give something back to this fantastic place, and fellow programmers.
Haven't really done a wiki like this before, and I am surely no authority on GSSAPI nor Kerberos, so please be kind, but more than that please contribute and correct my mistakes. Site Editors, I am counting on you to do your magic ;)
Getting your project completed successfully will require 3 specific things to be done correctly:
Setup of your test environment
Setup of your libraries
Your code
As I said already, such projects are beasts, just because all the three haven't been put together on the same page anywhere.
Ok So let's begin at the beginning.
Unavoidable theory for a newbie
GSSAPI helps a client application to provide credentials for a server to authoritatively identify the user. Extremely useful because the server applications can modulate their served responses if they wish to, as per the user. Very naturally therefore both - the client and the server applications must be kerberized, or as some would state kerberos-aware.
The kerberos based authentication, requires both the client and server applications, to be members of a Kerberos Realm. KDC (Kerberos Domain Controller) is the designated authority that rules the realm. Microsoft's AD servers are one of the most popularly experienced examples of a KDC, though you can of course be using a *NIX based KDC. But surely without a KDC there can be no Kerberos business at all. Desktops, Servers & workstations joined into the domain identify each other as long as all of them remain joined into the domain.
For your initial experiments, setup the client & server applications in the same realm.
Though Kerberos Authentication can surely be also used across realms by creating trusts between KDCs of these realms, or even merging keytabs from different KDCs that do not trust each other. Your code will not really need any change to accommodate such different and complex-sounding scenarios.
Kerberos Authentication basically works via "tickets (or tokens)". When a member joins the realm, the KDC "grants tokens" to each of them. These tokens are unique; time and FQDN are essential factors for these tickets.
Before you even think of the very first line of your code make sure you have got these two right:
Pitfall #1 When you setup your development and test environment, make sure everything is tested and addressed as FQDN. For example if you want to check connectivity, ping using FQDN, not IP. Needless to say therefore, they must necessarily have the same DNS service configuration.
Pitfall #2 Make sure all the host systems - that are running your KDC, client software, server software have the same time server. Time synchronization is something that one forgets, and realizes to be amiss after a lot of hair-splitting, and head-banging!
Both, the client and server applications NEED kerberos keytabs. So if your application is going to run in a *NIX host, and be a part of a Microsoft Domain, you have to get a kerberos keytab generated, before we start to look at the remaining preparatory steps for gss programming.
Step-by-Step Guide to Kerberos 5 (krb5 1.0) Interoperability at is an absolute must-read.
GSS-API Programming Guide is an excellent bookmark.
Depending upon your *NIX distribution you can install the headers & libraries for building your code. My suggestion however is to download the source and build it yourself. Yes, you might not get it right at one go, but it surely is worth the trouble.
Pitfall #3 Make sure that your application is running in an Kerberos aware environment.
I really learnt this the hard way, but maybe because I am not so smart. In my earliest stages of gssapi programming struggle, I had discovered that kerberos keytabs were absolutely necessary for making my application kerberos-aware. But I simply couldn't find anything about how to load these keytabs in my application. You know why?!! Because no such api exists!!!
Because: The application is to be run in an environment which is aware of the keytabs.
Ok, let me make this simple: Your application that is supposed to do the GSSAPI / Kerberos things has to run after you have set environment variable KRB5_KTNAME to the path where you have stored the keytabs. So either you do something like:
export KRB5_KTNAME=<path/to/your/keytab>
or make use of setenv to set KRB5_KTNAME in your application sufficiently before the very first line of your code that uses gssapi is run.
We are now ready to do the necessary things in the application's code.
I understand there are quite a few other aspects that must be reviewed by the application developer, to write and test an application. I know of a few environment variables, that can be important.
Can anybody please shed some more light upon that?
Is it possible that a database (connected to ColdFusion 9 via a datasource connection) being unavailable could cause ColdFusion to become unresponsive? (The database is used for a singular one-off lightly-trafficked app.)
Recently, maintenance on a connected Oracle database (oracle jdbc) has caused that database to be unavailable two different times. Coincidentally, at both these times, ColdFusion pages on our site became unavailable or terribly slow to load (static HTML pages seemed to load fine, for the most part). Restarting the ColdFusion application server service would fix the problem, but only for minutes. The first time, during a time the application server was responsive, we unchecked the "Maintain connections" checkbox. I'm not sure this had any effect, then shortly after the Oracle database came back online, and we didn't seem to have the problem any more.
The second time that database was offline, we experienced a very similar issue with our website - ColdFusion pages becoming reaaaally slow or unavailable altogether. During one of the times when I could access the CF administrator, I updated the datasource and checked "Disable connections". Then I stopped and restarted both the CF ODBC agent and ODBC server services. After that, the problem seemed to stop, but I don't know enough to know if this is causation or coincidence.
Anyone have insights on this?
Server setup: Windows Server 2003 SP2, ColdFusion 9, IIS 6
There are a number of ways to slow a database to a crawl if not stop it completely. If you have hackers for example attacking your database through Port 1433 with attempted logins several times a second that can slow it down and if they get in they can of course do whatever they want. When this happened to me I found a record of attacks in the Event logs; the solution is better network security intercepting such attacks and never letting them actually talk to the database. Or say if your site is vulnerable to SQL injection attacks hackers could be messing with your database that way too but network security wouldn't necessarily work in that case. It doesn't require hackers to degrade the performance of your database however, you could be having a problem with allocated disk space for transaction logs or indexes filling up, or heaven forbid an imminent hardware failure showing early symptoms. You're backing up your database often I hope, off the server. To answer your question yes ColdFusion can and will become unresponsive when pages are called that call the database, and will usually display error messages when the database finally times out and never sends the requested data to ColdFusion. You can protect against that to some extent with CFTRY tags around your queries that display clean and polite error messages instead of ColdFusion's ugly ones if the database fails to return data, at least your site continues to look professional that way. One project I worked used a shared SQL Server database that often got overloaded and slowed down terribly and there was nothing I could do about improving that situation. What I did to keep the site functioning was to maintain a DB backup in the form of a MS Access database (yeah it was inappropriate but it worked when SQL Server wouldn't) and anytime SQL Server failed I had the application set up to automatically use code that called the Access database instead.
These are some ideas for you to think about if you are continuing to have problems, I see nobody's even tried to answer your question in the last six months and that's kinda been my experience with the quality of assistance this site has offered me too. I hope my thoughts can be of some use to you.
I want to build the following back-end service:
For each call to the service, spawn a web browser that loads a webpage (including flash) and returns a screenshot of the page to the caller at intervals (ie every 3 seconds) until the caller disconnects. This needs to scale for many callers (thousands perhaps), each of which needs its own browser session.
When I decided I needed to build this program, I was surprised that I had basically no idea how I could do it.
On stackoverflow, I found the following link which looks promising: http://www.genuitec.com/about/labs.html
Any other ideas?
You can use XULRunner (Mozilla engine) on your server side. I'm in doubt though that this solution is scalable.