I'm looking to make heavy consumption of a RESTful API from a ColdFusion application.
I'm not a CF expert but I'm anticipating the repeated cfhttp calls will become a bottleneck as I believe that each results in a connection being established, request sent, response received and connection torn down.
I'm curious - is there a way to maintain a connection pool that requests could be sent through to avoid the repeated establish/tear down?
Does the ColdFusion server provide such a facility that I just don't know about (we're using CF 8) or can I write a java custom tag that could maintain the pool?
Certainly someone else has encountered this.
Unfortunately I think the answer is, "no", specifically because of your requirements. That's just not how REST works; and the limitation is the API side, not a ColdFusion issue.
You could do something similar, assuming you had control over the API end of things too, but it wouldn't be REST.
I think you might actually be able to do this by using the "Keep-Alive" request header with your cfhttp calls. For example:
<cfloop from="1" to="50" index="i">
<cfhttp url="http://mysite.com/getPage.cfm?i=#i#" method="get">
<cfif i LT 50>
<CFHTTPPARAM type="HEADER" name="Connection" value="Keep-Alive">
<cfelse>
<CFHTTPPARAM type="HEADER" name="Connection" value="close">
</cfif>
</cfhttp>
<cfdump var="#cfhttp.filecontent#">
</cfloop>
I haven't tested this, but in theory it should keep the connection to the back end open while you make each of those requests (assuming the backend allows for this and the delay between connections hasn't triggered a time-out). You should be sure that your API response includes a "Content-length" header so that the client (your cfhttp code) will know when each request has completed. You will want to issue an explicit "close" as I've shown to prevent unneeded open connections to the backend.
Related
This morning our Amazon EC2 Windows server became nearly unresponsive, throwing 500 errors intermittently. In the CF log I see a bunch of these errors:
The request has exceeded the allowable time limit Tag: CFQUERY
After rebooting the server, everything is back to normal.
While I hunt down the root cause for this, is it possible to configure CF Admin to send an email when a certain Log File error begins to appear several times in a row, like this one? We have already configured Health Status Checks for outages on EC2, but they never triggered because the server was still running, just incredibly slowly. I would like to be the first responder to situations like this but didn't know because I wasn't actively monitoring the server when it occurred - a user had to alert me to the problem a couple of hours later, which is embarrassing for me.
Thanks in advance!
Don't know if this is best practice, but what I do is utilize the OnError function in the application.cfc for reporting of unhandled errors.
<cffunction name="onError" returnType="void" hint="Runs when an uncaught exception occurs in the application.">
<cfargument name="Exception" required=true/>
<cfargument name="EventName" type="String" required=true/>
<cfmail to="myemail#mydomain.edu" from="myemail#mydomain.edu" subject="Error Alert" type="html">
<cfdump var="#session#">
<cfdump var="#CGI#">
<cfdump var="#Exception#">
</cfmail>
</cffunction>
You could obviously filter down to the types of errors you want reported.
If you just want to check specifically in the function (where the time out occurred), then you should use:
<cffunction name="yourfunction">
<cftry>
...
<cfcatch>
<cfmail to="myemail#mydomain.edu" from="myemail#mydomain.edu" subject="Error:yourfunction">
<cfdump var="#session#">
<cfdump var="#CGI#">
<cfdump var="#cfcatch#">
</cfcatch>
</cftry>
<cffunction>
Query timeouts are not the only source of overwhelmed servers. You might want something more global.
What you can do on the CF Administrator app is to use Alert Configuration. Not only does this allow you to notify yourself if certain conditions are met, but you can also kill the problematic threads.
What we have done in addition to that, is to schedule a script on our development server to attempt an http request to the production server with a one second timeout. If the attempt fails, mail gets sent to people, pagers, and cell phones.
Is there a way to output the raw html of a CFHTTP call? I am trying to see how some of the header authentication information is coming across.
I am open to browser plugins or code updates whichever helps me see what is going on during the cfhttp call.
So for example:
<cfhttp method="get" url="https://test-ows01.mywebsite.com/criminal_api//1.0/service/requests" result="orderList">
<cfhttpparam type="HEADER" name="Authorization" value="Basic #ToBase64("bearer:4EC8B09D3F911764B1DCD3EFA38DFB31")#">
</cfhttp>
what does the above call look like when it happens.
If I am understanding correctly, it sounds more like you want to view the http request sent to the remote server, rather than what is received. Installing a tool like Fiddler will provide very robust debugging, allowing you to view http requests as they happen. (See also the documentation for Enable HTTPS traffic decryption).
Tip for quick debugging, a low-tech hack is to switch the target URL to a separate .cfm script on your server. Inside the script, dump GetHTTPRequestData(), to display the request headers and body sent to the script.
test.cfm
<cfhttp method="get" url="http://localhost/receivingPage.cfm" result="orderList">
<cfhttpparam type="HEADER" name="Authorization"
value="Basic #ToBase64("bearer:4EC8B09D3F911764B1DCD3EFA38DFB31")#">
</cfhttp>
receivingPage.cfm
<cfdump var="#GetHTTPRequestData()#">
You can use requestcatcher.com for this.
It let you make a personal subdomain and then you can send your request to that URL. Very handy. Helped me alot for a complex SOAP integration.
We recently migrated our site from application.CFM to application.CFC. The CFM version could handle our excessive bot traffic, but our CFC version can't. We are trying to figure out why the CFC problem is. In the meantime, we are trying to limit bot traffic.
Currently, I am looking for a solution within the code base to slow bot traffic. We can do this by looking at the user agent as well as IP address.
We have used the code below to successfully stop many bots.
<cffunction name="OnRequestStart">
<cfif find("bot", cgi.httP_USER_AGENT)>
<cfabort>
</cfif>
</cffunction>
Obviously, we do want some bot traffic. But right now, we can't handle all of the bot traffic. It appears that as soon as we you abort to stop a request, another request is right behind it and eventually they bring down our server.
Instead of stopping the bots, what would the ramifications be of using CFTHREAD to slow the bots?
<cffunction name="OnRequestStart">
<cfif find("bot", cgi.httP_USER_AGENT)>
<cfthread action="sleep" duration="5"></cfthread>
</cfif>
</cffunction>
Would using CFTHREAD just stack up the requests and eventually kill our server or would the bots respond with fewer requests per hour?
Just to give you a brief intro on what am I trying to achieve.
I need to scrape a value for each month from a public website.
As an example, for a particular status in the month of May, 2014; the TOTAL value is 224,481 (at the bottom right part of the page). I need to capture this value and store in the database for each of the month.
The following URL will need to be run by changing the parameters for each month.
VEEC Website with params
To achieve this, I am trying to get the HTML using CFHTTP as follows and then using JSOUP will scrape the intended value from the returned HTML before storing into the database.
<cfset f_url = "https://www.veet.vic.gov.au/Public/PublicRegister/Search.aspx">
<cfhttp method="GET" url="#f_url#">
<cfhttpparam type="Header" name="Accept-Encoding" value="*">
<cfhttpparam type="Header" name="TE" value="deflate;q=0">
<cfhttpparam name="CreatedFrom" type="URL" value="#StartDate#">
<cfhttpparam name="CreatedTo" type="URL" value="#EndDate#">
<cfhttpparam name="Status" type="URL" value="PRP,PRV">
</cfhttp>
<cfdump var="#cfhttp#">
I'm getting the connection failure error when I try to run the code snippet.
Please advise if there is an alternate way of doing the same.
This has to be implemented just to gather data on the development side and not meant for production use.
Edit (Didn't want to Delete the question): I had to use the internet proxy to access the website in the CFHTTP tag to make it work. My bad for not checking it earlier. Hopefully, it might help someone in future. Thanks everyone.
Sounds like the classic secure cert thing.
CF has limited trust of secure certs. If the SSL cert at the destination you are calling is not one the CF likes, you will get a connection failure message. What you have to do is import the SSL cert in to the Java Keystore that CF is using.
Here is an old version of how to do it:
http://mkruger.cfwebtools.com/index.cfm?mode=entry&entry=8E44925A-B73D-E3AD-709D4E02FD6D4588
I have a web service setup for on a small part of a website and while the site overall gets a good amount of traffic this particular service does not. Once a day when I go to send a request through the web service it will fail on the first attempt, but retrying the request a second time works just fine. It's as if it was no longer cached in memory and times out while starting up.
Is there a way to keep this service active either on my end or on the web service provider's end which is also CF app (separate division of our company)? It's a bit hard to troubleshoot because it only happens once after a long period. And I don't want to setup a separate process just to keep pinging this service.
If the server is being restarted regularly between calls to the template, ensure the "save class files" setting is enabled in the administrator (under caching) to prevent the template from being recompiled after each server reload.
You can try to use following method on a webservice client side.
CF7+ got built-in coldfusion.server.ServiceFactory Java service.
Code can look like
<cftry>
<!--- here goes attempt to invoke service method, maybe dummy "ping" --->
<cfcatch type="any">
<!--- trying to refresh WSDL --->
<cfset createObject("java","coldfusion.server.ServiceFactory").XmlRpcService.refreshWebService(ServiceURL) />
</cfcatch>
</cftry>
<!--- usual code --->
Hope this helps.
Note: this factory contains a lot of useful methods, but almost zero documentation over the internet. Good idea would be to dump it and explore a bit.
Try increasing the requesttimeout and see if that helps.