I need to emulate a real http request via cfhttp.
I was getting rss feed with ColdFusion, but tonight they started to block my request and send an index page in response instead of rss fead.
I added useragent for cfhttp, but it doesn't help.
Opera, Firefox and Chrome open feed correctly from the same computer.
Yep, thanks. I sniffed all HTTP headers which browser sends to the site and then emulated them in cfhttp request. The solution is:
<cfhttp url="http://example.com/feed"
useragent="Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.7 (KHTML, like Gecko) Chrome/5.0.391.0 Safari/533.7"
result="httpresult"
redirect="false"
>
<cfhttpparam type="header" name="HTTP_REFERER" value="http://example.com/feed/" >
<cfhttpparam type="header" name="Accept-Encoding" value="gzip,deflate,sdch" >
<cfhttpparam type="header" name="Proxy-Connection" value="keep-alive" >
<cfhttpparam type="header" name="Accept" value="application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5">
<cfhttpparam type="header" name="Accept-Language" value="en-US,en;q=0.8">
<cfhttpparam type="header" name="Accept-Charset" value="ISO-8859-1,utf-8;q=0.7,*;q=0.3">
<cfhttpparam type="cookie" name="some-cookie" value="1">
I would guess that the site with the RSS feed is sniffing the User Agent still, and the CFHTTP one isn't set to one that the site is using. Use a HTTP Proxy Sniffer (ie Charles HTTP Proxy) to record the HTTP request of a browser that is displaying the RSS feed correctly, then try using CFHTTP with the same User Agent string as a previously successful request.
If it still doesn't work, use the 'proxyport' and 'proxyserver' attributes of CFHTTP to run the ColdFusion request through your HTTP sniffer and check to make sure the User Agent is being set correctly and compare to a working request.
Related
I have a question regarding sending a Web API call to sendgrid:
The following documentation is used to retrieve advanced stats:
http://sendgrid.com/docs/API_Reference/Web_API/Statistics/statistics_advanced.html
In the "Call" section, the following has been mentioned:
POST https://api.sendgrid.com/api/stats.getAdvanced.json
POST Data api_user=your_sendgrid_username&api_key=your_sendgrid_password&start_date=2013-01-01&end_date=2013-01-02&data_type=global
I understand that I need to use cfhttp for the Web API call and I should be
mentioning "https://api.sendgrid.com/api/stats.getAdvanced.json" in the url parameter
of the cfhttp tag.
Where do I need to mention api_user and api_key portion? I understand that I will have
to mention my sendgrid's account username and password here.
For Explanation Purpose let's consider the following dummy values for:
api_user = stack
api_key = 123456
Please let me know. I am trying to send an API call an get the data back in JSON format.
Thanks
ATTEMPT #1
I am using the following code:
<cfhttp url="https://api.sendgrid.com/api/stats.getAdvanced.json" method="POST" result="returnStruct">
<cfhttpparam name="api_user" value="stack" type="formfield">
<cfhttpparam name="api_key" value="123456" type="formfield">
<!--- <cfhttpparam name="days" value="5" type="formfield"> --->
<cfhttpparam name="start_date" value="2013-12-06" type="formfield">
<!--- <cfhttpparam name="end_date" value="2013-12-09" type="formfield"> --->
</cfhttp>
<cfdump var="#returnStruct#">
I am getting the following error in the form of Structure. Am I doing something wrong above? Please check my code below.
Charset [empty string]
ErrorDetail [empty string]
Filecontent {"error": "error in data_type: data_type is required"}
Header HTTP/1.1 400 Bad Request Content-Type: text/html Connection: close Date: Tue, 10 Dec 2013 06:21:19 GMT Server: nginx/1.4.2
Mimetype text/html
Responseheader
struct
Connection close
Content-Type text/html
Date Tue, 10 Dec 2013 06:21:19 GMT
Explanation Bad Request
Http_Version HTTP/1.1
Server nginx/1.4.2
Status_Code 400
Statuscode 400 Bad Request
Text YES
I am wondering, which data_type it is referring to, as the documentation doesn't talks about it.
Typically, you would use <cfhttpparam> for each of the keys:
<cfhttp url="https://api.sendgrid.com/api/stats.getAdvanced.json" method="POST">
<cfhttpparam name="api_user" value="some_username_goes_here" type="formfield">
<cfhttpparam name="api_key" value="some_api_key_goes_here" type="formfield">
...more params...
</cfhttp>
Using ColdFusion, I am trying to POST JSON to an API. Here is the code I have so far -
<cfhttp url="#url#" method="post" result="httpResp" timeout="60">
<cfhttpparam type="header" name="Content-Type" value="application/json" />
<cfhttpparam type="body" value="#serializeJSON(jsonStr)#">
</cfhttp>
An example of the JSON is here -
{
"booking":{
"username" : "#username#",
"password" : "#password#",
"customerEmail" : "#customer_email_address#",
"firstName" : "#customer_firstname#",
"lastName" : "#customer_surname#",
"telephoneNumber" : "#customer_mobile_number#",
"guestNumber" : #url.guests#,
"unitNumber" : #url.location#,
"eventDate" : "#LSDateFormat(url.when,'dd/mm/yyyy')#"
}
}
When I pass this JSON to the API URL with POSTMAN client in Chrome, everything is good! however when I process this in CF, I simply get a bad request error from the API. I realise that message is no use that is simple whats being set in the API.
If I remove the content-type from POSTMAN client in Chrome, I get the same message. So I am "assuming" That the content-type is not being sent or over written somehow in CF.
Can anyone point me in the right direction?
Thanks
Issue was with the API not accepting what I thought it required.
We are trying to interact with a RESTful web service that expects a file.
I set the name of the field to data (as required by the API) and then specify the file as an absolute path. When the file makes it to the server, the filename in the HTTP transaction is the complete absolute path.
This causes a problem with the API as the full path is then recorded as the "FileName".
How do I get ColdFusion to report only the file name rather than the full path?
We are using ColdFusion 9.
Here is the CFML:
<cfhttp url="http://server/testcode"
port="9876"
method="post"
result="Content">
<cfhttpparam type="file"
name="data"
file="c:\temp\testfile.txt">
</cfhttp>
Here are some examples of the HTTP interactions with different browsers:
CFHTTP 9
-------------------------------7d0d117230764
Content-Disposition: form-data; name="data"; filename="c:\temp\testfile.txt"
Content-Type: text/plain
This is the text, really long, well, not really.
-------------------------------7d0d117230764--
IE8
-----------------------------7db370d80e0a
Content-Disposition: form-data; name="FileField"; filename="C:\temp\testfile.txt"
Content-Type: text/plain
This is the text, really long, well, not really.
-----------------------------7db370d80e0a--
Chrome 13
------WebKitFormBoundaryDnpFVJwCsZkzTGDc
Content-Disposition: form-data; name="FileField"; filename="testfile.txt"
Content-Type: text/plain
This is the text, really long, well, not really.
Firefox 6
-----------------------------22798303036224
Content-Disposition: form-data; name="FileField"; filename="testfile.txt"
Content-Type: text/plain
This is the text, really long, well, not really.
-----------------------------22798303036224--
Apparently IE8 and CFHTTP both do the same thing (add "c:\temp" to the file name). I'm not sure what the spec for HTTP is, but it would be nice if there was a way to get CFHTTP to leave the path off.
Is there any way to do this?
I ran into a problem similar to yours, once. I didn't care about excluding the path, but I wanted to send a different filename than the name of the file on my server's filesystem. I could not find a way to do it using CF tags at all, but I was able to get it to work by dropping into Java. I used org.apache.commons.httpclient, which ships with CF9 IIRC. It goes something like this (pardon any typos, I'm transposing from more complicated code):
oach = 'org.apache.commons.httpclient';
oachmm = '#oach#.methods.multipart';
method = createObject('java', '#oach#.methods.PostMethod').init(post_uri);
filePart = createObject('java', '#oachmm#.FilePart').init(
'fieldname',
'filename',
createObject('java', 'java.io.File').init('filepath')
);
method.setRequestEntity(
createObject('java', '#oachmm#.MultipartRequestEntity').init(
[ filePart ],
method.getParams()
)
);
status = createObject('java', '#oach#.HttpClient').init().executeMethod(method);
method.releaseConnection();
I see that the content type is text/plain so first I think that you need to add the multipart property on the CFHTTP
<cfhttp url="http://server/testcode"
port="9876"
method="post"
result="Content"
multipart = "yes">
<cfhttpparam type="file"
name="data"
file="c:\temp\testfile.txt">
</cfhttp>
Could solve your issue.
The only difference I see between all of the posts is that CF is sending name="data" while everything else is sending name="FileField". If the other browser submissions are correct, then I would change your cfhttpparam:
<cfhttpparam type="file"
name="FileField"
file="c:\temp\testfile.txt">
or even try sending an additional FileName parameter:
<cfhttpparam type="file"
name="data"
file="c:\temp\testfile.txt" />
<cfhttpparam type="formField"
name="FileName"
value="testfile.txt" />
So I was able to get access to the API and made it work. Here is the code for this specific part (as I assume that you were able to login and get a document guid).
<!--- upload a document --->
<cfhttp method="post" url="<path to watchdox api upload>/#local.guid#/upload">
<cfhttpparam type="header" name="Content-type" value="multipart/form-data">
<cfhttpparam type="header" name="x-wdox-version" value="1.0">
<cfhttpparam type="header" name="x-wdox-ssid" value="#local.xwdoxssid#" >
<cfhttpparam type="formfield" name="filename" value="testfile.txt" >
<cfhttpparam type="file" file="c:\temp\testfile.txt" name="data" >
</cfhttp>
Hope it will help.
I'm searching for a way to create a new CouchDB user without using Futon or Curl... just a straight http request.
One way I found (http://stackoverflow.com/questions/3456256/error-creating-user-in-couchdb-1-0) puts a JSON doc to "http://localhost:5984/_users/org.couchdb.user:username" to create a user.
I have attempted the following:
<cfhttp url="http://127.0.0.1/_users/org.couchdb.user:xyz_company" port="5984" method="PUT" username="#variables.couch_username#" password="#variables.couch_password#">
<cfhttpparam type="header" name="Content-Type" value="application/json">
<cfhttpparam type='body' name='org.couchdb.user:xyz_company' value='{"roles":[],"name":"xyz_company","salt":"3B33BF09-26B9-D60A-8F469D01286E9590","id":"org.couchdb.user:xyz_company","password_sha":"096EA41A5A81EA1507F2C6F7EDC364C0B82694AC","type":"user"}'>
I keep receiving the following back from Couch:
cfhttp.statuscode = 405 Method Not Allowed
cfhttp.filecontent = Method Not Allowed; The requested method PUT is not allowed for the URL /_users/org.couchdb.user:xyz_company
Any thoughts or suggestions?
UPDATE:
I edited my code based on Marcello's suggestions. I still receive the same 405 Method Not Allowed error. Here is the code now:
<cfhttp url="http://127.0.0.1/_users/org.couchdb.user:xyz_company" port="5984" method="PUT" username="#variables.couch_username#" password="#variables.couch_password#"><cfhttpparam type="header" name="Content-Type" value="application/json;charset=UTF-8"><cfhttpparam type='body' value='{"roles":[],"name":"xyz_company","salt":"3B33BF09-26B9-D60A-8F469D01286E9590","_id":"org.couchdb.user:xyz_company","password_sha":"096EA41A5A81EA1507F2C6F7EDC364C0B82694AC","type":"user"}'></cfhttp>
Any more suggestions? Thank you!
curl is a straight http request. There are other ways to create such requests: you can craft them with your browser; you can use a different program (e.g. wget); or even write your own (e.g. in Python or in JavaScript with V8 or Rhino).
I'm trying to obtain credentials from ning network using Coldfusion 9, so first this is the curl syntax to test the api :
curl -k https://external.ningapis.com/xn/rest/mbdevsite/1.0/Token?xn_pretty=true -u devshare#megabase.tn:mbdev2011 -d "oauth_signature_method=PLAINTEXT&
oauth_consumer_key=741ab68b-63fb-4949-891c-9e88f5143034&oauth_signature=36da2ea8
-10fb-48cc-aaa4-c17c551c6b87%26"
and it returns :
{
"success" : true,
"entry" : {
"author" : "1o0butfek0b3p",
"oauthConsumerKey" : "741ab68b-63fb-4949-891c-9e88f5143034",
"oauthToken" : "46f1e137-549a-4d9d-ae05-62782debfd3d",
"oauthTokenSecret" : "9f778ab5-db8e-4f3e-b17f-61d249b91f0a"
},
"resources" : {
}
then i translated it to coldfusion like this :
<cfhttp
method="post"
url="https://external.ningapis.com/xn/rest/mbdevsite/1.0/Token"
username="devshare#megabase.tn"
password="mbdev2011">
<cfhttpparam type="header" name="content-type" value="application/x-www-form-urlencoded">
<cfhttpparam name="oauth_signature_method" type="FormField" value="PLAINTEXT"/>
<cfhttpparam name="oauth_consumer_key" type="FormField" value="741ab68b-63fb-4949-891c-9e88f5143034"/>
<cfhttpparam name="oauth_signature" type="FormField" value="36da2ea8-10fb-48cc-aaa4-c17c551c6b87%26"/>
</cfhttp>
<cfoutput>
#cfhttp.fileContent#
</cfoutput>
and the response is always :
{"success":false,"reason":"The oauth_signature is invalid. That is, it doesn't match the signature computed by the Service Provider.","status":401,"code":1,"subcode":12,"trace":"3d874587-072b-4877-b27e-b84ee2e2b537"}
does somebody have idea about what could be this error ??
url and login info are real for who wants to help by testing
Thank you..
Don't disclose your username & password in public forums. Better you change this user name & password after this issue completion :)
Your oauth_signature is 36da2ea8-10fb-48cc-aaa4-c17c551c6b87& not "36da2ea8-10fb-48cc-aaa4-c17c551c6b87%26"
I got the success response & it is working perfectly.
<cfhttp
method="post"
url="https://external.ningapis.com/xn/rest/mbdevsite/1.0/Token"
username="devshare#megabase.tn"
password="mbdev2011">
<cfhttpparam type="header" name="content-type" value="application/x-www-form-urlencoded">
<cfhttpparam name="oauth_signature_method" type="FormField" value="PLAINTEXT"/>
<cfhttpparam name="oauth_consumer_key" type="FormField" value="741ab68b-63fb-4949-891c-9e88f5143034"/>
<cfhttpparam name="oauth_signature" type="FormField" value="36da2ea8-10fb-48cc-aaa4-c17c551c6b87&"/>
</cfhttp>
Any specific reason why you're using cURL instead of cfhttp?
There's a nice library on RIAForge:
OAuth
that will help you with dealing with OAuth. The issue is probably with the parameters encoding.
Oh - and you shouldn't be posting your OAuth credentials.
UPDATE:
I'm afraid using OAuth isn't as simple as just calling cfhttp with params.
The parameters need to be in certain order, you need to sign the whole request using appropriate method (plain text in your case). The signing process also includes time stamp so you can't test your code with the values from the example as they definitely won't work.
If you download the RIAForge libraries there's an "\examples_external" folder there and twitter.cfm - you'll find all that I've mentioned there.
A bit of code from there to show what I mean:
<!--- Create empty token --->
<cfset oReq = CreateObject("component", "oauth.oauthrequest").fromConsumerAndToken(
oConsumer = oConsumer,
oToken = oToken,
sHttpMethod = "GET",
sHttpURL = sTokenEndpoint,stparameters= Parameters )>
<!--- Sign the request --->
<cfset oReq.signRequest(
oSignatureMethod = oReqSigMethodSHA,
oConsumer = oConsumer,
oToken = oToken)>
<!--- Get the request token --->
<cfhttp url="#oREQ.getString()#" method="get" result="tokenResponse"/>
Of course there's lots of bits missing before and after it.
You might check out Ben Nadel's blog post on OAuth. He covers some of the things you may be running into.