google apps script ==> UrlFetchApp, method GET and cookie - cookies

I use the UrlFetchApp to send the user and pwd (method POST). After get the cookie, and use in other request (method GET). But this new request not working, I think that this cookie not has correct use in this new request. Can anyone help me?
var opt ={
"method":"post",
"User-Agent" : "Mozilla/5.0",
"Accept" : "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language" : "en-US,en;q=0.5",
"payload": this.payload.toString(),
"followRedirects" : false
};
var response = UrlFetchApp.fetch("https://edas.info/addTopic.php?c=19349",opt);
var resp1=response.getContentText();
Logger.log(resp1);
response.getResponseCode();
var headers = response.getAllHeaders();
var cookies = headers['Set-Cookie'];
for (var i = 0; i < cookies.length; i++) {
cookies[i] = cookies[i].split( ';' )[0];
};
opt = {
"method" : "get",
"User-Agent" : "Mozilla/5.0",
"Accept" : "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language" : "en-US,en;q=0.5",
"headers": {
"Cookie": cookies.join(';')
},
"followRedirects" : false
};
response = UrlFetchApp.fetch("https://edas.info/addTopic.php?c=19349", opt);
var resp1=response.getContentText();
Logger.log(resp1);

First off, thanks you for the snippet of code, this got me started with processing cookies in such script. I encountered an issue that was possibly your problem. Sometimes a Web page returns an array of cookies, and then your code works fine. Sometimes it returns a single string (instead of an array of one string). So I had to disambiguate with a test like:
if ( (cookies != null) && (cookies[0].length == 1) ) {
cookies = new Array(1);
cookies[0] = headers['Set-Cookie'];
}

I cannot give you specific help for your problem, one pointer though, as found here
Cookie handling in Google Apps Script - How to send cookies in header?
As https://stackoverflow.com/users/1435550/thierry-chevillard put it:
Be aware also that GAS uses Google IPs. It can happen that two consecutive fetch use different IPs. The server your are connecting to may be session-IP dependant.
Does your code run on the local development server and only fail once deployed to App Engine?
Or does it fail locally, as well?

Related

Capturing Twilio's outgoing call status on google appscript

I am trying to get the status of my outgoing twilio calls and update them onto the spreadsheet. "Completed, Busy, No-Answer, Cancelled, Failed".
However, I do not know where to include the StatusCallBackEvent and how to access the retrieved status from Google Apps Script.
Here is the code which I have to initiate an outbound call.
function makeCall(to) {
var call_url = "https://api.twilio.com/2010-04-01/Accounts/" + TWILIO_ACCOUNT_SID + "/Calls.json";
var payload = {
"To": "+" + String(to),
"From" : TWILIO_NUMBER,
"Url": "http://a1fb888ec032.ngrok.io/" +"voice",
"Method": "GET"
};
var options = {
"method" : "post",
"payload" : payload
};
options.headers = {
"Authorization" : "Basic " + Utilities.base64Encode(TWILIO_ACCOUNT_SID + ":" + TWILIO_AUTH_TOKEN)
};
var response = UrlFetchApp.fetch(call_url, options);
UrlFetchApp.fetch(call_url, options);
return JSON.parse(response);
}
Twilio developer evangelist here.
To receive the status of the call, you need to set a StatusCallback URL when you create the call. The URL should point towards a URL which you control that can update your spreadsheet when it receives an HTTP request from Twilio.
var payload = {
"To": "+" + String(to),
"From" : TWILIO_NUMBER,
"Url": "http://a1fb888ec032.ngrok.io/" +"voice",
"Method": "GET",
"StatusCallback": "https://SOME_URL"
};
You can use Google Sheets and App Script to set up a web application that can receive webhooks like this. Briefly, you need to create script that has a doPost method (Twilio webhooks are POST requests by defaults) and returns a successful response. Twilio statusCallback webhooks don't expect any content in the response, so you can use an empty text output.
Something like this might work:
function doPost(event) {
var callSid = event.parameter("CallSid");
var status = event.parameter("CallStatus");
// update spreadsheet with call status
return ContentService.createTextOutput('');
}
Check out the documentation on turning scripts into web apps and Twilio call status callbacks for more details.

Cookies added in an OpenIdConnect middleware EventHandler do not persist when not on localhost

So I am using OpenId Connect and the AddOpenIdConnect configuration method to setup Authentication with Auth0. I have fully tested it locally using localhost as the domain of my cookies and everything works great! However, as soon as I release this to a production environment the cookies no longer seem to persist. The crazy thing is I am logging the context.Response.Headers directly after the context.Response.Cookies.Append(...) method and locally they are printed as expected, but in production they do not.
OnTokenValidated = (context) =>
{
var accessToken = context.TokenEndpointResponse.AccessToken;
var domain = HostingEnvironment.IsProduction() ? $"{cookieOptions.Domain}.{cookieOptions.Extension}" : "localhost";
context.Response.Cookies.Append("access_token", accessToken, new CookieOptions
{
Domain = domain,
SameSite = Microsoft.AspNetCore.Http.SameSiteMode.Lax,
Expires = DateTimeOffset.Now.AddHours(10),
HttpOnly = true,
Secure = HostingEnvironment.IsProduction(),
Path = "/"
});
var idToken = context.TokenEndpointResponse.IdToken;
context.Response.Cookies.Append("id_token", idToken, new CookieOptions
{
Domain = domain,
SameSite = Microsoft.AspNetCore.Http.SameSiteMode.Lax,
Expires = DateTimeOffset.Now.AddHours(10),
HttpOnly = false,
Secure = HostingEnvironment.IsProduction(),
Path = "/"
});
// Right here is where I can see the difference. Even though I am printing out the headers directly after appending them, they never are set in production.
context.HttpContext.RequestServices.GetService<ILogger<Startup>>().LogWarning($"[HEADERS] {JsonConvert.SerializeObject(context.Response.Headers)}");
return Task.CompletedTask;
},
I even went as far as looked at the implementation of IResponseCookies.Append(...) to see what could be going on but its very simple and I don't see any way it could be failing to add these to the Headers.
https://github.com/aspnet/HttpAbstractions/blob/bc7092a32b1943c7f17439e419d3f66cd94ce9bd/src/Microsoft.AspNetCore.Http/Internal/ResponseCookies.cs#L50
public void Append(string key, string value, CookieOptions options)
{
if (options == null)
{
throw new ArgumentNullException(nameof(options));
}
var setCookieHeaderValue = new SetCookieHeaderValue(
Uri.EscapeDataString(key),
Uri.EscapeDataString(value))
{
Domain = options.Domain,
Path = options.Path,
Expires = options.Expires,
MaxAge = options.MaxAge,
Secure = options.Secure,
SameSite = (Net.Http.Headers.SameSiteMode)options.SameSite,
HttpOnly = options.HttpOnly
};
var cookieValue = setCookieHeaderValue.ToString();
Headers[HeaderNames.SetCookie] = StringValues.Concat(Headers[HeaderNames.SetCookie], cookieValue);
}
Ok so I finally figured it all out. The Auth0 Quickstart for ASP .NET Core 3.x contains the following lines of code:
services.Configure<CookiePolicyOptions>(options =>
{
// This lambda determines whether user consent for non-essential cookies is needed for a given request.
options.CheckConsentNeeded = context => true;
options.MinimumSameSitePolicy = SameSiteMode.None;
});
This enforces "consent" to be required for all Cookies. I have yet to figure out what this means exactly as the MS docs seem to be very limited. After changing this to false it all started working again. false is the default, it was just the Auth0 Quickstart that added it for some reason.
What further lead to my confusion was I tried to look at the implementation of Cookies.Append(...) and I found the ResponseCookies class that implements IResponseCookies and ended up copy/pasting that implementation into my code to get it to work. After digging, I realized its not this implementation but instead this one for ResponseCookiesWrapper that is actually being used. As you can sees this implementation uses all the extra "consent" logic to make sure you are allowed to add a cookie which apparently I wasn't.
UPDATE:
Sorry what I mentioned was that I had this working only on localhost, but you might ask why it wasn't failing on localhost and PROD. That is because even though the Quickstart shows it hardcoded to true, the actual example code on GitHub uses HostingEnvironment.IsProduction() which is what I had as well.

Cannot login from Chrome extension to another site

I am trying to login into an online shopping site from Chrome extension, then scrape it.
There are three steps that happens when you do this from a browser.
Visit the site, the site gives you a cookie.
Go to the login page, send the aforementioned cookie, and username and password as params in POST. The site gives you 5 more cookies.
call GET to a path within the site, along with five + one = total six cookies.
Now that works well in browser. If I copy all the cookies from the browser into curl, the following call would work.
curl -X GET --header "Cookie: cookie1=value1; cookie2=value2; cookie3=value3; cookie4=value4; cookie5=value5; cookie6=value6" http://www.sitedomain.com/product_info.php?products_id=350799
However, how can I repeat this behavior in Google Chrome extension? Nothing I do seems to be working, yes I added the domain to the manifest.json and I've tried request, requestify (both browserified) and XMLHttpRequest. The requests seem fine (do include cookies) but the response I get seems to be the one I get when the site any receiving cookie.
manifest.json
{
"manifest_version": 2,
"name": "Getting started example",
"description": "This extension shows a Google Image search result for the current page",
"version": "1.0",
"browser_action": {
"default_icon": "icon.png",
"default_popup": "popup.html"
},
"permissions": [
"activeTab",
"cookies",
"https://sitedomain.com/*",
"http://sitedomain.com/*",
"https://www.sitedomain.com/*",
"http://www.sitedomain.com/*"
]
}
The code I use (request version):
var request = require('request');
request = request.defaults({jar: true});
var jar = request.jar();
var cookie1 = request.cookie('cookie1=value1');
var cookie2 = request.cookie('cookie2=value2');
var cookie3 = request.cookie('cookie3=value3');
var cookie4 = request.cookie('cookie4=value4');
var cookie5 = request.cookie('cookie5=value5');
var cookie6 = request.cookie('cookie6=value6');
var url = "http://www.sitedomain.com";
jar.setCookie(cookie1, url);
jar.setCookie(cookie2, url);
jar.setCookie(cookie3, url);
jar.setCookie(cookie4, url);
jar.setCookie(cookie5, url);
jar.setCookie(cookie6, url);
request({uri: http://www.sitedomain.com/product_info.php?products_id=350799, jar: jar}, function (error, response, html) {
if (!error && response.statusCode == 200) {
//breakpoint here and we did not receive the page after login
}
}
Any help would be appreciated.

Identity Server 3 Facebook Login Get Email

Identity server is implemented and working well. Google login is working and is returning several claims including email.
Facebook login is working, and my app is live and requests email permissions when a new user logs in.
The problem is that I can't get the email back from the oauth endpoint and I can't seem to find the access_token to manually request user information. All I have is a "code" returned from the facebook login endpoint.
Here's the IdentityServer setup.
var fb = new FacebookAuthenticationOptions
{
AuthenticationType = "Facebook",
SignInAsAuthenticationType = signInAsType,
AppId = ConfigurationManager.AppSettings["Facebook:AppId"],
AppSecret = ConfigurationManager.AppSettings["Facebook:AppSecret"]
};
fb.Scope.Add("email");
app.UseFacebookAuthentication(fb);
Then of course I've customized the AuthenticateLocalAsync method, but the claims I'm receiving only include name. No email claim.
Digging through the source code for identity server, I realized that there are some claims things happening to transform facebook claims, so I extended that class to debug into it and see if it was stripping out any claims, which it's not.
I also watched the http calls with fiddler, and I only see the following (apologies as code formatting doesn't work very good on urls. I tried to format the querystring params one their own lines but it didn't take)
(facebook.com)
/dialog/oauth
?response_type=code
&client_id=xxx
&redirect_uri=https%3A%2F%2Fidentity.[site].com%2Fid%2Fsignin-facebook
&scope=email
&state=xxx
(facebook.com)
/login.php
?skip_api_login=1
&api_key=xxx
&signed_next=1
&next=https%3A%2F%2Fwww.facebook.com%2Fv2.7%2Fdialog%2Foauth%3Fredirect_uri%3Dhttps%253A%252F%252Fidentity.[site].com%252Fid%252Fsignin-facebook%26state%3Dxxx%26scope%3Demail%26response_type%3Dcode%26client_id%3Dxxx%26ret%3Dlogin%26logger_id%3Dxxx&cancel_url=https%3A%2F%2Fidentity.[site].com%2Fid%2Fsignin-facebook%3Ferror%3Daccess_denied%26error_code%3D200%26error_description%3DPermissions%2Berror%26error_reason%3Duser_denied%26state%3Dxxx%23_%3D_
&display=page
&locale=en_US
&logger_id=xxx
(facebook.com)
POST /cookie/consent/?pv=1&dpr=1 HTTP/1.1
(facebook.com)
/login.php
?login_attempt=1
&next=https%3A%2F%2Fwww.facebook.com%2Fv2.7%2Fdialog%2Foauth%3Fredirect_uri%3Dhttps%253A%252F%252Fidentity.[site].com%252Fid%252Fsignin-facebook%26state%3Dxxx%26scope%3Demail%26response_type%3Dcode%26client_id%3Dxxx%26ret%3Dlogin%26logger_id%3Dxxx
&lwv=100
(facebook.com)
/v2.7/dialog/oauth
?redirect_uri=https%3A%2F%2Fidentity.[site].com%2Fid%2Fsignin-facebook
&state=xxx
&scope=email
&response_type=code
&client_id=xxx
&ret=login
&logger_id=xxx
&hash=xxx
(identity server)
/id/signin-facebook
?code=xxx
&state=xxx
I saw the code parameter on that last call and thought that maybe I could use the code there to get the access_token from the facebook API https://developers.facebook.com/docs/facebook-login/manually-build-a-login-flow
However when I tried that I get a message from the API telling me the code has already been used.
I also tried to change the UserInformationEndpoint to the FacebookAuthenticationOptions to force it to ask for the email by appending ?fields=email to the end of the default endpoint location, but that causes identity server to spit out the error "There was an error logging into the external provider. The error message is: access_denied".
I might be able to fix this all if I can change the middleware to send the request with response_type=id_token but I can't figure out how to do that or how to extract that access token when it gets returned in the first place to be able to use the Facebook C# sdk.
So I guess any help or direction at all would be awesome. I've spent countless hours researching and trying to solve the problem. All I need to do is get the email address of the logged-in user via IdentityServer3. Doesn't sound so hard and yet I'm stuck.
I finally figured this out. The answer has something to do with Mitra's comments although neither of those answers quite seemed to fit the bill, so I'm putting another one here. First, you need to request the access_token, not code (authorization code) from Facebook's Authentication endpoint. To do that, set it up like this
var fb = new FacebookAuthenticationOptions
{
AuthenticationType = "Facebook",
SignInAsAuthenticationType = signInAsType,
AppId = ConfigurationManager.AppSettings["Facebook:AppId"],
AppSecret = ConfigurationManager.AppSettings["Facebook:AppSecret"],
Provider = new FacebookAuthenticationProvider()
{
OnAuthenticated = (context) =>
{
context.Identity.AddClaim(new System.Security.Claims.Claim("urn:facebook:access_token", context.AccessToken, ClaimValueTypes.String, "Facebook"));
return Task.FromResult(0);
}
}
};
fb.Scope.Add("email");
app.UseFacebookAuthentication(fb);
Then, you need to catch the response once it's logged in. I'm using the following file from the IdentityServer3 Samples Repository, which overrides (read, provides functionality) for the methods necessary to log a user in from external sites. From this response, I'm using the C# Facebook SDK with the newly returned access_token claim in the ExternalAuthenticationContext to request the fields I need and add them to the list of claims. Then I can use that information to create/log in the user.
public override async Task AuthenticateExternalAsync(ExternalAuthenticationContext ctx)
{
var externalUser = ctx.ExternalIdentity;
var claimsList = ctx.ExternalIdentity.Claims.ToList();
if (externalUser.Provider == "Facebook")
{
var extraClaims = GetAdditionalFacebookClaims(externalUser.Claims.First(claim => claim.Type == "urn:facebook:access_token"));
claimsList.Add(new Claim("email", extraClaims.First(k => k.Key == "email").Value.ToString()));
claimsList.Add(new Claim("given_name", extraClaims.First(k => k.Key == "first_name").Value.ToString()));
claimsList.Add(new Claim("family_name", extraClaims.First(k => k.Key == "last_name").Value.ToString()));
}
if (externalUser == null)
{
throw new ArgumentNullException("externalUser");
}
var user = await userManager.FindAsync(new Microsoft.AspNet.Identity.UserLoginInfo(externalUser.Provider, externalUser.ProviderId));
if (user == null)
{
ctx.AuthenticateResult = await ProcessNewExternalAccountAsync(externalUser.Provider, externalUser.ProviderId, claimsList);
}
else
{
ctx.AuthenticateResult = await ProcessExistingExternalAccountAsync(user.Id, externalUser.Provider, externalUser.ProviderId, claimsList);
}
}
And that's it! If you have any suggestions for simplifying this process, please let me know. I was going to modify this code to do perform the call to the API from FacebookAuthenticationOptions, but the Events property no longer exists apparently.
Edit: the GetAdditionalFacebookClaims method is simply a method that creates a new FacebookClient given the access token that was pulled out and queries the Facebook API for the other user claims you need. For example, my method looks like this:
protected static JsonObject GetAdditionalFacebookClaims(Claim accessToken)
{
var fb = new FacebookClient(accessToken.Value);
return fb.Get("me", new {fields = new[] {"email", "first_name", "last_name"}}) as JsonObject;
}

Google Apps Script and cookies

I am trying to Post and get a cookie. I am a newbie and this is a learning project for me. My impression is that if you use 'set-cookie' one should be able to see an additional 'set-cookie' in the .toSource. (I am trying to accomplish this on Google Apps Site if that makes a difference.) Am I missing something? Here is my code:
function setGetCookies() {
var payload = {'set-cookie' : 'test'};
var opt2 = {'headers':payload, "method":"post"};
UrlFetchApp.fetch("https://sites.google.com/a/example.com/blacksmith", opt2);
var response = UrlFetchApp.fetch("https://sites.google.com/a/example.com/blacksmith")
var openId = response.getAllHeaders().toSource();
Logger.log(openId)
var AllHeaders = response.getAllHeaders();
for (var prop in AllHeaders) {
if (prop.toLowerCase() == "set-cookie") {
// if there's only one cookie, convert it into an array:
var myArray = [];
if ( Array.isArray(AllHeaders[prop]) ) {
myArray=AllHeaders[prop];
} else {
myArray[0]=AllHeaders[prop];
}
// now process the cookies
myArray.forEach(function(cookie) {
Logger.log(cookie);
});
break;
}
}
}
Thanks in advance! I referenced this to develop the code: Cookie handling in Google Apps Script - How to send cookies in header?
Open to any advice.
When you aren't logged in Google Sites won't set any cookies in the response. UrlFetchApp doesn't pass along your Google cookies, so it will behave as if you are logged out.
First the cookie you want to send whose name is 'test' does not have a value. You should send 'test=somevalue'.
Second I am wondering if you are trying to send the cookie to the googlesite server and ask it to reply with the same cookie you previously sent... ?
I am thinking you are trying to act as a HTTP server beside you are a HTTP client.
As a HTTP client your role is only to send back any cookies that the HTTP server have previously sent to you (respecting the domain, expiration... params).