I'd like to retrieve and parse multiple html pages within a Chrome extension.
Using Web Workers for each request seemed like a simple way to make them execute in parallel. Is that possible? My attempt failed, perhaps because it's a known permissions bug.
As a workaround, I guess I could have the main extension page do multiple asynchronous XmlHttpRequests, then have the callback send the result page to Web Workers for parallel parsing. But that method raises the question of how many asynchronous parallel requests can Chrome make at once? That question has been asked here, without answer.
The Chrome Extension I'm writing is a Browser Action.
Code for the worker:
// Triggered by postMessage in the page
onmessage = function (evt) {
var message = evt.data;
postMessage(message.count + " started");
update(message.count, message.url);
};
// parse the results page
function parseResponse(count, url, resp) {
var msg = count.toString() + " received response ";
postMessage(msg);
}
// read the Buganizer URL and parse the result page
var update = function(count, url) {
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function() {
if (xhr.readyState == 4 && xhr.status == 200) {
parseResponse(count, url, xhr.responseText);
}
}
xhr.onerror = function(error) {
var msg = "!>: " + count + ": error ";
postMessage(msg);
}
var url = "http://www.hotgrog.com"; // for testing (manifest has permissions for this url)
xhr.open("GET", url, true);
xhr.send();
postMessage(url);
}
Have you looked into trying asynchronous-loaders such as RequireJS or Curl?
Take a look at the authors explanation as to WHY we should use his product.
Related
I need to make requests to an API that accepts authentication tokens and I want to be able to use a dynamically generated token by running cmd.exe /c GenerateToken.bat instead of having to run my program and then manually paste the value in Postman every time.
I imagine something that looks like this:
How can I set the value of a HTTP header to contain the stdout output of a program or a batch file?
Short answer is, you can't. This is deliberate, both pre-request and test scripts (the only way, other than a collection runner, to make your environment dynamic) run in the postman sandbox, which has limited functionality.
More information of what is available is in the postman-sandbox Github repository page and in postman docs (scroll to the bottom to see what libraries you can import)
You do have a few options, as described in comments - postman allows sending requests and parsing the response in scripts, so you can automate this way. You do need a server to handle the requests and execute your script (simplest option is probably a small server suporting CGI - I won't detail it here as I feel it's too big of a scope for this answer. Other options are also available, such as a small PHP or Node server)
Once you do have a server, the pre-request script is very simple:
const requestOptions = {
url: `your_server_endpoint`,
method: 'GET'
}
pm.sendRequest(requestOptions, function (err, res) {
if (err) {
throw new Error(err);
} else if (res.code != 200) {
throw new Error(`Non-200 response when fetching token: ${res.code} ${res.status}`);
} else {
var token = res.text();
pm.environment.set("my_token", token);
}
});
You can then set the header as {{my_token}} in the "Headers" tab, and it will be updated once the script runs.
You can do something similar to this from Pre-request Scripts at the collection level.
This is available in postman for 9 different authorization and authentication methods.
this is a sample code taken from this article, that show how to do this in Pre-request Scripts for OAuth2
// Refresh the OAuth token if necessary
var tokenDate = new Date(2010,1,1);
var tokenTimestamp = pm.environment.get("OAuth_Timestamp");
if(tokenTimestamp){
tokenDate = Date.parse(tokenTimestamp);
}
var expiresInTime = pm.environment.get("ExpiresInTime");
if(!expiresInTime){
expiresInTime = 300000; // Set default expiration time to 5 minutes
}
if((new Date() - tokenDate) >= expiresInTime)
{
pm.sendRequest({
url: pm.variables.get("Auth_Url"),
method: 'POST',
header: {
'Accept': 'application/json',
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': pm.variables.get("Basic_Auth")
}
}, function (err, res) {
pm.environment.set("OAuth_Token", res.json().access_token);
pm.environment.set("OAuth_Timestamp", new Date());
// Set the ExpiresInTime variable to the time given in the response if it exists
if(res.json().expires_in){
expiresInTime = res.json().expires_in * 1000;
}
pm.environment.set("ExpiresInTime", expiresInTime);
});
}
I am trying to build a Firebase hosted page that will connect to a (Django) HTTP web service using XMLHttpRequest in the below script.
<script>
function UserAction() {
var req = createRequest(); // defined below
// Create the callback functions:
var handleResponse = function (status, response) {
alert("status " + status + " response: " + response)
}
var handleStateChange = function () {
switch (req.readyState) {
case 0 : // UNINITIALIZED
case 1 : // LOADING
case 2 : // LOADED
case 3 : // INTERACTIVE
break;
case 4 : // COMPLETED
alert("case 4" + req.responseType);
handleResponse(req.status, req.responseJson);
break;
default: alert("error");
}
}
req.onreadystatechange = handleStateChange;
req.open("GET", "https://foo.org:/bar/getid/?Id=" + document.getElementById('ID').value, true);
req.send();
function createRequest() {
var result = null;
if (window.XMLHttpRequest) {
// FireFox, Safari, etc.
result = new XMLHttpRequest();
}
else if (window.ActiveXObject) {
// MSIE
window.alert("windows");
result = new ActiveXObject("Microsoft.XMLHTTP");
}
else {
// No known mechanism -- consider aborting the application
window.alert("no known mechanism");
}
return result;
}
}</script>
Using Chrome FireBase throws 404 error. Django server does not register any connection.
Using FireFox FireBase throws 404 error. But Django server throws
'code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x00\xcc\x01\x00\x00\xc8\x03\x03\xdf\x04{\x9f\xe4\xb2\xc2ij\x8d\x14\xd5\xaa\xdcu\x14+&\xa4\xa1\xdf\xdc\xd8\x9b?\xea\xbdh\xb8')`
I did find this in the FireBase documentation here, making me think this is not possible. But hopefully I am wrong, or there is a way to do this in development/test, but not production.
SSL only: Firebase Hosting is SSL-only, meaning that content will only
be served over HTTPS. As you are developing your application make sure
that all external resources not hosted on Firebase Hosting are loaded
over SSL (HTTPS), including any external scripts. Most browsers do not
allow users to load "mixed content" (SSL and non-SSL traffic).
thanks
I know the question has been asked before and I agree with most answers that claim it is better to follow the way requests are made async with URLSession in Swift 3. I haver the following scenario, where async request cannot be used.
With Swift 3 and the ability to run swift on servers I have the following problem.
Server Receives a request from a client
To process the request the server has to send a url request and wait for the response to arrive.
Once response arrives, process it and reply to the client
The problem arrises in step 2, where URLSession gives us the ability to initiate an async data task only. Most (if not all) server side swift web frameworks do not support async responses. When a request arrives to the server everything has to be done in a synchronous matter and at the end send the response.
The only solution I have found so far is using DispatchSemaphore (see example at the end) and I am not sure whether that will work in a scaled environment.
Any help or thoughts would be appreciated.
extension URLSession {
func synchronousDataTaskWithURL(_ url: URL) -> (Data?, URLResponse?, Error?) {
var data: Data?
var response: URLResponse?
var error: Error?
let sem = DispatchSemaphore(value: 0)
let task = self.dataTask(with: url as URL, completionHandler: {
data = $0
response = $1
error = $2 as Error?
sem.signal()
})
task.resume()
let result = sem.wait(timeout: DispatchTime.distantFuture)
switch result {
case .success:
return (data, response, error)
case .timedOut:
let error = URLSessionError(kind: URLSessionError.ErrorKind.timeout)
return (data, response, error)
}
}
}
I only have experience with kitura web framework and this is where i faced the problem. I suppose that similar problems exist in all other swift web frameworks.
In Vapor, you can use the Droplet's client to make synchronous requests.
let res = try drop.client.get("https://httpbin.org")
print(res)
Additionally, you can use the Portal class to make asynchronous tasks synchronous.
let res = try Portal.open { portal in
asyncClient.get("https://httpbin.org") { res in
portal.close(with: res)
}
}
Your three-step problem can be solved via the use of a completion handler, i.e., a callback handler a la Node.js convention:
import Foundation
import Kitura
import HeliumLogger
import LoggerAPI
let session = URLSession(configuration: URLSessionConfiguration.default)
Log.logger = HeliumLogger()
let router = Router()
router.get("/test") { req, res, next in
let datatask = session.dataTask(with: URL(string: "http://www.example.com")!) { data, urlResponse, error in
try! res.send(data: data!).end()
}
datatask.resume()
}
Kitura.addHTTPServer(onPort: 3000, with: router)
Kitura.run()
This is a quick demo of a solution to your problem, and it is by no means following best Swift/Kitura practices. But, with the use of a completion handler, I am able to have my Kitura app make an HTTP call to fetch the resource at http://www.example.com, wait for the response, and then send the result back to my app's client.
Link to the relevant API: https://developer.apple.com/reference/foundation/urlsession/1410330-datatask
I am using phantomjs to print the webpage and create a pdf. As the UI needs the user's authentication before finding the data, I used persistent cookies to authenticate the user. But somehow I got login screen every time in the created PDF. I observed that the user authenticated successfully and also the result's webpage showing proper result (debug logs showing the proper data array) but while printing the web page or creating a PDF, it somehow gets the login screen. Sometimes I observed that I got two different cookies in my PHP code while getting the report data and in javascript 'document.cookies'.
Please let me know how can I fix this.
var page = require('webpage').create(),
system = require('system'), t, address;
page.settings.userName = 'myusername';
page.settings.password = 'mypassword';
if (system.args.length === 1) {
console.log('Usage: scrape.js ');
phantom.exit();
} else {
t = Date.now();
address = system.args[1];
page.open(address, function (status) {
if (status !== 'success') {
console.log('FAIL to load the address');
} else {
t = Date.now() - t;
var title = page.evaluate(function() { return document.title;})
console.log('Page title is ' + title);
console.log('Loading time ' + t + ' msec');
}
phantom.exit();
});
}
Another piece of code of sending a cookie file
bin/phantomjs --cookies-file=/tmp/cookies.txt --disk-cache=yes --ignore-ssl-errors=yes /phantomjs/pdf.js 'username' 'params' '/tmp/phantomjs_file' /tmp/phantom_pdf.pdf
And
phantomjs --cookies-file=cookies.txt examples/rasterize.js localhost:7000/reports /tmp/report.pdf
I am trying to Post and get a cookie. I am a newbie and this is a learning project for me. My impression is that if you use 'set-cookie' one should be able to see an additional 'set-cookie' in the .toSource. (I am trying to accomplish this on Google Apps Site if that makes a difference.) Am I missing something? Here is my code:
function setGetCookies() {
var payload = {'set-cookie' : 'test'};
var opt2 = {'headers':payload, "method":"post"};
UrlFetchApp.fetch("https://sites.google.com/a/example.com/blacksmith", opt2);
var response = UrlFetchApp.fetch("https://sites.google.com/a/example.com/blacksmith")
var openId = response.getAllHeaders().toSource();
Logger.log(openId)
var AllHeaders = response.getAllHeaders();
for (var prop in AllHeaders) {
if (prop.toLowerCase() == "set-cookie") {
// if there's only one cookie, convert it into an array:
var myArray = [];
if ( Array.isArray(AllHeaders[prop]) ) {
myArray=AllHeaders[prop];
} else {
myArray[0]=AllHeaders[prop];
}
// now process the cookies
myArray.forEach(function(cookie) {
Logger.log(cookie);
});
break;
}
}
}
Thanks in advance! I referenced this to develop the code: Cookie handling in Google Apps Script - How to send cookies in header?
Open to any advice.
When you aren't logged in Google Sites won't set any cookies in the response. UrlFetchApp doesn't pass along your Google cookies, so it will behave as if you are logged out.
First the cookie you want to send whose name is 'test' does not have a value. You should send 'test=somevalue'.
Second I am wondering if you are trying to send the cookie to the googlesite server and ask it to reply with the same cookie you previously sent... ?
I am thinking you are trying to act as a HTTP server beside you are a HTTP client.
As a HTTP client your role is only to send back any cookies that the HTTP server have previously sent to you (respecting the domain, expiration... params).