The AWS documentation here seems to have somewhat confusing, incomplete or contradictory information. It states that
CanonicalHeaders is a list of request headers with their values.
Which suggests that we'd put all request headers in the canonical request. However, later, they state
The CanonicalHeaders list must include the following:
HTTP host header
If the Content-Type header is present in the request, it must be added to the CanonicalHeaders list.
Any x-amz-* headers that you plan to include in your request must also be added. For example, if you are using temporary security credentials, you will include x-amz-security-token in your request. You must add this header in the list of CanonicalHeaders.
OK, the bit about the Content-Type and x-amz headers suggests that we don't actually take all headers, because otherwise they wouldn't need to state that they'd be must be included. So then perhaps, we only need to take the Host header, the Content-Type header, and any x-amz-* headers. But then below, it gets more confusing, because here's an example request:
GET /test.txt HTTP/1.1
Host: examplebucket.s3.amazonaws.com
Date: Fri, 24 May 2013 00:00:00 GMT
Authorization: SignatureToBeCalculated
Range: bytes=0-9
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20130524T000000Z
And here's the example canonical request created from it:
GET
/test.txt
host:examplebucket.s3.amazonaws.com
range:bytes=0-9
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20130524T000000Z
host;range;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
But this is inconsistent with both of the interpretations earlier: if we're supposed to only have Content-Type, Host and x-amz-* headers, then what is the range header doing in the list? And if we're just supposed to take all of the headers, then why isn't the Date header in the list?
Is the list of headers to put in a canonical request then arbitrary, as long as it contains at least the minimum headers? What, exactly, is the definitive set of rules to construct the canonical request headers?
if we're supposed to only have Content-Type, Host and x-amz-* headers,
then what is the range header doing in the list?
You are only required to have Content-Type, Host, and x-amz-*, but you can add other headers that you would like to add to the signature to be validated.
See the note in the docs that says: "For the purpose of calculating a signature, only the host and any x-amz-* headers are required; however, in order to prevent data tampering, you should consider including all the headers in the signature calculation."
And if we're just supposed to take all of the headers, then why isn't
the Date header in the list?
The Date header is special, because it is added by the browser according to client system time that may be incorrect. Because of that, you can use x-amz-date instead.
Is the list of headers to put in a canonical request then arbitrary,
as long as it contains at least the minimum headers?
Yes!
What, exactly, is the definitive set of rules to construct the
canonical request headers?
That would be those defined in the documentation of AWS signature version 4... but you got the idea: you must sign the minimal set of request data and can sign all headers that you'd like.
That said, avoid all this if you can. The SDK for [Javascript, Java, .NET, Python, Ruby, PHP,...] already sign requests for you, manages temporary credentials, credential chains, threading, retries and a lot more. If you can use that, it would probably save a lot of headache.
A website I am currently working on is currently responding to requests on http://www.mysite.com/dv with 301, where I would have expected a 404.
My site has no content item named DV, there is nothing in the site that corresponds to "DV", and I can't understand why on earth I am seeing this. I originally found these /DV/ URLs while looking at Google Webmaster Tools.
Then I used Fiddler to analyse the request and found
Response sent 26 bytes of Cookie data:
Set-Cookie: wwwwebsite#lang=DV; path=/
This appears to correspond to the Maldivian language. I did not enable this language, and it does not appear under /system/Languages.
I then noticed I can also navigate to http://www.mysite.com/de-CH which is not an enabled language on my site either, leading me to believe I will get results like this for every non-enabled language.
Is there a way to return 404 for unsupported languages in Sitecore?
Try seeing what your "languageEmbedding" value in the web.config is set to.
It can be set to the following: asNeeded | always | never
I would normally set it to "never" unless you have a site, which is set in multiplate languages, then set it to "always". The default is "asNeeded" which I frankly find kinda silly :)
The issue you are having sounds similar to http://sdn.sitecore.net/Forum/ShowPost.aspx?postid=51681 but I an't find DV in LanguageDefinitions.config. I am also able to replicate your error so I don't think it is environment specific to your implementation. Would suggest you contact Sitecore Support.
I don't believe this is actually related to languages at all. If you were to add any random characters in place of the DV, you'd still get a 302 redirect to a 404. Perhaps without the language cookie though.
You should follow the directions here to turn on server side error page redirects.
http://herskind.co.uk/blog/2012/03/sitecore-404-without-302
I've been searching everywhere trying to figure this one out. I'm trying to generate html pages from couchdb show and list functions. I'd like to leverage underscore.js's template solution. The part I'm getting stuck on is how to include html templates in my show and list functions.
Where do I store them? As attachments? And then how do I reference them in my show and list functions. I assume !json and !code macros are not being used, and I can't figure out how to use require() from common js to do it.
Any help would rock!
Thanks!
Extra Info: I'm using Kanso to push my apps, not CouchApp.
CouchDB attachments are, by definition, not accessible in show and list functions.
Show and list functions support CommonJS. So you simply need to include any libraries in the design doc.
{ "_id": "_design/example"
, "say_hi": "module.exports = function(person) { return 'Hello, ' + person }"
, "shows":
{ "hello": "function(doc, req) { var hi = require('say_hi'); return hi(req.query.me) }"
}
}
This view would look like this
GET /my_db/_design/example/_show/hello?me=Jason
HTTP/1.1 200 OK
Server: CouchDB/1.2.0 (Erlang OTP/R15B)
Date: Fri, 06 Apr 2012 11:02:33 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 12
Hello, Jason
I'm unfamiliar with Kanso, but before CouchDB 1.1, view/show etc. functions in CouchDB could not include anything. (The CouchApp tool had its own !include workarounds to solve this.) These are not necessary anymore. CouchDB 1.1 added CommonJS support.
Tll the templates and libraries must be part of the design document. You can access the raw values (as a string) by referencing this.some_key; or load them via CommonJS by executing require("some_key").
For example:
exports.example_view = {
map: function (doc) {
// this must be placed *inside* the map function
var example = require('views/lib/example');
if (doc.num) {
emit(doc._id, example.fn());
}
}
};
(Sharing code between views)
To render templates server-side, you'll need to encode them as string and require them like you require other JavaScript libraries. (For browser-side rendering, fetching attachments via AJAX works.)
What proven design patterns exist for batch operations on resources within a REST style web service?
I'm trying to be strike a balance between ideals and reality in terms of performance and stability. We've got an API right now where all operations either retrieve from a list resource (ie: GET /user) or on a single instance (PUT /user/1, DELETE /user/22, etc).
There are some cases where you want to update a single field of a whole set of objects. It seems very wasteful to send the entire representation for each object back and forth to update the one field.
In an RPC style API, you could have a method:
/mail.do?method=markAsRead&messageIds=1,2,3,4... etc.
What's the REST equivalent here? Or is it ok to compromise now and then. Does it ruin the design to add in a few specific operations where it really improves the performance, etc? The client in all cases right now is a Web Browser (javascript application on the client side).
A simple RESTful pattern for batches is to make use of a collection resource. For example, to delete several messages at once.
DELETE /mail?&id=0&id=1&id=2
It's a little more complicated to batch update partial resources, or resource attributes. That is, update each markedAsRead attribute. Basically, instead of treating the attribute as part of each resource, you treat it as a bucket into which to put resources. One example was already posted. I adjusted it a little.
POST /mail?markAsRead=true
POSTDATA: ids=[0,1,2]
Basically, you are updating the list of mail marked as read.
You can also use this for assigning several items to the same category.
POST /mail?category=junk
POSTDATA: ids=[0,1,2]
It's obviously much more complicated to do iTunes-style batch partial updates (e.g., artist+albumTitle but not trackTitle). The bucket analogy starts to break down.
POST /mail?markAsRead=true&category=junk
POSTDATA: ids=[0,1,2]
In the long run, it's much easier to update a single partial resource, or resource attributes. Just make use of a subresource.
POST /mail/0/markAsRead
POSTDATA: true
Alternatively, you could use parameterized resources. This is less common in REST patterns, but is allowed in the URI and HTTP specs. A semicolon divides horizontally related parameters within a resource.
Update several attributes, several resources:
POST /mail/0;1;2/markAsRead;category
POSTDATA: markAsRead=true,category=junk
Update several resources, just one attribute:
POST /mail/0;1;2/markAsRead
POSTDATA: true
Update several attributes, just one resource:
POST /mail/0/markAsRead;category
POSTDATA: markAsRead=true,category=junk
The RESTful creativity abounds.
Not at all -- I think the REST equivalent is (or at least one solution is) almost exactly that -- a specialized interface designed accommodate an operation required by the client.
I'm reminded of a pattern mentioned in Crane and Pascarello's book Ajax in Action (an excellent book, by the way -- highly recommended) in which they illustrate implementing a CommandQueue sort of object whose job it is to queue up requests into batches and then post them to the server periodically.
The object, if I remember correctly, essentially just held an array of "commands" -- e.g., to extend your example, each one a record containing a "markAsRead" command, a "messageId" and maybe a reference to a callback/handler function -- and then according to some schedule, or on some user action, the command object would be serialized and posted to the server, and the client would handle the consequent post-processing.
I don't happen to have the details handy, but it sounds like a command queue of this sort would be one way to handle your problem; it'd reduce the overall chattiness substantially, and it'd abstract the server-side interface in a way you might find more flexible down the road.
Update: Aha! I've found a snip from that very book online, complete with code samples (although I still suggest picking up the actual book!). Have a look here, beginning with section 5.5.3:
This is easy to code but can result in
a lot of very small bits of traffic to
the server, which is inefficient and
potentially confusing. If we want to
control our traffic, we can capture
these updates and queue them locally
and then send them to the server in
batches at our leisure. A simple
update queue implemented in JavaScript
is shown in listing 5.13. [...]
The queue maintains two arrays. queued
is a numerically indexed array, to
which new updates are appended. sent
is an associative array, containing
those updates that have been sent to
the server but that are awaiting a
reply.
Here are two pertinent functions -- one responsible for adding commands to the queue (addCommand), and one responsible for serializing and then sending them to the server (fireRequest):
CommandQueue.prototype.addCommand = function(command)
{
if (this.isCommand(command))
{
this.queue.append(command,true);
}
}
CommandQueue.prototype.fireRequest = function()
{
if (this.queued.length == 0)
{
return;
}
var data="data=";
for (var i = 0; i < this.queued.length; i++)
{
var cmd = this.queued[i];
if (this.isCommand(cmd))
{
data += cmd.toRequestString();
this.sent[cmd.id] = cmd;
// ... and then send the contents of data in a POST request
}
}
}
That ought to get you going. Good luck!
While I think #Alex is along the right path, conceptually I think it should be the reverse of what is suggested.
The URL is in effect "the resources we are targeting" hence:
[GET] mail/1
means get the record from mail with id 1 and
[PATCH] mail/1 data: mail[markAsRead]=true
means patch the mail record with id 1. The querystring is a "filter", filtering the data returned from the URL.
[GET] mail?markAsRead=true
So here we are requesting all the mail already marked as read. So to [PATCH] to this path would be saying "patch the records already marked as true"... which isn't what we are trying to achieve.
So a batch method, following this thinking should be:
[PATCH] mail/?id=1,2,3 <the records we are targeting> data: mail[markAsRead]=true
of course I'm not saying this is true REST (which doesnt permit batch record manipulation), rather it follows the logic already existing and in use by REST.
Your language, "It seems very wasteful...", to me indicates an attempt at premature optimization. Unless it can be shown that sending the entire representation of objects is a major performance hit (we're talking unacceptable to users as > 150ms) then there's no point in attempting to create a new non-standard API behaviour. Remember, the simpler the API the easier it is to use.
For deletes send the following as the server doesn't need to know anything about the state of the object before the delete occurs.
DELETE /emails
POSTDATA: [{id:1},{id:2}]
The next thought is that if an application is running into performance issues regarding the bulk update of objects then consideration into breaking each object up into multiple objects should be given. That way the JSON payload is a fraction of the size.
As an example when sending a response to update the "read" and "archived" statuses of two separate emails you would have to send the following:
PUT /emails
POSTDATA: [
{
id:1,
to:"someone#bratwurst.com",
from:"someguy#frommyville.com",
subject:"Try this recipe!",
text:"1LB Pork Sausage, 1 Onion, 1T Black Pepper, 1t Salt, 1t Mustard Powder",
read:true,
archived:true,
importance:2,
labels:["Someone","Mustard"]
},
{
id:2,
to:"someone#bratwurst.com",
from:"someguy#frommyville.com",
subject:"Try this recipe (With Fix)",
text:"1LB Pork Sausage, 1 Onion, 1T Black Pepper, 1t Salt, 1T Mustard Powder, 1t Garlic Powder",
read:true,
archived:false,
importance:1,
labels:["Someone","Mustard"]
}
]
I would split out the mutable components of the email (read, archived, importance, labels) into a separate object as the others (to, from, subject, text) would never be updated.
PUT /email-statuses
POSTDATA: [
{id:15,read:true,archived:true,importance:2,labels:["Someone","Mustard"]},
{id:27,read:true,archived:false,importance:1,labels:["Someone","Mustard"]}
]
Another approach to take is to leverage the use of a PATCH. To explicitly indicate which properties you are intending to update and that all others should be ignored.
PATCH /emails
POSTDATA: [
{
id:1,
read:true,
archived:true
},
{
id:2,
read:true,
archived:false
}
]
People state that PATCH should be implemented by providing an array of changes containing: action (CRUD), path (URL), and value change. This may be considered a standard implementation but if you look at the entirety of a REST API it is a non-intuitive one-off. Also, the above implementation is how GitHub has implemented PATCH.
To sum it up, it is possible to adhere to RESTful principles with batch actions and still have acceptable performance.
The google drive API has a really interesting system to solve this problem (see here).
What they do is basically grouping different requests in one Content-Type: multipart/mixed request, with each individual complete request separated by some defined delimiter. Headers and query parameter of the batch request are inherited to the individual requests (i.e. Authorization: Bearer some_token) unless they are overridden in the individual request.
Example: (taken from their docs)
Request:
POST https://www.googleapis.com/batch
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.20.0 (gzip)
Content-Type: multipart/mixed; boundary=END_OF_PART
Content-Length: 963
--END_OF_PART
Content-Length: 337
Content-Type: application/http
content-id: 1
content-transfer-encoding: binary
POST https://www.googleapis.com/drive/v3/files/fileId/permissions?fields=id
Authorization: Bearer authorization_token
Content-Length: 70
Content-Type: application/json; charset=UTF-8
{
"emailAddress":"example#appsrocks.com",
"role":"writer",
"type":"user"
}
--END_OF_PART
Content-Length: 353
Content-Type: application/http
content-id: 2
content-transfer-encoding: binary
POST https://www.googleapis.com/drive/v3/files/fileId/permissions?fields=id&sendNotificationEmail=false
Authorization: Bearer authorization_token
Content-Length: 58
Content-Type: application/json; charset=UTF-8
{
"domain":"appsrocks.com",
"role":"reader",
"type":"domain"
}
--END_OF_PART--
Response:
HTTP/1.1 200 OK
Alt-Svc: quic=":443"; p="1"; ma=604800
Server: GSE
Alternate-Protocol: 443:quic,p=1
X-Frame-Options: SAMEORIGIN
Content-Encoding: gzip
X-XSS-Protection: 1; mode=block
Content-Type: multipart/mixed; boundary=batch_6VIxXCQbJoQ_AATxy_GgFUk
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
Date: Fri, 13 Nov 2015 19:28:59 GMT
Cache-Control: private, max-age=0
Vary: X-Origin
Vary: Origin
Expires: Fri, 13 Nov 2015 19:28:59 GMT
--batch_6VIxXCQbJoQ_AATxy_GgFUk
Content-Type: application/http
Content-ID: response-1
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Date: Fri, 13 Nov 2015 19:28:59 GMT
Expires: Fri, 13 Nov 2015 19:28:59 GMT
Cache-Control: private, max-age=0
Content-Length: 35
{
"id": "12218244892818058021i"
}
--batch_6VIxXCQbJoQ_AATxy_GgFUk
Content-Type: application/http
Content-ID: response-2
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Date: Fri, 13 Nov 2015 19:28:59 GMT
Expires: Fri, 13 Nov 2015 19:28:59 GMT
Cache-Control: private, max-age=0
Content-Length: 35
{
"id": "04109509152946699072k"
}
--batch_6VIxXCQbJoQ_AATxy_GgFUk--
From my point of view I think Facebook has the best implementation.
A single HTTP request is made with a batch parameter and one for a token.
In batch a json is sent. which contains a collection of "requests".
Each request has a method property (get / post / put / delete / etc ...), and a relative_url property (uri of the endpoint), additionally the post and put methods allow a "body" property where the fields to be updated are sent .
more info at: Facebook batch API
I would be tempted in an operation like the one in your example to write a range parser.
It's not a lot of bother to make a parser that can read "messageIds=1-3,7-9,11,12-15". It would certainly increase efficiency for blanket operations covering all messages and is more scalable.
Great post. I've been searching for a solution for a few days. I came up with a solution of using passing a query string with a bunch IDs separated by commas, like:
DELETE /my/uri/to/delete?id=1,2,3,4,5
...then passing that to a WHERE IN clause in my SQL. It works great, but wonder what others think of this approach.