I'm trying to create GET URL that returns a resource located under 'public' folder, and i'm trying to use cache, to save redirects to the resource if not needed.
I have a table: 'ResourcesTable', that stores when was the last time that the resource was modified.
My route for the requested URL navigates to the controller that executes:
def handle_resource
resource_path = params(:resource)
last_updated_at = ResourcesTable.get_resource_timestamp(resource_path)
if stale?(:last_modified => last_updated_at.utc) then
redirect_to resource_path
The problem is, that its always goes to the 'redirect_to'.
I've checked with cURL, and the last_modify element in the header is the same as 'last_updated_at' returned from the DB.
Does anyone can help to make sure that it will do only one time the 'redirect_to', and takes the resource from cache the next time?
Thanks
Answered my own question.
In cURL - we should insert the date of the HTTP request:
curl -z 'Sun, 06 Nov 2024 08:49:37 GMT' -k -I -X GET URL_PATH
and that is how I can see when the 'stale?' method returns true or not.
In addition -
The controller should not return any message in case that 'stale?' returns False.
Related
Is it possible to update url of postman request in the pre-requisite script.
I want to edit the url based on dynamic environment input.
For example:
if (environment.someValue) {
request.url = request.url + "\" + environment.someValue
if (environment.anotherValue) {
request.url = request.url + "\" + environment.anotherValue
}
}
console.log(request.url);
The above code gives me prefect console log:
e.g. if url is https://someServer/someRequest, environment.someVar is x and environment.anotherVar is y the console.log(request.url) above prints:
https://someServer/someRequest/x/y
But the problem is (say if i am requesting a Get), even after logging the overridden request url, it only calls https://someServer/someRequest and does not override to https://someServer/someRequest/x/y.
Any ideas how to modify the url as asked above.
if your url in your request is set as a global, it should work.
ie. I have a get request :
GET http://{{myurl}}/etc. with myurl set as a global variable
In my prerequest script I do pm.globals.set("myurl", <new url>);
when I launch my request, it tries to do the GET request on my new url.
So it is possible to do it but you have to use global or environment variables to dynamically update your request:
set your 'someRequest' as a global that you can update in your prescript (instead of request.url), then it will be interpreted when you launch your request
https://someServer/{{someRequest}}
pm.request.url= "dynamic value"
you and update url using the above single line in postman prerequisite step
pm.request.url returns a object with fields host,path and params . But you can replace it with string.
Thanks for the tips: I used these lines of code to manipulate two values that appear in a RESTful API route:
// Ensure Payroll ID and Employee ID in request URL are correctly padded with zeros
pm.request.url.path[pm.request.url.path.indexOf("{{PayrollID}}")] = data.PayrollID.toString().padStart(6, '0');
pm.request.url.path[pm.request.url.path.indexOf("{{EmployeeID}}")] = data.EmployeeID.toString().padStart(7, '0');
I'm just getting started using the REST API to create pages.
I'm trying to configure a basic example and I thought of using libcurl.net to do it.
Does anyone see any reason why that wouldn't work?
UPDATE:
Here is what I have so far adapted from the curllib.net "bookpost" example
Option Explicit On
Imports System.IO
Imports System.Net
Imports SeasideResearch.LibCurlNet
Public Class CurlOperations
Public Shared Sub CurlPost()
Try
Dim credUserName As String = "username"
Dim credPassword As String = "password"
Dim response As String = Nothing
Dim outputStdErr As Stream = Nothing
Curl.GlobalInit(CURLinitFlag.CURL_GLOBAL_ALL)
Dim easy As Easy
easy = New Easy
' Set up write delegate
Dim wf As Easy.WriteFunction
wf = New Easy.WriteFunction(AddressOf OnWriteData)
easy.SetOpt(CURLoption.CURLOPT_WRITEFUNCTION, wf)
'Authentication
easy.SetOpt(CURLoption.CURLOPT_HTTPAUTH, CURLhttpAuth.CURLAUTH_BASIC)
easy.SetOpt(CURLoption.CURLOPT_USERPWD, credUserName & ":" & credPassword)
'disable ssl peer verification
easy.SetOpt(CURLoption.CURLOPT_SSL_VERIFYPEER, False)
'Header
easy.SetOpt(CURLoption.CURLOPT_HTTPHEADER, "Content-Type: application/json; charset=utf-8")
' Simple post - with a string
easy.SetOpt(CURLoption.CURLOPT_POSTFIELDS, WikiTools.CommREST.WebToCF.PostCurl())
' and the rest of the cURL options
easy.SetOpt(CURLoption.CURLOPT_USERAGENT, ".NET Framework Client")
easy.SetOpt(CURLoption.CURLOPT_URL, "https://domain.com/confluence/rest/api/content/")
easy.SetOpt(CURLoption.CURLOPT_POST, True)
response = easy.Perform().ToString
LoggingAndActivites.WriteLog("Response: " & response, GetFunctionName.GetCallerName, True, True)
Catch ex As Exception
LoggingAndActivites.WriteLog("Exception: " & ex.ToString, GetFunctionName.GetCallerName, True, True)
End Try
Curl.GlobalCleanup()
End Sub
' Called by libcURL.NET when it has data for your program
Public Shared Function OnWriteData(ByVal buf() As Byte, ByVal size As Int32, ByVal nmemb As Int32, ByVal extraData As Object) As Int32
LoggingAndActivites.WriteLog(System.Text.Encoding.UTF8.GetString(buf), GetFunctionName.GetCallerName, True, True)
Return size * nmemb
End Function
End Class
I am getting connected because if I remove the username and password I get a response through the "onWriteData" function as follows:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head>
<title>401 Unauthorized</title>
</head>
<body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
<hr>
<address>Apache Server at domain.com Port 7080</address>
</body>
</html>
The problem now is that if I correctly log on I'm not getting any response other than the "CURLE_OK" from the "easy.Perform()" function.
It's good because I know it's working to some degree.
According to the libcurl.net docs : http://www.libcurl.org/
libcurl also supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, and user+password authentication.
So I guess you should be able to make a REST API call with it. I have used curl (the linux version) to create and update pages, using something like this:
curl --globoff --insecure --silent -u username:password -X PUT -H 'Content-Type: application/json' --data #body.json confluenceRestAPIURL/pageId
where body.json is a file containing the data to update the page.
I wrote a blog about this here: https://javamemento.blogspot.no/2016/05/jira-confluence-3.html
You can get the code here: https://github.com/somaiah/jira-confluence-graphs
So it does work
Here is what I added/changed to make the code above work
'I added an Slist to store the header items (I only had one)
Dim slistHeaders As New Slist
slistHeaders.Append("Content-Type: application/json")
'Then I added the slist to the HTTPHEADER
easy.SetOpt(CURLoption.CURLOPT_HTTPHEADER, slistHeaders)
THINGS TO WATCH FOR:
(1) The URL of course is the number one thing
In my case I was using was https://domain.com/confluence/rest/api/content/ because the Confluence Documentation assumes you would be using the root domain name (as did I)
However, what I didn't know was that the URL I was given to test was already directing me into the "confluence" folder.
So my URI needed to be https://domain.com/rest/api/content/ instead
(2) An indicator that your HTTPHEADER needs to be put into an Slist is this return from the server: 415 Unsupported Media Type
(3) Be sure NOT to use the CURLOPT_HEADER property. If you are seeing this header in your responses you need to make sure it's not used:
HTTP/1.1 500 Internal Server Error
Date: Sun, 22 May 2016 17:50:32 GMT
Server: Apache
Content-Location: 500.en-GB.html
Vary: negotiate,accept-language
TCN: choice
Accept-Ranges: bytes
Content-Length: 7575
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Language: en-gb
Refer to my post here for an explanation of why: CURLOPT_HEADER
(4) Lastly, when you build your app if you receive this error:
An unhandled exception of type 'System.AccessViolationException' occurred in libcurl.NET.dll
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
This is caused by libcurl.net process not being cleaned up.
The "cleanup()" method isn't available in the DLL, despite being lsited in the examples.
So instead use EASY.Dispose() at the end of your procedure and this error will stop. (I kept the "GlobalCleanup()" method in just for good measure)
(5) Ironically I went the way of using CURL because I thought that the Confluence interface might require it.
But it appears now that it doesn't and that you can simply use the "HttpWebRequest" Class in .NET to get the same results.
However, the Jury is still out because the "lightweight" test server I was assigned to crashed and I'm waiting for them to fix it so I can verify this.
Either Way I hope all this helps someone
M
I always thought the per-site cache will not update until it expires, and other people thought so too. However I found different result after doing some test on my website. My website, as you can see, is a typical blog.
Here's how I tested it:
Launch memcached using memcached -vv, so that I can see what happened in memcached.
Then did some operations:
visit homepage -> visit homepage -> update an article on homepage -> visit homepage
cache stored nothing nothing cache stored(strange!)
Homepage is indeed updated in my last visit.
My cache expire time is 600s, so I can assure you the second cache stored operation has nothing to do with cache expiration.(In fact I repeated serveral times, all giving the same result).
So what's the explanation for this? The documentaion doesn't give much information. Or is it because I tested in a wrong way?
You can look up the source code of process_response in the middleware UpdateCacheMiddleware to figure it out:
def process_response(self, request, response):
"""Sets the cache, if needed."""
if not self._should_update_cache(request, response):
# We don't need to update the cache, just return.
return response
if response.streaming or response.status_code != 200:
return response
# Don't cache responses that set a user-specific (and maybe security
# sensitive) cookie in response to a cookie-less request.
if not request.COOKIES and response.cookies and has_vary_header(response, 'Cookie'):
return response
# Try to get the timeout from the "max-age" section of the "Cache-
# Control" header before reverting to using the default cache_timeout
# length.
timeout = get_max_age(response)
if timeout is None:
timeout = self.cache_timeout
elif timeout == 0:
# max-age was set to 0, don't bother caching.
return response
patch_response_headers(response, timeout)
if timeout:
cache_key = learn_cache_key(request, response, timeout, self.key_prefix, cache=self.cache)
if hasattr(response, 'render') and callable(response.render):
response.add_post_render_callback(
lambda r: self.cache.set(cache_key, r, timeout)
)
else:
self.cache.set(cache_key, response, timeout)
return response
First check if your blog is aimed to have authenticated users, as it is it won't work due to if not request.COOKIES and response.cookies and has_vary_header(response, 'Cookie').
If not authenticated then you have to look into learn_cache_key method which computes the key in which the view will be cached:
def learn_cache_key(request, response, cache_timeout=None, key_prefix=None, cache=None):
"""
Learns what headers to take into account for some request URL from the
response object. It stores those headers in a global URL registry so that
later access to that URL will know what headers to take into account
without building the response object itself. The headers are named in the
Vary header of the response, but we want to prevent response generation.
The list of headers to use for cache key generation is stored in the same
cache as the pages themselves. If the cache ages some data out of the
cache, this just means that we have to build the response once to get at
the Vary header and so at the list of headers to use for the cache key.
"""
if key_prefix is None:
key_prefix = settings.CACHE_MIDDLEWARE_KEY_PREFIX
if cache_timeout is None:
cache_timeout = settings.CACHE_MIDDLEWARE_SECONDS
cache_key = _generate_cache_header_key(key_prefix, request)
if cache is None:
cache = caches[settings.CACHE_MIDDLEWARE_ALIAS]
if response.has_header('Vary'):
is_accept_language_redundant = settings.USE_I18N or settings.USE_L10N
# If i18n or l10n are used, the generated cache key will be suffixed
# with the current locale. Adding the raw value of Accept-Language is
# redundant in that case and would result in storing the same content
# under multiple keys in the cache. See #18191 for details.
headerlist = []
for header in cc_delim_re.split(response['Vary']):
header = header.upper().replace('-', '_')
if header == 'ACCEPT_LANGUAGE' and is_accept_language_redundant:
continue
headerlist.append('HTTP_' + header)
headerlist.sort()
cache.set(cache_key, headerlist, cache_timeout)
return _generate_cache_key(request, request.method, headerlist, key_prefix)
else:
# if there is no Vary header, we still need a cache key
# for the request.build_absolute_uri()
cache.set(cache_key, [], cache_timeout)
return _generate_cache_key(request, request.method, [], key_prefix)
Note that Django will dig into response['Vary'] header, and will add to the key variations to the key for it.
So fist check if your responses add Vary header, and if so which value it has, if it's different from before saving the entry you'll have why is not cached.
Note also that if multilanguage is activated it will also generate different keys per language.
When Django adds or modifies Varyheader?
Here you can see when it happens or how it happens in the docs and how to control cache in views here.
Finally take into account that if you're using a third-party app for your blog then it probably make use of cache control mechanisms to refresh cache when varying data like updating an entry.
Django version
In Django >=1.7 fully qualified urls are used, so if you use these versions also take into account HOSTS:
Changed in Django 1.7:
Cache keys use the request’s fully-qualified URL rather than just the path and query string.
Well, turns out my testing has problem. I tested again carefully when no one else visits my site. Here's the output of memcached -vv:
# first visit
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..f384b899ecab7abd6fb0a567608b97b2.en-us.CST
>30 END
<30 set :1:views.decorators.cache.cache_header..f384b899ecab7abd6fb0a567608b97b2.en-us.CST 1 600 14
>30 STORED
<30 set :1:views.decorators.cache.cache_page..GET.f384b899ecab7abd6fb0a567608b97b2.d41d8cd98f00b204e9800998ecf8427e.en-us.CST 1 600 33215
>30 STORED
<30 connection closed.
# second visit
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..f384b899ecab7abd6fb0a567608b97b2.en-us.CST
>30 sending key :1:views.decorators.cache.cache_header..f384b899ecab7abd6fb0a567608b97b2.en-us.CST
>30 END
<30 get :1:views.decorators.cache.cache_page..GET.f384b899ecab7abd6fb0a567608b97b2.d41d8cd98f00b204e9800998ecf8427e.en-us.CST
>30 sending key :1:views.decorators.cache.cache_page..GET.f384b899ecab7abd6fb0a567608b97b2.d41d8cd98f00b204e9800998ecf8427e.en-us.CST
>30 END
<30 connection closed.
# modified and save
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..7029e9375fc4657a73dae1f9bddb73e5.en-us.CST
>30 END
<30 connection closed.
# visit again
<30 new auto-negotiating client connection
30: Client using the ascii protocol
<30 get :1:views.decorators.cache.cache_header..f384b899ecab7abd6fb0a567608b97b2.en-us.CST
>30 sending key :1:views.decorators.cache.cache_header..f384b899ecab7abd6fb0a567608b97b2.en-us.CST
>30 END
<30 get :1:views.decorators.cache.cache_page..GET.f384b899ecab7abd6fb0a567608b97b2.d41d8cd98f00b204e9800998ecf8427e.en-us.CST
>30 sending key :1:views.decorators.cache.cache_page..GET.f384b899ecab7abd6fb0a567608b97b2.d41d8cd98f00b204e9800998ecf8427e.en-us.CST
>30 END
<30 connection closed.
This output shows that memcached is working as expected. I'm not sure what's the explanation for previous output though.
Is there any way I can set a variable in settings.py to point to the current url ?
For example, If I'm running a debug server on http://0.0.0.0:8080 or http://127.0.0.1:9000 or https://www.mydomain.com.
I basically need to generate full path for image fields returned from an API. The trick is that I don't always have a request object (POST on DRF for example - The request does not exist in the context when transform_xxx is called).
Appreciate any help with this.
Assuming you're using the contrib.sites framework you can pull the site URL from there.
There's a good example in the docs for when you don't have access to the request:
current_site = Site.objects.get_current()
if current_site.domain == 'foo.com':
# Do something
pass
else:
# Do something else.
pass
Hopefully that's what you need.
UPDATE: I see from the comments that used correctly you would have the request available. I'll leave this here answering the challenge for when you don't.
I've written a Django app that uses DataTables. The problem is when I delete a row from the table it's still displayed in the table when running against nginx/gunicorn. However, it works correctly when I'm running against the Django test server. So if I start a server with this command line:
python manage.py runserver 192.168.0.1:8000
everything works fine. That is, I delete the row, the table refreshes, and the deleted row is not displayed.
This is a summary of the HTTP calls:
// An initial GET command to populate the table
GET /myapp/get_list (returns a list to display)
// I now select a row and delete it which causes this POST to fire
POST /myapp/delete (deletes a row from the list)
// After the POST the code automatically follows up with a GET to refresh the table
GET /myapp/get_list (returns a list to display)
The problem is when I use nginx/gunicorn the second GET call returns the same list as the first GET including the row that I know has been deleted from the backend database.
I'm not sure it's a caching problem either because this is the response header I get from the first GET:
Date Fri, 23 Dec 2011 15:04:16 GMT
Last-Modified Fri, 23 Dec 2011 15:04:16 GMT
Server nginx/0.7.65
Vary Cookie
Content-Type application/javascript
Cache-Control max-age=0
Expires Fri, 23 Dec 2011 15:04:16 GMT
The problem can be solved also by sending an added parameter to the server so that the browser doesn't cache the call. With jQuery you can simply use:
$.ajaxSetup({ cache: false});
Otherwise you must creat manually the parameter. Usually you create a timestamp
var nocache = new Date().getTime();
//send nocache as a parameter so that the browser thinks it's a new call
You can use Django's add_never_cache_headers or never_cache decorator to tell the browser not to cache given request. Documentation is here. I thinks it's cleaner solution than forcing the cache off in the javascript.
from django.utils.cache import add_never_cache_headers
def your_view(request):
(...)
add_never_cache_headers(response)
return response
from django.views.decorators.cache import never_cache
#never_cache
def your_other_view(request):
(...)
try this
oTable.fnDeleteRow( anSelected, null, true );
and b.t.w what version of datatables do you use?
I fixed the problem by changing
GET /myapp/get_list
to
POST /myapp/get_list
After doing this, I no longer get a cached response.