How to include only specific paths that match a pattern for GZipHandler in Jetty 9.4. For example, I have a request mapping to path /api/employees/{id}/payments. I need to apply GZip compression only to this path. Jetty only supports exact match , or prefix (/*) or suffix (*.) in include paths. Can I configure GZipHandler to compress only requests to path /api/employees/{id}/payments like /api/employees/1/payments, /api/employees/42/payments etc?
The Jetty GzipHandler supports standard Servlet URL-Pattern mappings.
Your "request mapping" of /api/employees/{id}/payments is not a valid servlet url-pattern string.
It looks like a REST API string, which is done entirely within your REST library, without the servlet spec or servlet api really being involved.
Instead of attempting to control GzipHandler from paths (include / exclude), consider instead setting up the Mime-Type mappings that it should respond to instead. That way you can setup the include path to /api/* and have the mime-types includes decide what content should be compressed or not. Keep in mind that GzipHandler will not compress if the http client has not indicated that it can handle gzip (it checks the Accept and Accept-Encoding request headers), so it would be safe to cast a larger net.
Also consider setting the minimum size configuration setMinGzipSize(int) so that tiny/small responses do not get needlessly gzipped.
Related
I have a JETTY_HOME directory containing the unpacked Jetty distribution.
I want to disable the DefaultHandler, to achieve three things:
Prevent directory listings which would otherwise be controlled by the dirAllowed init parameter (the DefaultServlet is configured by this parameter).
Prevent web app context listings, which may reveal sensitive information such as server directory paths or other web app contexts running within the same Jetty instance.
Be sure that DefaultHandler doesn't provide access to any sensitive files in the event that I botch a web app deployment. I'm happy to implement my own static file serving servlet where necessary as an alternative to using DefaultHandler.
I could simply edit JETTY_HOME/etc/jetty.xml and remove the DefaultHandler from there. However, JETTY_HOME is supposed to be read-only, and I'm only supposed to make changes in my JETTY_BASE folder. Only modifying JETTY_BASE comes with the advantage of not having to repeatedly modify JETTY_HOME when upgrading to a newer release of Jetty.
How to I make this change from inside JETTY_BASE?
DefaultHandler is necessary, don't remove it.
Lets address each point.
Prevent directory listings which would otherwise be controlled by the dirAllowed init parameter (the DefaultHandler is configured by this parameter).
The DefaultHandler doesn't do directory listings.
That's the role of the DefaultServlet in a WebAppContext, or a ResourceService / ResourceHandler in an embedded scenario.
If you want to prevent directory listings presented by the DefaultServlet in a WebAppContext you need to configure the DefaultServlet.
You can do that with one of the following choices.
Declare the <servlet> entries in your WEB-INF/web.xml to configure the named servlet default to have a init-param of dirAllowed set to false.
This is a change in the individual webapp's own WEB-INF/web.xml
Declare a servlet context init-parameter (not servlet specific, whole context), where the key org.eclipse.jetty.servlet.Default.dirAllowed is set to value false.
This is a change in either the individual webapp's own WEB-INF/web.xml or the XML deployable (ie: ${jetty.base}/webapps/<name>.xml) for each webapp.
Provide an alternate webdefault.xml for the defaultDescriptor that configures the default behavior to be dirAllowed=false.
This is a change to the either the individual webapp XML deployable to set the WebAppContext.setDefaultDescriptor(), or the overall deployable defaults for the chosen DeploymentManager / AppProvider combo you are using. This will also require a custom ${jetty.base}/etc/<name>.xml which is your new default descriptor.
Provide a override descriptor xml that can be applied after your defaultDescriptor + webapp descriptor to configure dirAllowed=false.
This is a change to the either the individual webapp XML deployable to set the WebAppContext.setOverrideDescriptor(), or the overall deployable defaults for the chosen DeploymentManager / AppProvider combo you are using. This will also require a custom ${jetty.base}/etc/<name>.xml which is your new override descriptor.
If you are using ResourceService / ResourceHandler in an embedded-jetty scenario, you can just call ResourceService.setDirAllowed(false).
Prevent web app context listings, which may reveal sensitive information such as server directory paths or other web app contexts running within the same Jetty instance.
That's controlled by the showContexts configuration on the DefaultHandler.
There's 2 ways to control this behavior.
Option A: configure the DefaultHandler
You can add 2 files to your ${jetty.base}
New file: etc/tweak-defaulthandler.xml
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "https://www.eclipse.org/jetty/configure_10_0.dtd">
<Configure id="DefaultHandler" class="org.eclipse.jetty.server.handler.DefaultHandler">
<Set name="showContexts">false</Set>
</Configure>
Add a line to ${jetty.base}/start.d/tweaks.ini to use this XML
$ cat start.d/tweaks.ini
etc/tweak-defaulthandler.xml
Option B: just declare a ROOT context with default behavior for ROOT.
Create a ${jetty.base}/webapps/ROOT directory.
Add a ${jetty.base}/webapps/ROOT/index.html with whatever content you want.
That will be served instead of DefaultHandler creating the list of contexts.
Be sure that DefaultHandler doesn't provide access to any sensitive files in the event that I botch a web app deployment. I'm happy to implement my own static file serving servlet where necessary as an alternative to using DefaultHandler.
The DefaultHandler only serves 3 things.
/favicon.ico requests (only if a GET request)
/ (show context listing) (only if a GET request and showContexts is true)
404 Errors - all other requests that reach this handler.
You are confusing DefaultServlet / ResourceService / ResourceHandler with DefaultHandler a totally different thing.
How do I get the raw request path (everything after the host name and the port) in Django?
I tried request.get_full_path(), but it doesn’t work for some URLs.
For example, when the URL is http://localhost:8000/data/?, the result is /data/ instead of /data/?.
I know that the server receives the full string because it show "GET /data/? HTTP/1.1" 200 642 in the terminal.
You can use request.build_absolute_uri()
Depends on source code:
Builds an absolute URI from the location and the variables available in
this request. If no location is specified, the absolute URI is
built on request.get_full_path(). Anyway, if the location is
absolute, it is simply converted to an RFC 3987 compliant URI and
returned and if location is relative or is scheme-relative (i.e.,
//example.com/), it is urljoined to a base URL constructed from the
request variables.
Getting access to the raw URL depends on your setup. I'm not aware of a way to do this with Django's development server.
HTTPRequest.META is described as follows:
A dictionary containing all available HTTP headers. Available headers depend on the client and server, but here are some examples:
If you're running gunicorn, request.META["RAW_URI"] might give you what you need.
So I am having a bit of trouble.
I have many products with the same part o URL that I recently changed:
https://www.website.com/shop/category-sample/product1/
https://www.website.com/shop/category-sample/product2/
https://www.website.com/shop/category-sample/product3/
https://www.website.com/shop/category-sample/product4/
I need the "category-sample" to be "category"
So the new URLS would look like this:
https://www.website.com/shop/category/product1/
And etc.
Thank you!
Assuming that you are using the typical apache http server with loaded rewriting module this should do what you are looking for:
RewriteEngine on
RewriteRule ^/?shop/category-sample/(.*)$ /shop/category/$1 [R=301,QSA,END]
In case "category" actually is a dynamic value, not a fixed literal this variant should do what you ask for:
RewriteEngine on
RewriteRule ^/?shop/(.+)-sample/(.*)$ /shop/$1/$2 [R=301,QSA,END]
That rule will work likewise in the http servers host configuration of in a dynamic configuration file (".htaccess" style file) if you have to use those.
If you receive an "internal server error" using those rules (http status 500) then chances are that you operate a very old version of the apache http server. Have a try using the L flag instead of the newer END flag then. You will find a corresponding hint in your http servers error log file in that case.
And a general remark: you should always prefer to place such rules in the http servers host configuration instead of using dynamic configuration files (".htaccess"). Those dynamic configuration files add complexity, are often a cause of unexpected behavior, hard to debug and they really slow down the http server. They are only provided as a last option for situations where you do not have access to the real http servers host configuration (read: really cheap service providers) or for applications insisting on writing their own rules (which is an obvious security nightmare).
I am using IIS7 request Filtering (WebSite level) with 'Allow unlisted file name extensions' set to un-checked. After this setting when I access my web application say, _http:/machine_name/app1 then it fails. After adding correct file extensions under 'file name extensions' it starts functioning but it later fails when I surf other links inside my application. I found that I need to add URL's under 'Allow URL' but I do have huge number of URL list which need to be added for e.g.:
'/app1/prop1', '/app1/prop1/services/', '/app1/prop2/repository'..... and more
So, I see a pattern like /app1/* but 'allow URL don't allow any pattern matching. Is there any way to add Regular expression somewhere in IIS so that the matching URL's can be allowed.
Thanks in advance.
I think URLs like '/app1/prop1', '/app1/prop1/services/', '/app1/prop2/repository' don't work with 'Allow unlisted file name extensions' set to false because you didn't allow extensionless requests.
To allow extensionless requests, add <add fileExtension="." allowed="true" /> to your web.config as below:
<system.webServer>
<security>
<requestFiltering>
<fileExtensions allowUnlisted="false">
<add fileExtension="." allowed="true" />
</fileExtensions>
</requestFiltering>
</security>
</system.webServer>
Please let me know if this helped or the issue still occurs.
OK, at last I got the solution. The trick was not using 'requestfiltering' but 'URL rewrite'. It has the option to allow or deny URL based on the regular expression or Wildcard with the option 'Does not match the pattern'. I created rule for the URL/file which does not matches my application's pattern and deny those by redirecting a 404 type of error.
#alexander-abakumov is correct. When allowUnlisted=false, one must include the provided xml (<add fileExtension="." allowed="true" />) to allow extension-less requests (e.g. /App/ or /Folder/).
Request Filtering Limitations
You're correct to use URL Rewrite for achieving RegEx pattern matching. Below are the limitations of Request Filtering.
Stateless - Request Filtering has no knowledge of application or session state. Each request is processed individually regardless of whether a session has or has not been established.
Request Header Only - Request Filtering can only inspect the request header. It has no visibility into the request body or any part of the response.
Basic Logic - Regular expressions and wildcard matches are not available. Most settings consist of establishing size constraints while others perform simple string matching.
Still use Request Filtering
I would still recommend using Request Filtering since it operates so early in the IIS request pipeline; even before URL Rewrite. This gives you the opportunity to immediately discard bad requests. URL Rewrite can then be a second layer of request filtering.
IISRFBaseline
The information above is based on my PowerShell module IISRFBaseline. It helps establish an IIS Request Filtering baseline by leveraging Microsoft Logparser to scan a website's content directory and IIS logs.
For the fileExtensions setting there are two methods for establishing a baseline:
IIS Logs - by parsing prior successful requests and determining the extension of the request.
Content Directory - scanning the file system of the website content directory to see what file extensions are in use.
There are caveats to these two approaches which are described in more detail here. With two techniques, a markdown file is provided describing the methodology of each as denoted by the suffix IIS or FS.
fileExtensions_FS
fileExtensions_IIS
The result of these two techniques are combined, after removing bad requests, to create a baseline for this Request Filtering setting.
I have a Django application and I use nginx to serve static content. Unfortunately, all registered MIME types get displayed in client browser, while I would like to give an ability to download the same content, along with usual behaviour. Say, I have JPEG file under /media/images/image01.jpg and I want that nginx serves this file in usual way, with standard image/jpeg header, but additionally I want the same image to be served by nginx with content-disposition: attachment (effectively forcing content download) when accessed as, say, /downloads/images/image01.jpg. Anybody can suggest a solution?
Make sure you have the http_headers_module compiled in. (should be by default, if it isn't in the core)
Use "add_header content-disposition attachment;"
I recommend using a url like "/download?file=/downloads/images/image01.jpg" combined with a rewrite rule to avoid some annoying bug later.
Http Headers Module Documention