I have setup a Django application, in which user can upload his image and it is served by Nginx and Gunicorn.
I have a problem with uploading large image files which does not get appropriate permissions to be served by Nginx
location /medias/images/ {
root /var/www/html;
}
When uploading files, the larger ones only get read permissions for the user, not for group/other:
-rw------- 1 user1 user1 4.9M Mar 15 14:35 File1.jpg
-rw------- 1 user1 user1 3.7M Mar 15 14:31 File2.jpg
-rw-r--r-- 1 user1 user1 110K Mar 15 14:44 File3.pdf
-rw-r--r-- 1 user1 user1 34K Mar 15 09:17 File4.docx
-rw-r--r-- 1 user1 user1 136K Mar 15 14:45 File5.jpg
-rw-r--r-- 1 user1 user1 92K Mar 15 14:22 File6.doc
-rw------- 1 user1 user1 4.4M Mar 15 14:25 File7.jpg
However the smaller images get their permissions fine and are served properly.
The point is that both uploading small and semi-large (3mb) image files are done by a same process.
Any ideas?
Set the FILE_UPLOAD_MAX_MEMORY_SIZE parameter in your Django settings, in Bytes.
For example FILE_UPLOAD_MAX_MEMORY_SIZE = 20971520 equals 20MB.
Related
I am trying to connect to my LND node running on AWS (I know it is not the best case scenario for an LND node but this time I had no other way of doing it) from my local running Django Rest Api. The issue is that it cannot find the admin.macaroon file even though the file is in the mentioned directory. Below I am giving some more detailed information:
view.py
class GetInfo(APIView):
def get(self, request):
REST_HOST = "https://ec2-18-195-111-81.eu-central-1.compute.amazonaws.com"
MACAROON_PATH = "/home/ubuntu/.lnd/data/chain/bitcoin/mainnet/admin.macaroon"
# url = "https://ec2-18-195-111-81.eu-central-1.compute.amazonaws.com/v1/getinfo"
TLS_PATH = "/home/ubuntu/.lnd/tls.cert"
url = f"https//{REST_HOST}/v1/getinfo"
macaroon = codecs.encode(open(MACAROON_PATH, "rb").read(), "hex")
headers = {"Grpc-Metadata-macaroon": macaroon}
r = requests.get(url, headers=headers, verify=TLS_PATH)
return Response(json.loads(r.text))
The node is running with no problem on AWS. This is what I get when I run lncli getinfo:
$ lncli getinfo:
{
"version": "0.15.5-beta commit=v0.15.5-beta",
"commit_hash": "c0a09209782b1c62c3393fcea0844exxxxxxxxxx",
"identity_pubkey": "mykey",
"alias": "020d4da213770890e1c1",
"color": "#3399ff",
"num_pending_channels": 0,
"num_active_channels": 0,
"num_inactive_channels": 0,
"uris": [
....
and the permissions are as below:
$ ls -l
total 138404
-rwxrwxr-x 1 ubuntu ubuntu 293 Feb 6 09:38 admin.macaroon
drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 5 14:48 bin
drwxr-xr-x 6 ubuntu ubuntu 4096 Jan 27 20:17 bitcoin-22.0
drwxrwxr-x 4 ubuntu ubuntu 4096 Feb 1 16:39 go
-rw-rw-r-- 1 ubuntu ubuntu 141702072 Mar 15 2022 go1.18.linux-amd64.tar.gz
drwxrwxr-x 72 ubuntu ubuntu 4096 Feb 1 16:36 lnd
-rw-rw-r-- 1 ubuntu ubuntu 0 Jan 27 20:13 screenlog.0
The error I get is [Errno 2] No such file or directory:'/home/ubuntu/.lnd/data/chain/bitcoin/mainnet/admin.macaroon'
I guess the problem should be how I need to access the node from my API, but I have no idea how to access an EC2 instance from an external api.
Thank you in advance
I'm working with OpenEdX, it has a plugin system, called XBlocks, that in this case allows importing content created by third party "studio apps." This content can be uploaded as a zip file. it is then processed by the following code:
#XBlock.handler
def studio_submit(self, request, _suffix):
self.display_name = request.params["display_name"]
self.width = request.params["width"]
self.height = request.params["height"]
self.has_score = request.params["has_score"]
self.weight = request.params["weight"]
self.icon_class = "problem" if self.has_score == "True" else "video"
response = {"result": "success", "errors": []}
if not hasattr(request.params["file"], "file"):
# File not uploaded
return self.json_response(response)
package_file = request.params["file"].file
self.update_package_meta(package_file)
# First, save scorm file in the storage for mobile clients
if default_storage.exists(self.package_path):
logger.info('Removing previously uploaded "%s"', self.package_path)
default_storage.delete(self.package_path)
default_storage.save(self.package_path, File(package_file))
logger.info('Scorm "%s" file stored at "%s"', package_file, self.package_path)
# Then, extract zip file
if default_storage.exists(self.extract_folder_base_path):
logger.info(
'Removing previously unzipped "%s"', self.extract_folder_base_path
)
recursive_delete(self.extract_folder_base_path)
with zipfile.ZipFile(package_file, "r") as scorm_zipfile:
for zipinfo in scorm_zipfile.infolist():
default_storage.save(
os.path.join(self.extract_folder_path, zipinfo.filename),
scorm_zipfile.open(zipinfo.filename),
)
try:
self.update_package_fields()
except ScormError as e:
response["errors"].append(e.args[0])
return self.json_response(response)
where the code
default_storage.save(
os.path.join(self.extract_folder_path, zipinfo.filename),
scorm_zipfile.open(zipinfo.filename),
)
is the origin of the following (Django) error trace:
cms_1 | File "/openedx/venv/lib/python3.5/site-packages/openedxscorm/scormxblock.py", line 193, in studio_submit
cms_1 | scorm_zipfile.open(zipinfo.filename),
cms_1 | File "/openedx/venv/lib/python3.5/site-packages/django/core/files/storage.py", line 52, in save
cms_1 | return self._save(name, content)
cms_1 | File "/openedx/venv/lib/python3.5/site-packages/django/core/files/storage.py", line 249, in _save
cms_1 | raise IOError("%s exists and is not a directory." % directory)
cms_1 | OSError: /openedx/media/scorm/c154229b568d45128e1098b530267a35/a346b1db27aaa89b89b31e1c3e2a1af04482abad/assets exists and is not a directory.
I posted the issue on github too
exception FileExistsError
Raised when trying to create a file or directory which already exists. Corresponds to errno EEXIST.
I don't really understand what is going on. It's based on a hairball of javascript in layered docker containers, so I can't readily hack&print for extra info.
The only thing I found was that some of the folders in the zip file are written to the docker volume as files instead of directories at the moment the error is thrown. This may however be expected and these files might be rewritten as or changed to directories later (?) on Linux (?).
The error lists the assets folder
root#93f0d2b9667f:/openedx/media/scorm/5e085cbc04e24b3b911802f7cba44296/92b12100be7651c812a1d29a041153db5ba89239# ls -la
total 84
drwxr-xr-x 2 root root 4096 Aug 2 22:17 .
drwxr-xr-x 3 root root 4096 Aug 2 22:17 ..
-rw-r--r-- 1 root root 4398 Aug 2 22:17 adlcp_rootv1p2.xsd
-rw-r--r-- 1 root root 0 Aug 2 22:17 assets
-rw-r--r-- 1 root root 0 Aug 2 22:17 course
-rw-r--r-- 1 root root 14560 Aug 2 22:17 imscp_rootv1p1p2.xsd
-rw-r--r-- 1 root root 1847 Aug 2 22:17 imsmanifest.xml
-rw-r--r-- 1 root root 22196 Aug 2 22:17 imsmd_rootv1p2p1.xsd
-rw-r--r-- 1 root root 1213 Aug 2 22:17 ims_xml.xsd
-rw-r--r-- 1 root root 1662 Aug 2 22:17 index.html
-rw-r--r-- 1 root root 0 Aug 2 22:17 libraries
-rw-r--r-- 1 root root 1127 Aug 2 22:17 log_output.html
-rw-r--r-- 1 root root 481 Aug 2 22:17 main.html
-rw-r--r-- 1 root root 759 Aug 2 22:17 offline_API_wrapper.js
-rw-r--r-- 1 root root 0 Aug 2 22:17 player
-rw-r--r-- 1 root root 1032 Aug 2 22:17 popup.html
root#93f0d2b9667f:/openedx/media/scorm/5e085cbc04e24b3b911802f7cba44296/92b12100be7651c812a1d29a041153db5ba89239# cd assets
bash: cd: assets: Not a directory
How do I add resources (preferably programmatically) to the classpath of my program ?
I am using compojure.route.resources like the following :
A route for serving resources on the classpath. Accepts the following
keys: :root - the root prefix path of the resources, defaults
to 'public' :mime-types - an optional map of file extensions to mime
types
(defroutes routes
(resources "/js" {:root "js"})
(resources "/css" {:root "css"}))
Now I would like to add files to be served, but I don't know where to generate them.
Note : in case it matters, I am using boot and the example is taken from this template.
In regular boot based project, you can have a resources folder at the root:
-rw-r--r--# 1 niko staff 173 Dec 18 10:19 boot.properties
-rw-r--r--# 1 niko staff 2796 Dec 30 09:55 build.boot
drwxr-xr-x# 17 niko staff 578 Dec 30 10:49 resources
drwxr-xr-x# 4 niko staff 136 Nov 16 09:52 src
drwxr-xr-x# 17 niko staff 578 Jan 14 11:50 target
In that folder you can create the css and the js folder and include your files there:
drwxr-xr-x# 5 niko staff 170 Dec 14 15:33 resources/css/
drwxr-xr-x# 4 niko staff 136 Nov 30 18:01 resources/js/
when uploading files in the admin backend of SilverStripe, I always get permission denied.
The apache2 is running under www-data:www-data (checked with "whoami" and in the envvars).
The permissions for the assets folder are the following:
drwxrwxr-x+ 5 git www-data 4.0K Oct 2 08:18 .
drwxrwsr-x+ 17 git users 4.0K Oct 2 09:53 ..
-rwxrwxr-x+ 1 git www-data 1.5K Oct 2 08:18 .htaccess
drwxrwsr-x+ 2 git www-data 4.0K Oct 2 07:28 Uploads
drwxrwsr-x+ 2 git www-data 4.0K Oct 2 08:18 _combinedfiles
drwxrwsr-x+ 2 git www-data 4.0K Oct 2 07:57 books
-rwxrwxr-x+ 1 git www-data 4.5K Oct 1 14:48 error-404.html
-rwxrwxr-x+ 1 git www-data 4.3K Oct 1 14:48 error-500.html
-rwxrwxr-x+ 1 git www-data 4.4K Oct 2 08:18 web.config
Even if I set them to 777, I still get permission denied. It works locally and I can't get my head around why it doesnt work.
Apache ErrorLog:
[Fri Oct 02 10:24:18.364138 2015] [:error] [pid 16008] [client 62.218.34.90:44300] PHP Warning: copy(/var/www/X/assets/books/Mike.png): failed to open stream: Permission denied in /var/www/X/framework/filesystem/Upload.php on line 195, referer: http://x.amazonaws.com/index.php/admin/assets/add/?ID=7
[Fri Oct 02 10:24:18.364615 2015] [:error] [pid 16008] [client 62.218.34.90:44300] PHP Warning: Requirements_Backend::process_combined_files(): Couldn't create '/var/www/X/assets/_combinedfiles//lib.js' in /var/www/X/framework/view/Requirements.php on line 1214, referer: http://x.amazonaws.com/index.php/admin/assets/add/?ID=7
Is this an AWS specific problem? Any clues?
The problem were ACLs.
Using getfacl ./assets/ I could determine, that the ACLs were set to undermine the normal permission model.
With setfacl I was able to advance the ACLs to include the webuser (www-data) into the assets directory with rwx.
Hope this helps someone,
Best regards. G
I would recommend you use a module and store the assets in S3
https://github.com/markguinn/silverstripe-cloudassets
I'm on a shared linux server and I only have access to http://domain.com/~username/ which maps to ~/public_html.
I would like to use the Taffy framework for ColdFusion in this context. As far as I can tell, to use the framework you must have an Application.cfc that extends the Taffy framework component taffy.core.api.
https://github.com/atuttle/Taffy
https://github.com/atuttle/Taffy/wiki/Installing-Taffy
https://github.com/atuttle/Taffy/wiki/So-you-want-to:-Create-a-dead-simple-CRUD-API
The only directories I have access to are not sub-directories of the web root and therefore (as I understand it) not subsets of the ColdFusion path.
In my particular case I neither have access to CFADMIN nor are the server admins going to install a component I need to extend in a system-wide context where it is already on the path and accessible via global dot notation.
The instructions say you should unzip the taffy folder into your web root, and if you can't do that, you should make it a subfolder of your api. The former is not a possibility for me and when I do the latter I get "Could not find the ColdFusion Component or Interface taffy.core.api."
More Detail:
My api is at http://domain.com/~username/api/, so I unzipped /taffy to ~/public_html/api/. If I copy the Taffy example at taffy/examples/api to ~/public_html/api so that going to http://domain.com/~username/api/ should access the example, I get "Could not find the ColdFusion Component or Interface taffy.core.api" even if have taffy/core/api.cfc beneath that directory (~/public_html/api).
On this server I have successfully made cfc's that extend a cfc in another directory using <cfset THIS.mappings["/subdir"]= getDirectoryFromPath(getCurrentTemplatePath()) & "subdir/"> and <cfobject name="parentObj" component="subdir.parent">.
I have also successfully made an Application.cfc that extends a cfc in the same directory.
I just haven't successfully made an Application.cfc that can extend a cfc in another directory, even if it's a subdirectory.
I did try to use grep & related tools to strip every reference of "taffy.core" out of Taffy's source code so I could just dump all the taffy cfc's into my root directory along with Application.cfc so I could extend api.cfc, but I got different errors and didn't pursue that hacky solution any further.
<cfdump var=#expandPath('/mapping')# /> outputs /var/www/html/mapping.
uname#domain $>ls -la /var/www/html
drwxr-xr-x 3 root root 4096 Sep 16 00:34 .
drwxr-xr-x 7 root root 4096 May 28 2012 ..
lrwxrwxrwx 1 root root 19 Sep 16 00:34 cfide -> /var/www/html/CFIDE
drwxrwxr-x 10 apache root 4096 Sep 16 00:32 CFIDE
~/public_html/api/resources/successesCollection.cfc:
<cfcomponent extends="taffy.core.resource" taffy_uri="/successes">
<cffunction name="get" access="public" output="false">
<cfreturn representationOf('success').withStatus(200) />
</cffunction>
</cfcomponent>
~/public_html/api/Application.cfc:
<cfcomponent extends="taffy.core.api">
<!--- doesn't work
<cfset THIS.mappings["/taffy"]= getDirectoryFromPath(getCurrentTemplatePath()) & "taffy/">
<cfset THIS.mappings["/core"]= getDirectoryFromPath(getCurrentTemplatePath()) & "taffy/core/">
--->
<cfscript>
this.name = hash(getCurrentTemplatePath());
// do your onApplicationStart stuff here
function applicationStartEvent(){}
// do your onRequestStart stuff here
function requestStartEvent(){}
// this function is called after the request has been parsed and all request details are known
function onTaffyRequest(verb, cfc, requestArguments, mimeExt){
// this would be a good place for you to check API key validity and other non-resource-specific validation
return true;
}
// called when taffy is initializing or when a reload is requested
function configureTaffy(){
setDebugKey("debug");
setReloadKey("reload");
setReloadPassword("true");
// Usage of this function is entirely optional. You may omit it if you want to use the default representation class.
// Change this to a custom class to change the default for the entire API instead of overriding for every individual response.
setDefaultRepresentationClass("taffy.core.genericRepresentation");
}
</cfscript>
</cfcomponent>
Output of http://domain.com/~uname/api/index.cfm/successes/: Could not find the ColdFusion Component or Interface taffy.core.api.
Adding this to my Application.cfc doesn't fix the issue:
<cfcomponent extends="taffy.core.api">
<cfscript>
this.name = hash(getCurrentTemplatePath());
this.mappings = StructNew();
this.mappings['/taffy'] =
expandPath('./taffy');
In addition, adding the following to ~/public_html/api/Application.cfc doesn't fix the issue either:
<cfset this.mappings["/taffy"] =
expandPath(getDirectoryFromPath(getCurrentTemplatePath()) & "taffy")>
Check out the following sequence of commands and let me know if I've overlooked something. I'm still left with "Could not find the ColdFusion Component or Interface taffy.core.api" upon browsing to "http://domain/~uname/api".
[uname#domain ~]$ cd ~/public_html
[uname#domain ~/public_html]$ rm -rf api
[uname#domain ~/public_html/api]$ wget -O taffy.zip https://github.com/atuttle/Taffy/zipball/master
[uname#domain ~/public_html/api]$ unzip taffy.zip
[uname#domain ~/public_html/api]$ mv atuttle-Taffy-35df54e/ taffy
[uname#domain ~/public_html/api]$ mv taffy/examples/api .
[uname#domain ~/public_html/api]$ mv taffy api/
[uname#domain ~/public_html/api]$ tree -d ~/public_html/api/
~/public_html/api/
|-- resources
`-- taffy
|-- bonus
|-- core
|-- examples
| |-- ParentApplication
| | |-- config
| | |-- mixin
... etc
[uname#domain ~/public_html/api]$ ls -la ~/public_html/api/
total 8
drwxr-xr-x 4 uname ugroup 1024 Dec 9 11:00 .
drwxr-xr-x 10 uname web 1024 Dec 9 10:57 ..
-rw-r--r-- 1 uname ugroup 1188 Dec 9 11:00 Application.cfc
-rw-r--r-- 1 uname ugroup 172 Sep 20 13:04 .htaccess
-rw-r--r-- 1 uname ugroup 218 Sep 20 13:04 index.cfm
drwxr-xr-x 2 uname ugroup 1024 Sep 20 13:04 resources
drwxr-xr-x 8 uname ugroup 1024 Sep 20 13:04 taffy
[uname#domain ~/public_html/api]$ ls -la ~/public_html/api/taffy/
total 15
drwxr-xr-x 8 uname ugroup 1024 Sep 20 13:04 .
drwxr-xr-x 4 uname ugroup 1024 Dec 9 11:00 ..
drwxr-xr-x 2 uname ugroup 96 Sep 20 13:04 bonus
-rw-r--r-- 1 uname ugroup 4096 Sep 20 13:04 build.xml
drwxr-xr-x 2 uname ugroup 1024 Sep 20 13:04 core
drwxr-xr-x 15 uname ugroup 1024 Dec 9 10:57 examples
-rw-r--r-- 1 uname ugroup 99 Sep 20 13:04 .gitignore
drwxr-xr-x 2 uname ugroup 96 Sep 20 13:04 lib
-rw-r--r-- 1 uname ugroup 1356 Sep 20 13:04 LICENSE.TXT
-rw-r--r-- 1 uname ugroup 2490 Sep 20 13:04 ReadMe.md
drwxr-xr-x 3 uname ugroup 96 Sep 20 13:04 snippets
drwxr-xr-x 5 uname ugroup 1024 Sep 20 13:04 tests
[uname#domain ~/public_html/api]$ ls -la ~/public_html/api/taffy/core/
total 72
drwxr-xr-x 2 uname ugroup 1024 Sep 20 13:04 .
drwxr-xr-x 8 uname ugroup 1024 Sep 20 13:04 ..
-rw-r--r-- 1 uname ugroup 42382 Sep 20 13:04 api.cfc
-rw-r--r-- 1 uname ugroup 4574 Sep 20 13:04 baseRepresentation.cfc
-rw-r--r-- 1 uname ugroup 2572 Sep 20 13:04 dashboard.cfm
-rw-r--r-- 1 uname ugroup 1756 Sep 20 13:04 dashboard.css
-rw-r--r-- 1 uname ugroup 4538 Sep 20 13:04 docs.cfm
-rw-r--r-- 1 uname ugroup 3030 Sep 20 13:04 factory.cfc
-rw-r--r-- 1 uname ugroup 179 Sep 20 13:04 genericRepresentation.cfc
-rw-r--r-- 1 uname ugroup 3516 Sep 20 13:04 mocker.cfm
-rw-r--r-- 1 uname ugroup 389 Sep 20 13:04 nativeJsonRepresentation.cfc
-rw-r--r-- 1 uname ugroup 3765 Sep 20 13:04 resource.cfc
You do have a couple of options.
Mapping (per-application or otherwise)
As Taffy is a development framework, your sysadmin/host may be willing to install it in a central location for all developers to make use of. They could put the Taffy folder in the web root, or create a server-level mapping to wherever the folder may be.
Relative Path
It should be possible to run Taffy from a relative path. It sounds like this is the approach you're trying to take, but you may not have the files in the right locations.
In order to use relative paths, you need a directory structure that resembles:
~uname/api/
~uname/api/taffy/core/api.cfc <- Framework contents
~uname/api/taffy/core/factory.cfc
~uname/api/taffy/core/dashboard.cfm
~uname/api/taffy/core/...
~uname/api/taffy/...
~uname/api/Application.cfc <- your api code
~uname/api/index.cfm
~uname/api/resources <- where you put your resource CFC's
It sounds like you're missing the "taffy" folder and either put the CFC's directly in your api folder or put the "core" folder in your api folder.
The "taffy" folder is required. Think of the dot-notation path you're putting in your Application.cfc's extends attribute as a filesystem path. Since it's taffy.core.api then your filesystem needs to contain taffy/core/api.cfc.
Have you tried using a relative path proxy, as Ben Nadel explains here: http://www.bennadel.com/blog/2115-Extending-The-Application-cfc-ColdFusion-Framework-Component-With-A-Relative-Path-Proxy.htm
The idea is that your application.cfc would extend a local proxy (rootProxy.cfc) that resides in the same directory. That proxy then cfincludes the cfc you are interested in. Since cfinclude takes a relative path, you needn't worry about global dot notation or mappings.