I need to download an xml file from AWS-S3.
I tried using get_contents_to_filename(fname) , it worked.
But i need to download the file without specifying fname, because if i specify the fname my downloaded file gets saved tofname.
I want to save the file as it is, with its name.
this is my current code
k = Key(bucket)
k.set_contents_from_filename(fname)
can someone please help me to download and fetch the file without using key.
Thanks in advance!
I'm not sure which library you're using, but if k is the AWS key you want to download, then k.name is probably the key name, so k.get_contents_to_filename(k.key) would probably do more or less what you want.
The one problem is that the key name might not be a legal file name, or it may have file path separators. So if the key name were something like '../../../../somepath/somename' the file would be saved somewhere you don't expect. So copy k.name to a string and either sanitize it by changing all dangerous characters to safe ones, or just extract the part of the key name you want to use for the file name.
Related
I have a use case where I want to read the filename from a metadata table, I have written a pipeline function to read the metadata table, but I am not sure how can I pass this information to ReadFromText as it only takes string as input, Is it possible to assign this value to ReadFromText(). Please suggest some workarounds or ideas how to achieve this, Thanks
code: pipeline | 'Read from a File' >> ReadFromText(I want to pass the file path here?,
skip_header_lines=1)
Note: There will be various folders and files in storage, files are in csv format, but in my use case I can't directly pass the storage location or filename to file path in ReadFromText. I want to read it from metadata and pass the value. Hope I am clear, Thanks
I don't understand why you need to read the metadata. If you want to read all the files inside a folder, you can just provide a blob. This solution working in python, not sure about java.
p|readfromtext("./folder/*.csv")
"*" is the blob here, which allows pipeline to read all the patterns matching .csv. You can also add something at the starting.
What you want is textio.ReadAllFromText which reads from a PCollection instead of taking a string directly.
I'm trying to understand why there needs to be two arguments here:
key which is basically the filename
path argument, the path of the file
Isn't it redundant to still pass the key?
Can't we just pass the bucket and the filename?
key in S3 could be a long string, like /my-prefix/YYYY/MM/DD/UUID.txt. It can, and usually will, contain things like slash characters. So it makes sense to have to specify the local filename argument separately, because you may not want to save the file in the same path that it is saved in S3, and you also may not want to save it using the same name that was used in S3.
You can download the file to the Filename which has different name than the Key from where you are downloading, e.g. Key = 'hello.txt', Filename = '/tmp/MylocalCopyOfTheHello.txt'
Is that possible to shorten a file path and get the file with that shortened path?
Ex: In my rails public folder I have a file like
/users/000/001/991/image.png
I want to shorten it like /<dynamic user_name>/image.png. But when I use that shortened url I want to refer to the original file path. I googled but couldn't find a way. Appreciate if I get any reference.
Logic depends as per your need and current DB structure. some possible solution could be
-If your path is fix, just store image name. and while creating full path join +
-If path is no fixed and storing path in DB, you can create one column more column and store unique id(any integer or alpanumeric as per logic but unique value) and use that unique value to replace path
Guy
now i want to upload file with rails 4
my problem now i can't check the file extension before upload it
Note : I can upload the file well but i want to get the file kind before upload it
because i need the extension in another step in my App.
I'm tried to use the commands
File.extname(params[:Upload])
but always got the error
can't convert ActionDispatch::Http::UploadedFile into String
also how i can get the file base name before upload it ??
when i tryed to use
File.basename(params[:Upload])
i got the same error
can't convert ActionDispatch::Http::UploadedFile into String
also when i tried to convert the name to Sting i don't get any thing
That's because File.extname expects a string file name, but an uploaded file (your params[:upload] is an object, it's an instance of the ActionDispatch::Http::UploadedFile class (kind of a temporary file)
To fix the problem you need to call the path property on your params[:upload] object, kind of like that
File.extname(params[:Upload].path)
Btw, if you're trying to get the type of the uploaded file, I'd encourage you to check for the params[:Upload].content_type instead, it's harder to spoof
You can use this:
params[:Upload].original_filename.split('.').last
The original_filename contains full filename with extension.
so you split it based on '.' and last index will contain the file extension.
For Example:
"my_file.doc.pdf".split('.').last # => 'pdf'
You can check this for more info ActionDispatch::Http::UploadedFile.
at the moment I'm writing a kind of lib, which gets from outside the file name 'sFilename'. With it data were written to a file it will be created, data were append to an existing file with data, data were updated in an existing file with data or the data were read from an existing data.
The user of the application with my lib should get as much as possible on information about errors of file handling.
For this purpose I code a method FileExists(...) which uses _stat() to determine if a file exists or not and a method "bool checkPermission(std::string sFilename, CFile::EOpenmode iOpenmode)" which gives back a bool if the specified file (sFilename) with the iOpenmode (Read, Write, Readwrite) have the permission to be read, written or read and written.
This method works with _stat(sFilename.c_str(), &buf) too and gives the desired information of the file back in buf.
Before checking any file I want to check if the directory containing the specified file has the desired permissions and for that I want to use the checkPermission method [works with _stat()] for the directory!
Now the problem: how can I determine easyly the containing directory? If the user just give a filename "test.txt" the file will be created or read in working directory. So its easy to get the up-directory. Its the same like the working directory. (And there its simple to use checkPermission to get detailed information about the directory).
But what about when the user not only give the file name? For exaample "....\test.txt" or "dir1\dir2\test.txt". How to combine the working directory with a specific path to gain the up-directory of the file and then to check the permissions?
Phew, I hope all is clear and it was'nt too long ;-)
Rumo
I'd suggest using the Boost FileSystem library at www.boost.org. In particular, check out the path class, which has methods such as make_absolute and parent_path.
This is Windows example code GetFileNameFromHandle to show you how to get the path from a HANDLE. I think it is what you are looking for.
http://msdn.microsoft.com/en-us/library/aa366789%28v=vs.85%29.aspx
I found out that _stat() and _access() doesn't really works for the permissions of the directories. See this stackoverflow page.
With _stat() you can't use ".\" to get information about the current directory. But _access() at least can check if a directory exists as well ".\" or "..\".
In conclusion I use _access() to check the existence of a directory and _stat() to check the permissions of an existing file. If a file should be created I'll check it by doing.
And by the way ;-) I don't need to combine working directory with the user specified file because I can use the specified file alone in _access() to determine if directory exists.
Rumo