So far to manage getting data from the bucket I use download_to_file() to get it downloaded on the instance that it is using and access the files/folders locally. Though what I want to achieve is being able to just read from the cloud. How can I go about doing that? There doesn't seem to be a way for me create a relative path from the ML Job instance and the google cloud bucket.
You can use TensorFlow's file_io.FileIO class to create file_like objects to read/write files on gcs, local, or any other supported file system.
See this post for some examples.
Related
I use google cloud storage to store static data. at the start of the build I was able to create the folder without any problems. some time ago I wanted to create a new folder on my cloud storage and an error appeared as below:
Even though as I recall, I didn't make any changes before, at that time I was able to create folders and now why can't I create folders?
I've tried adding my email to cloud storage permissions with the role as storage admin and storage legacy bucket owner in the hope of creating a folder. but when I save the configuration and try to create the folder again, the same error still appears and still can't create the folder. but if uploading files to cloud storage is not a problem, the files can be uploaded properly. But why can't I create a folder.
Can anyone help me?
As mentioned in the comments and the linked question, you cannot create new buckets due to having a delinquent billing account. You can reactivate it by following these steps. Keep in mind that you will need the required permissions and resolve any delayed/declined payments. You can also contact the Billing team.
I have been able to mount Google Cloud Bucket using
gcsfuse --implicit-dirs " production-xxx-appspot /mount
or equally
sudo mount -t gcsfuse -o implicit_dirs,allow_other,uid=1000,gid=1000,key_file=service-account.json production-xxx-appspot /mount
Mounting works fine.
What happens is that when I execute the following commands after mounting, they also work fine :
mkdir /mount/files/
cp -rf /home/files/* /mount/files/
However, when I use :
mcedit /mount/files/a.txt
or
vi /mount/files/a.txt
The output says that there is no file available which makes sense.
Is there any other way to cover this situation, and use applications in a way that they can directly create files on the mounted google cloud bucket rather than creating files locally and copying afterwards.
If you do not want to create files locally and upload later, you should consider using a file storage system like Google Drive
Google Cloud storage is an object Storage system that means objects cannot be modified, you have to write the object completely at once. Object storage also does not work well with traditional databases, because writing objects is a slow process and writing an app to use an object storage API is not as simple as using file storage.
In a file storage system, Data is stored as a single piece of information inside a folder, just like you would organize pieces of paper inside a manila folder. When you need to access that piece of data, your computer needs to know the path to find it. (Beware—It can be a long, winding path.)
If you want to use Google Cloud Storage, you need to create your file locally and then push it to your bucket.
Here are an example of how to configure Google Cloud Storage with Node.js: File Upload example
Here is a tutorial on How to mount Object Storage on Cloud Server using s3fs-fuse
If you want to know more about storage formats please follow this link
More information about reading and writing to Cloud Storage in this link
I'm trying to get the file that was uploaded to Google Cloud Storage, do some work with its content, and move it to a different bucket using Google Cloud Functions with python3.7. Following their documentation I was only able to get file name. I tried using import cloudstorage but it errors module 'cloudstorage' has no attribute 'NotFoundError', and googling did not get me anywhere.
Does any one have a sample code that could do what I need?
The cloudstorage library is specific to the Standard environment of App Engine.
A library compatible with Cloud Storage would be google-cloud-storage. You must declare it in your requirements.txt file for your function.
This example on how to copy from one bucket to another should suffice. After copying it, you can just do source_blob.delete() to get rid of it.
I am trying to grasp how to store video files. I know I can store .mp4's on Google Cloud Store. However, I have had a hard time interfacing my application to stream these video files.
I have found video URLs like:
http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4
Versus what the file on the Cloud Store is, which probably somehow refers to the mp4 I uploaded (right?)
https://firebasestorage.googleapis.com/v0/b/packfeed-e027b.appspot.com/o/Stories%2F0%2FM41WiOceiQTs3ELETIT5evcfsJm1_1520646187885.mp4?alt=media&token=201a831b-c239-4563-8178-cec3c4567212
Is there a difference between these two URLs, one points directly to the mp4, and then the other URL which is a "downloadlink"? is there a difference?
Are there any options to store files in the Google Cloud Platform like this?
Your first link points to the file stored on clips.vorwaerts-gmbh.de server. The second link points to the file stored on Google Cloud Storage server.
You can upload your files to Google Cloud Storage, then share the file publicly by checking the box "Share publicly" on the file. The "Public link" appeared will be the link available to public, similar to the second link you posted.
https://cloud.google.com/storage/docs/access-control/making-data-public#objects
Just getting started with Wowza Streaming Engine.
Objective:
Set up a streaming server which live streams existing video (from S3) at a pre-defined schedule (think of a tv channel that linearly streams - you're unable to seek through).
Create a separate admin app that manages that schedule and updates the streaming app accordingly.
Accomplish this with as a little custom Java as possible.
Questions:
Is it possible to fetch / update streamingschedule.smil with the Wowza Streaming Engine REST API?
There are methods to retrieve and update specific SMIL files via the REST API, but they only seem to be applicable to those created through the manager. After all, streamingschedule.smil needs to be created manually by hand
Alternatively, is it possible to reference a streamingschedule.smil that exists on an S3 bucket? (In a similar way footage can be linked from S3 buckets with the use of the MediaCache module)
A comment here (search for '3a') seems to indicate it's possible, but there's a lot of noise in that thread.
What I've done:
Set up Wowza Streaming Engine 4.4.1 on EC2
Enabled REST API documentation
Created a separate S3 bucket and filled it with pre-recorded footage
Enabled MediaCache on the server which points to the above S3 bucket
Created a customised VOD edge application, with AppType set to Live and StreamType set to live in order to be able to point to the above (as suggested here)
Created a StreamPublisher module with a streamingschedule.smil file
The above all works and I have a working schedule with linearly streaming content pulled from an S3 bucket. Just need to be able to easily manipulate that schedule without having to manually edit the file via SSH.
So close! TIA
To answer your questions:
No. However, you can update it by creating an http provider and having it handle the modifications to that schedule. Should you want more flexibility here you can even extend the scheduler module to not require that file at all.
Yes. You would have to modify the ServerListenerStreamPublisher solution to accomplish it. Currently it solely looks a the local filesystem to read teh streamingschedule.smil file.
Thanks,
Matt