I am receiving csv files from different users (from the same organisation) over Microsoft Teams. I have to download each file and import them into a bucket on Google Cloud Storage.
What would be the most efficient way to directly store those files directly into Google Cloud Storage everytime I am receiving a file from a given user over Teams? Files must be imported using Microsoft Teams.
I was thinking to trigger from Pub/Sub using Cloud Run but I am a bit confused how to connect this with teams.
I imagine you should be able to do this fine using Power Automate, but it might depend on how you're receiving the files (for instance are users sending them 1-1 to you directly, or uploading them into a Files tab in a specific Team/Channel).
Here's an example template for moving files from OneDrive for Business to Google Drive, that sounds like it should help: https://flow.microsoft.com/en-us/galleries/public/templates/02057296acac46e9923e8a842ab9911d/sync-onedrive-for-business-files-to-google-drive-files/
Related
My teams heavily uses spacy, bert, and other NLP tools based on models. Where should I store these models (en_core_web_lg and such), so that:
It is only stored once (pricing reasons)
Multiple Notebook projects can access it
I have tried uploading it in a Cloud Storage bucket because pandas could open files directly from the bucket, but this is not the case for spacy.
I would like to avoid solutions like having the notebooks downloading locally models from the bucket every time it is run.
I want to copy the data from my User_Log kind from my Test GCP project to my Live project. I have exported the User_Log kind from the Datastore to the Google Cloud Storage bucket for the Test project. But when I go to import it using the GCP GUI into the Live project I can see the Test project buckets - even though I have given Storage Admin access to testProject#appspot.gserviceaccount.com in my Live project and vice versa Storage Admin access to LiveProject#appspot.gserviceaccount.com in the Test project.
From what I have read it should be possible to transfer files from one project's bucket to another.
Thanks
TimN
It looks like you can't import/export from one project to another using the GCP Console GUI, but you can if you use gcloud using the commands in the post: Export GCP Datastore and import to a different GCP Project
You are correct, the Cloud Console UI only allows you to select the buckets that exist in you current project. However, if the overall_export_metadata file is located in another project, you'll have to use other methods like gcloud tool or REST for the import - link
I am setting up a relationship where two Google App Engine applications (A and B) need to share data. B needs to read data from A, but A is not directly accessible to B. Both A and B currently use Google Datastore (NOT persistent disk).
I have an idea where I take a snapshot of A's state and upload it to a separate Google Cloud Storage location. This location can be read by B.
Is it possible to take a snapshot of A using Google App Engine and upload this snapshot (perhaps in JSON) to a separate Google Cloud Storage location to be read from by B? If so, how?
What you're looking for is the Datastore managed export/import service:
This page describes how to export and import Cloud Firestore in
Datastore mode entities using the managed export and import service.
The managed export and import service is available through the gcloud
command-line tool and the Datastore mode Admin API (REST,
RPC).
You can see a couple of examples described in a bit more details in these more or less related posts:
Google AppEngine Getting 403 forbidden trying to update cron.yaml
Transferring data from product datastore to local development environment datastore in Google App Engine (Python)
You may need to take extra precautions:
if you need data consistency (exports are not atomic)
to handle potential conflicts in entity key IDs, especially if using manually-generated ones or referencing them in other entities
If A is not directly accessible to B isn't actually something intentional and you'd be OK with allowing B to access A then that's also possible. The datastore can be accessed from anywhere, even from outside Google Cloud (see How do I use Google datastore for my web app which is NOT hosted in google app engine?). It might be a bit tricky to set it up, but once that's done it's IMHO a smoother sharing approach than the export/import one.
I want to view files Such as excel or zip or any other files in the browser without getting downloaded.
I am able to display image and pdf files in the browser but unable to view any other format's such as zip or xls.
I am storing my files in S3.
What should i do?
Web browsers are not able to natively display most file types. They can render HTML and can display certain types of images (eg JPG, PNG), but only after these files are actually downloaded to your computer.
The same goes for PDFs -- they are downloaded, then a browser plug-in renders the content.
When viewing file (eg Excel spreadsheets and PDF files) within services like Gmail and Google Drive, the files are typically converted into images on the server-end and those images are sent to your computer. Amazon S3 is purely a storage service and does not offer a conversion service like this.
Zip files are a method of compressing files and also storing multiple files within a single archive file. Some web services might offer the ability to list files within a Zip, but again Amazon S3 is purely a storage service and does not offer this capability.
To answer your "What should I do?" question, some options are:
Download the files to your computer to view them, or
Use a storage service that offers these capabilities (many of which store the actual files in Amazon S3, but add additional services to convert the files for viewing online)
I might be a bit too late but did you try Filestash? (I made it)
That's what it looks like when you open a xls document on S3:
I aim to support all the common formats and the already supported list is rather big already
So far to manage getting data from the bucket I use download_to_file() to get it downloaded on the instance that it is using and access the files/folders locally. Though what I want to achieve is being able to just read from the cloud. How can I go about doing that? There doesn't seem to be a way for me create a relative path from the ML Job instance and the google cloud bucket.
You can use TensorFlow's file_io.FileIO class to create file_like objects to read/write files on gcs, local, or any other supported file system.
See this post for some examples.