I am trying to pull data from the google playstore console through the cli.
The documentation that I am using as a reference is this.
I have retrieved my cloud storage URI from the playstore console and used the command:
gsutil cp -r gs://pubsite_prod_xxxxxx/stats/installs/installs_com.my.package_2021* .
I get an error message like this:
zsh: no matches found: gs://pubsite_prod_xxxxx/stats/installs/installs_com.my.package_2021*
I am trying to understand what the issue could be. Any pointers would be of great help.
Other details:
Though my playstore account is two or three years old, I just created my google cloud platform account. (So, could the issue be that there are no reports written onto my bucket, though there is a URI mentioned on the play console?)
I am unable to see this bucket through the google cloud platform console. The list of buckets are empty. (I am however not sure whether I am looking at the right place). Attaching a screenshot of the project on the console that I am looking into.
screenshot - empty list of buckets
This is an issue related to ZSH You may could change your shell, via sudo chsh --shell=/bin/bash $USER will change the shell field in /etc/passwd
Or you may use ' around a special character to escape the special character in order to let the shell recognize it. like:
gsutil cp -r 'gs://pubsite_prod_xxxxxx/stats/installs/installs_com.my.package_2021*' .
Related
I'm trying to Google's open source Android emulator for Mac. They're very picky about which versions of things (e.g., Xcode, python) are installed. One specific requirement is MacOS10.13sdk, which the Mac instructions say should be retrieved with
gsutil cp gs://emu-dev-development/MacOSX10.13.sdk.tar.gz .
but that returns
AccessDeniedException: 403 <myaccount> does not have storage.objects.list access to the Google Cloud Storage bucket.
I'm more of an AWS user, and although I've used Google Cloud from the console for some limited things, this is my first time using the CLI, so I'm sure I'm missing something basic.
I installed gsutil per the instructions. I installed their recommended Python3.7:
% which python3
/usr/local/opt/python#3.7/bin/python3
I've run gcloud init twice. Once I selected an existing project from the list, and once I selected "Create a new project". Authentication has clearly worked, because said new project is now visible from the web console.
What am I missing to download from what I'm guessing must be a public bucket?
It is weird to see this as the result, please tell me what should I do now.
I cannot find anything helpful through Google, please help me, thanks
When I execute the command:
gcloud ml speech recognize 'gs://cloud-smaples-test-01/speech/test.wav' --language-code='en-US'
on my computer, the only response that I can see is this:
ERROR: (gcloud.ml.speech.recognize) Invalid audio source ['gs://cloud-smaples-test-01/speech/test.wav']. The source must either be a local path or a Google Cloud Storage URL (such as gs://bucket/object).
smaples is correct, I do change the order to avoid the same name.
However, when I execute the same command on Google Cloud Shell, I can see the result of speech to text. I do not know what happened exactly.
I use the same Google account to execute command whatever on my computer or Google Cloud Shell. I also set the file even the whole storage can be read by anyone.
What could cause this problem?
result on my computer
result on google cloud shell
You seem to be running Windows in your computer. The problem is that Windows interprets quotation marks as part of the string.
Removing the quotes in both your bucket path and the language code tag will resolve the issue.
Example:
gcloud ml speech recognize gs://cloud-smaples-test-01/speech/test.wav --language-code=en-US
This question seems to have been asked before, but I am doing it a different way and so I am posting this since I can't find a specific question that addressed this. Is there a way to get a list of files (actually just the latest file) in a public Amazon bucket without the use of special tools or Amazon CLI, etc? I am not experienced with this whatsoever and after 3 hours I am ready to pull my hair out. I have a simple project where I need the latest file in a bucket but I do not know the filename. (Weather radar data). This is the only requirement - and I do have a small amount of programming knowledge but it seems I could not figure out the python/Amazon tools so I am just trying to find an easier route as I am rapidly running out of time. I am using Windows and command-line tools. On a similar host, I was pulling the data from a server by using wget and parsing the index.html file using a simple C++ console app I wrote to get the filename, then launching wget to download the file since the filename was easily found in the index.
With Amazon I can't seem to figure this out.
This is the main listing : https://s3.amazonaws.com/noaa-nexrad-level2/index.html
Since the filenames are listed by date, then radar site - I can figure most of the URL out. The example filename would be:
https://noaa-nexrad-level2.s3.amazonaws.com/2018/08/07/KEWX/KEWX20180807_094339_V06
For a more precise example - I need the latest file for https://noaa-nexrad-level2.s3.amazonaws.com/2018/08/07/KEWX.
If I could get the XML returned of a directory, I could do it by using the method I did before... but I can't seem to figure that out.
I would be OK with writing a simple program to do this, even if I had to learn some python (it seems to be the most popular way), but I don't understand what I am doing regarding AWS authentication, buckets, etc. and have wasted way too much time on this to start over yet again unless I get some serious help. I hope to find some assistance. I am not trying to be lazy, I am just running out of time and ideas.
You will need some form of programmatic handling of the results, either in PowerShell or the AWS Command-Line Interface (CLI).
This seems to work:
aws s3api list-objects --bucket noaa-nexrad-level2 --prefix 2018/08/07/KEWX/ --query 'sort_by(Contents, &LastModified)[-1].Key' --output text
2018/08/07/KEWX/KEWX20180807_234734_V06
It's basically saying: Sort by LastModified, return the last record, only show the Key (filename)
(This worked on a Mac. You might need some fiddling with the quotes on Windows.)
Some good videos about the AWS CLI:
AWS re:Invent 2017: AWS CLI: 2017 and Beyond (DEV307) - YouTube
AWS re:Invent 2016: The Effective AWS CLI User (DEV402) - YouTube
The JMESPath Tutorial is also very useful for understanding how to use --query parameters.
I'm having trouble when uploading folders onto my google-cloud-storage bucket: They wont show up in the mounted drive /mnt/willferbucket (with gcsfuse)
It doesnt matter if i upload them through the webfrontend or by using gsutil, the only folders that show up, are the ones i create (not upload!, doesnt matter via webfrontend or directly on the mounted drive, in the example below: "canvas" and "prints", so this is working)
So.. "ls" on the mounted drive looks like this:
root#ubuntu-2:/mnt/willferbucket# ls
canvas helloWorld.py helloWorldSimple.py prints test.txt
But as you can see, when using gsutil:
There are my uploaded folders showing up and i'm able to download (same in the webfrontend: they show up):
root#ubuntu-2:/mnt/w# gsutil ls gs://willferbucket
gs://willferbucket/helloWorld.py
gs://willferbucket/helloWorldSimple.py
gs://willferbucket/test.txt
gs://willferbucket/canvas/
gs://willferbucket/prints/
gs://willferbucket/test/
gs://willferbucket/testfolder/
gs://willferbucket/tst/
I couldn't find out what the reason for this behaviour is :(
Maybe someone can help or is facing the same problem
Thanks for your reply
I have done the same test as you, using gcsfuse to mount one of my Cloud Storage buckets into my local system, and it also appears incomplete to me.
I have notified this situation to the Google Cloud Storage engineering team on your behalf and the issue is currently being handled by them. You can keep track on any progress they have made by following this link.
Please click on the star button next to the issue number to get email notifications on how the situation is being handled.
UPDATE
The Google Cloud Storage engineering team has come back to me and pointed out that the reason not all files and directories are listed when using gcsfuse is because of what would be called implicit directories, which are not set by just using gcsfuse (like an mdkir operation inside the mounted bucket), but through other means such as the Cloud Storage console. These directories may not be recognized by gcsfuse and therefore not added to the mounted bucket in your file system.
There is a way to solve this. Whenever you run the gcsfuse command to mount a Cloud Storage bucket into your local file system, add the --implicit-dirs flag so that all implicit directories are included. This would be an example for it:
gcsfuse --implicit-flags [YOUR_BUCKET] /path/to/your/local/dir/
Linux n00b here having trouble pulling a file from the server to my local Windows 7 professional 64 bit machine. I am using Wowza to stream live video and I am recording these live videos to my Google Cloud instance located here:
/usr/local/WowzaStreamingEngine/content/myStream.mp4
When I ssh:
gcutil --project=”myprojectname” pull “my instance”
“/usr/local/WowzaStreamingEngine/content/myStream.mp4” “/folder1”
I receive a permission denied error. When I try saving another folder deep on my local machine i.e "/folder1/folder2" the error returned is file or directory not found. I've checked that I have write permisions set on my local Windows 7 machine so I do not think it is a permissions error. Again, apologize for the n00b question, I'm just been stuck here for hours.
Thx,
~Greg
Comment added 7/18:
I enter the following through ssh:
gcutil --project=”Myproject” pull “instance-1” "/usr/local/WowzaStreamingEngine/content/myStream.mp4” “/content"
By entering this I'm expecting the file mystream.mp4 to be copied to my C:/content folder. The following is returned: Warning: Permanently added '107.178.218.8' (ECDSA) to the list of known hosts. Enter passphrase for key '/home/Greg/.ssh/google_compute_engine':
Here I enter the passphrase and the following error is returned: /content: Permission denied Have write set up on this folder. Thanks! – Greg
-=-=-==->
To answer the question about using Cygwin, I'm not familiar with Cygwin and I do not believe it was used in this instance. I ran these commands through the Google Cloud SDK shell which I installed per the directions found here: https://developers.google.com/compute/docs/gcutil/.
What I am doing:
After setting up my google cloud instance I open Google CLoud SDK and enter the following:
gcutil --service_version="v1" --project="myproject" ssh --zone="us-central1-a" "instance-1"
I then am prompted for a passphrase which I create and then run the following:
curl http://metadata/computeMetadata/v1/instance/id -H "X-Google-Metadata-Request:True"
This provides the password I use to login to the Wowza live video streaming engine. All of this works beautifully, I can stream video and record the video to the following location: /usr/local/WowzaStreamingEngine/content/myStream.mp4
Next I attempt to save the .mp4 file to my local drive and that is where I'm having issues. I attempt to run:
gcutil --project=”myproject” pull “instance-1” “/usr/local/WowzaStreamingEngine/content/myStream.mp4” “C:/content”
also tried, C:/content C:\content and C:\content
These attempts threw the following error:
Could not resolve hostname C: Name or service not known
Thanks again for your time, I know it is valuable, I really appreciate you helping out a novice.
Update I believe I am close thanks to your help. Switched to local C drive, entered the command as you displayed in your Answer update. Now returning a new, not before seen error:
Error: API rate limit exceeded
I did some research on S.O. and some suggestions made were that billing is not enabled or the relevant API is not enabled and I could solve by turning on Google Compute Engine. Billing has been enabled for a few weeks now on my project. In terms of Google Compute Engine, below are what I believe to be the relevant items turned on:
User Info: Enabled
Compute: Read Write
Storage: Full
Task Queue: Enabled
BigQuery: Enabled
Cloud SQL: Enabled
Cloud Database: Enabled
The test video I recorded was short and small in size. I also have not done anything else with this instance so at a loss as to why I am getting the API rate exceeded error.
I also went to the Google APIs console. I see very limited usage reported so, again, not sure why I am exceeding the API limit. Perhaps I do not have something set appropriately in the APIs console?
I'm guessing you're using Cygwin here (please correct me if I'm wrong).
The root directory for your Cygwin installation is most likely C:\cygwin (see FAQ) and not C: so when you say /content on the command line, you're referring to C:\cygwin\content and not C:\content.
Secondly, since you're likely running as a regular user (and not root) you cannot write to /content so that's why you're getting the permission denied error.
Solution: specify the target directory as C:/content (or C:\\content) rather than /content.
Update: from the update to the question, you're using the Google Cloud SDK shell, not Cygwin, so the above answer does not apply. The reason you're seeing the error:
Could not resolve hostname C: Name or service not known
is because gcutil (like ssh) parses destinations which include : as having the pattern [hostname]:[path]. Thus, you should avoid : in the destination, which means we need to drop the drive spec.
In this case, the following should suffice, assuming that you're currently at a prompt that looks like C:\...>:
gcutil --project=myproject pull instance-1 /usr/local/WowzaStreamingEngine/content/myStream.mp4 \content
If not, first switch to the C: drive by issuing the command:
C:
and then run the above command.
Note: I removed the quotes from the command line because you don't need it in the case where parameters don't have spaces in them.