I started using Google Colab through the GCE VM, but I have not been able to initialize or upload any notebooks.
The error I am getting is The user has exceeded their Drive storage quota, but upon looking into my Storage, I have 15 GB of available storage, 0 bytes of 15 GB used.
Trying to fix it, I log in as the admin account and go to Storage, and then what shows is that 0 bytes of shared 0 bytes (100%) and it is in red. So I believe the is something wrong happening with the storage allocation or the store quota?
The virtual machine was ran with 200 GB space, so I cannot see the issue.
Full error when trying to upload a notebook:
The user has exceeded their Drive storage quota
GapiError: The user has exceeded their Drive storage quota
at YI.YA [as constructor] (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:1433:2101)
at YI.iH [as constructor] (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:2270:222)
at new YI (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:2334:151)
at JGa (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:2396:348)
at xa.program_ (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:2410:131)
at za (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:21:57)
at xa.throw_ (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:20:201)
at Aa.throw (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:22:89)
at c (https://ssl.gstatic.com/colaboratory-static/common/456ec723e499489081e770703ac07ab8/external_polymer_binary.js:22:343)
I have not found a way to troubleshoot it or resolve it. There is nothing broken shown in the log, it says that its working as intended?
Related
i try in GCE to create a copy of a running system, i did a snapshot from the running system and set up a new instance. On starting it showed to be running but the log from console one says:
SeaBIOS (version 1.8.2-google)
Total RAM Size = 0x0000000100000000 = 4096 MiB
CPUs found: 2 Max CPUs supported: 2
found virtio-scsi at 0:3
virtio-scsi vendor='Google' product='PersistentDisk' rev='1' type=0 removable=0
virtio-scsi blksize=512 sectors=20971520 = 10240 MiB
drive 0x000f22f0: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=20971520
Sending Seabios boot VM event.
Booting from Hard Disk 0...
So the system is not reachable. I Tryed to check the UUID of the drive but it seems to be the right one. Can someone tell me how to fix this.
Best regards
Alex
What you're experiencing looks very much like a bug. You may try to stop the VM, take a snapshot and try creating new one.
My recommendation would be to contact GCP support to get more immediate help (however it's a paid service) or open up a new issue at Google IssueTracker to get help for free but there are not ETA's for this.
I just implemented cloud run to process/encode video for my mobile application. I have recently gotten an unknown 503 error: POST 503 Google-Cloud-Tasks: The request failed because the HTTP connection to the instance had an error.
My process starts when a user uploads a video to cloud storage, then a function is triggered and sends the video source path to cloud tasks to be enqueued for encoding. Finally cloud run downloads the video, processes it via ffmpeg, and uploads everything to a separate bucket (all downloaded temp files are deleted).
I know video encoding is a cpu heavy task, but my application only allows up to ~3 minute videos to be encoded (usually around 100 MB). It works perfectly fine for shorter videos, but ones on the longer end flag the 503 error after processing for 2+ minutes
My instances are only used for video encoding and only allow 1 concurrent request/instance. Here are my services settings:
CPU - 2 cpu
Memory - 2 Gb
Concurrency - 1
Request Timeout - 900 seconds (15 minutes)
The documentation states that it is because of heavy cpu tasks so it's clear it is caused by the processing of heavier files, but I'm unsure what I can do to fix this given the max settings. Is it possible to set a cap on the CPU so it doesn't go overboard? Or is cloud run not a good solution for this kind of task?
I deployed a sample HelloWorld app on Google Cloud Run, which is basically k-native, and every call to the API takes 1.4 seconds at best, in an end-to-end manner. Is it supposed to be so?
The sample app is at https://cloud.google.com/run/docs/quickstarts/build-and-deploy
I deployed the very same app on my localhost as a docker container and it takes about 22ms, end-to-end.
The same app on my GKE cluster takes about 150 ms, end-to-end.
import os
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
target = os.environ.get('TARGET', 'World')
return 'Hello {}!\n'.format(target)
if __name__ == "__main__":
app.run(debug=True,host='0.0.0.0',port=int(os.environ.get('PORT', 8080)))
I am a little experience in FaaS and I expect API calls would get faster as I invoked them in a row. (as in cold start vs. warm start)
But no matter how many times I execute the command it doesn't go below 1.4 seconds.
I think the network distance isn't the dominant factor here. The round-trip time via ping to the API endpoint is only 50ms away, more or less
So my questions are as follows:
Is it potentially an unintended bug? Is it a technical difficulty which will be resolved eventually? Or maybe nothing's wrong, it's just the SLA of k-native?
If nothing's wrong with Google Cloud Run and/or k-native, what is the dominant time-consuming factor here for my API call? I'd love to learn the mechanism.
Additional Details:
Where I am located at: Seoul/Asia
The region for my Cloud Run app: us-central1
type of Internet connection I am testing under: Business, Wired
app's container image size: 343.3MB
the bucket location that Container Registry is using: gcr.io
WebPageTest from Seoul/Asia (warmup time):
Content Type: text/html
Request Start: 0.44 s
DNS Lookup: 249 ms
Initial Connection: 59 ms
SSL Negotiation: 106 ms
Time to First Byte: 961 ms
Content Download: 2 ms
WebPageTest from Chicago/US (warmup time):
Content Type: text/html
Request Start: 0.171 s
DNS Lookup: 41 ms
Initial Connection: 29 ms
SSL Negotiation: 57 ms
Time to First Byte: 61 ms
Content Download: 3 ms
ANSWER by Steren, the Cloud Run product manager
We have detected high latency when calling Cloud Run services from
some particular regions in the world. Sadly, Seoul seems to be one of
them.
[Update: This person has a networking problem in his area. I tested his endpoint from Seattle with no problems. Details in the comments below.]
I have worked with Cloud Run constantly for the past several months. I have deployed several production applications and dozens of test services. I am in Seattle, Cloud Run is in us-central1. I have never noticed a delay. Actually, I am impressed with how fast a container starts up.
For one of my services, I am seeing cold start time to first byte of 485ms. Next invocation 266ms, 360ms. My container is checking SSL certificates (2) on the Internet. The response time is very good.
For another service which is a PHP website, time to first byte on cold start is 312ms, then 94ms, 112ms.
What could be factors that are different for you?
How large is your container image? Check Container Registry for the size. My containers are under 100 MB. The larger the container the longer the cold start time.
Where is the bucket located that Container Registry is using? You want the bucket to be in us-central1 or at least US. This will change soon with when new Cloud Run regions are announced.
What type of Internet connection are you testing under? Home based or Business. Wireless or Ethernet connection? Where in the world are you testing from? Launch a temporary Compute Engine instance, repeat your tests to Cloud Run and compare. This will remove your ISP from the equation.
Increase the memory allocated to the container. Does this affect performance? Python/Flask does not require much memory, my containers are typically 128 MB and 256 MB. Container images are loaded into memory, so if you have a bloated container, you might now have enough memory left reducing performance.
What does Stackdriver logs show you? You can see container starts, requests, and container terminations.
(Cloud Run product manager here)
We have detected high latency when calling Cloud Run services from some particular regions in the world. Sadly, Seoul seems to be one of them.
We will explicitly capture this as a Known issue and we are working on fixing this before General Availability. Feel free to open a new issue in our public issue tracker
we experienced problems with our RDS instance.
RDS stops running. RDS are in state of "green"(on the AWS console) but we cannot connect to the RDS instance.
Cloud Logs we found following errors:
2018-03-07 8:52:31 47886953160896 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
InnoDB: Set innodb_force_recovery to ignore this error.
2018-03-07 8:52:32 47886953160896 [ERROR] Plugin 'InnoDB' init function returned error.
2018-03-07 8:53:46 47508779897024 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
InnoDB: Set innodb_force_recovery to ignore this error.
2018-03-07 8:53:46 47508779897024 [ERROR] Plugin 'InnoDB' init function returned error.
When we tried to reboot RDS instance its take almost 2 hours to reboot. After rebooting its working fine again!.
Can someone help us to know the root cause of this incident.
As t2.small provides 2G of RAM. However you might be knowing, most DB engines tend to use up 75% of the memory for caching purposes such as queries, temporary tables, table scans to make things go faster.
For our Maria DB engine, following parameters are by default set to below pre-optimized values :
innodb_buffer_pool_size (DB instance size *3/4= 1.5 Gb)
key_buffer_size (16777216 = 16.7 Mb)
innodb_log_buffer_size (8388608 =8.3Mb)
Apart from that the OS and the RDS Processes will also use some amount of RAM to do their own operations. Hence to summarize, around 1.6 Gb approximately is utilized by DB engine and the actual usable memory which will be getting after taking out these values innodb_buffer_pool_size, key_buffer_size, innodb_log_buffer_size will be around 400 MB.
Overall a decrease in your Freeable Memory as low as ~137MB. As a result, Swap Usage increased drastically in the same time period to 152MB approximately.
FreeableMemory was quite low and there was a high swap utilization. Further, due to the memory pressure ( insufficient memory and high swap usage), RDS internal monitoring system was not able to proceed with host communication which in turn resulted into underlying host replacement.
I'm trying to upload a video from a Cordova app to an Amazon AWS S3 bucket from an Android/iPhone. But it's failing sometimes, giving sporadic reports of this error from AWS bucket:
http_status:400,
<Code>EntityTooLarge</Code>
Some of the files are tiny, some around 300mb or so.
What can I do to resolve this at the AWS end?
The 400 Bad Request error is sometimes used by S3 to indicate conditions that make the request in some sense invalid -- not just syntactically invalid, which is the traditional sense of 400 errors.
EntityTooLarge
Your proposed upload exceeds the maximum allowed object size.
400 Bad Request
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
Note the word "proposed." This appears to be a reaction to the Content-Length request header you are sending. You may want to examine that. Perhaps the header is inconsistent with the actual size of the file, or the file is being detected as larger than it actually is.
Note that while the maximum object size in S3 is 5 TiB, the maximum upload size is 5 GiB. (Objects larger than 5 GiB have to be uploaded in multiple parts.)
413 errors occur when the request body is larger than the server is configured to allow. I believe its not the error which AWS S3 is throwing because they support 5 TB size of object.
If you are first accepting this video in your app and from there you are making request to amazon S3, then your server is not configure to accept the large entities in request.
Refer -set-entity-size for different servers. if your server is not listed here, then you need to figure out how to increase entity size for your server.