I have an App that consumes my own API (Google Cloud Functions) and my own Storage (there are images).
Now, I have a couple of clients, that wants to consume my API and my Storage (A Google Cloud Bucket).
The Cloud Storage is a bucket that contains a lot of photo that have Public Read Access.
I'm trying to define a tier pricing model, in which the price depends on 2 things:
The number of API calls,
The Cloud Storage Bandwidth
Meaning, I want to set some pricing in relation to the costs they are consuming on my Google Cloud account.
To give an example:
If a client does between 1 and 500.000 API calls, I'll change them 10 dollars. Between 500.001 and 1.000.000, I'll charge 18 dollars, etc, etc.
Same thing for the Cloud Storage Bandwidth, if they consume between 0GM and 10GB, it's going to cost 10 dollars. If they consume between 10GB and 100GB, it's going to cost 18 dollars, etc, etc.
How can I do it with Google Cloud? How can I know how my clients are consuming? And is there a way to share that information with them, so they are able to monitor the usage every day?
I'm thinking that measuring the API usage is not going to be THAT hard, because I can just save a value in the DB every time the user calls the API, but if there is a way to avoid it, will be good, due to Google Cloud is going to charge me for that DB write action (that I use to track the API usage).
On the other hand, for measuring the Cloud Storage, I was thinking something like this:
Let's suppose I have a Public Bucket with photos in the URL: buckets.google.com/photos.
If my client wants to get the /cats/ugly-cat.jpg photo, I can ask them to call A FUNCTION in /api/get-photo/?url=/cats/ugly-cat.jpg, so there in that Function a can track that the user just get a photo, and then I redirect the call to the real URL where the user is going to see the photo (buckets.google.com/photos/cats/ugly-cat.jpg). As you can see, this idea seems to be too ppor performant, due to it's going to charge the Function usage, the DB write, and also the Storage bandwidth usage. And even, that way doesn't track the Bandwidth. It only tracks the number of photos that the client wants to show.
As you can see, both ideas are a bit ugly, with poor performance.
There should be something already done that makes it beautiful.
Obviously, the API call (and also the photo link) may have the client API-KEY, to help to measure the usage. Something like:
functions.google.com/api/search-photos/?api-key=111, and
bucket.google.com/photos/cats/ugly-cat.jpg?api-key=111
Where 111 identifies the client 111.
So, the question: Do you know if there is a "best-known" way to do measure those usages?
I think Cloud Endpoints is the best solution for you because managing your API as you suggest might get unwieldy quickly.
Endpoints provides all the tools to control authentication, quota and cost management and a developer portal so your users can access documentation and interact with your API. It also integrates with all Cloud Platform products including Cloud Functions.
Related
I have made a Business Card ordering portal and its backend is in NodeJS.
I am currently using t2.micro and I am getting like 50 daily users and 15-20 concurrent users but in some time the user count would go up to 300 daily users and 100 concurrent users. I don't want to spend much either.
It has single database and we don't use threads.
I am confused whether I should change my instance type or should use Auto Scaling Groups.
I am not a pro in AWS. Please help!!
Nobody can give you an answer to your question because every application is different.
Some applications need more CPU (eg for encoding/encrypting). Some need lots of RAM (for calculations). Some need lots of disk access (eg for file manipulation).
Only you know how your application behaves and what resources it would need.
You could either pick something and then monitor it (CPU, RAM, Disk) in production to see where the 'bottleneck' lies, or you could create a test that simulates the load of users and pushes it to breaking point to discover the bottleneck.
We are planning to use extensively Google APIs to transcribe audio in text from a single computer (about 10.000 requests approximately at the same time).
Except quota issues, are you aware of any other limitations on APIs calls coming from the same address ?
Thanks a lot !
If you look at the following article, you can see that there are some restrictions and limits when it comes to the speech-to-text api.
Other than the ones mentioned in the article, there are no restrictions or limits. When making a call to an API, it depends on the amount of requests, not where it comes from. The API does not care if the requests come from one or multiples addresses. As long as the requests do not go over the Quota that the API can handle, it will be okay.
I've been trying for a while to figure out if Google Cloud has a mechanism for a "crowbar" limit on API usage, as a security measure.
The scenario I'm concerned about is, say I have an API keyfile on a server, and a hacker steals it and plugs it into their system that (for the purposes of some nebulous scam) is running an unbounded number of translate requests as quickly as possible. I then receive a $25,000 bill. I don't see any mechanisms to really prevent this.
I can limit the roles for that key, but if the hacker's interest is in the same API I use for my project, that doesn't help. I can set a billing limit, but that just sends me an email, and I am not a person who notices incoming email, so in practice it would take me days to realize this had happened.
I can set a quota, but all the quotas in the dashboard seem to be per-minute or per-day limits. I need a per-month quota, and I don't see any way to do it. Am I just missing something, or does Google simply provide no option for this?
I understand that "well just don't let your API key get compromised" is probably the common wisdom, but that's unacceptable to my use cases for a number of reasons, so I'm not interested in discussing it. Thanks!
Edit: I should note, Google's documentation says "you can set requests per day caps" - but there are no instructions on that page for how to do this, and I can see no mechanism for it. I don't want a per-day cap, I want a per-month cap, but I'd take a per-day cap if I could find one. These are the only quotas I see for Cloud Vision, for instance:
Quotas part 1
Quotas part 2
As per Link 1 there is no Hard quota limit for Vision API on a monthly basis . If you need to use this feature you can request for this feature using the link 2.
In the meantime or as workaround you can control your vision API budget by using the Cloud Billing Budget API by following the link 3.
The company which I work right now planning to use AWS to host a new website for a client. Their old website had roughly 75,000 sessions and 250,000 page views per year. We haven't used AWS before and I need to give a rough cost estimate to my project manager.
This new website is going to be mostly content-driven with a cms backend (probably WordPress) + a cost calculator for their services. Can anyone give me a rough idea about the cost to host such kind of a website in aws?
I have used simple monthly calculator with a single Linux t2.small 3 Year upfront which gave me around 470$.
(forgive my English)
The only way to know the cost is to know the actual services you will consume (Amazon EC2, Amazon EBS, database, etc). It is not possible to give an accurate "guess" of these requirements because it really does depend upon the application and usage patterns.
It is normally recommended that you implement the system and run it for a while before committing to Reserved Instances so that you have a chance to measure performance and test a few different instance types.
Be careful using T2 instances for production workloads. They are very powerful instances, but if the CPU Credits run out, the amount of CPU is limited.
Bottom line: Implement, measure, test. Then you'll know what is right for your needs.
Take Note
When you are new in AWS you have a 1 year free tier on a single t2.micro
Just pulled it out, looking into your requirement you may not need this
One load balancer and App server should be fine (Just use route53 to serve some static pages from s3 while upgrading or scalling )
Use of email subscription and processing of Some document can be handled with AWS Lambda, SNS and SWQ which may further reduce the cost ( you may reduce the server size and do all the hevay lifting from Lambda)
A simple webpage with 3000 request/monthly can be handled by T2 micro which is almost free for one year as mentioned above in the note
You don't have a lot of details in your question. AWS has a wide variety of services that you could be using in that scenario. To accurately estimate costs, you should gather these details:
What will the AWS storage be used for? A database, applications, file storage?
How big will the objects be? Each type of storage has different limits on individual file size, estimate your largest object size.
How long will you store these objects? This will help you determine static, persistent or container storage.
What is the total size of the storage you need? Again, different products have different limits.
How often do you need to do backup snapshots? Where will you store them?
Every cloud vendor has a detailed calculator to help you determine costs. However, to use them effectively you need to have all of these questions answered and you need to understand what each product is used for. If you would like to get a quick estimate of costs, you can use this calculator by NetApp.
I have a web app which serves media files (in other words pretty large) with public access. The files are hosted on S3. I'm wondering if AWS offers any kind of abuse-protection, for example detection or prevention against download hogs via some type of rate limiting. A scenario might be a single source re-downloading the same content repeatedly. I was hoping there might be some mechanism to detect that behavior and either take preventative action or notify me.
I'm looking at AWS docs and don't see anything but perhaps I'm not looking smartly enough.
How do folks who host files which are available publicly handle this?
S3 is mostly a file storage service, with elementary web server capabilities. I would highly recommend you place a CDN between your end users and S3. A good CDN will provide protection from the sort of abuse you are talking about, while also serving the files to the user more quickly.
If you are mostly worried about how the abuse will affect your bills (and they can get very large so its good to be concerned about this), I would suggest that you put in some billing alerts on your account that alarm when certain thresholds are reached.
I have a step alarms set on my account so that I know when it hits 25%, 50%, 75% and 100% of what I budget each month. That way, for example, if I hit an alarm that tells me I have used 25% of my budget in the first two days of the month, I know I better look into it.