how to find GCP services interdependencies - google-cloud-platform

Before I can use a particular service, I have to enable the API in my Google project. Sometimes when I do this, more services are enabled than the one I specified. for example, enabling cloudfunctions service actually enabling multiple APIs in the background like pubsub, storage etc.
I followed this article to find the service dependencies https://binx.io/blog/2020/10/03/how-to-find-google-cloud-platform-services-dependencies/. The command output in the above article shows the dependent services as below
gcloud services list \
--available --format json | \
jq --arg service cloudfunctions.googleapis.com \
'map(select(.config.name == $service)|
{
name: .config.name,
dependsOn: .dependencyConfig.dependsOn
}
)'
the result is:
{
"name": "cloudfunctions.googleapis.com",
"dependsOn": [
"cloudfunctions.googleapis.com",
"logging.googleapis.com",
"pubsub.googleapis.com",
"source.googleapis.com",
"storage-api.googleapis.com",
"storage-component.googleapis.com"
]
}
when i am executing this command, i am not getting the output as above. the ""dependsOn":" section is showing as "null"
below is the output of the same command when i executed in cloudshell.
{
"name": "cloudfunctions.googleapis.com",
"dependsOn": null
}
]
does there any alternate way to identify the service interdependencies. can someone help on this please

I do not see the dependencyConfig section output by the command on my system.
When I look at the API documentation, the Service resource does not contain the section dependencyConfig.
It appears that Google has removed that item from the API that the command gcloud services list uses to display service information.

Related

I'm getting an error creating a AWS AppSync Authenticated DataSource

I working through the Build On Serverless|S2 E4 video and I've gotten to the point of creating an authenticated HTTP datasource using the AWS CLI. I'm getting this error.
Parameter validation failed:
Unknown parameter in httpConfig: "authorizationConfig", must be one of: endpoint
I think I'm using the same information provided in the video, repository and gist, updated for my own aws account. It seems like it's some kind of formatting or missing information error, but, I'm just not seeing the problem.
When I remove the "authorizationConfig" property from the state-machine-datasource.json the command works.
I've reviewed the code against the information in the video as well as documentation and examples here and here provided by aws
This is the command I'm running.
aws appsync create-data-source --api-id {my app sync app id} --name ProcessBookingStateMachine
--type HTTP --http-config file://src/backend/booking/state-machine-datasource.json
--service-role-arn arn:aws:iam::{my account}:role/AppSyncProcessBookingState --profile default
This is my state-machine-datasource.json:
{
"endpoint": "https://states.us-east-2.amazonaws.com",
"authorizationConfig": {
"authorizationType": "AWS_IAM",
"awsIamConfig": {
"signingRegion": "us-east-2",
"signingServiceName": "states"
}
}
}
Thanks,
I needed to update my aws cli to the latest version. The authenticated http datasource is something fairly new I guess.

AWS Amplify: How to delete the environment, when resources are already partially deleted?

TL;DR: How to delete an amplify environment, when some resources of the service have been deleted manually in the console?
So, I took a course on egghead to learn the aws amplify CLI. Unfortunately, it doesn't teach you how to delete the environment (otherwise it's great though!). My google search back then said you will have to delete the resources manually. I tried (/did) that for the resources I used. I deleted the user account for the CLI (🤦🏻‍♂️), "deleted" the cognito user pool (it still shows up in amplify status), deleted the DynamoDB and the AppSync API (also still shows up).
Now as I mentioned when I run amplify status I get:
| Category | Resource name | Operation | Provider plugin |
| -------- | --------------- | --------- | ----------------- |
| Auth | cognito559c5953 | No Change | awscloudformation |
| Api | AmplifyTodoApp | No Change | awscloudformation |
I wondered - since I thought I deleted them - do they still exist?
So I googled some more. Now it turns out there is also the command amplify delete which automatically deletes all resources associated with your amplify project. Since I deleted the account that I used for the project, that command throws:
The security token included in the request is invalid.
Is there any way I can delete these resources without the user? Are these resources even still online (since I manually deleted them and they do not show up in the console online - even in the CloudFront console)? Or will I have to delete my whole AWS account? I don't want to end up with a big bill one day for these resources.
EDIT: I also deleted the S3 bucket.
EDIT 2: So I managed to use another profile (by changing local-aws-info.json) so I don't get the security request failed error any more. Now I get the error:
Missing region in config
amplify status still yields the same response.
amplify cli determines the status by diffing amplify/#current-cloud-backend and amplify/backend folder inside your project. So what you see when you run amplify status you see isn't accurate in your case.
If you have created multiple environments (in different regions) make sure that you delete them too. The easiest way to delete them if you can't use amplify delete is to go to cloud formation in the region where you have created the environment and deleting the root stack, which ensures that all the resources created by that stack are removed.
PS: The cli creates roles for auth and unauth users when initialized and creates policies for the resources (they don't cost anything if they exist). You could delete them if you don't want them hanging around.
When some resources have been deleted manually (S3 & Cloudformation) then
$amplify delete
Gives Following :
Unable to remove env: dev because deployment bucket amplify-amplifyAPPName-dev-XYZ-deployment does not exist or has been deleted.
Stack has already been deleted or does not exist
Please look at this:
C:user\samadhan\Amplify-Projects\amplifyapp-demo>amplify delete
? Are you sure you want to continue? This CANNOT be undone. (This will delete all the environments of the project from the cloud and wi
pe out all the local files created by Amplify CLI) Yes
- Deleting resources from the cloud. This may take a few minutes...
Deleting env: dev.
Unable to remove env: dev because deployment bucket amplify-
amplifyinitdemo-dev-131139-deployment does not exist or has been deleted.
Stack has already been deleted or does not exist
\ Deleting resources from the cloud. This may take a few minutes...App
dfwx13s2bgtb1 not found.
App dfwx13s2bgtb1 not found.
√ Project already deleted in the cloud.
Project deleted locally.
App Amplify App still showing in Console Unable to delete from Console.
Please Take a look :
Solution:
Using AWS CLI You Can be Fixed This Issue.
Step 1 ) Make Sure AWS CLI is configured with the Same AWS Account if Not Please Create IAM User & Configure it with the same Region.
C:user\samadhan\Amplify-Projects\amplifyapp-demo>aws configure
AWS Access Key ID [****************HZHF]: ****************ICHK
AWS Secret Access Key [****************iBJl]:****************SnaX
Default region name [ap-south-1]: ap-south-1
Default output format [json]: json
Step 2 ) Use Following AWS CLI Commands.
C:user\samadhan\Amplify-Projects\amplifyapp-demo>>aws amplify help
Available Commands
******************
* create-app
* create-backend-environment
* create-deployment
* delete-app
* delete-backend-environment
* get-app
* list-apps
* list-backend-environments
C:user\samadhan\Amplify-Projects\amplifyapp-demo>aws amplify list-apps
{
"apps": [
{
"appId": "d39pvb2qln4v7l",
"appArn": "arn:aws:amplify:ap-south-1:850915XXXXX:apps/d39pvb2qln4v7l",
"name": "react-amplify-demo-project",
"tags": {},
"platform": "WEB",
"createTime": 1640206703.371,
"updateTime": 1640206703.371,
"environmentVariables": {
"_LIVE_PACKAGE_UPDATES": "[{\"pkg\":\"#aws-amplify/cli\",\"type\":\"npm\",\"version\":\"latest\"}]"
},
{
"appId": "d2jsl78ex1asqy",
"appArn": "arn:aws:amplify:ap-south-1:85091xxxxxxxx:apps/d2jsl78ex1asqy",
"name": "fullstackapp",
"tags": {},
"platform": "WEB",
"createTime": 1640250148.974,
"updateTime": 1640250148.974,
"environmentVariables": {
"_LIVE_PACKAGE_UPDATES": "[{\"pkg\":\"#aws-amplify/cli\",\"type\":\"npm\",\"version\":\"latest\"}]"
}
}
Step 3) Use Following CLI Command to Delete App Or App Env
C:user\samadhan\Amplify-Projects\amplifyapp-demo>aws amplify delete-app --app-id d39pvb2qln4v7l
{
"app": {
"appId": "d39pvb2qln4v7l",
"appArn": "arn:aws:amplify:ap-south-1:8509xxxxx:apps/d39pvb2qln4v7l",
"name": "react-amplify-demo-project",
"repository": "https://gitlab.com/samadhanfuke/react-amplify-demo-project",
"platform": "WEB",
"createTime": 1639077857.194,
"updateTime": 1639077857.194,
"iamServiceRoleArn": "arn:aws:iam::850915xxxx:role/amplifyconsole-backend-role",
"environmentVariables": {
"_LIVE_UPDATES": "[{\"name\":\"Amplify CLI\",\"pkg\":\"#aws-amplify/cli\",\"type\":\"npm\",\"version\":\"latest\"}]"
},
"defaultDomain": "d39pvb2qln4v7l.amplifyapp.com",
"enableBranchAutoBuild": false,
"enableBranchAutoDeletion": false,
"enableBasicAuth": false,
"customRules": [
{
"source": "/<*>",
"target": "/index.html",
"status": "404-200"
}
],
"productionBranch": {
"lastDeployTime": 1639078272.607,
"status": "SUCCEED",
"branchName": "preview"
},
"buildSpec": "version: 1\nbackend:\n phases:\n # IMPORTANT - Please verify your build commands\n build:\n commands:\n - '# Execute Amplify CLI with the helper script'\n - amplifyPush --simple\nfrontend:\n phases:\n build:\n commands: []\n artifacts:\n # IMPORTANT - Please verify your build output directory\n baseDirectory: /\n files:\n - '**/*'\n cache:\n paths: []\n",
"customHeaders": "",
"enableAutoBranchCreation": false
}
}
Amplify App With Environment Successfully Deleted.
Check-in Amplify Console.
As of 9/26/2022 several updates have been released that fix issues deleting apps/backends, including issues where the s3 bucket or cloudformation stack was already deleted
Note that deleting the amplify application as documented here, does not remove the resources created in S3. You need to delete these manually.
The content in the bucket amplify-{project name}-{env name}-{some id}-deployment is created and updated when you run amplify init, amplify push among others. It appears to be used as the remote synchronisation directory.
The S3 buckets will be recreated by the amplify root CloudFormation stack, whenever you create a new env or run amplify init.

Deploy Google Cloud Function from Cloud Function

Solved/invalid - see below
I'm trying to deploy a Google Cloud Function from a Google Cloud Function on demand.
However, whatever I try, I get a 403 Forbidden:
HttpError 403 when requesting https://cloudfunctions.googleapis.com/v1/projects/MY_PROJECT/locations/MY_REGION/functions?alt=json returned "The caller does not have permission"
I ended up granting the cloud function service account Project Owner role to make sure it can do anything, yet still I get the same error.
Is this limited intentionally (for example to avoid fork bombs or something) or am I doing something wrong?
Has anyone been able to make this work?
For the record: I ran the same (Python) function locally with Flask using my own account and then it will deploy the new cloud function perfectly, so the code itself seems to be ok.
Update
Code snippet of how I'm trying to deploy the cloud function:
cf_client = discovery.build('cloudfunctions', 'v1')
location = "projects/{MYPROJECT}/locations/europe-west1"
request = {
"name": "projects/{MYPROJECT}/locations/europe-west1/functions/hopper--2376cd24d318cd2d42f000f4f1c31a8f",
"description": "Hopper hopper--2376cd24d318cd2d42f000f4f1c31a8f",
"entryPoint": "pubsub_trigger",
"runtime": "python37",
"availableMemoryMb": 256,
"timeout": "60s",
"sourceArchiveUrl": "gs://staging.{MYPROJECT}.appspot.com/deployment/hopper.zip",
"eventTrigger": {
"eventType": "providers/cloud.pubsub/eventTypes/topic.publish",
"resource": "projects/{MYPROJECT}/topics/hopper-test-input"
},
"environmentVariables": {
"HOPPER_ID": "hopper--2376cd24d318cd2d42f000f4f1c31a8f"
}
}
response = cf_client.projects() \
.locations() \
.functions() \
.create(location=location, body=req) \
.execute()
Update
I feel like such an idiot... it turns out that for some reason I deployed the master function in a different project then the project I gave permissions on. No wonder it didn't work.
The correct answer should be: check that everything is indeed running how/where you expect it to be. Everything was configured correctly and deploying a CF in a CF is not a problem. The project was incorrect, due to a different default project being set on the gcloud utility.

Getting the list of binded services for an app

Referring the docs: https://docs.cloudfoundry.org/devguide/services/managing-services.html#list
How to get the list of services that are bind to an app?
We can do it using
cf env <app_name>
It returns a list of applications that are bound to this app
System-Provided:
{
"VCAP_SERVICES": {
"external-dependency-service": [ .. ],
"sso": [ .. ],
"<any other service type>": [<list of services>]
}
}
There is no straight-forward cf CLI command to list the services instances bound to an app.
You could use cf create-app-manifest YOUR_APP, which will create a manifest file with a services block listing all bound service instances, if any exist.
You can also call the CC API (using cf curl /v2/apps/:guid/service_bindings). See API documentation here.

API Gateway not importing exported definition

I am testing my backup procedure for an API, in my API Gateway.
So, I am exporting my API from the API Console within my AWS account, I then go into API Gateway and create a new API - "import from swagger".
I paste my exported definition in and create, which throws tons of errors.
From my reading - it seems this is a known issue / pain.
I suspect the reason for the error(s) are because I use a custom authorizer;
"security" : [ {
"TestAuthorizer" : [ ]
}, {
"api_key" : [ ]
} ]
I use this on each method, hence, I get a lot of errors.
The weird thing is that I can clone this API perfectly fine, hence, I assumed that I could take an exported definition and re-import without issues.
Any ideas on how I can correct these errors (preferably within my API gateway, so that I can export / import with no issues).
An example of one of my GET methods using this authorizer is:
"/api/example" : {
"get" : {
"produces" : [ "application/json" ],
"parameters" : [ {
"name" : "Authorization",
"in" : "header",
"required" : true,
"type" : "string"
} ],
"responses" : {
"200" : {
"description" : "200 response",
"schema" : {
"$ref" : "#/definitions/exampleModel"
},
"headers" : {
"Access-Control-Allow-Origin" : {
"type" : "string"
}
}
}
},
"security" : [ {
"TestAuthorizer" : [ ]
}, {
"api_key" : [ ]
} ]
}
Thanks in advance
UPDATE
The error(s) that I get when importing a definition I had just exported are:
Your API was not imported due to errors in the Swagger file.
Unable to put method 'GET' on resource at path '/api/v1/MethodName': Invalid authorizer ID specified. Setting the authorization type to CUSTOM or COGNITO_USER_POOLS requires a valid authorizer.
I get the message for each method in my API - so there is a lot.
Additionality, right at the end of the message, I get this:
Additionally, these warnings were found:
Unable to create authorizer from security definition: 'TestAuthorizer'. Extension x-amazon-apigateway-authorizer is required. Any methods with security: 'TestAuthorizer' will not be created. If this security definition is not a configured authorizer, remove the x-amazon-apigateway-authtype extension and it will be ignored.
I have tried with Ignoring the errors, same result.
Make sure you are exporting your swagger with both integrations and authorizers extensions.
Try exporting your swagger using AWS CLI:
aws apigateway get-export \
--parameters '{"extensions":"integrations,authorizers"}' \
--rest-api-id {api_id} \
--stage-name {stage_name} \
--export-type swagger swagger.json
The output will be sent to swagger.json file.
For more details about swagger custom extensions see this.
For anyone that may come across this issue.
After LOTS of troubleshooting and eventually involving the AWS Support Team, this has been resolved and identified as an AWS CLI client bug (confirmed from AWS Support Team);
Final response.
Thank you for providing the details requested. After going through the AWS CLI version and error details, I can confirm the error is because of known issue with Powershell AWS CLI. I apologize for inconvenience caused due to the error. To get around the error I recommend going through the following steps
Create a file named data.json in the current directory where the powershell command is to be executed
Save the following contents to file {"extensions":"authorizers,integrations"}
In Powershell console, ensure the current working directory is the same as the location where data.json is present
Execute the following command aws apigateway get-export --parameters file://data.json --rest-api-id APIID --stage-name dev --export-type swagger C:\temp\export.json
Using this, finally resolved my issue - I look forward to the fix in one of the upcoming versions.
PS - this is currently on the latest version:
aws --version
aws-cli/1.11.44 Python/2.7.9 Windows/8 botocore/1.5.7