I am trying to create a default Composer 2 Instance on GCP and get the Errors:
CREATE operation on this environment failed 32 minutes ago with the following error message:
Composer Backend timed out. Currently running tasks are [stage: CP_COMPOSER_AGENT_RUNNING
description: "No agent response published."
...
or
CREATE operation on this environment failed 32 minutes ago with the following error message:
Environment couldn't be created, but no error was surfaced. This can be cause by a lack of
proper permissions. Check if this environment's service account ... .iam.gserviceaccount.com
has the 'roles/composer.worker' role and there is no firewall inhibiting internal
communications set.
I already tried to add the Composer Worker role to the service account and all other required roles (e.g. Cloud Composer v2 API Service Agent Extension) in https://cloud.google.com/composer/docs/composer-2/access-control (for public as well as for private, eventhough instance is public).
I looked into the GKE instance and found the Pod composer-agent failing with:
Traceback (most recent call last): File "composer_agent.py", line 467, in <module> main() File "composer_agent.py", line 292, in main responses = pubsub_subscriber.pull() (...)
oauth2client.client.HttpAccessTokenRefreshError: Failed to retrieve
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/(...)
compute#developer.gserviceaccount.com/token from the Google Compute Enginemetadata service.
Response:
{'date': 'Thu, 17 Feb 2022 10:29:46 GMT', 'status': '403', 'content-length': '668', 'content-
type': 'text/plain; charset=utf-8', 'x-content-type-options': 'nosniff'}
So I assume there is still some permission issue but I can not figure out what, Composer 1 Instances can be created without a problem, as well for a different project Composer 2 Instances with the same permissions on the service accounts.
I also tried to create different than default compute service account with the required permissions but also without success. I also checked that the service account I am adding permissions to is actually the service account sending the request in the composer-agent and is sending the environment creation request to the GKE cluster.
I hope anyone can help, who faced similar issues or knows more about the error occuring in composer-agent, thank you very much!
After being in contact with the Google Support Team, the solution was to manually enable the "IAM Service Account Credentials API". There was no issue in Service Account Rights or Firewall settings.
Related
I already have an environment running, but want to scale up the machine sizes. But keep getting a very similar error to this:
UPDATE operation on this environment failed 3 minutes ago with the
following error message:
Composer Backend timed out. Currently running tasks are [stage:
CP_COMPOSER_AGENT_RUNNING
description: "No agent response published."
response_timestamp {
seconds: 1618203503
nanos: 291000000
}
].
The gcp docs say that the service account doesn't have required permissions. I check required permissions also in the docs. My personal account and service already have the role Environment and Storage Object Administrator. And the service account already has the role Cloud Composer v2 API Service Agent Extension. I don't understand which permission am I missing.
Service account that the composer uses has roles:
Cloud Composer v2 API Service Agent Extension
Editor
Environment and Storage Object Administrator
Service Account User
And my personal account has roles:
Editor
Environment and Storage Object Administrator
Service Account User
Storage Admin
Viewer
I've searched all over the documentation and google without luck. I changedt he IAM permissions and granted the 'Cloud Composer v2 API Service Agent Extension' role to the Compute Engine default service account. I'm getting the following error:
CREATE operation on this environment failed x minutes ago with the following error message:
Composer Backend timed out. Currently running tasks are [stage: CP_GKE_CLUSTER_CREATING
description: "GKE cluster creation requested."
response_timestamp {
seconds:x
nanos:x
}
],
I am trying to create basic Composer environment:
image version: 1.17.8/2.1.4
using service account with composer.worker permission
my own user has project.owner permission
public ip
All my attempts failed with following error:
Http error status code: 400
Http error message: BAD REQUEST
Errors in: [Web server]; Error messages:
The caller does not have permission
Required 'deploymentmanager.typeProviders.create' permission for 'projects/<my-project>/global/typeProviders/europe-west2-<name-id>-addons-gke-typer'
deploymentmanager.typeProviders.create is covered by Deployment Manager Type Editor, so I added this permission to both my account and service account, but the error remains the same.
Cloud Composer Service Agent account is present in the project without any modifications to its permissions.
Is there anything else I can check or something that I missed during the set up?
For an account (whether User Account or Service Account) to be able to create a Composer Environment, the account must have a composer.environments.create permission.
And according to Google Cloud's documentation on Cloud Composer Access Control,
The Composer Worker role provides the permissions necessary to run a Cloud Composer environment
VM and intended for service accounts.
The Composer Worker role is not intended for creation of environments thus, it does not have the composer.environments.create permission.
If you want your service account to be able to create a Composer environment, you will need to assign the role Composer Administrator and this has the composer.environments.create permission needed.
You may refer to Access Control for Cloud Composer for the complete list of permission for Composer Worker, Composer Administrator and other Composer related roles.
I tried to deploy an OpenVPN Access Server to Google Compute Engines and received the following error message:
openvpn-access-server-1-vm: {"ResourceType":"compute.v1.instance","ResourceErrorCode":"EXTERNAL_RESOURCE_NOT_FOUND","ResourceErrorMessage":"The resource 'PROJECT_ID-compute#developer.gserviceaccount.com' of type 'serviceAccount' was not found."}
PROJECT_ID is just a placeholder for my own PROJECT_ID.
In the cloud console, I can't find the "compute engine default service account" (I think, I accidentally deleted it last year). In the log files, I found in 2020 it's ACCOUNT_ID, so I tried to undelete it with the following command:
gcloud beta iam service-accounts undelete ACCOUNT_ID
I had no success, I received:
ERROR: (gcloud.beta.iam.service-accounts.undelete) NOT_FOUND: Not found; Not found AccountDataType for <numeric_id>
<numeric_id> was a 12-digit number.
I tried to disable and enable compute service to restore the default service account, but it wasn't successful, I received:
response:
'#type': type.googleapis.com/google.iam.admin.v1.ServiceAccount
serviceName: iam.googleapis.com
status:
code: 6
message: ALREADY_EXISTS
receiveTimestamp: '2021-08-05T06:45:55.798772716Z'
Because of this error, I tried to delete it, but this didn't work too.
Now I don't know what to do, to get the default service account back.
Is it still existing or not?
Why isn't it working?
Keep in mind, I'm talking about PROJECT_ID-compute#developer.gserviceaccount.com.
service-PROJECT_ID#compute-system.iam.gserviceaccount.com is existing and recreated each time I disable and enable the Compute Engine API again.
Thanks for helping.
Since the Service Account was deleted an year ago it cannot be undeleted using the following command,
gcloud beta iam service-accounts undelete ACCOUNT_ID
This only works for Service Accounts deleted fewer than 30 days ago. Undeleting a service account for more information.
Instead, we can create a new Service Account and grant an ‘Editor’ role to it. As a Default Compute Engine Service Account has the same role by default. Compute Engine default service account for more information.
Now, we can create a new Compute Engine VM using the new Service Account. Setting up a new instance to run as a service account for more information.
If we already have a running VM and the Service Account got deleted, As #John Hanley suggested, we can edit the VM instance in the Google Cloud Console and assign the new Service Account to the instance. Changing the service account and access scopes for an instance for more information.
To set the new Service Account as the Compute Engine Default Service Account on the project, we can use the following command,
gcloud alpha compute project-info set-default-service-account
But since the command is in the ‘alpha’ launch stage, it is not available for everyone.
Another workaround would be creating a new project and deploying our instance there.
Below are the steps that I performed for setup :
Created a new windows server 2012 EC2 instance and then connected
with it using the remote desktop connection.
Installed EC2 config service which automatically installs
AmazonSSMAgent service.
Both EC2 and SSM services are running and in error logs, it is throwing some 'no chain provider exception' which is fine because I have not provided aws credentials yet.
Created a new IAM user and give him administrative policy access and
created three environment variables providing access_key,
secret_access_key and region_id.
Then I restarted AmazonSSMAgent service, then the exception changed to below exception:
status code: 400, request id: 400b4c75-eae2-11e7-a120-d33083d55d1f 2017-12-27 08:45:13 ERROR [HandleAwsError # awserr.go.48] [instanceID=i-084646fb32d8a7b6d] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AuthorizationFailureException:
Please suggest what i am doing wrong or any steps that i have missed or any other way to do this.