GCP Composer didn't apply new value in Environment Variables
As I have a connection string and put it under a new key in Environment Variables then save it. And after a while, I'd changed the host name in the connection id and save again, but this time the Composer still used the old config even though the Composer itself already stated that "This environment is running" which also means it finished the changing Environment Variables completely.
I already refreshed and checked for what I changed, the new value was there too. But running a new task with composer and it still used the old connection id.
There is no code that contain that connection id.
So weird, or I missed something.
This issue can be resolved by disabling and then enabling the Cloud Composer API.
Also, the AIRFLOW_GPL_UNIDECODE has to be set to yes.
Related
On Windows, in the VSCode Terminal (which I believe is Powershell), I am attempting to work with the aws cli, and as part of that, I need to setup and refer to a profile that has credentials to connect to my AWS account. A month or so ago, I was using a profile named xxxx_AWSAdministratorAccess (where xxxx is my AWS account number), and all was working as expected. Today, I want to use different credentials, so I setup a new profile named CICD. If I set my profile using the aws configure command, it will work for that session. However if I create a new Terminal session, or start a new instance of VSCode, it reverts back to the xxxx_AWSAdministratorAccess profile.
I noticed in the Terminal there is an environment variable named AWS_PROFILE and this is set to xxxx_AWSAdministratorAccess. I have changed this using $env:AWS_PROFILE = 'CICD' as well as [System.Environment]::SetEnvironmentVariable('AWS_PROFILE','CICD'). Both work as they set the environment variable to CICD, and the latter command is supposed to persist the value according to what I read. It sort of works as I can open up a PowerShell session away from VSCode and it will also report the correct AWS profile, but as soon as I close the VSCode Terminal and re-open it reverts back to the xxxx_AWSAdministratorAccess value. It appears VSCode or an extension or something is caching this value, but I can't figure out where. Thoughts?
Per the comment from #rioV8 above, I opened a new Powershell window, set the AWS_PROFILE to CICD, and then launched VSCode from there. When I launched a new Terminal window inside VSCode, and ran dir env: the AWS_PROFILE variable was still set to CICD, so that is interesting. I then went to the Environment Variables in Control Panel and deleted AWS_PROFILE. I then re-launched VSCode and the environment variable is no longer there, and the aws configure command works as expected to update the profile.
So...long story short, I am not sure where that AWS_PROFILE environment variable came from, but deleting it as opposed to updating the value to something else, seems to have resolved my issue. Thanks for the suggestion!
I am having some of my GCP instances behave in a way similar to what is described in the below link:
Google Cloud VM Files Deleted after Restart
The session gets disconnected after a small duration of inactivity at times. On reconnecting, the machine is as if it is freshly installed. (Not on restarts as in the above link). All the files are gone.
As you can see in the attachment, it is creating the profile directory fresh when the session is reconnected. Also, none of the installations I have made are there. Everything is lost including the root installations. Fortunately, I have been logging all my commands and file set ups manually on my client. So, nothing is lost, but I would like to know what is happening and resolve this for good.
This has now happened a few times.
A point to note is that if I get a clean exit, like if I properly logout or exit from the ssh, I get the machine back as I have left, when I reconnect. The issue is there only when the session disconnects itself. There have been instances where the session disconnected and I was able to connect back as well.
The issue is not there on all my VMs.
From the suggestions from the link I have posted above:
I am not connected to the cloud shell. i am taking ssh of the machine using the chrome extension
Have not manually mounted any disks (afaik)
I have checked the logs from gcloud compute instances get-serial-port-output --zone us-east4-c INSTANCE_NAME. I could not really make much of it. Is there anything I should look for specifically?
Any help is appreciated.
Please find the links to the logs as suggested by #W_B
Below is from 8th when the machine was restarted and files deleted
https://pastebin.com/NN5dvQMK
It happened again today. I didn't run the command immediately then. The below file is from afterwards though
https://pastebin.com/m5cgdLF6
The below one is after logout today.
[4]: https://pastebin.com/143NPatF
Please note that I have replaced the user id, system name and a lot of numeric values in general using regexp. So, there is a slight chance that the time and other values have changed. Not sure if that would be a problem.
I have added the screenshot of the current config from the UI
Using locally attached SDD seems to be the cause ... here it is explained:
https://cloud.google.com/compute/docs/disks/local-ssd#data_persistence
You need to use a "persistent disk" - else it will behave just as you describe it.
I recently added a few new DAGs to production airflow and as a result decided to scale up the number of nodes in the Composer pool. After doing so I got the error: Can't decrypt _val for key=<KEY>, invalid token or value. This happens now for every single DAG that uses variables. It's not the same key either, it depends on what variables the DAG needs.
I immediately scaled Composer back down to 3 nodes and the problem persisted.
I have tried re-saving all of the Variables, recreating them in the UI (which says they are all valid), recreating them in the CLI (which lists invalid for every single one).
I have also tried updating configuration to try and reboot the server, and manually stopping the VM instances.
Composer also seems to negate the ability to update the Fernet Key, so I can't try and use a new one. For some reason it appears that the permanent one Composer has assigned is now invalid.
Is there anything else that can be done to remedy that problem short of recreating the environment?
I managed to fix this problem by adding a new python package. It seems that adding a package is the only way to really "reboot" the environment. The reboot invalidated all of my variables and connections when it had finished but I was able to just add those back in rather than having to recreate the entire environment.
Heard back about this issue: According to Google, Composer creates a custom image for the environment and passes one to each node, and if that got corrupted during scaling then the only way to fix it is by adding a new python package so it rebuilds the image. Incidentally, version 1.3.0 of Composer is much better as the scheduler is restarted every 10 minutes which should solve some of the latter issues I experienced.
In Google shell which is a part of Google cloud, I set environment variable GOOGLE_APPLICATION_CREDENTIALS because It is need it for PHP NLP project [info: https://cloud.google.com/natural-language/docs/quickstart-client-libraries#client-libraries-install-php]. My project worked fine, but I notice that variable GOOGLE_APPLICATION_CREDENTIALS lasts on my sistem only one day. This is my third time that I am setting it. My project doesn't work when I am missing required variable. Am I doing something wrong?
EDIT:
It is default OS (Debian) when you create new App on Google App engine.
When I type help in Google shell I get info with:
Your 5GB home directory will persist across sessions, but the VM is ephemeral and will be reset
approximately 20 minutes after your session ends. No system-wide change will persist beyond that.
You are completely right, Cloud Shell is running on an ephemeral instance that resets some minutes after the session has ended, reason why you are losing the content of the environment variable you mentioned.
The documentation about limitations in Cloud Shell clearly states that it is intended for interactive use only, and any non-interactive session or intensive usage can be automatically terminated with (or without) a warning.
Therefore, and understanding from your question that you have a background script that is working with Cloud Natural Language, I would strongly advise you to move to a "real" instance of Compute Engine, in which you will have much more control about what is happening. This will allow more flexibility and you will be able to use a bigger machine type, given that Cloud Shell runs on a g1-small GCE instance which, in general, is not enough to run an application. Also, depending on your use case, you may even consider App Engine.
That being said, I have found that when constructing the LanguageClient instance, you may also not use Application Default Credentials and, instead, use the keyFile or keyFilePath variables (explained in the PHP Client Library reference) to pass the path to the JSON key directly to your code, instead of reading it from the environment variable.
Lets assume you are using Linux, make sure that:
The system is not being restarted, and if it is, make sure to set the environment variables accordingly (see how to set permantent environment variables)
Let's say I would have one app with two instances.
I would change an environment variable (cf set-env) and not perform a cf restage.
Eventually one of the two instances would crash and restart. Would it take the new environment variable or the old?
In general if an instance crashes (say the app runs out of memory) and is restarted by Diego (the runtime that's actually running the container instances), the restarted instance will still have the environment variables it was originally "desired" (created) with.
If you explicitly cf restart or cf stop && cf start the app it will pickup the new environment variables which out needing to be restaged.
As user152468 said above, if the environment variables are used during the staging process you will need to cf restage the app for them to functionally take effect.
Edge Case Scenario
If the Diego runtime goes away/loses data for some catastrophic reason, the Cloud Controller will re-sync it and recreate the apps that are meant to be running. In this event the behavior is similar to a cf restart and the app will pick up the new environment variables. This is definitely uncommon, but would also count as a "crash" scenario.
EDIT:
After reading tcdowney's answer below, I tried it out. And tcdowney is right. When the app is restarted due to a crash, it will not pick up the new environment variable, so both of your app instances will share the same environment.
In contrast when you do a cf restart my-app then it will pick it up. Sorry for the confusion!
========================================================
It would take the new value of the environment variable. See here: https://docs.run.pivotal.io/devguide/deploy-apps/start-restart-restage.html
Only if the variable is relevant for the buildpack, you will need to restage.