I define some variables in the env/variables, then make changes to the value in phases/pre_build. I want to use the variable down in artifacts, but it looks like the changes are not persisted.
This is a legacy Windows .NET Framework 4.7.2 application getting deployed to IIS.
My buildspec.yml file:
version: 0.2
env:
variables:
APPNAME: DummyApp
BRANCH: manual
phases:
pre_build:
commands:
- echo "start BRANCH = ${BRANCH}"
- echo "CODEBUILD_WEBHOOK_HEAD_REF = ${env:CODEBUILD_WEBHOOK_HEAD_REF}"
# CODEBUILD_WEBHOOK_HEAD_REF is null when build is triggered from console as opposed to a webhook
- if (${CODEBUILD_WEBHOOK_HEAD_REF}) { ${BRANCH} = ($CODEBUILD_WEBHOOK_HEAD_REF.replace('refs/heads/', '')) }
- echo "after BRANCH = ${env:BRANCH}"
build:
commands:
- echo "build commands happen here"
artifacts:
files:
- .\Dummy\bin\Debug\*
# not sure why this doesnt work down here, are changes in the phases section above not propagated?
name: ${env:APPNAME}/${env:APPNAME}-${env:BRANCH}.zip
discard-paths: yes
The value of $CODEBUILD_WEBHOOK_HEAD_REF = "refs/head/develop".
The value of $BRANCH after the replace statement = "develop".
The value of my artifact in S3 is "DummyApp/DummyApp-manual.zip".
I want the artifact named "DummyApp/DummyApp-develop.zip".
Some sort of scoping issue?
Saw various indications that this is not possible.
https://blog.shikisoft.com/define-environment-vars-aws-codebuild-buildspec/
The crucial thing you should note here is that you can only assign literal values to the environment variables declared this way. You cannot assign dynamic values at runtime. If you would like to change the value of the <...> variable above, you have to change your buildspec file and push your changes to your repository again. So it is like hardcoding parameter values. But it is better than typing the in all commands needed in the phases section.
In addition to trying to simply set the local var in pre_build, I tried a number of approaches, including
running a custom powershell script to parse the Branch name as the first step in the pre_build
running the command in the variable declaration itself
calling the prsh SetEnvironmentVariable method
The thing that seems to work is using the replace command down in the artifact/name itself:
artifacts:
files:
- .\Dummy\bin\Debug\*
name: ${env:APPNAME}/${env:APPNAME}-$($CODEBUILD_WEBHOOK_HEAD_REF.replace('refs/heads/', '')).zip
discard-paths: yes
created this artifact: DummyApp\DummyApp-develop.zip
I (try to) deploy my current application using CDK pipelines.
In doing so, I stumbled across an unexpected behavior (here if interested) which now I am trying to resolve. I have a Lambda function for which the asset is a directory that is dynamically generated during a CodeBuild step. The line is currently defined like this in my CDK stack :
code: lambda.Code.fromAsset(process.env.CODEBUILD_SRC_DIR_BuildLambda || "")
The issue is that locally, this triggers the unexpected and undesired behaviour because the environment variable does not exist and therefore goes to the default "".
What is the proper way to avoid this issue?
Thanks!
Option 1: Set the env var locally, pointing to the correct source directory;
CODEBUILD_SRC_DIR_BuildLambda=path/to/lambda && cdk deploy
Option 2: Define a dummy asset if CODEBUILD_SRC_DIR_BuildLambda is undefined
code: process.env.CODEBUILD_SRC_DIR_BuildLambda
? lambda.Code.fromAsset(process.env.CODEBUILD_SRC_DIR_BuildLambda)
: new lambda.InlineCode('exports.handler = async () => console.log("NEVER")'),
We are trying to return some metrics from our Vertex Pipeline, such that they are visible in the Run Comparison and Metadata tools in the Vertex UI.
I saw here that we can use this output type Output[Metrics], and the subsequent metrics.log_metric("metric_name", metric_val) method to add the metrics, and it seemed from the available documentation that this would be enough.
We want to use the reusable component method as opposed to python function based components, around which the example is based. So we implemented it within our component code like so:
We added the output in the component.yaml:
outputs:
- name: metrics
type: Metrics
description: evaluation metrics path
then added the output to the command in the implemenation:
command: [
python3, main.py,
--gcs-test-data-path, {inputValue: gcs_test_data_path},
--gcs-model-path, {inputValue: gcs_model_path},
--gcs-output-bucket-id, {inputValue: gcs_output_bucket_id},
--project-id, {inputValue: project_id},
--timestamp, {inputValue: timestamp},
--batch-size, {inputValue: batch_size},
--img-height, {inputValue: img_height},
--img-width, {inputValue: img_width},
--img-depth, {inputValue: img_depth},
--metrics, {outputPath: metrics},
]
Next in the components main python script, we parse this argument with argparse:
PARSER.add_argument('--metrics',
type=Metrics,
required=False,
help='evaluation metrics output')
and pass it to the components main function:
if __name__ == '__main__':
ARGS = PARSER.parse_args()
evaluation(gcs_test_data_path=ARGS.gcs_test_data_path,
gcs_model_path=ARGS.gcs_model_path,
gcs_output_bucket_id=ARGS.gcs_output_bucket_id,
project_id=ARGS.project_id,
timestamp=ARGS.timestamp,
batch_size=ARGS.batch_size,
img_height=ARGS.img_height,
img_width=ARGS.img_width,
img_depth=ARGS.img_depth,
metrics=ARGS.metrics,
)
in the declaration of the component function, we then typed this metrics parameter as Output[Metrics]
from kfp.v2.dsl import Output, Metrics
def evaluation(gcs_test_data_path: str,
gcs_model_path: str,
gcs_output_bucket_id: str,
metrics: Output[Metrics],
project_id: str,
timestamp: str,
batch_size: int,
img_height: int,
img_width: int,
img_depth: int):
finally, we implement the log_metric method within this evaluation function:
metrics.log_metric('accuracy', acc)
metrics.log_metric('precision', prec)
metrics.log_metric('recall', recall)
metrics.log_metric('f1-score', f_1)
When we run this pipeline, we can see this metric artifact materialised in the DAG:
And Metrics Artifacts are listed in the Metadata UI in Vertex:
However, clicking through to view the artifacts JSON, there is no Metadata listed:
In addition, No Metadata is visible when comparing runs in the pipeline UI:
Finally, navigating to the Objects URI in GCS, we are met with 'Requested entity was not found.', which I assume indicates that nothing was written to GCS:
Are we doing something wrong with this implementation of metrics in the reusable components? From what I can tell, this all seems right to me, but it's hard to tell given the docs at this point seem to focus primarily on examples with Python Function based components.
Do we perhaps need to proactively write this Metrics object to an OutputPath?
Any helps is appreciated.
----- UPDATE ----
I have since been able to get artifact metadata and URI To update. In the end we used kfp sdk to generate a yaml file based on a #component decorated python function, we then adapted this format for our reusable components.
Our component.yaml now looks like this:
name: predict
description: Prepare and create predictions request
implementation:
container:
args:
- --executor_input
- executorInput: null
- --function_to_execute
- predict
command:
- python3
- -m
- kfp.v2.components.executor_main
- --component_module_path
- predict.py
image: gcr.io/PROJECT_ID/kfp/components/predict:latest
inputs:
- name: input_1
type: String
- name: intput_2
type: String
outputs:
- name: output_1
type: Dataset
- name: output_2
type: Dataset
with this change to the yaml, we can now successfully update the artifacts metadata dictionary, and uri through artifact.path = '/path/to/file'. These updates are displayed in the Vertex UI.
I am still unsure why the component.yaml format specified in the Kubeflow documentation does not work - I think this may be a bug with Vertex Pipelines.
As I can see in the code you are running, everything should work without a problem; but, as you commented, I would recommend you to write the metrics object into a path so that it can reach somewhere within your project.
How do I specify a different "Executed Function" in my cloudbuild.yaml file than the name of the actual function name in GCP?
For example:
I have a cloud function, written in python called hello_world
In my GCP deployment, I want to name the function hello-world-dev, and hello-world-prod which is passed in the Triggers variables dynamically on build.
Build fails because it was expecting the function to be called hello-world-dev or whatever
I'm sure there's a flag to specify the executing function, but I haven't found it.
My cloudbuild.yaml file looks like this:
#hello-world
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
args:
- gcloud
- functions
- deploy
- hello-world-${_ENV}
- --region=us-west2
- --source=./demo/hello-world/
- --trigger-http
- --runtime=python39
Steps I've Tried
I've tried the following flags:
--function
--call
--deploy
Looking at this documentation: https://cloud.google.com/functions/docs/deploying
The executed function is the function name in your code. By default, and it's your error, the executed function, if not specified, must be the same as the function name.
If your function name has a different name as the executed function, you need to specify the entrypoint of your function (the function to run in your code). Use the parameter --entry-point=
I have these config files serverless.yml and env.yml and when I try to deploy, I get an error. The Lambda functions cannot be deployed.
serverless.yml
---omitted---
provider:
environment: ${file(env.yml):${self:custom.stage}}
---omitted---
env.yml
---omitted---
dev:
keyzero: "valuezero"
keyone:
keyoneone: "valueoneone"
keyonetwo: "valueonetwo"
keyonethree: "valueonethree"
---omitted---
ERROR:
Serverless: Operation failed!
Serverless Error ---------------------------------------
An error occurred: PingLambdaFunction - Value of property Variables
must be an object with String (or simple type) properties.
You need to specify which value from env.yml you want to use.
In your example, if you want to get the value of keyonetwo, you'd use
${file(env.yml):${opt:stage}.keyone.keyonetwo}
Which would yield valueonetwo
Also, checkout the documentation and how they reference environment variables.
You need to set each environment variable, so you'd need
provider:
environment:
keyoneone: ${file(env.yml):${opt:stage}.keyone.keyoneone}
keyonetwo: ${file(env.yml):${opt:stage}.keyone.keyonetwo}
Environment variables cannot be an object. They are simply key-value pairs where value should be of primitive types (i.e. string/number/boolean/null).
Your keyone variable is an object which is why it throws the error "Variables must be an object with String (or simple type) properties".