Airflow won't let me convert try_number macros to an integer - airflow-scheduler

My issue is rather simple, but yet no real solution is really offered. The DAG I am trying to build will have a critical environment variable that might change depending on the number of retries that it goes through. For example, after 3 retries we will switch the value of the environment variable "critical_value" value mentioned in the code below.
So when I push the code below, I get the following broken DAG error:
Broken DAG: [/usr/local/airflow/dags/sync/dags/example_dag.py] invalid literal for int() with base 10: '{{ task_instance.try_number }}'
This is an error we get when we try to convert a string to an int. In this case, at runtime '{{ task_instance.try_number }}' will take the appropriate value of the try number of the run. I have tried also removing the int() converter but then get an error of comparing an integer and a string.
Any help will be immensely appreciated! Thank you in advance.
# Required DAG arguments
default_args = {
'owner':airflow,
"depend_on_past":False,
'start_date':datetime(2021, 6, 4),
"retries": 3
}
# Instantiate DAG
with DAG(
example_dag,
default_args=default_args,
schedule_interval=#once,
) as dag:
# Run python script of loader job
full_script_task = KubernetesPodOperator(
name=example_task,
task_id=example_task_id,
namespace=Variable.get("kubernetes_namespace"),
image=some_image,
configmaps=CONFIG_MAPS,
secrets=SECRETS,
volumes=[logging_volume],
volume_mounts=[logging_volume_mount],
env_vars={
"critical_value": "value_if_true" if int('{{ task_instance.try_number }}') >= 3 else "value_if_false"
}
)

Aiflow can handle the if expression and correct typing evaluation in Jinja but the whole value needs to be in the template syntax. Try this (env_vars needs to be a list based on the documentation of the KubernetesPodOperator otherwise the value will be a stringified list):
env_vars=[
{
"critical_value": "{{ 'value_if_true' if task_instance.try_number >= 3 else 'value_if_false' }}"
}
],

As we confirmed with your test, you cannot use jinja templated variables in an if clause inside the parameter. You tried this and the variable was read properly:
env_vars={
"critical_value": "{{ task_instance.try_number }}"
}
What you can do is to process the try_number inside the operator and instead of acting upon the critical_value, just use the try_number.

Related

AWS CDK - StepFunction - Modify input before passing to the next step

I have a 3 step state machine for a step function.
InputStep -> ExecuteSparkJob -> OutputLambda
ExecuteSparkJob is a glue task. Since it cannot pass its output to the step function, it writes it output to an S3 bucket. OutputLambda reads it from there and passes it on to the step function.
The idea of InputStep is simply to define a common S3 URI that the following steps can use.
Below is the code I have for the Input Step.
const op1 = Data.stringAt("$.op1");
const op2 = Data.stringAt("$.op2");
const inputTask = new Pass(this, "Input Step", {
result: Result.fromString(this.getURI(op1, op2)),
resultPath: "$.s3path"
});
getURI(op1: string, op2: string): string {
return op1.concat("/").concat(op2).concat("/").concat("response");
}
However, the string manipulation that I am doing in getURI is not working. The values in inputTask.result are not being substituted by the value in Path.
This is the input and output to the Input Step
{
"op1": "test1",
"op2": "test2"
}
Output
{
"op1": "test1",
"op2": "test2"
"responsePath": "$.op1/$.op2/response"
}
Is it possible to do some string manipulations using parameters in the Path in Step Function definition? If yes, what am I missing?
Thanks for your help.
You can use one or more EvaluateExpression Tasks - it's still a bit clunky.
You can find examples here.
API doc here.
Use a Lambda function instead of a Pass state to build the string.
Step Functions doesn't currently support string concatenation with reference paths. The Result field of a Pass state doesn't allow reference paths either. It has to be static value.
The Pass state's Parameters field supports the intrinsic functions and substitutions you need to do this natively, without a Lambda task. The Result field doesn't.
Compose a string from the execution inputs with the Format intrinsic function:
const inputTask = new Pass(this, "Input Step", {
parameters: {
path: JsonPath.format(
"{}/{}/response",
JsonPath.stringAt("$.op1"),
JsonPath.stringAt("$.op2")
),
},
outputPath: "$.s3",
});
The resulting string value test1/test2/response will be output to $.s3.path.

Azure Pipeline dynamic parameters to template file from YAML pipeline

I am currently working with Azure Devops Build Pipelines, and am trying to call a template file to do some tasks from my build yaml.
I am facing some difficulties to pass parameters to the template file. Let assume that this is my template file (simplified) which works fine :
parameters:
iterations: []
steps:
- ${{ each i in parameters.iterations }}:
- task: PowerShell#2
displayName: "Get key values ${{i}}"
name: getKeyValues_${{i}}
inputs:
targetType: 'inline'
script: |
$item = "${{i}}"
Write-Host "item : $($item)"
$keyVal = $item -split "_"
Write-Host $keyVal
Write-Host "key: $($keyVal[0]) | value: $($keyVal[1])"
echo "##vso[task.setvariable variable=key;isOutput=true]$($keyVal[0])"
echo "##vso[task.setvariable variable=value;isOutput=true]$($keyVal[1])"
So I want my iterations parameter contain something like this :
iterations: ["1_60", "2_40"]
Inside my Yaml pipeline I have the following code(also simplified) :
Not working scenario
- task: PowerShell#2
displayName: Calculate iterations for $(copies) copies
name: calculateIterations
inputs:
targetType: 'inline'
script: |
# Do some stuf here to get the arrow below from int value = 100
$iterations = ["1_60, "2_40"]
echo "##vso[task.setvariable variable=iterations;isOutput=true]$($iterations)"
- template: container-template.yml
parameters:
iterations: $(calculateIterations.iterations)
Working scenario
- task: PowerShell#2
displayName: Calculate iterations for $(copies) copies
name: calculateIterations
inputs:
targetType: 'inline'
script: |
# Do some stuf here to get the arrow below from int value = 100
$iterations = ["1_60, "2_40"]
echo "##vso[task.setvariable variable=iterations;isOutput=true]$($iterations)"
- template: container-template.yml
parameters:
iterations: ["1_60, "2_40"]
As you can see, the problem is that I am unable to use the output variable of my script to pass it as parameter to my template.
When I run the not working scenario I have the following error :
I have found this post, but no solutions yet...
As what 4c74356b41 said, this is the dilemma at present. In another word, the Not working scenario you mentioned does not support to achieve until now.
Now, we must let the template know the clear text at compile time. Because during this compile time, we have difficulty to do 2 or more things in one step at same time, especially contain compiling variable value, passing to corresponding template dynamic arguments and etc.
Expected a sequence or mapping. Actual value
'$(calculateIterations.iterations)'
More detailed, during compile time (after you click Run but before pipeline start truely):
1) Firstly we map the value that come from YAML pipeline, to make sure the - ${{ each i in parameters.iterations }} has clear value to start.
2) After it is done, then parse exact value on name: getKeyValues_${{i}} in script order.
In your scenario, it cannot even satisfy the first step since what you passed is a variable, and we do not has parse value process here. That's why you saw error said Expected a sequence or mapping. Actual value '$(calculateIterations.iterations)'.
Another expression of this error message is: we(template) are looking forward to exact value(s) to map our dynamic parameters, but what you give is a unrecognized content $(calculateIterations.iterations). Sorry, we cannot start to run.

Create Inventory Adjustment giving an error

I am trying to create an inventory adjustment from a MAP/REDUCE script. the record from which i am setting the values is getting ftom a search in getInputData(). In map function i am loading that custom record and setting these values
var newcase_inv_Adj = record.create({type:'inventoryadjustment',isDynamic:true});
newcase_inv_Adj.setValue({fieldId:'account',value:creel_account});
newcase_inv_Adj.setValue({fieldId:'custbody_cp_adjreasoncode',value:creel_reasoncode});
newcase_inv_Adj.setValue({fieldId: 'custbody_c_from', value: name});
newcase_inv_Adj.selectNewLine({sublistId:'inventory'});
newcase_inv_Adj.setCurrentSublistValue({sublistId:'inventory',fieldId:'item',value:creel_item});
newcase_inv_Adj.setCurrentSublistValue({sublistId:'inventory',fieldId:'location',value:creellocation});
newcase_inv_Adj.setCurrentSublistValue({sublistId:'inventory',fieldId:'adjustqtyby',value:creel_weigh_oh});
var create_inv_detail = newcase_inv_Adj.getCurrentSublistSubrecord({sublistId: 'inventory',fieldId: 'inventorydetail'});
create_inv_detail.selectNewLine({sublistId:'inventoryassignment' });
create_inv_detail.setCurrentSublistValue({sublistId:'inventoryassignment',fieldId:'receiptinventorynumber',value: creel_casenumber});
create_inv_detail.setCurrentSublistText({sublistId:'inventoryassignment',fieldId:'binnumber',value: creel_bin });
create_inv_detail.setCurrentSublistValue({sublistId:'inventoryassignment',fieldId:'quantity',value: creel_weigh_oh });
create_inv_detail.commitLine('inventoryassignment');log.debug("N","commited inventoryassignment");
newcase_inv_Adj.commitLine({sublistId:'inventory'});log.debug("N","commited inventory");
var invAdjID = newcase_inv_Adj.save();log.debug("N","invAdjID : "+invAdjID);
But, I am getting this error as {"type":"error.SuiteScriptError","name":"UNEXPECTED_ERROR"
Is it because of Map/Reduce script?
I don't find anything wrong in the code.
Few suggestions:
- Follow the sequence: Set subsidiary value before you setvalue for account or any other mandatory field.
- First try with inventory items without serial numbers and see if you continue to face same problem.
- You need to make sure you all your variables has valid value (not null or undefined).
- First try doing this in a schedule script.

How to get the name of the arguments passed into a block?

In Ruby, you can do this:
prc = lambda{|x, y=42, *other|}
prc.parameters #=> [[:req, :x], [:opt, :y], [:rest, :other]]
In particular, I'm interested in being able to get the names of the parameters which are x and y in the above example.
In Crystal, I have the following situation:
def my_method(&block)
# I would like the name of the arguments of the block here
end
How would one do this in Crystal?
While this already sounds weird in Ruby, there's no way to do it in Crystal since in your example the block already takes no arguments. The other problem is that such information is already lost after compilation. So we would need to access this at compile time. But you cannot access a runtime method argument at compile time. However you can access the block with a macro and that then even allows arbitrary signatures of the block without explicitly giving them:
macro foo(&block)
{{ block.args.first.stringify }}
end
p foo {|x| 0 } # => "x"
To expand on the great answer by Jonne Haß, the equivalent of the Ruby parameters method would be something like this:
macro block_args(&block)
{{ block.args.map &.symbolize }}
end
p block_args {|x, y, *other| } # => [:x, :y, :other]
Note that block arguments are always required in Crystal and can't have default values.

OTRS Bug parsing zeroes on email

I have just setup my first installation of OTRS and I was setting up my first filtering when I found a 'possible' bug:
When the regex matches a zero (as in '0') and I try to assign it to a DynamicField (type text) using '[***]' I get an empty value (maybe it believes it's NULL??)
The actual regex works:
Node users: (.*?)\,
And what I'm parsing on the body:If I have a '1' it works fine
"...3.250. Node users: 1, Backend use..."
log:
Filter: 'variable name 4' Set param 'X-OTRS-DynamicField-variable4' to '1'
If I have a '0' OTRS doesn't see a value
"...3.250. Node users: 0, Backend use..."
log:
Filter: 'variable name 4' Set param 'X-OTRS-DynamicField-variable4' to ''
Any ideas? I'm no perl expert, but it looks like it's both successfully matching and unsuccessfully inserting in DB.
Without seeing some code, I'm left with guessing. My guess would be there's some logic:
if ( $variable ) { # do something };
Which means when the variable is present, but 0 it's evaluating as false when it shouldn't be. This can be easily fixed by tracking down the offending line and changing it to:
if ( defined $variable ) { #do something };