We have a pipeline job with 2 stages and we have declared an environment variable.
We are changing value of environment variable in stage 1 and now I want to used changed value of environment variable in stage 2.
But it is not using a changed value in stage 2.
We have tried using environment variable on Pipeline level, job level and stage level.
Can anyone please help ?
There isn't a way to pass an environment variable from stage 1 to stage 2. The only way that I have been dealing with this situation is to echo the environment variable to a text file and pas the text file along as an artifact between the two stages.
Related
I am using the step functions data science SDK using python. I have a task that runs every day and the path of the data that is to be accessed in certain steps of the step functions keeps changing every day as it has the date parameter.
How can I pass the date parameter when I execute the step function and
use it so that I can access new data every day automatically.
This is an example of a step I am adding to the workflow.
etl_step = steps.GlueStartJobRunStep(
'Extract, Transform, Load',
parameters={"JobName": execution_input['GlueJobName'],
"Arguments":{
'--S3_SOURCE': data_source,
'--S3_DEST': 's3a://{}/{}/'.format(bucket, project_name),
'--TRAIN_KEY': train_prefix + '/',
'--VAL_KEY': val_prefix +'/'}
}
)
I want to add the date variable to the S3_DEST. If I use execution_input, the type isn't string so I cannot concatenate it for the path.
Edit
If the date is a datetime object you can use datetime.strftime('%Y-%m-%d')` to output it as a string.
Original
Step functions support input into them.
If you're using the SDK for start_execution then you can use the input parameter.
If you have CloudWatch event you can specify a constant from the console.
I have an API request that I need to run in Postman-Collection-Runner thru multiple iterations. The API request uses Variable.
How can I make this variable to automatically increase with each iteration (or maybe set the iteration value as another Variable)?
If I understand your question correctly, you would like to assign different values to a variable in the request in different iterations which is achievable in 2 ways.
a) Using data files
https://learning.getpostman.com/docs/postman/collection_runs/working_with_data_files/
The data files could be in JSON or CSV format. Unfortunately, there is no way in Postman to tie the variable values to another variable unless you want to do it in a hacky way!
b) Pre-request & Tests scripts
1- Initialise the environment variable in the Pre-request Scripts like this:
var value = pm.environment.get("var");
if( !value) {
pm.environment.set("var", 1);
}
2- Increment the variable value in Tests
var value = pm.environment.get("var");
pm.environment.set("var", value+1);
This creates an environment variable and increments it after each iteration. depending on how you structure your collection you might need to consider flushing/resetting the environment variable to be ready for the next run
It worth mentioning that Pre-request Scripts and Tests running before and after the requests respectively, so you can write any scripts that would like to run after the request in the Tests. It shouldn't be necessarily a test!
1. Using Global pm.* functions and Variables in Pre-Request Scripts/Tests
Pre-Request script - runs before executing the request
Tests - runs after executing the request
a.
pm.variables.set("id", pm.info.iteration);
Ex: example.com/{{id}}/update gives
example.com/0/update
example.com/1/update etc...
Number of iterations is set in Collection Runner. pm.info.iteration key has the current iteration number, starting at 0.
b.
var id = +pm.globals.get("id");
pm.globals.set("id", ++id);
The variables can be in any scope - globals/collection/environment/local/data.
In Collection Runner, check the Keep Variable Values checkbox, to persist the final value of the variable in the session (here id).
Note: If the variable is accessed via individual scopes (via pm.globals.* or pm.environment.* or pm.collectionVariables.*), then the mentioned checkbox should be toggled as required. Else if accessed via local scope (pm.variables.*), the value will not be persisted irrespective of the checkbox.
Ex: Same as above
More on variables and scoping
2. Using Dynamic variables
These variables can be used in case random values are needed or no specific order is necessary.
a. $randomInt - gives a random Integer within 1 - 1000.
Ex: example.com/{{$randomInt}}/update gives
example.com/789/update,
example.com/265/update etc...
b. $timestamp - gives current UNIX timestamp in seconds.
Ex: example.com/{{$timestamp}}/update gives
example.com/1587489427/update
example.com/1587489434/update etc...
More on Dynamic variables
Using Postman 7.22.1, while answering this. New methods may come in future.
Using Stata, I define a local macro (macro_name) as a variable (macro_variable) in one data file.
After reading in a new file (in the same do file), I'm no longer able to reference that macro.
Instead, I receive the error:
. di `macro_name'
macro_variable not found
I am learning how to use macros, so please bear with me. However, shouldn't I be able to still display or call on that macro in a single do file even if I load in a new data set?
For example:
use "newdata.dta", clear
This problem occurs regardless of whether I define the macro as a global or local. Additionally, I attempted to solve the problem by creating a separate locals.do file that include in the preamble of my master do file as:
include locals.do
But, I still receive the error listed above.
Do macros (local or global) disappear immediately upon reading in a new file? That doesn't seem right based on what I've read.
Thanks in advance for any clarification.
Consider the following, which points to the source of your problem, and in the last command, reproduces precisely the error message you received.
. do "/var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//SD08491.000000"
. local macro_name macro_variable
. macro list _macro_name
_macro_name: macro_variable
. display "`macro_name'"
macro_variable
. display `macro_name'
macro_variable not found
r(111);
end of do-file
Added in edit: The above was run from the do-file editor window. When I instead launch Stata and paste the four commands into the command window, running them a line at a time, the following are what results.
. local macro_name macro_variable
. macro list _macro_name
_macro_name: macro_variable
. display "`macro_name'"
macro_variable
. display `macro_name'
macro_variable not found
r(111);
.
At the risk of over-explaining, the point to my original answer is that the error message the displayed in the original post, and in the final command in both of my examples, was due to the failure to include quotation marks in the display command, which caused display to believe that "macro_variable", which was the value assigned to the local macro "macro_name" was not a character string constant, but rather a variable name or scalar, and display was unable to locate a variable or scalar by that name.
Let me add as a bonus explanation that the use of locals.do described in the original post has no hope of working, because local macros are local to the do-file in which they are executed, and vanish at the termination of that do-file. In particular, if you submit a local command by selecting a subset of the lines in the do-file editor window, those lines are copied into a temporary do-file and the values of the local macros vanish at the termination of the temporary do-file.
Generalizing what I wrote in my comment above to Nick:
Macros only maintain the connection between the variable/varlist assigned to a macro name and, therefore, the variable/varlist to which the macro's name refers to must be in memory (i.e. the dataset that contains the variable/varlist has to be in memory) in order to access it via the macro.
Assigning a variable/varlist to a macro does not persist the actual value(s)/element(s) in memory, but rather maintain the connection between the variable/varlist and the macro name assigned to it/them.
I am setting up a dynamic model in Stata 13 by using the xtabond command. I need to add interaction between the lagged dependent variable and other variables, as attached here Formula
My attempts:
xtabond depvariable c.L1.depvariable#c.indvariable, lags(1) artests(2)
xtabond depvariable c.L1.depvariable*c.indvariable, lags(1) artests(2)
xtabond depvariable L1.depvariable*indvariable, lags(1) artests(2)
but they do not work.
Can someone please help me with the syntax? Or does some dirty alternative exist (for instance, creating interaction variable by hand)?
I am having trouble running a foreach loop. The loop runs without error but gives no output. Can someone tell me what they think might be going on? Many thanks in advance!
Here is the code:
cd "O:\RESEARCH\ikhilko\Subway Big Data project"
local datafiles : dir . files "*.txt"
foreach file in `datafiles' {
insheet using `file',
clear
insheet using `file',
drop v9-v43
save date1, replace
}
UPDATE:
Interestingly, the code runs when I just type it into the command line, rather than doing it from the .do file, any idea what might be going on there?
It is important to note that local macros are precisely that, i.e. defined and visible only locally.
Locally means within
the same interactive session
or
the same program
or
the same do file (or do file editor contents)
or
the same part of the do file (or ...) executed by selection
Locality is, it seems, biting you here. A local macro defined in one place is not visible in another. A local macro reference will evaluate to missing, i.e. an empty string, if the macro is not visible.
Some code for the debugging. display the contents of your local datafiles to see what's going into the loop:
local datafiles : dir . files "*.txt"
display `"`datafiles'"'
local wordx : word 1 of `datafiles'
display `"`wordx'"'
foreach file in `datafiles' {
display "`file'"
}
(The code does not format well in the comments section.)