Manipulating certain lines of a file in Perl - regex

This is basically a question regarding file manipulation. I'm working on a Perl script that will run as a cron job that sends an email when certain values shows up on a web page, but does so only once until the value resets (it's a page of reports containing audits that show an alert when a report requires attention). I've figured out how to parse the page for alerts, I know how to send emails via Perl, I'm just having a hard time figuring out the logic for WHEN to send the email. I gather that a separate file (alerts.txt) will need to be created to store the reports that had an alert during the previous execution of the script. In that case you have 4 scenarios to handle in the script. 1) The report has an alert and isn't in alerts.txt (needs to be added). 2) The report doesn't have an alert but is in alerts.txt from a previous execution of the script (needs to be removed). 3) The report doesn't have an alert and isn't in alerts.txt (do nothing). 4) The report has an alert and is already in alerts.txt (do nothing).
Is this the best way to go about this? If so, what's the best way to implement scenarios #1 and #2 above?

I am assuming you've chosen not to use a database - which is valid in some situations but you may change your mind as your application grows.
The simplest method, is to use the Storable modules which comes built in with perl. Note, this produces non human readable formats but can store quite complex data structures;
#!/usr/bin/env perl
use Storable;
# Create some alerts data, and store it in a data file
my $alerts = {};
$alerts->{report_01} = 'alert';
$alerts->{report_02} = 'ok';
store $alerts, 'alerts.data';
# Read it back
my $prev_alerts = retrieve('alerts.data');
foreach my $key (keys %$prev_alerts) {
print "$key=$prev_alerts->{$key}\n";
}
If you desire a human readable format there are various options that support file based storage;
ini; Config::Tiny
xml; XML::Simple
csv; Text::CSV

Related

How does multi-line logging work in Lambda -> CloudWatch

My multi-line logging events all end up multi-events - one event per line. According to the documentation:
Each call to LambdaLogger.log() results in a CloudWatch Logs event...
but then:
However, note that AWS Lambda treats each line returned by System.out
and System.err as a separate event.
Looking inside LambdaAppender's source code, it seems that it proceeds to log the event to System.out anyway. So does that mean multi-line messages will always be broken down into multiple event?
I have read about configuring the multi_line_start_pattern, but that seems only applicable when you get to deploy a log agent, which isn't accessible in Lambda.
[Edit] LambdaAppender logs to LambdaLogger which logs to System.out.
[Edit] I found some post where a workaround was suggested - use '\r' for the eol when printing the messages. This seems to work for messages that my code produces. Stack traces logged everywhere are still a problem.
[Edit] I have been using two workarounds:
Log complex data structures (e.g. sizable maps) in JSON. CloudWatch actually recognizes JSON strings in log events, and pretty print them.
Replace '\n' with '\r'. For stack traces I created a utility method (this is in Kotlin, but the idea is generic enough):
fun formatThrowable(t: Throwable): String {
val buffer = StringWriter()
t.printStackTrace(PrintWriter(buffer))
return buffer.toString().replace("\n", "\r")
}
I think in the long run a more ideal solution would be an Appender implementation that decorates ConsoleAppender, which would do the \r replacement on all messages passing through.
Best practice is to use json in your logs. Instead of sending multiline outputs, send a formatted json (regardless of the language you are using, you will find a lib that already does that for you)
You will be amazed how easy it gets to browse your logs from there. For instance, aws cloudwatch insights automatically detects your fields, it allow to parse them and query them within seconds
I suggest to use the project slf4j-simple-lambda and to refer to this blog for more explanations.
Using slf4j and slf4j-simple-lambda is solving elegantly your problem and the solution stay lightweight. The project includes the usage of the parameter org.slf4j.simpleLogger.newlineMethod which is there to solve this problem. By default, its value is auto and should be able to detect automatically the need for manual newline handling.
Discloser: I am co-author of slf4j-simple-lambda and author of the blog.

Create complex-type variables for a Web Service Task

In SSIS, I already have a Web Service Task using a WSDL for sending SMS. I am indeed able to send SMS using this task.
I want supply values to this task from the database, such as Mobile Number, Message body, User ID, etc.
How can I create a complex type user variable that can be passed as input to a Web Service task?
It looks like the only answer is to change the web service to accept only simple types as parameters. I have scoured the web and there seems to be no way to dynamically create complex types for consumption by the input values in the web service task.
The more 'easy' way is to use the script component for bypassing variables to a web service. Check http://amolpandey.com/2016/09/26/ssis-script-task-to-obtain-geo-cordinates-from-address-text-via-google-api/ & http://www.sqlmusings.com/2011/03/25/geocode-locations-using-google-maps-v3-api-and-ssis/.
Tested and working. Using this task you can bypass the SSIS variables/parameters.
Example: Getting ID, addreess, zipcode, city, country from a table with an execute SQL Task. Change Resultset:Full result set on General tab. Then on resultset tab add Result_Name:0 & Variable_Name: User::YourObject. Then the next task will be a Forlooptask editor (Foreach ADO Enumerator ,Collection tab - Ado object source variable: User::YourObject, enumeration mode: rows in the first table, variable Mapping tab - Variable User::Id, 0 | address,1 etc.). Inside the Forlooptask editor you add a data flow task, which the source of this task will be a script component. If you be more specific about your logic,we may assist you more.
Okay so I came across the same problem. I needed to pass one parameter as complex type.
Create a Web Service task in your package.
Fill all the needed properties at General tab: HttpConnection and WSDFile
Fill properties in Input tab: Service, Method
Below click on Value, manually enter the value you need (mine is 2021-11-15)
Deploy and execute package to be sure everything is OK
After this easy steps go into folder where package is localated. Right click on package file (Package.dtsx) and select Open with > Notepad. With find function in notepad search the value you manually inserted.
The part which we are looking for looks in my case like this
<WSTask:ComplexValue>
<WSTask:ComplexProperty
WSTask:Name="date"
WSTask:Datatype="dateTime"
WSTask:ParamType="Primitive">
<WSTask:PrimitiveValue>2021-11-15</WSTask:PrimitiveValue>
</WSTask:ComplexProperty>
</WSTask:ComplexValue>
Finally I found what I was looking for. Now for the second part I needed to be that parameter changing by current date when I execute that package. In powershell I managed to write a code that change date part in string: <WSTask:PrimitiveValue>2021-11-15</WSTask:PrimitiveValue> to current date, everytime when the package is executed. The code looks like this:
$Now = Get-Date -Format "yyyy-MM-dd"
$Yesterday = (Get-Date).AddDays(-1).ToString("yyyy-MM-dd")
$file = ((Get-Content -path "C:\Package.dtsx" -Raw) -replace "<WSTask:PrimitiveValue>$Yesterday</WSTask:PrimitiveValue>", "<WSTask:PrimitiveValue>$Now</WSTask:PrimitiveValue>")
[System.IO.File]::WriteAllText("C:\Package.dtsx",$file)
# This part will execute the package #
dtexec.exe /f "C:\Package.dtsx"
After all this, I planned this script in Task Scheduler and it works.
In my case changing the type of request from complex to simple wasn't an option and all I needed was just one parameter to pass.
Hopes it gonna help somebody.

Load testing with SOAP UI

I have a SOAP UI 4.5.1, I have made a load test, it is working fine. My problem is that I run the same request every time and I need to change the values of the soap request I am sending.
For e.g. I have a block of my soap request:
<ns:Assessment>
<ns:Project>
<ns:ProviderId>SHL</ns:ProviderId>
<ns:ProjectId>SampleAssessment</ns:ProjectId>
</ns:Project>
</ns:Assessment>
Provider ID: SHL
Project ID: SampleAssessment
Is there a way to make those values changing from some kind of interval?
For e.g.: Provider IDs [SHL, SLH, LHS]
Project IDs [SampleAssessment, TestAssessment, AnotherAssessment]
And with a load test I am making three request so that for the first request values looks like this:
<ns:Assessment>
<ns:Project>
<ns:ProviderId>SHL</ns:ProviderId>
<ns:ProjectId>SampleAssessment</ns:ProjectId>
</ns:Project>
</ns:Assessment>
for the second like this:
<ns:Assessment>
<ns:Project>
<ns:ProviderId>SLH</ns:ProviderId>
<ns:ProjectId>TestAssessment</ns:ProjectId>
</ns:Project>
</ns:Assessment>
and so on...
Is there a way to make this happen with SOAP UI?
From my experience, you will need to use a Groovy Script step.
For example, if you have a step before your request that is a script, you can use something like:
context.setProperty("ProviderId", "SHL")
Then in your request, use:
<ns:ProviderId>${ProviderId}</ns:ProviderId>
Of course, this doesn't buy you much by itself. There are few ways to vary what the context.setProperty("ProviderId", "SHL") line will set. You can create a collection and iterate over it using something like:
def providers = ['ABC', 'DEF', 'GHI', 'JKL']
providers.each() {
context.setProperty("ProviderId", it)
testRunner.runTestStepByName( "nameofteststep" )
}
Where "nameofteststep" is the name of the Soap Request test step. This might sound odd, but if you right click the test step and disable it, the groovy script will still be able to execute it but it will not run sequentially. By that I mean that the groovy script will run it 4 times, but it won't run a fifth time when the script is complete because it is after the script. Then you just need to keep in mind that each load test thread makes four requests, but I am pretty sure that the SoapUI statistics will take this into account for you... might want to keep an eye out for it, though.
Alternatively, you could check the 'threadIndex' and set a the context variable based on that. A bit like this here: Log ThreadCount.
You could also use a collection without a loop and increment an index that you save as a testcase property and send the string corresponding to the index.
Personally, I think the first way is the most straightforward but I can provide an example of the other ones if you like.
There is a simple way of doing this without writing a groovy script.
After creating a test case you should include the below test steps:
1-Data source
2-Request
3-Loop
Data source will read an excel file (or other data source methods such as XML, groovy, JDBC, gird .. however the excel is the simplest one).
You should include the datas (that you need to change within the request)
Within the test request you need the right click and select "get data" . please notice that your test request should be as below
<ns:ProviderId>${ProviderId}</ns:ProviderId>
Then the last step is the "Loop" . This for returning to the first step until the data ends.
I hope this helps.

Mediawiki mass user delete/merge/block

I have 500 or so spambots and about 5 actual registered users on my wiki. I have used nuke to delete their pages but they just keep reposting. I have spambot registration under control using reCaptcha. Now, I just need a way to delete/block/merge about 500 users at once.
You could just delete the accounts from the user table manually, or at least disable their authentication info with a query such as:
UPDATE /*_*/user SET
user_password = '',
user_newpassword = '',
user_email = '',
user_token = ''
WHERE
/* condition to select the users you want to nuke */
(Replace /*_*/ with your $wgDBprefix, if any. Oh, and do make a backup first.)
Wiping out the user_password and user_newpassword fields prevents the user from logging in. Also wiping out user_email prevents them from requesting a new password via email, and wiping out user_token drops any active sessions they may have.
Update: Since I first posted this, I've had further experience of cleaning up large numbers of spam users and content from a MediaWiki installation. I've documented the method I used (which basically involves first deleting the users from the database, then wiping out up all the now-orphaned revisions, and finally running rebuildall.php to fix the link tables) in this answer on Webmasters Stack Exchange.
Alternatively, you might also find Extension:RegexBlock useful:
"RegexBlock is an extension that adds special page with the interface for blocking, viewing and unblocking user names and IP addresses using regular expressions."
There are risks involved in applying the solution in the accepted answer. The approach may damage your database! It incompletely removes users, doing nothing to preserve referential integrity, and will almost certainly cause display errors.
Here a much better solution is presented (a prerequisite is that you have installed the User merge extension):
I have a little awkward way to accomplish the bulk merge through a
work-around. Hope someone would find it useful! (Must have a little
string concatenation skills in spreadsheets; or one may use a python
or similar script; or use a text editor with bulk replacement
features)
Prepare a list of all SPAMuserIDs, store them in a spreadsheet or textfile. The list may be
prepared from the user creation logs. If you do have the
dB access, the Wiki_user table can be imported into a local list.
The post method used for submitting the Merge & Delete User form (by clicking the button) should be converted to a get method. This
will get us a long URL. See the second comment (by Matthew Simoneau)
dated 13/Jan/2009) at
http://www.mathworks.com/matlabcentral/newsreader/view_thread/242300
for the method.
The resulting URL string should be something like below:
http: //(Your Wiki domain)/Special:UserMerge?olduser=(OldUserNameHere)&newuser=(NewUserNameHere)&deleteuser=1&token=0d30d8b4033a9a523b9574ccf73abad8%2B\
Now, divide this URL into four sections:
A: http: //(Your Wiki domain)/Special:UserMerge?olduser=
B: (OldUserNameHere)
C: &newuser=(NewUserNameHere)&deleteuser=1
D: &token=0d30d8b4033a9a523b9574ccf73abad8%2B\
Now using a text editor or spreadsheet, prefix each spam userIDs with part A and Suffix each with Part C and D. Part C will include the
NewUser(which is a specially created single dummy userID). The Part D,
the Token string is a session-dependent token that will be changed per
user per session. So you will need to get a new token every time a new
session/batch of work is required.
With the above step, you should get a long list of URLs, each good to do a Merge&Delete operation for one user. We can now create a
simple HTML file, view it and use a batch downloader like DownThemAll
in Firefox.
Add two more pieces " Linktext" to each line at
beginning and end. Also add at top and at
bottom and save the file as (for eg:) userlist.html
Open the file in Firefox, use DownThemAll add-on and download all the files! Effectively, you are visiting the Merge&Delete page for
each user and clicking the button!
Although this might look a lengthy and tricky job at first, once you
follow this method, you can remove tens of thousands of users without
much manual efforts.
You can verify if the operation is going well by opening some of the
downloaded html files (or by looking through the recent changes in
another window).
One advantage is that it does not directly edit the
MySQL pages. Nor does it require direct database access.
I did a bit of rewriting to the quoted text, since the original text contains some flaws.

How to record Script and keywordTest in Testcomplete together?

My question is, Can TestComplete record script and keywordTest together? Because when I test my application, it either record script or keywordTest but not both in parallel. How Can I do this?
You can switch between keyword test recording and script recording on the fly by clicking the corresponding test type on the Recording toolbar:
(Image taken from http://www.automatedqa.com/products/testcomplete/record-playback/.)
Can we record both types in parallel? i.e when it records KeywordTest it also record the corresponding script.
No, this is not possible. You can, however, manually convert the recorded keyword test into its script equivalent.