Friends:
I have to process a CSV file, using Perl language and produce an Excel as output, using the Excel::Writer::XSLX module. This is not a homework but a real life problem, where I cannot download whichever Perl version (actually, I need to use Perl 5.6), or whichever Perl module (I have a limited set of them). My OS is UNIX. I can also use (embedding in Perl) ksh and csh (with some limitation, as I have found so far). Please, limit your answers to the tools I have available. Thanks in advance!
Even though I am not a Perl developer, but coming from other languages, I have already done my work. However, the customer is asking for extra processing where I am getting stuck on.
1) The stones in the road I found are coming from two sides: from Perl and from Excel particular styles of processing data. I already found a workaround to handle the Excel, but -as mentioned in the subject- I have difficulties while processing zeroes found in CSV input file. To handle the Excel, I am using the '0 way which is the final way for data representation that Excel seems to have while using the # formatting style.
2) Scenario:
I need to catch standalone zeroes which might be present in whichever line / column / cell of the CSV input file and put them as such (as zeroes) in the Excel output file.
I will go directly to the point of my question to avoid loosing your valuable time. I am providing more details after my question:
Research and question:
I tried to use Perl regex to find standalone "0" and replace them by whichever string, planning to replace them back to "0" at the end of processing.
perl -p -i -e 's/\b0\b/string/g' myfile.csv`
and
perl -i -ple 's/\b0\b/string/g' myfile.csv
Are working; but only from command line. They aren't working when I call them from the Perl script as follows:
system("perl -i -ple 's/\b0\b/string/g' myfile.csv")
Do not know why... I have already tried using exec and eval, instead of system, with the same results.
Note that I have a ton of regex that work perfectly with the same structure, such as the following:
system("perl -i -ple 's/input/output/g' myfile.csv")
I have also tried using backticks and qx//, without success. Note that qx// and backticks have not the same behavior, since qx// is complaining about the boundaries \b because of the forward slash.
I have tried using sed -i, but my System is rejecting -i as invalid flag (do not know if this happens in all UNIX, but at least happens in the one at work. However is accepting perl -i).
I have tried embedding awk (which is working from command line), in this way:
system `awk -F ',' -v OFS=',' '$1 == \"0\" { $1 = "string" }1' myfile.csv > myfile_copy.csv
But this works only for the first column (in command line) and, other than having the disadvantage of having extra copy file, Perl is complaining for > redirection, assuming it as "greater than"...
system(q#awk 'BEGIN{FS=OFS=",";split("1 2 3 4 5",A," ") } { for(i in A)sub(0,"string",$A[i] ) }1' myfile.csv#);
This awk is working from command line, but only 5 columns. But not in Perl using #.
All the combinations of exec and eval have also been tested without success.
I have also tried passing to system each one of the awk components, as arguments, separated by commas, but did not find any valid way to pass the redirector (>), since Perl is rejecting it because of the mentioned reason.
Using another approach, I noticed that the "standalone zeroes" seem to be "swallowed" by the Text::CSV module, thus, I get rid off it, and turned back to a traditional looping in csv line by line and a spliter for commas, preserving the zeroes in that way. However I found the "mystery" of isdual in Perl, and because of the limitation of modules I have, I cannot use the Dumper. Then, I also explored the guts of binaries in Perl and tried the $x ^ $x, which was deprecated since version 5.22 but valid till that version (I said mine is 5.6). This is useful to catch numbers vs strings. However, while if( $x ^ $x ) returns TRUE for strings, if( !( $x ^ $x ) ) does not returns TRUE when $x = 0. [UPDATE: I tried this in a devoted Perl script, just for this purpose, and it is working. I believe that my probable wrong conclusion ("not returning TRUE") was obtained when I did not still realize that Text::CSV was swallowing my zeroes. Doing new tests...].
I will appreciate very much your help!
MORE DETAILS ON MY REQUIREMENTS:
1) This is a dynamic report coming from a database which is handover to me and I pickup programmatically from a folder. Dynamic means that it might have whichever amount of tables, whichever amount of columns in each table, whichever names as column headers, whichever amount of rows in each table.
2) I do not know, and cannot know, the column names, because they vary from report to report. So, I cannot be guided by column names.
A sample input:
Alfa,Alfa1,Beta,Gamma,Delta,Delta1,Epsilon,Dseta,Heta,Zeta,Iota,Kappa
0,J5,alfa,0,111.33,124.45,0,0,456.85,234.56,798.43,330000.00
M1,0,X888,ZZ,222.44,111.33,12.24,45.67,0,234.56,0,975.33
3) Input Explanation
a) This is an example of a random report with 12 columns and 3 rows. Fist row is header.
b) I call "standalone zeroes" those "clean" zeroes which are coming in the CSV file, from second row onwards, between commas, like 0, (if the case is the first position in the row) or like ,0, in subsequent positions.
c) In the second row of the example you can read, from the beginning of the row: 0,J5,alfa,0, which in this particular case, are "words" or "strings". In this case, 4 names (note that two of them are zeroes, which required to be treated as strings). Thus, we have a 4 names-columns example (Alfa,Alfa1,Beta,Gamma are headers for those columns, but only in this scenario). From that point onwards, in the second row, you can see floating point (*.00) numbers and, among them, you can see 2 zeroes, which are numbers. Finally, in the third line, you can read M1,0,X888,Z, which are the names for the first 4 columns. Note, please, that the 4th column in the second row has 0 as name, while the 4th column in the third row has ZZ as name.
Summary: as a general picture, I have a table-report divided in 2 parts, from left to right: 4 columns for names, and 8 columns for numbers.
Always the first M columns are names and the last N columns are numbers.
- It is unknown which number is M: which amount of columns devoted for words / strings I will receive.
- It is unknown which number is N: which amount of columns devoted for numbers I will receive.
- It is KNOWN that, after the M amount of columns ends, always starts N, and this is constant for all the rows.
I have done a quick research on Perl boundaries for regex ( \b ), and I have not found any relevant information regarding if it applies or not in Perl 5.6.
However, since you are using and old Perl version, try the traditional UNIX / Linux style (I mean, what Perl inherits from Shell), like this:
system("perl -i -ple 's/^0/string/g' myfile.csv");
The previous regex should do the work doing the change at the start of the each line in your CSV file, if matches.
Or, maybe better (if you have those "standalone" zeroes, and want avoid any unwanted change in some "leading zeroes" string):
system("perl -i -ple 's/^0,/string,/g' myfile.csv");
[Note that I have added the comma, after the zero; and, of course, after the string].
Note that the first regex should work; the second one is just a "caveat", to be cautious.
I'm converting some Access VBA functionality to DB2 and found a vital difference. VBA lets you specify the starting point in the character string you're working on. DB2 doesn't have that option. It starts from position 1 and replaces whatever you want to be replaced in the whole string. How can I make DB2 start the replace at a specified place in the string? For example, my string is "Incongruent Plastics Incorporated" and I want to replace the second "Inc" at position 22 with "Inc". I'm doing this in a WHILE loop, going through long strings, replacing parts of them until they are less than a specified maximum (15 or 30 depending on the field).
I looked at the Locate function, but I'm not sure that's right.
Replace(a.PAYEE_STD_NAME, B.FullWord, B.abbreviation, B.mLastWord)
Where a.PAYEE_STD_NAME is the string I'm looking at, B.FullWord is what I want to replace, B.abbreviation is what I want to replace it with, and B.mLastWord is the position where I want to start replacing. Something like Replace("Incongruent Plastics Incorporated","Incorporated","Inc",22)
I expect the characters to be replaced starting in the position I need, towards the back of the string, not in the beginning.
Thanks!
Not that good at DB2, but that limitation can generally be worked around by using SUBSTR
The equivalent of Replace(a.PAYEE_STD_NAME, B.FullWord, B.abbreviation, B.mLastWord) would be:
CONCAT(SUBSTR(a.PAYEE_STD_NAME, 1, B.mLastWord - 1), Replace(SUBSTR(a.PAYEE_STD_NAME, b.mLastWord), B.FullWord, B.abbreviation))
This assumes b.mLastWord is greater than 1, if it's 1 you can use a normal REPLACE.
Maybe consider using REGEXP_REPLACE https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.sql.ref.doc/doc/r0061496.html
and possibly consider recusrive SQL rather than looping logic
I am making an expect script to check memory usage and can only proceed to the next steps if the mem usage is less than 65%.
#!/usr/bin/expect -f
spawn telnet $serverip
send "show performance\r"
expect {
timeout { send_user "\nCPU Usage is too high.\n";exit 1}
"0-65%" # i need to expect 0-65%
}
then proceed to other commands.
output is :
CPU used MEM used RX(Kbps) TX(Kbps) RX(Kbps) TX(Kbps)
1.0% 51.2% 0.000 0.000 1.620 2.426
i need to make sure that mem used is less than 65%. How can i do this in EXPECT SCRIPT?
Thanks for the help. Its been killing me.
You have to use regular expression in the expect itself with the help of -re flag.
There can be two ways to get this done.
Match all the show performance command output till the prompt and then apply the tcl's legacy regexp in that output
Match only the required value (i.e. the mem used % value) alone directly.
I assume your device's prompt will be #. But, there are some devices whose prompt may vary. So, in order to handle this, we can come up with generalized prompt pattern as
set prompt "#|>|\\\$";
If your device's prompt is not available in this, then please include the same.
#!/usr/bin/expect -f
#This is a common approach for few known prompts
#If your device's prompt is missing here, then you can add the same.
set prompt "#|>|\\\$"; # We escaped the `$` symbol with backslash to match literal '$'
spawn telnet $serverip
# Add code for login here
expect -re $prompt; # Using '-re' flag here to match one one of the prompt.
# Your some other code here to something if any
# This is to clean up the previous expect_out(buffer) content
# So that, we can get the exact output what we need.
expect *;
send "show performance\r"; # '\r' used here to type 'return' .i.e new line
expect -re $prompt; # Matching the prompt with regexp
#Now, the content of 'expect_out(buffer)' has what we need
set output $expect_out(buffer);
# Applying the tcl's regexp here
if {[regexp {%\s+([^%]+)} $output ignore mem]} {
puts "Memory used : $mem"
}
I have used the pattern as {%\s+([^%]+)}. In your output, we have 2 percentage symbols. The first one corresponds to the CPU used and second one is for the memory used. So, basically I am trying to match the text % 51.2%
Let me decode the pattern.
% - to match the first percentage sign
\s+ - to match the more than one spaces.
[^%]+ - Match anything other than % (This is where we are getting the required value i.e. the value 51.2)
Then what is the need of parenthesis here ? Well ,that is for grouping. Expect will save the matched output into expect_out(0,string). For the nth sub match, it will be saved on expect_out(n, string). i.e. For 1st sub match expect_out(1,string) and for 2nd sub match expect_out(2,string) and so on. Expect will store all the matched and unmatched input to a variable called expect_out(buffer). So, that is the short story. One more thing might bother you. What is this expect *` doing here ? You can have a look at here to know more about the same.
That's all about the 1st way. Now, what about the second approach which I have described above ? That is bit more easy.
send "show performance\r";
expect {
-re {%\s+([^%]+)} { set mem $expect_out(1,string); puts "Memory used : $mem" }
timeout { puts timeout_happened }
}
This looks more comfortable and no need of applying separate regexp additionally. That is one advantage of it. You can use whichever you find it comfortable and whichever is much needed as per your requirement.
Once your get the value, you can simply compare it with a if loop if it is less than 65%.
Hey I'm trying to create a function that parses a string passed via a browser protocol. It's a "callto://" protocol and it is in this format: "callto://5551234567/" with the persons phone number inside there. I need to extract the number and pass it to another program that dials the number. The syntax for that other program is like this: "CallClerk.exe dial=5551234567=".
I'm a beginner to batch however, and can't figure out exactly what to do. Here's my current code:
#echo off
set var=%1
set number=theirphone
FindStr /R "callto://(..........)/" %var% > %number%
start C:\Program Files (x86)\CallClerk\CallClerk.exe dial=%number%=
Exit /B
Thanks for the help!
#echo off
FOR /f "tokens=2 delims=/" %%i IN ('echo %~1') DO start "" "C:\Program Files (x86)\CallClerk\CallClerk.exe" dial=%%i=
Exit /B
should work for you (untested) - assuming your input parameter is callto://5551234567/
Note the use of quoting - the .exe needs to be quoted since it contains a space in the path. The extra pair of quotes in the window-name. If you like, you could replace that pair with "Calling %%i". This parameter is optional, but inserting it ensures that START doesn't get confused between window-title, executable-name and parameter-to-executable.
This works to extract numbers from a string.
It uses two for loops, the first one gathers all the non-numeric characters and they are used as delimiters in the second for loop to gather the numerics and dial the number.
Strings of variable lengths can be handled, as long as all numbers are used in the desired telephone number.
If you want to keep the + as a valid telephone character then include it in the first for command in the delims with the numbers.
#echo off
set "var=callto://5551234567/"
for /f "delims=0123456789" %%a in ("%var%") do set "delims=%%a"
for /f "delims=%delims%" %%a in ("%var%") do (
start "" "C:\Program Files (x86)\CallClerk\CallClerk.exe" dial=%%a=
)
You should be able to use a regex along the lines of (?<=callto:\/\/)[\d]+(?=\/) to grab the number itself. This uses a positive look ahead and look behind to make sure you are matching at least one number that is preceded by the callto:// and followed by a /.
If you left it as something like callto:\/\/[\d]+\/, then it is matching the entire string and will return back with the callto text included. If you are intending to pass just the numbers along to the next part of you code, extract them using the look ahead to guarantee the before and after conditions are met.
I did a quick test using the strings you used in your example. You can see the regex in action here.
Intro
I work in a facility where we have microscopes. These guys can be asked to generate 4D movies of a sample: they take e.g. 10 pictures at different Z position, then wait a certain amount of time (next timepoint) and take 10 slices again.
They can be asked to save a file for each slice, and they use an explicit naming pattern, something like 2009-11-03-experiment1-Z07-T42.tif. The file names are numbered to reflect the Z position and the time point
Question
Once you have all these file names, you can use a regex pattern to extract the Z and T value, if you know the backbone pattern of the file name. This I know how to do.
The question I have is: do you know a way to automatically generate regex pattern from the file name list? For instance, there is an awesome tool on the net that does similar thing: txt2re.
What algorithm would you use to parse all the file name list and generate a most likely regex pattern?
There is a Perl module called String::Diff which has the ability to generate a regular expression for two different strings. The example it gives is
my $diff = String::Diff::diff_regexp('this is Perl', 'this is Ruby');
print "$diff\n";
outputs:
this\ is\ (?:Perl|Ruby)
Maybe you could feed pairs of filenames into this kind of thing to get an initial regex. However, this wouldn't give you capturing of numbers etc. so it wouldn't be completely automatic. After getting the diff you would have to hand-edit or do some kind of substitution to get a working final regex.
First of all, you are trying to do this the hard way. I suspect that this may not be impossible but you would have to apply some artificial intelligence techniques and it would be far more complicated than it is worth. Either neural networks or a genetic algorithm system could be trained to recognize the Z numbers and T numbers, assuming that the format of Z[0-9]+ and T[0-9]+ is always used somewhere in the regex.
What I would do with this problem is to write a Python script to process all of the filenames. In this script, I would match twice against the filename, one time looking for Z[0-9]+ and one time looking for T[0-9]+. Each time I would count the matches for Z-numbers and T-numbers.
I would keep four other counters with running totals, two for Z-numbers and two for T-numbers. Each pair would represent the count of filenames with 1 match, and the ones with multiple matches. And I would count the total number of filenames processed.
At the end, I would report as follows:
nnnnnnnnnn filenames processed
Z-numbers matched only once in nnnnnnnnnn filenames.
Z-numbers matched multiple times in nnnnnn filenames.
T-numbers matched only once in nnnnnnnnnn filenames.
T-numbers matched multiple times in nnnnnn filenames.
If you are lucky, there will be no multiple matches at all, and you could use the regexes above to extract your numbers. However, if there are any significant number of multiple matches, you can run the script again with some print statements to show you example filenames that provoke a multiple match. This would tell you whether or not a simple adjustment to the regex might work.
For instance, if you have 23,768 multiple matches on T-numbers, then make the script print every 500th filename with multiple matches, which would give you 47 samples to examine.
Probably something like [ -/.=]T[0-9]+[ -/.=] would be enough to get the multiple matches down to zero, while also giving a one-time match for every filename. Or at worst, [0-9][ -/.=]T[0-9]+[ -/.=]
For Python, see this question about TemplateMaker.