I have a bunch of Fortran 77 files that use a non-standard structure notation (described here):
STRUCTURE /item/
INTEGER id
CHARACTER(LEN=200) description
REAL price
END STRUCTURE
This is apparently a syntax that was used by some old compilers but was later standardized in Fortran 90 to the following:
TYPE item
INTEGER id
CHARACTER(LEN=200) description
REAL price
END TYPE
What I want to do is use sed or some other appropriate tool to automatically process my source files to locate all uses of the old syntax and patch them to use the updated syntax (i.e. change all instances of "structure" to "type" and remove the slashes). I'm sure that I can build up some set of regular expressions to do this, but I'm not skilled enough in that art to get started.
On a side note, there is a similar incompatibility with record notation, also described in the link:
RECORD /item/ pear, store_catalog(100)
becomes:
TYPE(item) pear, store_catalog(100)
I assume similar techniques can be used to fix those instances also.
I'm not very familiar with Fortran syntax so I don't know how to generalize this, but your example can be tackled with
sed 's%^STRUCTURE /\(.*\)/$%TYPE \1%;
s%^END STRUCTURE$%END TYPE%'
Note the use of % to separate the command (s, substitute), pattern and replacement. Normally you would use /, but that's part of the pattern to be matched.
Your RECORD problem should indeed have a similar solution.
I have some Fortran 77 source files that I'm trying to convert from a non-standard STRUCTURE and RECORD syntax to the standardized Fortran 90 TYPE syntax. One tricky aspect of this is the different way that structure members are addressed.
Non-standard:
s.member = 1
Standard:
s%member = 1
So, I need to trap all uses of periods in these sort of scenarios and replace them with % characters. Not too bad, except when you think about all of the ways that periods can be used (decimal points in numbers, filenames in include statements, punctuation in comments, Fortran 77 relational operators, maybe others). I've done some preprocessing to fix the relational operators to use the Fortran 90 symbols, and I don't really care about mangling the grammar of comments, but I haven't come up with a good approach to translate the . to % for the cases above. It seems like I should be able to do this with sed, but I'm not sure how to match the instances I need to fix. Here are the rules that I've thought of:
On a line-by-line basis:
If the line begins with <whitespace>include, then we shouldn't do anything to that line; pass it through to the output, so we don't mess up the filename inside the include statement.
The following strings are operators that don't have symbolic equivalents, so they must be left alone: .not. .and. .or. .eqv. .neqv.
Otherwise, if we find a period that is surrounded by 2 non-numeric characters (so it's not a decimal point), then it should be the operator that I'm looking to replace. Change that period to a %.
I'm not a native Fortran speaker myself, so here are some examples:
include 'file.inc' ! We don't want to do anything here. The line can
! begin with some amount of whitespace
if x == 1 .or. y > 2.0 ! In this case, we don't want to touch the periods that
! are part of the logical operator ".or.". We also don't
! want to touch the period that is the decimal point
! in "2.0".
if a.member < 4.0 .and. b.othermember == 1.0 ! We don't want to touch the periods
! inside the numbers, but we need to
! change the "a." and "b." to "a%"
! and "b%".
Any good way of tackling this problem?
Edit: I actually found some additional operators that contain a dot in them that don't have symbolic equivalents. I've updated the rule list above.
You can't do this with a regexp, and it's not that easy.
If I had to do what you have to, I would probably do it by hand, unless the codebase is huge. If the former applies, first replace all [a-zA-Z0-9].[a-zA-Z] to something very weird that is guaranteed never to compile, something like "#WHATEVER#", then proceed to search all these entries and replace them by hand after manual control.
If the amount of code is huge, then you need to write a parser. I would suggest you to use python to tokenize basic fortran constructs, but remember that fortran is not an easy language to parse. Work "per routine", and try to find all variable names used, using them as a filter. If you encounter something like a.whatever, and you know that a is in the list of local or global vars, apply the change.
Unless the codebase is really HUUGE (and do think very hard whether this is indeed the case), I'd just take an editor like Vim (vertical select & block select are your friends) a*nd set aside an afternoon to do this by hand*. In one afternoon, my guess is you'll be done with most of it, if not all. Afternoon is a lot of time. Just imagine how many cases you could've covered in these 2 hours alone.
Just by trying to write a parser for something like this, will take you much longer than that.
Of course, the question begs itself ... if the code if F77 which all compilers still support, and the code works ... why are you so keen on changing it?
I'm not that versed in regexps, so I guess I'd try tackling one this from other side. If you grep for the STRUCTURE keyword, you get the list of all the STRUCTURES used in the code. Once you have it, for each STRUCTURE S then you can just replace all instances of S. by S%.
This way you don't have to worry about things like .true., .and., .neq. and their relatives. The main worry then would be to be able to parse the STRUCTURE declarations.
Although the regex below :
(?<!')\b([^.\s]+)(?<!\.(?:not|and|or|eqv|neqv))(?<=\D)\.(?=\D)(?!(?:not|and|or|eqv|neqv)\.)([^.\s]+)\b(?!')
Replace $1%$2
Works perfectly for your examples, I would not recommend using it with your current task. It will definitely not cover all your cases. Now if you care for a 80% coverage or something you could use it but you should probably back up your sources. With the limited set of input cases I had , I am sure that there will be cases that the regex would replace something that it shouldn't.
Good luck :)
This sed oneliner might be a start
sed -r '/^\s*include/b;/^\s*! /b;G;:a;s/^(\.(not|and|or|eqv|neqv)\.)(.*\n.*)/\3\1/;ta;s/^\.([^0-9]{2,})(.*\n.*)/\2%\1/;ta;s/^(.)(.*\n.*)/\2\1/;ta;s/\n//'
Based on your examples, I am guessing it would be enough to protect quoted strings, then replace periods with alphabetics on both sides.
perl -pe '1 while s%(\x27[^\x27]+)\.([^\x27]+\x27)%$1##::##$2%;
s/([a-z])\.([a-z])/$1%$2/g;
s/##::##/./g' file.f
I offer this Perl solution not because sed is not a good enough tool for this, but because it avoids the issue of minor but pesky differences between sed dialects. The ability to use a hex code for the single quotes is a nice bonus.
I am trying to change all names that looks like this: thisForExample and change it with: this_for_example in Xcode with regex. Does anyone know how to do that?
I have tried with this: ([a-z][A-Z])*[a-z]?
but it does not find anything.
You just need to find [a-z][A-Z][a-z].
Automating the replacement process will be tricky though - how do you plan on changing an arbitrary upper case letter to its lower case equivalent ?
Perl would be a good tool for this (if you insist on using Regex)as it supports case modification in substitution patterns:
\l => change first char (of following match variable) to lower case
\L => change all chars (of following match variable) to lower case
\u => change single char (of following match variable) to upper case
\U => change all chars (of following match variable) to upper case
If all you care is to convert (simple/trivial!) variables and method names à la thisForExample into this_for_example.
For this a single regex like this would be sufficient:
echo 'thisForExample = orThisForExample();' \
| perl -pe 's/(?<=[^A-Z])([A-Z]+)(?=[^A-Z])/_\L\1/g;'
//output: "this_for_example = or_this_for_example();"
As soon however as you're expecting to come across (quite common) names like…
fooURL = URLString + URL + someURLFunction();
…you're in trouble. Deep trouble.
Let's see what our expression does with it:
echo 'fooURL = URLString + URL + someURLFunction();' \
| perl -pe 's/(?<=[^A-Z])([A-Z]+)(?=[^A-Z])/_\L\1/g;'
//output: "foo_url = _urlstring + _url + some_urlfunction();"
Pretty bad, huh?
And to make things even worse:
It is syntactically impossible to distinguish between a (quite common) variable name "URLString" and a class name "NSString".
Conclusion: Regex alone is pretty hack-ish and error prone for this kind of task. And simply unsufficient. And you don't want a single shell call to potentially mess up your entire code base, do you?
There is a reason why Xcode has a refactor tool that utilizes clang's grammar tree to differentiate between syntactically identical (pattern-wise at least) variable and class names.
This is a problem for context free languages, not regular languages. Hence regular expressions cannot deal with it. You'd need a contect free grammar to generate a language tree, etc. (and at that time you've just started building a compiler)
Also: Why use under_scores anyway? If you're using Xcode then you're probably coding in ObjC(++) or similar anyway, where it's common sense to use camelCase. I and probably pretty much everybody else would hate you for making us one day deal with your underscored ObjC/C/… code.
Alternative Answer:
In a comment answer to Paul R you said you were basically merging two projects, one with under_scored naming, one with camelCased naming.
I'd advise you then to switch your under_scored code base to camelCase. For two reasons:
Turning under_scored names into camelCased is way less error prone then vice versa. (That is: in a camelCase dominated environment only, of course! it would be just as error prone if you'd mainly deal with under_scored code in Xcode. Think of it as "there's simply less code to potentially break" ;) )
Quoting my own answer:
[…] I and probably pretty much
everybody else would hate you for making us one day deal with your
underscored ObjC/C/… code. […]
Here is a simple regex for converting under_score to camelCase:
echo 'this_for_example = _leading_under_score + or_this_for_example();' \
| perl -pe 's/(?<=[\w])_([\w])/\u\1/g;'
//output: "thisForExample = _leadingUnderScore + orThisForExample();"
Something like ([a-zA-Z][a-z]+)+?
The process could like this:
You get all the names to a file, there you automatically forge the replacement, and make (automatically) sed script to change one to another.
I am pretty new to scipting languages (Perl in particular), and most of the code I write is an unconscious effort to convert C code to Perl.
Reading about Perl, one of the things that is often mentioned as the biggest difference is that Perl is a dynamic language. So, it can do stuff at runtime that the other languages (static ones) can only do at compiletime, and so be better at it because it can have access to realtime information.
All that is okay, but what specific features should I, with some experience in C and C++, keep in mind while writing code in Perl to use all the dynamic programming features that it has, to produce some awesome code?
This question is more than enough to fill a book. In fact, that's precisely what happened!
Mark Jason Dominus' excellent Higher-Order Perl is available online for free.
Here is a quote from it's preface that really grabbed me by the throat when I first read the book:
Around 1993 I started reading books
about Lisp, and I discovered something
important: Perl is much more like Lisp
than it is like C. If you pick up a
good book about Lisp, there will be a
section that describes Lisp’s good
features. For example, the book
Paradigms of Artificial Intelligence
Programming, by Peter Norvig, includes
a section titled What Makes Lisp
Different? that describes seven
features of Lisp. Perl shares six of
these features; C shares none of them.
These are big, important features,
features like first-class functions,
dynamic access to the symbol table,
and automatic storage management.
A list of C habits not to carry over into Perl 5:
Don't declare your variables at the top of the program/function. Declare them as they are needed.
Don't assign empty lists to arrays and hashes when declaring them (they are empty already and need to be initialized).
Don't use if (!(complex logical statement)) {}, that is what unless is for.
Don't use goto to break deeply nested loops, next, last, and redo all take a loop label as an argument.
Don't use global variables (this is a general rule even for C, but I have found a lot of C people like to use global variables).
Don't create a function where a closure will do (callbacks in particular). See perldoc perlsub and perldoc perlref for more information.
Don't use in/out returns, return multiple values instead.
Things to do in Perl 5:
Always use the strict and warnings pragmas.
Read the documentation (perldoc perl and perldoc -f function_name).
Use hashes the way you used structs in C.
Use the features that solve your problem with the best combination of maintainability, developer time, testability, and flexibility. Talking about any technique, style, or library outside of the context of a particular application isn't very useful.
Your goal shouldn't be to find problems for your solutions. Learn a bit more Perl than you plan on using immediately (and keep learning). One day you'll come across a problem and think "I remember something that might help with this".
You might want to see some of these book, however:
Higher-Order Perl
Mastering Perl
Effective Perl Programming
I recommend that you slowly and gradually introduce new concepts into your coding. Perl is designed so that you don't have to know a lot to get started, but you can improve your code as you learn more. Trying to grasp lots of new features all at once usually gets you in trouble in other ways.
I think the biggest hurdle will not be the dynamic aspect but the 'batteries included' aspect.
I think the most powerful aspects of perl are
hashes : they allow you to easily express very effective datastructures
regular expressions : they're really well integrated.
the use of the default variables like $_
the libraries and the CPAN for whatever is not installed standard
Something I noticed with C converts is the over use of for loops. Many can be removed using grep and map
Another motto of perl is "there is more than one way to do it". In order to climb the learning curve you have to tell yourself often : "There has got to be a better way of doing this, I cannot be the first one wanting to do ...". Then you can typically turn to google and the CPAN with its riduculous number of libraries.
The learning curve of perl is not steep, but it is very long... take your time and enjoy the ride.
Two points.
First, In general, I think you should be asking yourself 2 slightly different questions:
1) Which dynamic programming features of Perl can be used in which situations/to solve which problems?
2) What are the trade-offs, pitfalls and downsides of each feature.
Then the answer to your question becomes extremely obvious: you should be using the features that solve your problem better (performance or code maintainability wise) than a comparable non-DP solution, and which incurs less than the maximum accaptable level of downsides.
As an example, to quote from FM's comment, string form of eval has some fairly nasty downsides; but it MIGHT in certain cases be an extremely elegant solution which is orders of magnitude better than any alternate DP or SP approach.
Second, please be aware that a lot of "dynamic programming" features of Perl are actually packaged for you into extremely useful modules that you might not even recognize as being of the DP nature.
I'll have to think of a set of good examples, but one that immediately springs to mind is Text template modules, many of which are implemented using the above-mentioned string form of eval; or Try::Tiny exception mechanism which uses block form of eval.
Another example is aspect programming which can be achieved via Moose (I can't find the relevant StackOverflow link now - if someone has it please edit in the link) - which underneath uses access to symbol table featrue of DP.
Most of the other comments are complete here and I won't repeat them. I will focus on my personal bias about excessive or not enough use of language idioms in the language you are writing code in. As a quip goes, it is possible to write C in any language. It is also possible to write unreadable code in any language.
I was trained in C and C++ in college and picked up Perl later. Perl is fabulous for quick solutions and some really long life solutions. I built a company on Perl and Oracle solving logistics solutions for the DoD with about 100 active programmers. I also have some experience in managing the habits of other Perl programmers new and old. (I was the founder / ceo and not in technical management directly however...)
I can only comment on my transition to a Perl programmer and what I saw at my company. Many of our engineers shared my background of primarily being C / C++ programers by training and Perl programmers by choice.
The first issue I have seen (and had myself) is writing code that is so idiomatic that it is unreadable, unmaintainable, and unusable after a short period of time. Perl, and C++ share the ability to write terse code that is entertaining to understand at the moment but you will forget, not be around, and others won't get it.
We hired (and fired) many programmers over the 5 years I had the company. A common Interview Question was the following: Write a short Perl program that will print all the odd numbers between 1 and 50 inclusive separated by a space between each number and terminated with a CR. Do not use comments. They could do this on their own time of a few minutes and could do it on a computer to proof the output.
After they wrote the script and explained it, we would then ask them to modify it to print only the evens, (in front of the interviewer), then have a pattern of results based on every single digit even, every odd, except every seventh and 11th as an example. Another potential mod would be every even in this range, odd in that range, and no primes, etc. The purpose was to see if their original small script withstood being modified, debugged, and discussed by others and whether they thought in advance that the spec may change.
While the test did not say 'in a single line' many took the challenge to make it a single terse line and with the cost of readability. Others made a full module that just took too long given the simple spec. Our company needed to delver solid code very quickly; so that is why we used Perl. We needed programmers that thought the same way.
The following submitted code snippets all do exactly the same thing:
1) Too C like, but very easy to modify. Because of the C style 3 argument for loop it takes more bug prone modifications to get alternate cycles. Easy to debug, and a common submission. Any programmer in almost any language would understand this. Nothing particularly wrong with this, but not killer:
for($i=1; $i<=50; $i+=2) {
printf("%d ", $i);
}
print "\n";
2) Very Perl like, easy to get evens, easy (with a subroutine) to get other cycles or patterns, easy to understand:
print join(' ',(grep { $_ % 2 } (1..50))), "\n"; #original
print join(' ',(grep { !($_ % 2) } (1..50))), "\n"; #even
print join(' ',(grep { suba($_) } (1..50))), "\n"; #other pattern
3) Too idiomatic, getting a little weird, why does it get spaces between the results? Interviewee made mistake in getting evens. Harder to debug or read:
print "#{[grep{$_%2}(1..50)]}\n"; #original
print "#{[grep{$_%2+1}(1..50)]}\n"; #even - WRONG!!!
print "#{[grep{~$_%2}(1..50)]}\n"; #second try for even
4) Clever! But also too idiomatic. Have to think about what happens to the annon hash created from a range operator list and why that creates odds and evens. Impossible to modify to another pattern:
print "$_ " for (sort {$a<=>$b} keys %{{1..50}}), "\n"; #orig
print "$_ " for (sort {$a<=>$b} keys %{{2..50}}), "\n"; #even
print "$_ " for (sort {$a<=>$b} values %{{1..50}}), "\n"; #even alt
5) Kinda C like again but a solid framework. Easy to modify beyond even/odd. Very readable:
for (1..50) {
print "$_ " if ($_%2);
} #odd
print "\n";
for (1..50) {
print "$_ " unless ($_%2);
} #even
print "\n";
6) Perhaps my favorite answer. Very Perl like yet readable (to me anyway) and step-wise in formation and right to left in flow. The list is on the right and can be changed, the processing is immediately to the left, formatting again to the left, final operation of 'print' on the far left.
print map { "$_ " } grep { $_ & 1 } 1..50; #original
print "\n";
print map { "$_ " } grep { !($_ & 1) } 1..50; #even
print "\n";
print map { "$_ " } grep { suba($_) } 1..50; #other
print "\n";
7) This is my least favorite credible answer. Neither C nor Perl, impossible to modify without gutting the loop, mostly showing the applicant knew Perl array syntax. He wanted to have a case statement really badly...
for (1..50) {
if ($_ & 1) {
$odd[++$#odd]="$_ ";
next;
} else {
push #even, "$_ ";
}
}
print #odd, "\n";
print #even;
Interviewees with answers 5, 6, 2 and 1 got jobs and did well. Answers 7,3,4 did not get hired.
Your question was about using dynamic constructs like eval or others that you cannot do in a purely compiled language such as C. This last example is "dynamic" with the eval in the regex but truly poor style:
$t='D ' x 25;
$i=-1;
$t=~s/D/$i+=2/eg;
print "$t\n"; # don't let the door hit you on the way out...
Many will tell you "don't write C in Perl." I think this is only partially true. The error and mistake is to rigidly write new Perl code in C style even when there are so many more expressive forms in Perl. Use those. And yes, don't write NEW Perl code in C style because C syntax and idiom is all you know. (bad dog -- no biscuit)
Don't write dynamic code in Perl just because you can. There are certain algorithms that you will run across that you will say 'I don't quite know how I would write THAT in C' and many of these use eval. You can write a Perl regex to parse many things (XML, HTML, etc) using recursion or eval in the regex, but you should not do that. Use a parser just like you would in C. There are certain algorithms though that eval is a gift. Larry Wall's file name fixer rename would take a lot more C code to replicate, no? There are many other examples.
Don't rigidly avoid C stye either. The C 3 argument form of a for loop may be the perfect fit to certain algorithms. Also, remember why you are using Perl: assumably for high programmer productivity. If I have a completely debugged piece of C code that does exactly what I want and I need that in Perl, I just rewrite the silly thing C style in Perl! That is one of the strengths of the language (but also its weakness for larger or team projects where individual coding styles may vary and make the overall code difficult to follow.)
By far the killer verbal response to this interview question (from the applicant who wrote answer 6) was: This single line of code fits the spec and can easily be modified. However, there are many other ways to write this. The right way depends on the style of the surrounding code, how it will be called, performance considerations, and if the output format may change. Dude! When can you start?? (He ended up in management BTW.)
I think that attitude also applies to your question.
At least IME, the "dynamic" nature isn't really that big of a deal. I think the biggest difference you need to take into account is that in C or C++, you're mostly accustomed to there being only a fairly minor advantage to using library code. What's in the library is already written and debugged, so it's convenient, but if push comes to shove you can generally do pretty much the same thing on your own. For efficiency, it's mostly a question of whether your ability to write something a bit more specialized outweighs the library author's ability to spend more time on polishing each routine. There's little enough difference, however, that unless a library routine really does what you want, you may be better off writing your own.
With Perl, that's no longer true. Much of what's in the (huge, compared to C) library is actually written in C. Attempting to write anything very similar at all on your own (unless you write a C module, of course) will almost inevitably come out quite a bit slower. As such, if you can find a library routine that does even sort of close to what you want, you're probably better off using it. Using pre-written library code is much more important than in C or C++.
Good programming practices arent specific to individual languages. They are valid across all languages. In the long run, you may find it best not to rely on tricks possible in dynamic languages (for example, functions that can return either integer or text values) as it makes the code harder to maintain and quickly understand. So ultimately, to answer your question, I dont think you should be looking for features specific to dynamicly typed languages unless you have some compelling reason that you need them. Keep things simple and easy to maintain - that will be far more valuable in the long run.
There are many things you can do only with dynamic language but the coolest one is eval. See here for more detail.
With eval, you can execute string as if it was a pre-written command. You can also access variable by name at runtime.
For example,
$Double = "print (\$Ord1 * 2);";
$Opd1 = 8;
eval $Double; # Prints 8*2 =>16.
$Opd1 = 7;
eval $Double; # Prints 7*2 =>14.
The variable $Double is a string but we can execute it as it is a regular statement. This cannot be done in C/C++.
The cool thing is that a string can be manipulated at run time; therefore, we can create a command at runtime.
# string concatenation of operand and operator is done before eval (calculate) and then print.
$Cmd = "print (eval (\"(\$Ord1 \".\$Opr.\" \$Ord2)\"));";
$Opr = "*";
$Ord1 = "5";
$Ord1 = "2";
eval $Cmd; # Prints 5*2 => 10.
$Ord1 = 3;
eval $Cmd; # Prints 5*3 => 15.
$Opr = "+";
eval $Cmd; # Prints 5+3 => 8.
eval is very powerful so (as in Spiderman) power comes with responsibility. Use it wisely.
Hope this helps.
One of the more interesting "programming languages" I've been stuck with lately is MediaWiki templates. You can do a surprising amount of stuff with the limited syntax they give you, but recently I've run into a problem that stumps me: using string functions on template arguments. What I'd like to do (somewhat simplified) is:
{{myTemp|a=1,2,3,4}}
then write a template that can do some sort of magic like
You told me _a_ starts with {{#split:{{{a}}}, ",", 0}}
At present, I can do this with embedded javascript, capturing regexp matching, and document.write, but a) it's huge, b) it's hacky, and c) it will break horribly if anybody turns off javascript. (Note that "split" is merely an example; concatenate, capturing-regexp matching, etc., would be even better)
I realize the right solution is to have the caller invoke the template with separate arguments, but for various reasons that would be hard in my particular case. If it's simply not possible, I guess that's the answer, but if there is some way to have templates do string-manipulation on the back end, that'd be great.
Concatenate is easy. To assign x = y concat z
{{#vardefine:x|{{{y}}}{{{z}}}}}
And, to add to Mark's answer, there are also RegexParserFunctions
Ceterum censeo: MediaWiki will never be not hacky.
You can do this with extensions, e.g. StringFunctions. But see also ParserFunctions and ParserFunctions/Extended. (You'll find a lot more examples in the Category:Parser function extensions.)
A great overview Help:Extension:ParserFunctions.